Deep Conditional Gaussian Mixture Model for Constrained Clustering
Authors: Laura Manduchi
Kieran Chin-Cheong
Holger Michel
Sven Wellmann
Julia E Vogt
Venue: NeurIPS 2021 Poster (2021)
Keywords: constrained clustering
deep generative models
representation learning

Abstract: Constrained clustering has gained significant attention in the field of machine learning as it can leverage prior information on a growing amount of only partially labeled data. Following recent advances in deep generative models, we propose a novel framework for constrained clustering that is intuitive, interpretable, and can be trained efficiently in the framework of stochastic gradient variational inference. By explicitly integrating domain knowledge in the form of probabilistic relations, our proposed model (DC-GMM) uncovers the underlying distribution of data conditioned on prior clustering preferences, expressed as \textit{pairwise constraints}. These constraints guide the clustering process towards a desirable partition of the data by indicating which samples should or should not belong to the same cluster. We provide extensive experiments to demonstrate that DC-GMM shows superior clustering performances and robustness compared to state-of-the-art deep constrained clustering methods on a wide range of data sets. We further demonstrate the usefulness of our approach on two challenging real-world applications.

Deep Conditional Gaussian Mixture Model for Constrained Clustering
Laura Manduchi
Kieran Chin-Cheong
Department of Computer Science
Department of Computer Science
ETH Zürich muura.manduch@@m ethz ch
ETH Zürich
kieran. chincheongeinf ethz. ch
Holger Michel
Department of Neonatology
University Children's Hospital Regensburg (KUNO)
University of Regensburg, Germany
holger mi
de
Sven Wellmann
Julia E. Vogt
Department of Neonatology
Department of Computer Science
University Children's Hospital Regensburg (KUNO)
ETH Zürich
University of Regensburg, Germany
julia. vogt@inf wtw.e ch
sven. wee..welllmmn@k uni regensburg..ee
Abstract
Constrained clustering has gained significant attention in the field of machine learning it can leverage prior information growing amount of only partially labeled data. Following recent advances in deep generative models, we propose a novel framework for constrained clustering that is intuitive, interpretable, and be trained efficiently in the framework of stochastic gradient variational inference. By explicitly integrating domain knowledge in the form of probabilistic relations, our proposed model (DC-GMM) uncovers the underlying distribution of data conditioned on prior clustering preferences, expressed as pairwise constraints. These constraints guide the clustering process towards a desirable partition of the data by indicating which samples should or should not belong the same cluster. We provide extensive experiments to demonstrate that DC-GMM shows superior clustering performances and robustness compared to state-of-the-ar deep constrained clustering methods on a wide range of data sets. We further demonstrate the usefulness our approach on two challenging real-world applications.
1 Introduction
The ever-growing amount of data and the time cost associated with its labeling has made clustering relevant task in the field of machine learning. Yet, in many cases, a fully unsupervised clustering algorithm might naturally find solution that not consistent with the domain knowledge (Basu et al. 2008). In medicine, for example, clustering could be driven by unwanted bias, such as the type of machine used record the data, rather than more informative features. Moreover, practitioners often have access to prior information about the types of clusters that are sought, and principled method guide the algorithm towards desirable configuration then needed. Therefore. constrained clustering has long history in machine learning as it enforces desirable clustering properties by
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

incorporating domain knowledge, in the form of instance-level constraints Wagstaft & Cardie 2000).
into the clustering objective.
variety of methods have been proposed to extend deterministic deep clustering algorithms, such as DEC (Xie et al., 2016), to force the clustering process to be consistent with given constraints (Ren et al., 2019; Zhang et al. 2019). This results in wide range of empirically motivated loss functions that are rather obscure in their underlying assumptions. Further, they are unable uncover the distribution of the data, preventing them from being extended to other tasks beyond clustering, such as Bayesian model validation, outlier detection, and data generation (Min et al., 2018). Thus, we restrict our search for constrained clustering approach to the class of deep generative models. Although these models have been successfully used in the unsupervised setting (Jiang al.) 2017 Dilokthanakul et al. 2016). their application to constrained clustering has been under-explored.
In this work we propose a novel probabilistic approach constrained clustering, the Deep Conditional Gaussian Mixture Model (DC-GMM), that employs a deep generative model to uncover the underlying data distribution conditioned on domain knowledge, expressed in the form of pairwise constraints. Our model assumes a Conditional Eexture-of-Gausiial prior on the latent representation of the data. That is, a Gaussian Mixture Model conditioned on the user's clustering preferences, based e.g. on domain knowledge. These preferences are expressed as Bayesian prior probabilities with varying degrees of uncertainty. By integrating prior information the generative process of the data, our model can guide the clustering process towards the configuration sought by the practitioners. Following recent advances variational inference (Kingma & Welling 2014 Rezende et al. 2014). we derive scalable and efficient training scheme using amortized inference.
Our main contributions are as follows: (i) We propose new paradigm for constrained clustering (DC-GMM) to incorporate instance-level clustering preferences, with varying degrees of certainty, within the Variational Auto-Encoder (VAE) framework. (ii) We provide a thorough empirical assessment of our model. In particular, we show that (a) a small fraction of prior information remarkably increases the performance of DC-GMM compared to unsupervised variational clustering methods, (b) our model shows superior clustering performance compared to state-of titte--fftheeeee the-hhhtttt art deep constrained clustering models on wide range of data sets and, (c) our model proves to be robust against noise as it can casily incorporate the uncertainty of the given constraints. (iii) Additionally, we demonstrate on two challenging real-world applications that our model can drive the clustering performance towards different desirable configurations, depending on the constraints used.
2 Deep Conditional Gaussian Mixture Model
In the following section, we propose a probabilistic approach to constrained clustering (DC-GMM) that incorporates clustering preferences, with varying degrees of certainty, in VAE-based setting. In particular, we first describe the generative assumptions of the data conditioned on the domain knowledge, for which VaDE (Jiang et al. 2017) and GMM-VVE (Dilokthanakul et al., (016) are special cases. We then define concrete prior formulation to incorporate pairwise constraints and derive new objective, the Conditional ELBO, to train the model in the framework of stochastic gradient variational Bayes. Finally, we discuss the optimization procedure and the computational complexity of the proposed algorithm.
2.1 The Generative Assumptions
W
Let us consider a data set X [rr) consist- ing of N samples with x, E RM that we wish cluster into K groups according to instance-level prior information encoded as W E R.NNN For example, we may know priori that certain sam- ples should (or should not) be clustered together. However, the prior information often comes from different sources with different noise levels. As an example, the instance-level annotations could be obtained from both very experienced domain ex- perts and less experienced users. Hence, W should
Z
μ,
N
Figure 1: The proposed graphical model.

---

encode both our prior knowledge of the data set, expressed in the form of constraints, and its degree of confidence.
We assume the data is generated from a random process consisting of three steps, as depicted in Fig. I
First, the cluster assignments {c; with e, {I,..., K), are sampled from distribution conditioned on the prior information:
c c~ p(c[W;;]]
The prior distribution of the cluster assignments without domain knowledge W, i.c. p(c). follows a categorical distribution with mixing parameters я. Second, for each cluster assignment C; continuous latent embedding, Z; IR is sampled from Gaussian distribution, whose mean and variance depend on the selected cluster Finally, the sample is generated from distribution conditioned on Z,. Given C₁ the generative process can be summarized as:
2, p(z))c)) NZ; [4cc T.=T
x, ~ px( = [N U2,, of 1) real-valued (Ber(aa;)) if binary
(3)
where Me; and o are mean and variance of the Gaussian distribution corresponding to cluster e; the latent space, In the case where real-valued then [Mx,, - fzz; 0). x is binary then Mz. f(;;; 0). The function (=; 0) denotes neural network, called decoder, parametrized by O.
is worth noting that, given W, the cluster assignments are not necessarily independent, i.e. there might be certain j E {l,.... .., N} for which (c, JKC cj (:). This important detail prevents the use of standard optimization procedure
will be explored in the following Sections.
contrary, if there is no prior information, that is ()) = p(c) [I. p(c,) II. (al(c. rT), the cluster assignments are independent and identical distributed. In that particular case, the generative assumptions described above are equal those of Jiang et al. (2017); Dilokthanakul et al. (2016) and the parameters of the model can learned using the unsupervised VaDE method. As a result, VaDE (or RMM-VAED can be seen as special case of our framework.
2.2 Conditional Prior Probability with Pairwise Constraints
We incorporate the clustering preference through the conditional probability p(c W). We focus on pairwise constrains consisting of must-links, if two samples are believed to belong to the same cluster, and cammot- links, otherwise. However, different types of constraints can be included, for example, triple-constraints (see Appendix C
Definition 1. Given a dataset X = ( the pairwise prior information W E NxN is defined as a symmetric matrix containing the pairwise preferences and confidence. In particular
> 0 if there is must-link constraint between x; and xz
= 0 if there is no prior information on samples &; and Wi
O
if there is cannot-link constraint between æ; and Dj
where the value Weal E [0, oo) reflects the degree of certainty in the constraint.
Definition 2. Given the pairwise prior information W, the conditional prior probability p(c W; T) defined as:
he W) P[c TT) EIII, ne (c W) (((T) II ac, W),
1
(4)
where π = [ are the weights associated to each cluster, e; is the cluster assignment of sample x,, ((T)) is the normalization factor and hy W) is a weighting function of the form
h, W)
II exp (Wi
(5)
follows that (e, W) assumes large values if c; agrees with our belief with respect to and low values otherwise. If W., -oo then e; and Φy must be assigned to different clusters otherwise
3

---

P[c[W) + 0 (hard constraint). On the other hand, smaller values indicate soft preference as they admit some degree of freedom in the model. An heuristic select [W, presented in See
The conditional prior probability with pairwise constraints has been successfully used by traditional clustering methods in the past (Lu & Leen 2004), but the best of our knowledge, it has never been applied in the context of deep generative models. It can also be seen as the posterior of the superparamagnetic clustering method (Blatt al. 1996), with loss function given by fully connected Potts model Wu 1982).
2.3 Conditional Evidence Lower Bound
Given the data generative assumptions illustrated in Sec. 2.1 the objective is to infer the parameters of the model, 0, π, and V/ {Mt, o k c= given both the observed data X and the pairwise prior information the cluster assignments W. This could be achieved by maximizing the marginal log-likelihood conditioned on W, that is:
logp X W) = log
Σ 2
p(X,
(6)
where Z = { is the collection of the latent embeddings corresponding to the data set X. The conditional joint probability derived from Eq. and Eq. and can factorized as:
p(X, Z, po (
W π I (2(
Since the conditional log-like ligllikeli intractable, we derive an alternative tractable objective. Definition 3. Lc, the Conditional ELBO (C-ELBO), defined as Co Φ, v, n W := (Z. [log pe (X) -DI Dxc.l///Z.cC/)llzzzcc.....
(8)
with 9o (Z, c(X) being the following amortized mean-field variational distribution:
9 (, c/X) 9o((4/))pclZzz..) II 9olls//.))pcc//..
(9)
The first term of the Conditional ELBO is known as the reconstruction term, similarly to the VAE. The second term, on the other hand, is the Kullback-Leiler (KL) divergence between the variational posterior and the Conditional Gaussian Mixture prior. By maximizing the C-ELBO, the variational posterior mimics the true conditional probability of the latent embeddings and the cluster assignments. This results in enforcing the latent embeddings to follow Gaussian mixture that agrees on clustering preferences.
From Definition 3 and Equations and we can directly derive Lemma Lemma 1. It holds that
1. The C-ELBO Le a lower bound of the marginal log- likelihood conditioned W, that log pl X W) 2 Le Φb X(W).
2. log logp(X(W) Lc(0, Φ0 n, X if and only if gob Z - c,
For the proof we refer the Appendix B does not depend on W. This approximation is used to retain a mean-field variational distribution when the cluster assignments, conditioned on the prior information, are independent (Sec2.1)
is worth noting that in Eq. g the variational distribution
that is when p(c) (c * P(c; W). Additionally, the probability p(c (2;) can be easily computed using the Bayes Theorem, yieldingg
(2, Mage o )Te
p(c. (c.ZZ, Σ. ((z pkx m
(10)
while we define the variational distribution qs (x,) be Gaussian distribution with mean ps and variance (zx)) parametrized by neural network, also known as encoder.

---

2.4 Optimisation & Computational Complexity
The parameters of the generative model and the parameters of the variational distribution are optimised by maximising the C-ELBO. From Definitions 2 and Bl we derive Lemma
Lemma 2. The Conditional ELBO Le factorizes as follow:
(o U, = log ((T) +
Bo (n, x..) logpo(x;|zz)) - log 90 (2/1x)))
1=1
(z)(1//x) (x.)
pkklzz)
log logp(z;1k) + log Te log logl(k))))) (11)
k=
N
Σ
Eng
Σ
pkk p(( W.y
ifj=1
where p(k|z,) pp(c k/2;; M, T) and p p(z. (k) -p(zicc k;
For the complete proof we refer to the Appendix Maximizing Eq. w.r.t π poses computational problems due the normalization factor 52 1.) Crude approximations are investigated in (Basu et al. 2008), however we choose to fix the parameter T 1/ K to make z uniformly distributed in the latent space, as in previous works (Dilokthanakul al. 2016). Hence the normalization factor be treated as a constant. The Conditional ELBO can then be approximated using the SGVB estimator and the reparameterization trick (Kingma & Welling 2014] to be trained efficiently using stochastic gradient descent. We refer to the Appendix [ for the full derivation. We observe that the pairwise prior information only affects the last term, which scans through the dataset twice. To allow for fast iteration we simplify it by allowing the search of pairwise constraints to be performed only inside the considered batch, yielding
B Σ +zj=1k=1
Σ
plc: - k/2 >(c; = (
(12)
T
where L denotes the number of Monte Carlo samples and B the batch size. By doing so, the overhead in computational complexity of single joint update of the parameters (LBBKKC)) where Cp the cost of evaluating p(c; = kllz;) with z; E R... The latter is O(KD).
Related Work
Constrained Clustering. A constrained clustering problem differs from the classical clustering scenario as the user has access to some pre-existing knowledge about the desired partition of the data expressed as instance-level constraints (Lange et al. (005). Traditional clustering methods, such as the well-knows K-means algorithm, have been extended to enforce pairwise constraints (Wagstaff)
al. (001). Bilenko et al. (2004)) Several methods also proposed constrained version of the
Gaussian Mixture Models (Shental et al. 2003 Law et al., 2004] (005). Among them, penalized probabilistic clustering (PPC, Lu & Leen 2004 is the most related our work expresses the
pairwise constraints as Bayesian priors over the assignment of data points to clusters, similarly to our model. However, all previous mentioned models shows poor performance and high computational complexity on high-dimensional and large-scale data sets.
Constrained Deep Clustering. To overcome the limitations of the above models, constrained clustering algorithms have lately been used in combination with deep neural networks (DNNs). Hsu Kira (2015) train DNN to minimize the Dullback-Leible (KL) divergence between similar pairs of samples, while Chen 2015) performs semi- -upervised maximum margin clustering on the learned features of DNN. More recently, many extensions of the widely used DEC model (Xie al.) 2016)
have been proposed to include a variety of loss functions to enforce pairwise constraints. Among them, SDEC, (Ren et al. 2019) includes a distance loss function that forces the data points with must-link to be close in the latent space and vice-versa. Constrained IDEC ZZhang et al. 2019). uses a KI. divergence loss instead, extending the work of Shuuka et al. (2018). [Smieja et al. (2020) focuses on discriminative clustering methods by self-generating pairwise constraints from Siamese networks.
5

---

As none of these approaches are based on generative models, the above methods fail to uncover the underlying data distribution.
Deep Generative Models. Although a wide variety of generative models have been proposed in the
literature to perform unsupervised clustering (Li al. 2019 Yang et al. 2019 Manduchi et al. 202T
Jiang ct al. 2017). not much effort has been directed towards extending them to incorporate domain knowledge and clustering preferences. Nevertheless, the inclusion of prior information on growing amount of unlabelled data is of profound practical importance in a wide range of applications (Kingma et al. 2014). The only exception the work of Luo et al. (2018), where the authors proposed the SDCD algorithm, which has remarkably lower clustering performance compared to state-of-the-an constrained clustering models, as we will show in the experiments. Different from our approach, SDCD models the joint distribution of data and pairwise constraints. The authors adopts the two-coin Dawid-Skene model from Raykar et al. (2010) to model p( ([(c), resulting in a different graphical model (see Fig. T. Instead, we consider simpler and more intuitive scenario, where we assume the cluster assignments are conditioned on the prior information, p(c|W:: πT) (see Definition 2 In other words, we assume that different clustering structures might be present within a data set and the domain knowledge should indicate which one is preferred over the other. It is also worth mentioning the work of Tang et al. (2019), where the authors propose to augment the prior distribution of the VAE to take into account the correlations between samples. Contrarily to the DC-GMM, their proposed approach requires additional lower bounds the ELBO cannot be directly applied. Additionally, it also requires post-hoc clustering of the learnt latent space to compute the cluster assignments.
4 Experiments
In the following, we provide a thorough empirical assessment of our proposed method (DC-GMM) with pairwise constraints using a wide range of data sets, First, we evaluate our model's 's performance compared to both the unsupervised variational deep elustering method and state-of-the-art constrained clustering methods. As a next step, we present extensive evidence of the ability of our model to handle noisy constraint information. Additionally, we perform experiments on a challenging real medical data set consisting of pediatric heart ultrasound videos, as well as face image data set to demonstrate that our model can reach different desirable partitions of the data, depending on the constraints used, with real-world, noisy data.
Baselines & Implementation Details. As baselines, we include the traditional pairwise constrained K-means (PCKmeans, Basu et al. (2004)) and two recent deterministic deep constrained clustering methods based on DEC (SDEC, Ren et al. (019), and Constrained IDEC, Zhang et al. (2019) as they achieve state-of-the-art performance in constrained clustering. For simplicity, we will refer to the latter as C-IDEC, We also compare our method to generative models, the semi-supervised SCDC (Lo et al. 2018) and the unsupervised VaDE (Jiang et al.) 2017). To implement were careful in maintaining a fair comparison with the baselines. In particular, we adopted the same encoder and decoder feed-forward architecture used by the basclines: four layers of 500, 500, 2000, D units respectively, where D- = 10 unless stated otherwise. The VAE is pretrained for 10 epochs while the DEC-based baselines need more complex layer-wise pretraining of the autoencoder which involves 50 epochs of pretraining for each layer and 100 epochs of pretraining as finetuning. Each data set is divided into training and test sets, and all the reported results are computed on the latter. We employed the same hyper-parameters for all data sets, see Appendix F.T for details. The pairwise constraints are chosen randomly within the training set by sampling two data points and assigning a must-link if they have the same label and a cannot-link otherwise. Unless stated otherwise, the values of W are set to 10 for all data sets, and 6000 pairwise constraints are used for both our model and the constrained clustering baselines. Note that the total amount of pairwise annotations in a data set of length N is O(V2). Given that N is typically larger than 10000, the number of pairwise constraints used in the experiments represents a small fraction of the total information.
model,
Constrained clustering. We first compare the clustering performance of our model with the baselines on four different standard data sets: MNIST (LeCun et al., 2010), Fashion MNIST (Xiao) jet al., 2017), Reuters (Xie et al., 2016) and STL-10 (Coates et al. (011)) (see Appendix A). More complex data sets will be explored in the following paragraphs. Note that we pre-processed the Reuters data by computing the tf-idf features on the 2000 most frequent words on a random subset of 10 000 documents and by selecting 4 root categories (Xie et al. 2016). Additionally, we extracted

---

Table 1: Clustering performances (%) of our proposed method DC-GMM compared with baselines. All methods use 6000 pairwise constraints except the unsupervised VaDE and the SCDC. Means and standard deviations are computed across 10 runs with different random model initialization. *VaDE results are different from (Jiang et al. 2017) as they only report their best performance. **Results taken from (uo al. 2018).
Dataset
Metric
VaDE*
PCKmeans
SDEC
C-IDEC
SCDD**
DC-GMM (ours)
MNIST
Acc
89.0 #5.0
56.4 12.0
86.2 20.1
96.3 =0.2
84.2
96.6 10.1
NMI
82.8 #3.0
50.5
#1.3
84.2 #0.1
91.8 #1.0
81.2
91.5 #0:2
ARI
80.9 #5.0
38.8 #1.9
80.1 20.1
92.1 =0.4
92.7 #0.3
FASHION Acc
55.1
#2.
53.9 2299
54.0 2.02
68.1 #3.0
80.0 41.0
NMI
57.9 #2.7
50.8 +1.3
57.3 #0.1
66.7 #2.0
71.8 #0.5
ARI
41.6
36.1 #1.7
40.2
#0.1
52.3 #3.0
65.8
REUTERS Acc
76.0 10.7
71.5 12.4
82.1 #0.1
94.7 20.6
95.4 1:022
NMI
50.1 #1.3
48.2 +3.8
62.3 #0.1
81.4
#0.7
82.7 #0.7
ARI
58.0 #1.4
46.5 44.22
66.7 #0.1
87.7 #0.9
89.0 #0.8
STL-10
Acc
77.3 20.5
70.3 442
79.2 20.1
81.6 =3.8
89.5 10.5
NMI
70.6 20.4
71.6 #1.3
78.6 20.1
77.3
#1.7
80.2 10.7
ARI
62.7 #0.4
58.4 42.1
71.0 #0.1
71.8 #3.4
78.4 4:0.9
features from the STL-10 image data set using a ResNet-50 (He et al. 2016 as in previous works (Jiang al. 2017). Accuracy, Normalized Mutual Information (NMI), and Adjusted Rand Index (ARI) are used as evaluation metrics. In Table | we report the mean and standard deviation of the clustering performance across 10 runs of both method and the baselines. The only exception the SDCD, for which we only report their original results (Luo et al, 2018) computed with a higher number of constraints. The provided code with 6000 constraints produced highly unstable and sub-optimal results (see Appendix D).
We observe that our model reaches state-of-the-arr clustering performance in almost all metrics and data sets. As C-IDEC turns out to be the strongest baseline, we performed additional comparison to investigate the difference in performance under different settings. In Figure 2 we plot the clustering performance in terms of ARI using a varying number of constraints, Ne. For additional metrics, we refer the Appendix E.T We observe that our method outperforms the strongest baseline C-IDEC by a large margin on all four data sets when fewer constraints are used. When N₂ gets close to N, the two methods tend to saturate on the Reuters and STL data sets.
0.35
0.255
3.65
0340
0.55
0.35
GMM
035
DC- CIIDEC
0.70
DC
DC
0.23
CIDEC
350
62
da 14 n.
0. ss 10
8. 64 as
(a) MNIST
(b) Fashion MNIST
(c) Reuters
(d) STL10
Figure 2: Comparison of clustering performance of our proposed DC-GMM model and the strongest baseline C-IDEC on four different data sets. The number of constraints vary between 0.1 x N and N, where N the length of the data set. ARI is used as evaluation metric.
Constrained clustering with noisy labels. In real-world applications it is often the case that the additional information comes from different sources with different confidence levels. Hence, the ability to integrate constraints with different degrees of certainty into the clustering algorithm is of significant practical importance. this experiment, we consider the case in which the given pairwise constraints have three different noise levels, q {0.1, 0.2, 0.3). where q determines the fraction
7

---

of pairwise constraints with flipped signs (that is, when must-link is turned into cannot-ling link and vice-versa). In Fig. B we show the ARI clustering performance of our model compared the strongest baseline derived from the previous section, C-IDEC. For all data sets, we decrease the value
of the pairwise confidence of our method using the heuristic [Wayl
log ( with a = 1000.
Also, we use grid search to choose the hyper-paramettrs of C-IDEC for the different noise levels (in particular we set the penalty weight of their loss function to 0.01, 0.005, and 0.001 respectively). Additionally, report Accuracy and NMI in Appendix
B. DC-GMM clearly achieves better performance on all three noise levels for all data sets. In particular, the higher the noise level, the greater the difference in performance. We conclude that our model is more robust than its main competitor on noisy labels and it can easily include different sources of information with different degrees of uncertainty.
1.40
0.55
1.0
0.50
DC GMM
DC-GMM
DC. DC-GHM
DC-GMM
3.00
C-IDEC
C-IDEC
0.5
C-DEC
0.85
C-IDEC
3.50
0.65
0.7
9.05
0.65
0.55
0.6
0.85
0:50
0.00
0.5
Noise level
Noise level
Noise level
Noise level
(a) MNIST
(b) Fashion MNIST
(c) Reuters
(d) STL10
Figure 3: Comparison of clustering performance of our proposed DC-GMM model and the strongest
baseline C-IDEC on four different data sets with noisy labels. ARI is used as evaluation metric.
Heart Echo. We evaluate the capability of our model in real-world application by using data set consisting of 305 infant echo cardiogram videos obtained from the Hospital Barmherzige Brüder Regensburg. The videos taken from five different angles (called views), denoted by [LA, KAKL, KAPAP, KAAP, 4CV]. Preprocessing of the data includes cropping the videos, resizing them to 64 x 64 pixels and splitting them into total of 20000 individual frames. We investigated two different constrained clustering settings. First, we cluster the echo video frames by view (Zhang et al. 2018). Then, cluster the echo video frames by infant maturity birth, following the WHO definition of premature birth categories ("Preterm") "). We believe that these two clustering tasks demonstrate that our model admits degree of control in choosing the underlying structure of the learned clusters. For both experiments, we compare the performance of our method with the unsupervised VaDE and with C-IDEC. Additionally, we include a variant of both our method and VaDE in which we use VGG-like convolutional neural network (Simonyan & Zisserman 2015) for details on the implementation we refer the Appendix F2 The results are shown in Table The DC-GMM outperforms both baselines by significant margin in accuracy, NMI, and ARI, as well as in both clustering experiments. We also observe that C-IDEC performs poorly on real-world noisy data. We believe this is due to the heavy pretraining of the autoencoder, required by DEC-based methods, it
Table 2: Clustering performance (%) using the heart echo cardiogram data with fully connected layers on the left and convolutional layers (CNN-) on the right. All methods use 6000 pairwise constraints except the VaDE. Mcans and standard deviations computed across 10 runs with different random model initialization.
Clustering
Metric
VaDE
C-IDEC
DC-GMM
CNN-VaDE
CNN-DC-GMM
View
Acc
33.4 13.3
55.1 116.0
83.2 114
41.7 15.2
92.5 11.4
NMI
8.9 12.7
33.3 115.9
64.9 12.3
19.7 47.4
82.6 43.0
ARI
6.5 #2.5
31.2 416.1
63.7 42.7
13.7 46.3
83.1 43.1
Preterm
Acc
41.4 45.4
69.6 #1.1
72.3 11.5
36.7 42.5
73.3 11.1
NMI
6.4 21.
8.3 211.8
25.1 23.4
6.5 24.0
32.0 42.4
ARI
2.3 42.6
13.7 4:1.3
45.1 44.1
3.9 43.8
48.1 42.8
8

---

does not always lead to a learned latent space that is suitable for the clustering task. Additionally, we illustrate a PCA decomposition of the embedded space learned by both the DC-GMM and the unsupervised VaDE baseline for both tasks in Figure 4 Our method is clearly able to learn an embedded space that clusters both the different views and the different preterms more effectively than the unsupervised VaDE. This observation, together with the quantitative results, demonstrates that adding domain knowledge is particularly effective for medical purposes.
Looa
(a) DC-GMM: Acc 0.925
(b) VaDE: Aec 0.417
(c) DC-GMM: Ace 0.733
(d) VaDE: Acc 0.367
Figure 4: PCA decomposition of heart ultrasound imaging test set examples in the embedded space and generative samples using the unsupervised VaDE and our model DC-GMM with 6000 constraints for (a)-(b) View, (c)-(d) Preterm.
Face Images. We further evaluate the performance of our model using the UTKFace data set Zhang al., 2017). This data set contains over 20000 images of male and female faces, aged from I to 118 years old, with multiple ethnicities represented. We use VGG nets (Simonyan & Zisserman 2015) for the VAE (the implementation details are described in the Appendix F.3 As in the Heart
Echo experiment, we chose two different clustering tasks. First we cluster the data using the gender prior information, then we select a sub- sub-sample of individuals between 18 and 50 years of age (approx. 11000 samples) and cluster by ethnicity (White, Black, Indian, Asian) In Fig. 5 we illustrate the PCA decomposition of the embedded space learned by the VaDE and our model. For both tasks we use 2N pairwise constraints where N is the length of the data set, which requires labels for 1.5% of the entire data set. Specifically, on the gender task and the ethnicity task our model achieves an accuracy of 0.89 and 0.85, which outperforms VaDE with relative increase ratio of 74.5% and 49.1%. In terms of NMI, the unsupervised VaDE performance is close to in both tasks, while our model performance is 0.52 and 0.54 respectively. Visually, we observe neat division of the selected clusters in the embedding space with the inclusion of domain knowledge. The unsupervised approach is not able to distinguish any feature of interest. We conclude that it is indeed possible guide the clustering process towards preferred configuration, depending on what the practitioners are seeking in the data, by providing different pairwise constraints. Finally, we tested the generative capabilities of our model by sampling from the learnt generative process of Sec For a visualization of the generated sample refer the Appendix
F
Label
Label
Female
Block
White
Asian
Indian
(a) DC-GMM: Acc 0.89
(b) VaDE: Acc 0.51
(c) DC-GMM: Acc 0.85
(d) VaDE: Acc 0.57
Figure 5: PCA decomposition of test set examples in the embedded space and generative samples using VaDE and DC-GMM for (a)-() Gender, (c)-(d) Ethnicity. In this configuration, DC-GMM obtains a NMI of 0.52 (gender) and 0.58 (ethnicity) while VaDE obtains a NMI close to 0 for both tasks.

---

Conclusion
In this work, we present a novel constrained deep clustering method called DC-GMM, that incor- porates clustering preferences in the form of pairwise constraints, with varying degrees of certainty. In contrast to existing deep clustering approaches, DC-GMM uncovers the underlying distribution of the data conditioned prior clustering preferences. With the integration of domain knowledge, we show that our model can drive the clustering algorithm towards the partitions of the data sought by the practitioners, achieving sfate-of-the-ar constrained clustering performance in real-world and complex data sets, Additionally, our model proves to be robust to noisy constraints as it can efficiently include uncertainty into the clustering preferences. As result, the proposed model can be applied to a variety of applications where the difficulty of obtaining labeled data prevents the use of fully supervised algorithms.
Limitations & Future Work The proposed algorithm requires that the mixing parameters of the clusters are chosen a priori. This limitation is mitigated by the prior information W. which permits more flexible prior distribution if enough information is available (see Definition 2 The analysis of different approaches to learn the weights и represents a potential direction for future work. Additionally, the proposed framework could also be used in a self-supervised manner, by learning W from the data using, e.g. contrastive learning approaches (Chen et al. 2020
Wu et al. 2018
6 Code and Data Availability
The code is available in GitHub repository: datasets are publicly available except the Heart Echo data. The latter is not available due to medical confidentiality.
All
Acknowledgments and Disclosure of Funding
We would like to thank Luca Corinzia (ETH Zurich) for the fruitful discussions that contributed shape this work. We would also like thank Vincent Fortuin (ETH Zurich), Thomas M. Sutter (ETH Zurich), Imant Daunhawer (ETH Zurich), and Ričards Marcinkevičs (ETH Zurich) for their helpful comments and suggestions. This project has received funding from the PHRT SHFN grant # #1-000018-0577 SWISSHEART.
References
Basu, S., Banerjee, A., and Mooney, R. Active semi-supervision for pairwise constrained clustering. In SDM, 2004.
Basu, S., Davidson, I., and Wagstaff, K. Constrained clustering: Advances in algorithms, theory, and applications. 2008
Bilenko, M., Basu, S., and Mooney, R. Integrating constraints and metric learning in sumi-supervised clustering. In /CML '04, 2004.
Blatt, Wiseman, and Domany. Superparamagnetic clustering of data. Physical review letters, 76 18: 3251-3254, 1996.
Chen, G. Deep transductive sumi-super maximum margin clustering. ArXiv, 0ass/1501.062377 2015.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, simple framework for contrastive learning of visual representations. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 Proceedings of Machine Learning Research, pp- 1597-1607. PMLR, 13-18 Jul 2020.
Coates, A., Ng, A., and Lee, H. An analysis of single-layer networks in unsupervised feature learning. In AISTATS, 2011.
10

---

Dilokthanakul, N., Mediano, P. A. M.. Garnelo, M., Lee, M. C. H., Salimbeni, H., Arulkumaran, K., and Shanahan, M. Deep unsupervised clustering with gaussian mixture variational autoencoders. CoRR, abs/16/ 12//61115 1.02448, 2016.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Hsu, Y.-. and Kira, Z. Neural network-based clustering using pairwise constraints. CoRR, abs/1511.063211 2015.
Jiang, Z., Zheng, Y., Tan, H.., Tang, B., and Zhou, H. Variational deep embedding: An unsupervised and generative approach to clustering. In ICCAI, 2017.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. In Bengio, Y. and LeCun, Y. (eds.), 2nd International Conference on Learning Representations, ICLR, 2014.
Kingma, D. P.. Mohamed. S., Rezende. D. J., and Welling, M. Semi-supervised learning with deep generative models. In NIPS, 2014.
Lange, T., Law, M. H. C., Jain, A. K., and Buhmann, J. Learning with constrained and unlabelled data. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 05), 1:731-738 vol. 1, 2005.
Law, M. H. C., Topchy, A., and Jain, A. K. Clustering with soft and group constraints. In SSPR/SPR, 2004.
Law, M. H. C., Topchy, A., and Jain, A. K. Model-based clustering with probabilistic constraints. In SDM, 2005.
LeCun, Y., Cortes, C., and Burges, C. Mnist handwritten digit database. ATT Labs Availal: 2, 2010.
Available:
Lewis, D. D., Yang. Y., Rose, T. G., and Li, Revl: A new benchmark collection for text categorization research. J. Mach. Learn. Res., 5:361-397, December 2004. ISSN 1532-4435.
Li, X., Chen, Z., Poon, L. K. M., and Zhang, N. L. Learning latent superstructures in variational autoencoders for deep multidimensional clustering. In ICLR, 2019.
Lu, Z. and Leen, T. Semi-supervised learning with penalized probabilistic clustering, In NIPS, 2004.
Luo, Y., TIAN, T., Shi, J.. Zhu, J., and Zhang, B. Semi-crowdsourced clustering with deep generative models. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 31, pp. 3212-3222 Curran Associates,
Inc., 2018
Manduchi, L., Hüser, M., Faltys, M., Vogt, J., Rätsch, G., and Fortuin, V. T-dpsom: An interpretable clustering method for unsupervised learning of patient health states. In Proceedings of the Conference on Health, Inference, and Learning. CHIL '21, pp. 236-245, New York, NY, USA, 2021. Association for Computing Machinery.
Min, E., Guo, X., Liu, Q., Zhang, G., Cui, J., and Long, J. A survey of clustering with deep learning: From the perspective of network architecture. IEEE Access, 6:39501-39514 2018.
Raykar, V. C., Yu, S., Zhao, H., Valadez, G. H., Florin, C., Bogoni, L., and Moy, L. Learning from crowds. Journal of Machine Learning Research, 1 1443): 2010.
Ren, Y., Hu, K., Dai, X., Pan, L., Hoi, S. C. H., and Xu, Z. Semi-supervised deep embedded clustering. Neurocomputing, 325:121-130, 2019.
Rezende, D. Mohamed, S., and Wierstra, D. Stochastic backpropagation and approximate inference in deep generative models. ICML, 2014.
Shental, N., Hir-Hillel, A., Hertz, T., and Weinshall, D. Computing gaussian mixture models with using equivalence constraints. In NIPS, 2003.

---

Shukla, A., Cheema, G. S., and Anand, S. Semi-supervised clustering with neural networks. arXiv: Learning, 2018.
Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409 1556, 2015.
Smieja, M., Struski, L.., and Figueiredo, M. A. T. A classification-basel approach to semi-supervised clustering with pairwise constraints. Neural networks the official journal of the International Neural Network Society, 127:193-203. 2020.
Tang, D., Liang, D., Jebara, T., and Ruozzi, N. Correlated variational auto-encoders, ICML, 2019.
Wagstaff, K. and Cardie, Clustering with instance-level constraints. In AAAI/IAAI, 2000.
Wagstaff, K., Cardie, C., Rogers, S., and Schrödl, Constrained k-means clustering with background knowledge. In ICML, 2001.
Wu, F. Y. The potts model. Rev. Mod. Phys., 54:235-268, Jan 1982. doi: 10.1 10.110//rvMo// 235. https::::://inn aps org/ 11t..0///1/1//////...eee.e. 54. 235
Wu, Z., Xiong, Y. Yu, S., and Lin, D. Unsupervised feature learning via non-parametric instance discrimination. 2018 IEFE/CVE Conference Computer Vision and Pattern Recognition, pp- 3733-3742, 2018.
Xiao, H., Rasul, K., and Vollgraf, R. fushion-mniste novel image dataset for benchmarking machine learning algorithms, 2017.
Xie, J., Girshick, R., and Farhadi, A. Unsupervised deep embedding for clustering analysis. volume 48 of Proceedings of Machine Learning Research, pp. 478-487, New York, New York, USA, 20-22 Jun 2016. PMLR.
Yang, L., Cheung, N., Li, and Fang. J. Deep clustering by gaussian mixture variational autoencoders with graph embedding. 2019 IEEE/CVF International Conference Computer Vision (ICCV), pp. 6439-6448, 2019.
Zhang, H., Basu, S., and Davidson, I. A framework for deep constrained clustering algorithms and advances. In ECML/PKDD, 2019.
Zhang, J., Gajjala, S., Agrawal, P., Tison, G., Hallock, I.., Beussink, L.,, Lassen, M., Fan, E.,, Aras, M., Jordan, C., Fleischmann, K., Melisko, M., Qasim, A., Shah, S., Bajesy, R., and Deo, R. Fully automated echocardiogram interpretation in clinical practice: Feasibility and diagnostic accuracy. Circulation, 138: 138:1623-166 10 2018. doi: 10. COL.IG//CRCCLLATTONAAAAA....
Zhang, Z., Song. Y.. and Qi, H. Age progressionnregressinn by conditional adversarial autoencoder. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2017.
12


Reviews:
[{"review_id": "a8pTX6ATE2f", "reviewer": "Reviewer_JUWY", "summary": "This paper deals with a constrained clustering problem which differs from the vanilla clustering problem in the extra pre-existing knowledge. We can also regard this problem as clustering with some supervision. The prior information known here is the pairwise relationship among input samples (a small portion). The method introduces a latent categorical variable of corresponding group. Experimental results show superior performance over the other baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*originality*\n\nDesigning a mixture prior based on the given pairwise sample relations is pretty novel. The experiments include several real-world datasets as well as standard benchmarks like MNIST. The claims are also supported by proofs or experiments.\n\n*quality*\n\n- some notions like $\\delta_{c_ic_j}$ is not explained in the paper\n- How $W$ is defined specifically is very important in this method but it is not clarified. I don't quite understand why \"the values of |$W_{i,j}$| are set to $10^4$ for all data sets\". $W$ is very important in the optimization. How it is defined directly affects the training.\n- The objective function involves all the samples. Do you still use mini-batch training? I can see Eq. (12) is adopting the mini-batching. But what about other terms. \n- Eq. (11) is the final objective. I think the authors could give some insights or explanations why the Eq. (8) is decomposed to Eq. (11). The transition is not very smooth. Is it for the convenience of the computation?\n- Is K pre-defined? With a fixed K, the problem would generally be easier.\n- How is acc calculated? Do you directly use variable $c$?\n- Some examples could be given to better motivated and validate the use of $W$.\n\n*clarity*\n\nConstrained clustering problem is not quite the same as the standard clustering problem. Some supervision is known a priori. I think the authors should explain the problem more clearly, stating explicitly which is know and which is not. Besides, the evaluation protocol should be more clear, e.g. which variable is used for prediction and how you obtain the accuracy.\n\n*significance*\n\nThe real-world datasets are novel. So the technique might be useful in the healthcare domain. The method is also generally novel, taking advantage of the VAE framework.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors mentioned some limitations in the \"Limitations & Future Work\" section. I agree that obtaining $W$ is complicated and might not be straightforward. The paper should improve the clarity of this part.\n\nBesides, the explanation of the objective function should be elaborated. Some insights and training details should be given.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "_tATHzBuqv", "reviewer": "Reviewer_bU1a", "summary": "Authors propose a probabilistic method for representation learning for clustering problem. Authors took a generative perspective and proposed an interesting algorithm for the problem at hand. Authors also evaluated their proposal on a set up problems.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Authors propose a probabilistic method for representation learning for clustering problem. Authors took a generative perspective and proposed an interesting algorithm for the problem at hand. Authors also evaluated their proposal on a set up problems.\n\nThe paper is well written and proposed method is novel. Intuitively integrating domain knowledge is making sense. However, I have several concerns about the method and problem setting. I have given my concerns below.\n•\tWhy do we need domain expertise for deep learning? One can also design features.\n•\tDomain expertise is implemented as a pairwise relation matrix i.e., W \\in R^{NxN} where N is the number of data points. To get full domain knowledge, one needs O(N^2) relations. Please note that this is quantity scale quadratically. I am aware that authors don’t claim the need of full information however this will not decrease the complexity. In other words, if one increases the data set size for a given number of relations, the ratio will decrease quadratically. How will authors address this point?\n•\tConflict between experts: Authors assume that there exists a panel of experts which can create W \\in R^{NxN}. In principle I agree that domain knowledge exits however one needs to address the conflicts between experts. Let’s assume we have two experts and each supplied relations on k points (the same points). By chance they give opposite responses. How will the algorithm handle these cases? \n•\tDoes W \\in R^{NxN} define a metric on data points? If yes, would you please elaborate how to check this? If no, can one use any function for generating W? \n•\tAuthors assume the existence of Gaussian latent space. How realistic is this assumption for practical problems, i.e., Categorical distributions? \n•\tIn the experiments, there are 6000 constraints used in all datasets. One can consider labelling data as domain knowledge or constraint. One can ask experts to label 6000 data points and use a semi-supervised method. What is the benefit of current setup compared to semi-supervised methods?\n•\tCan the proposed algorithm scale to millions of points?\n•\tCan the proposed algorithm scale to high dimensional data e.g. large resolution images ?\nIn general, I am positive about the method however I would like to some of my concerns to be addressed.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please see above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "WjHSn9En7tI", "reviewer": "Reviewer_3xdu", "summary": "The authors propose a novel framework for constrained clustering via a Gaussian Mixture Model on the latent space of a Deep Variational Autoencoder. To impose the constraint information, the proposed paradigm incorporates instance-level clustering preferences via Bayesian prior probabilities with varying degrees of certainty. The authors have derived the Conditional Evidence Lower Bound in variational inference for their problem setup. They have also provided comprehensive experiments compared with state-of-the-art methods DEC, VaDE and IDEC on classic benchmark datasets of MNIST, FMNIST, Reuters and STL-10. They have also explored the performance of their method on a more challenging dataset, which is the Face image dataset.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors provide a novel framework for constrained clusterin via deep generative models by incorporating instance-level clustering preferences via Bayesian prior probabilities with varying degrees of certainty. The paper is technically sound and comprehensive experiments have been conducted for multiple classic benchmark datasets for clustering including MNIST, FMNIST, Reuters, STL-10 and more challenging cases such as the face image dataset. The proposed method has outperformed state-of-the-art clustering methods such as DEC, VaDE and IDEC. The paper is also well written and easy to follow. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. The authors have adequately discussed the limitations of their method. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "TCUP1q7WEj3", "reviewer": "Reviewer_2jYB", "summary": "This paper proposes a deep generative model with latent variables drawn from a Gaussian mixture model.  Moreover, the prior over the components is conditioned on prior information that specifies clustering constraints, allowing the user to specify one of three cases: x_i and x_j (1) must be in the same cluster, (2) must not be in the same cluster, or (3) no information.  An ELBO is derived.  Experiments demonstrate clustering performance on a variety of data sets ranging from MNIST to STL-10 to heart ultrasounds. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:  I find the paper novel, as I am aware of no other work that combines deep Gaussian mixture models with clustering constraints\n\nClarity:  I find the paper to be well-written, clear, and the results well-presented.\n\nSignificance:  Given that clustering is a pervasive methodology, there is potential for wide adoption.  However, I don't think that Gaussian mixture DGMs have found the popularity of other DGM types.  Moreover, while the combination is novel, there does not seem to be any outstanding originality or insight (i.e. this paper didn't make me view clustering or DGMs from a new perspective).\n\nQuality:  I find the paper's execution to be well done.  The model is reasonable and practical.  The results use sensible benchmarks and baselines.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Addressed in Section 5", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper describes a way of incorporating prior information in a deep Gaussian mixture model in the form of pairwise relation constraints.

The reviewers carefully considered the merits of the paper and the response of the authors. One of the primary concerns was around the implementation of domain expertise --- the method requires experts to express all pairwise relations leading to O(N^2) relations. The authors responded that the complexity of the algorithm does not scale quadratically, but instead with the batch size. It would be expected that the computational time of running a quadratic algorithm is dwarfed by the effort require to elucidate all pairwise relations from one or more experts. Therefore, the questions around the practicality of such an approach raised by the reviewer remains. It seems that there many be better data sets and problems for which the method is appropriate. The chosen evaluation data sets MNIST, etc, are very standard and the constrained clustering using expertise on pairwise relations may be better suited to other application domains.

In summary, the remaining concerns about practicality of the method and a demonstrable application domain outweigh the benefits of the novelty of the approach.

Decision: Accept (Poster)