Deep Goal-Oriented Clustering
Authors: Yifeng Shi
Christopher M Bender
Linnea Olsson
Melissa Troester
Katherine A Hoadley
Junier Oliva
Marc Niethammer
Venue:  (2021)
Keywords: clustering
variational inference

Abstract: Clustering and prediction are two primary tasks in the fields of unsupervised and supervised learning, respectively. Although much of the recent advances in machine learning have been centered around those two tasks, the interdependent, mutually beneficial relationship between them is rarely explored. One could reasonably expect appropriately clustering the data would aid the downstream prediction task and, conversely, a better prediction performance for the downstream task could potentially inform a more appropriate clustering strategy. In this work, we focus on the latter part of this mutually beneficial relationship. To this end, we introduce Deep Goal-Oriented Clustering (DGC), a probabilistic framework that clusters the data by jointly using supervision via side-information and unsupervised modeling of the inherent data structure in an end-to-end fashion. We show the effectiveness of our model on a range of datasets by achieving prediction accuracies comparable to the state-of-the-art, while, more importantly in our setting, simultaneously learning congruent clustering strategies.

Under review as a conference paper at ICLR 2021
DEEP GOAL-ORIENTED CLUSTERING
Anonymous authors
Paper under double-blind review
ABSTRACT
Clustering and prediction are two primary tasks in the fields of unsupervised and supervised machine learning. Although much of the recent advances in machine learning have been centered around those two tasks, the interdependent, mutually beneficial relationship between them is rarely explored. One could reasonably expect appropriately clustering the data would aid the downstream prediction task and, conversely, a better prediction performance for the downstream task could potentially inform more appropriate clustering strategy. In this work, we focus on the latter part of this mutually beneficial relationship. To this end, we introduce Deep Goal-Oriented Clustering (DGC), probabilistic framework that clusters the data by jointly using supervision via side-infe informaffomoon and unsupervised modeling of the inherent data structure an end-to-end fashion. We show the effectiveness our model on range of datasets by achieving prediction accuracies comparable to the state-of-the-an while, more importantly in our setting, simultancously learning congruent clustering strategies. We also apply DGC to real-world breast cancer dataset, and show that the discovered clusters carry clinical significance.
INTRODUCTION
Much of the advances in supervised learning in the past decade are due to the development of deep neural networks (DNN), a class of hierarchical function approximators that are capable of learning complex input-output relationships. Prime examples of such advances include image recognition (Krizhessky et al. 2012), speech recognition (Nassif et al. 2019), and neural transla- tion (Bahdanau et al. (013). However, with the explosion of the size of modern datasets, it becomes increasingly unrealistic to manually annotate all available data for training. Hence, understanding inherent data structure through unsupervised clustering is of increasing importance.
Several approaches to apply DNNs to unsupervised clustering have been proposed in the past few years (Caron et al. 2018: Law et al. 2017,; Xie et al., 2016 [Shaham et al. 2018), centering around the concept that the input space in which traditional clustering algorithms operate is of importance. Hence, learning this space from data is desirable, in particular, for complex data. Despite the improvements these approaches have made on benchmark clustering datasets, the ill-defined, ambiguous nature of clustering still remains a challenge. Such ambiguity is particularly problematic in scientific discovery, sometimes requiring researchers to choose from different, but potentially equally meaningful clustering results when little information is available priori (Ronan et al. 2016).
When facing such ambiguity, using direct side-information to reduce clustering ambivalence proves be a fruitful direction (Xing al.) 2002 Khashabi et al. 2015 Jin et al. 2013). Direct side- information is usually available in terms of constraints, such as the must-link and the cannot-link constraints (Wang & Davidson 2010 Wagstaff & Cardie 2000). or via pre-conceived notion of similarity (Xing et al.] 2002). However, defining such direct side-informa inforrfffottnnnn requires human expertise, which could be labor intensive and potentially vulnerable to labeling errors. On the contrary, indirect, but informative, side-information might exist in abundance, and may not require human expertise to obtain. Being able to learn from such indirect information to form congruous clustering strategy is thus immensely valuable.
Main Contributions We propose Deep Goal-Oriented Clustering (DGC), a probabilistic model that is capable of using indirect. but informative, inde-information to form a pertinent clustering strategy. Specifically: 1) We combine supervision via and-inforrmttin and unsupervised data structure modeling in probabilistic manner; 2) We make minimal assumptions what form the

---

Under review as a conference paper at ICLR 2021
supervised side-information might take, and assume explicit correspondence between the side- information and the clusters; 3) We train DGC end-to-end so that the model simultaneously learns from the available side-nfformation while forming desired clustering strategy.
2 RELATED WORK
Most related work in the literature can be classified into two categories: 1) Methods that utilize extra intermfforattonn to form better, less ambiguous clusters; however, such iide-information needs to be provided beforehand and cannot be learned: 2) Methods that can learn from the provided labels to lessen the ambiguity in the formed clusters, but these methods rely on the cluster assumption (detailed below), and usually assume that the provided labels discrete and the ground truth labels. This excludes the possibility of learning from indirectly related, but informative side-information. We propose a unified framework that allows using informative side-information directly or indirectly to arrive at better formed clusters. Latent space sharing among different tasks has been studied in a VAE setting (Le et al. 2018 Xie Ma 2019). In this work we utilize this latent space sharing framework, but instead focus on clustering with the aid of general, indirect side-information.
Side-information as constraints Using side-information to form better clusters is well-studied. Wagstaff & Cardie (2000) consider both must-link and cannot-link constraints in the context of
K-means clustering. Motivated by image segmentation, OObanz & Buhmann (2007) proposed a
probabilistic model that can incorporate muuttllink constraints. Khashabi et al. (2015) proposed
nonparametric Bayesian hierarchical model to incorporate noisy andeinnformatio as soft-constraints.
Vu et al. (2099) utilize constraints and cluster labels side information. [aaummaar & Saha (007))
give complexity bounds when provided with an oracle that can be queried for side information. Wasid
& Ali (2019) incorporate side information through the use of fuzzy sets. In supervised clustering, the side-information is the a priori known complete clustering for the training set, which is being used a constraint to learn a mapping between the data and the given clustering (Finley & Joachims 2005). In contrast, we do not assume that the constraints are given a priori. Instead, we let the side-information guide the clustering procedure during the training process.
Semi-supervised methods & the cluster assumption Semi-supervised clustering approaches gen- erally assume that they only have access to a fraction of the true cluster labels. Via constraints as the ones discussed, the available labels are propagated to unlabeled data, which can help mitigate the ambiguity in choosing among different clustering strategies (Bair 2013). The generative approach to semi semi-superve supervise learning introduced in Kingma et al. (2014) is based on a hierarchical generative model with two variational layers. Although it was originally meant for semi-supervised classification tasks, it can also be used for clustering. However, if used for clustering, it has to strictly rely on the cluster assumption, which states that there exists direct correspondence between labels/classes and clusters (Färber et al., 2010; Chapelle al.] (2006). We show that this approach is a special case of our framework without the probabilistic ensemble component (see Sec. 4.2) and when certain distributional assumptions are made. Sansone al. (2016) proposed a method for joint classification and clustering to address the stringent cluster assumption most approaches make by modeling the cluster indices and the class labels separately, underscoring the possibility that each cluster may consist of multiple class labels. Deploying a mixture of factor analysers as the underlying probabilistic framework, they also used a variational approximation to maximize the joint log-likeliine -iceeiikddoodd.
In this work, generalize the notion of learning from discrete, ground truth labels to learning from indirect, but informative side-information. We make virtually no assumptions on the form of y nor its relations the clusters. This makes our approach more applicable to general settings.
3 BACKGROUND & PROBLEM SETUP
3.1
DEEP EMBEDDING
The starting point for DCC is the variational auto encoder (VAE) (Kingma & Welling 2014) with the prior distribution of the latent code chosen Gaussian mixture distribution. This is introduced in Jiang et al. (2017) as VaDE. We briefly review the generative VaDE approach here to provide the background for DGC. We adopt the notation that lower case letters denote samples from their
2

---

Under review as a conference paper at ICLR 2021
corresponding distributions; bold, lower case letters denote random variables/vectors; and bold upper case letters denote random matrices.
Assume the prior distribution of the latent code, belongs the family of Gaussian mixture distributions, i.c. p(z) = E. P(z)))p(() = Ee (,, o²1) where с a random variable, with prior probability Ne. indexing the normal component with mean μi and variance o². VaDE allows for the clustering of the imput data in the latent space, with each component of the Gaussian mixture prior representing an underlying cluster. VAE-based model can be efficiently described in terms of its generative process and inference procedure. Given an imput R the following decomposition of the joint probability p(x, z, e) details VaDE's generative process: p(x, Z, c) p(x|z)p(z c)p(c)- In words, we first sample the component index e from prior categorical distribution p(c), then sample the latent code z from the component p(zlc), and lastly reconstruct the input x through the reconstruction network px)z.). To perform inference and learn from the data, VaDE constructed to maximize the log- ligelikelihood of the input data x by maximizing its evidence lower bound (ELBO):
log 2 (x((ix) logp(x(z) Eg(els) log q(clx) Bg(x,e(x)) log q(12x) p(z|c)
p(c)
where, given the input x, q(z, c/x) denotes the variational posterior distribution over the latent variables, and Ex denotes the expectation wrt. distribution d. With proper assumptions on the prior and variational posterior distributions, the ELBO in Eq. [admits closed-form expression in terms of the parameters of those distributions. We refer readers to Joang et al. (2017) for additional details.
3.2 PROBLEM SETUP
Unlike the unsupervised settings, we do assume we have response variable y. and our goal is to leverage y to inform a better clustering strategy. Abstractly, given the input-output random variable pair (x, y), we seek to divide the probability space of into oon-ovarlapinn subspaces that are meaningful in explaining the output y. In other words, we want to use the prediction task of mapping data points, x, sampled from the probability space of x to their corresponding sampled outcomes y as a teaching agent, to guide the process of dividing the prob- ability space of x into subspaces that optimally explain y. Since our goal is to discover the subspace-struc structure without knowing priori whether such structure indeed exists, probabilistic framework is more appropriate due to its ability to incorporate and extend the
y
Figure 1: The Bayesian network that underlies the generative pro- cess of DGC. 0 and π together con- stitute the generative parameters.
and reason with uncertainty. To this end, VaDE framework, with the following assumption imposed on the latent code that specifically caters to our setting, Assume the input x carries predictive information with respect to the output y. Since the latent code should inherit sufficient information from which the input x can be reconstructed, it is reasonable to assume that z also inherits that predictive information. This assumption implies that x and y are conditionally independent given z, i.e. p(x, ylz) = p pxx)z)p(y(z).
DEEP GOAL-ORIENTED CLUSTERING
4.1 GENERATIVE PROCESS
In order to incorporate y into a probabilistic model, recall from our previous discussion that might manifest with respect to the input differently across different subspaces of the input space. Viewing p(ylz) conditional probability distribution over y resulting from a functional transformation from z to the space of probability distributions over y, we can assume that the ground truth transformation function, 9c, is different for each subspace indexed by If p(z|c) for some index assume p(y|z, c) x (z) for some subspace-spent specific gc- As a result, we learn a different mapping function for each subspace.
The overall generative process of our model is as follows: 1. Generate e ~ Cat(TT: 2. Generate p(z]c); ; 3. Generate p(x|z); 4. Generate y p(ylz,c). The Bayesian network that
3

---

Under review as a conference paper at ICLR 2021
underlies DGC is shown in Fig. 1 and the joint distribution of x, 2, z, and can be decomposed as: p(x.y, z, c) = ply|z, c)(x(x))))xp
4.2 INFERENCE & VARIATIONAL LOWER BOUND
We first note that the joint variational posterior distribution c/x, y) be factorized as q(z, y) g(c]x, 2, y) ((Z)x y). Since we assume we do not have access to the side-information y at test time, we do not use to compute q(z]x.y y) (in reality, this entails that the encoder only takes x as input to compute the latent code z). We omit the variable in g(z]x, y) for the rest of the paper for notation convenience. See Sec. for how compute q(c/x, Z. y. at test time when y is not available. With this setup, we have the following variational lower bound (see the Appendix for detailed derivation)
log p(x. 2 E Eo log poyp(yy,,, c) B log D(X,2,0 Olz, c)x, y)
LELBO
Probabilistic Ensemble
ELBO for VAR with GMM prior
The first term in LELBO allows for probabilistic ensemble of classifiers based on the subspace index. This can be seen as follows
M
Eggx,els,)) log p(yl(,), = Eg(((x)) Σ logplylee,
1 M
Σ
A log plylz k))
where d = =c(c= k/xx and indexes the Monte Carlo samples used to approximate the expectation with respect to g(2 x). The probabilistic ensemble allows the model to maintain necessary uncertainty with respect to the discovered subspace structure until an unambiguous structure is captured.
is also worth noting that the variational lower bound described in Eq, 2 holds regardless of the prior distribution we choose for the latent code z. Although we choose the mixture distribution the prior in this work, choosing z ~ NO,, I) and disregarding the probabilistic ensemble component would recover the exact model introduced in [Kingma et al. (2014) (when all labels are missing), and hence is a special case of our proposed framework.
4.3 MEAN-FIELD VARIATIONAL POSTERIOR DISTRIBUTIONS
Following VAE (Kingma h is parameterized by feed-forwar neural network with weights 6. See the Appendix for a detailed discussion of why using a unimodal distribution (i.e. q(zlx)) to approximate a multimodal distribution (p(z)) is appropriate in our setting.
2014), we choose ((l)x) to Ne (z),z, 1) where [ h(x; 0).
Choosing q(e|x, z, y) appropriately requires us to analyze the proposed LELOO in greater detail based on the following decomposition (see the Appendix for detailed derivation):
p(x, z) LELBO Ey(n, plgl(y/,,c) + (x)x)) log ((z(x)
Eqcats KL (g(c)
(3)
We observe that since 2 does not depend on c, q(c|x, y, should be chosen maximize (Q) Moreover, the expectation over (z|xx) does not depend c, and thus has influence over choice of q(c|x, y. Casting finding q(c|x, z, y) as an optimization problem, we have
min <(cxxzyy)
Sol4) = KL (q(c)x, z, l)l((()))))) Egtelx2y)) logp(yyz,,c) c),
S.t. Σ _(((x22)) = 1, d(clx, y 2 0, Ak.
(4)
The objective functional fo convex over the probability space of q, as the Kullback-Leibe divergence is convex in g and the expectation is linear in q. Analytically solving the convex program (see the Appendix for detailed derivation), we obtain
q(c = k,),,,,) = p(ylz, c = k) -)pppc- = k
(5)
Exp(y/z,c= kk) .ppc ki
4

---

Under review as a conference paper at ICLR 2021
First note that since the solution ((c|x,z,y) does not depend x, omit in q(c)x, 2, for the remainder of the paper for notational convenience. To better facilitate understanding, we interpret Eq. [ in two extremes. If y is evenly distributed across the different subspaces, i.c the ground truth transformations Je are the same for all c, then q(clz,)) y) = p(c = k[z), which is what one would choose for unsupervised clustering (Jiang al. 2017). However, if the supervised task is informative while the unsupervised task is not, i.c. p(c is uniform distribution, the likelihoods {p(ylz, c h}}& will dominate q. Therefore, one could interpret any in-between scenario as balance that automatically weights the supervised and the unsupervised tasks based on how strong their signals are with respect to grouping the latent probability space into different subspaces.
4.4 EVALUATING ON UNLABELED DATA
We first introduce some notations that we will adopt in this section. We write p(y z, c) to denote the likelihood value, which requires specific value of y to compute. We write Pylz,c to refer to the entire distribution (in the context of the entropy (HI) and the expectation (E) operators we refer to distributions, otherwise we refer to specific likelihoods). When presented with the response variable y. Eq. gives the optimal choice of g(c/z, y) that allows the network to incorporate both the supervised and the unsupervised signals when weighting the clusters. In practice, we do not have access to y on newly collected (test) data points, which prohibits us from evaluating (cll,, y). One easy remedy to this would be to use p(clz) when y is not available; however, having an ensemble of well-trained conditional likelihood mappings, {p(y/z, data points seems wasteful. We thus add a regularization term to LELBO, so that DGC can naturally generalize to unlabeled testing samples. The regularized ELBO is:
k)},- and not utilizing them when evaluating new
Ceggu ELBO
LELBO Eggz,e/s,y) HH (Prin.))
(6)
where Hamax (Pyl = (H ())p),)).))))))))))) 0} and (pyla,))) = - -Epsa.. log Pylz,e which is the entropy of the task network distributions. If y is a discrete random variable, Hmax (Pylz.e = H(Pptk.)), which is the entropy of Pylz.e and always non-negative; on the other hand, when y is continuous, although the differential entropy of Pyle.c can take any sign, the max operator ensures that Hmax (Py will remain non-negative Therefore, adding (a convex combination of) negative entropies preserves the inequality, and thus EELOO remains proper lower bound. Additionally, solving similar convex program provides the optimal choice of q(c|z, y) in the presence of the regularizer
clon,(s/x.A)) (Pyia))) p(k|z)
q(c = klz,y) = Σ, eogp (P9ia)) p(jlz))
This form of regularization penalizes the entropies of the conditional distributions Pylz.e- More specifically, clusters with higher posterior weights are penalized more towards having low entropies. This aligns with our intuition: the most suitable cluster to explain a given sample should be relatively more certain in how it is distributed. We thus use Eq. 7 to weight the clusters during training. It is creu worth noting that although Eq. 7 provides the optimal choice of q(clz, y) for maximizing ELBO any long as it maintains a proper probability distribution, would satisfy the fact that C is proper lower bound. Based on the previously stated intuition, when evaluating on an unlabeled data point, we use
choice of qíc = & ll.z.)
Hom ppkkzz) klz,y) E, - (p)) p(jlz)
to weight the clusters. This aligns with our previous reasoning: the cluster that corresponds to plylz, c) with the lowest entropy will be weighted most heavily. This allows the model to use the tuned conditional likelihood mappings when the side-information y is not available.
5 EXPERIMENTS
We investigate the efficacy of DGC range of datasets. We refer the reader to the Appendix for the experimental details, e.g. the train/validationtess split, the chosen network architecture, the choices of learning rate and optimizer.
5

---

Under review as a conference paper at ICLR 2021
5.1 NOISY MNIST
We introduce synthetic data experiment using the MNIST dataset, which we name the noisy MNIST, to illustrate that the supervised part of DGC can enhance the performance of an otherwise well-performing unsupervised counterpart. Further, unsupervised part to demonstrate the importance of capturing the inherent data structure. We extract images that correspond to the digits and 7 from MNIST. For each digit, we randomly select half of the images for that digit and superpose noisy backgrounds onto those images (see the Appendix for image samples). The binary random variable y indicates what digit each image belongs to. Our goal is to cluster the images into 4 clusters: digits 2 and 7, with and without background. However, we are only using the binary responses for supervision and have no direct knowledge of the background. We therefore parameterize the task networks, (p(ylz, 2 c = k] k= as Bernoulli distributions where we learn the parameters (the probabilities).
explore the behavior of DGC without its
As baseline, the unsupervised approach, VaDE, already performs well on this dataset, achieving a clustering accuracy of 95.6% when the desired number of clusters is set to 4. Fig. shows that VaDE distinguishes well based on the presence or absence of the noisy background, and incorrectly clustered samples are mainly due to VaDE's 's inability to differentiate the underlying digits. This is reasonable behavior: if the background signal dominates, the network may focus on the background for clustering as it has no explicit knowledge about the digits.
DGC performs nearly perfectly (with a clustering accuracy of 99.6%) in this setting with the help of the added supervision. We see that DGC miti- gates the difficulty of distinguishing between digits under the presence of strong, noisy backgrounds (as shown in Fig. [b] where DGC makes almost no mistakes in distinguishing between digits even in the presence of noisy backgrounds). This added supervi- sion does not overshadow the original advantage of VaDE (i.e. distinguish- ing whether the images contain back- ground or not). Instead, enhances the overall model in cases where the unsupervised part, i.c. VaDE, struggles. lier sections, most existing approaches that take advantage of available labels rely on the cluster assumption, which assumes a one-to-one correspondence between the clusters and the labels used for supervision. This experiment is concrete example that demonstrates DGC does not need to rely on such assumption to form sound clustering strategy. Instead, DGC is able to work with class labels that are only partially indicative of what the final clustering strategy should be, potentially making DGC more applicable to more general settings.
514
124
(%)
(3.0%)
484
32
5155
0
78
10.0%)
(23.49%4
10.0%)
11
1.55
Truth
20
40 09:44
(25.0 0%5
0%
(0.0%)
502
(0.24%%
44 12.1.9%)
470
51.
(%)
10.0%%
122.8153
60 0%k
(0 (44%)
(0 0%)
(24 94.9%%
2
26
/
I8
2
Z8 Prediction
/B
Prediction
(a) Confusion Matrix VaDc
(b) Confusion Matrix DGC
Figure 2: Confusion matrices abbreviated, 2B/7B, in the row/column labels denotes digits 2/7 with background. Rows represent the predicted clusters, and columns represent the ground truth.
Furthermore, as detailed in Sansone et al. (2016) and ear-
Ablation study To further test the importance of each part of our model, we ablate the probabilistic components (i.e. get rid of the decoder and the loss terms associated with it, that only the supervision will inform how the clusters are formed in the latent space) and perform clustering using only the supervised part of our model. We find that clustering accuracy degrades from the nearly-perfect accuracy obtained by the full model to 50%. Coupled with the improvements over VaDE, this indicates that each component of our model contributes to the final accuracy and that our original intuition that supervision and clustering may reinforce each other is correct.
5.2 PACMAN
In this experiment we test DGC's ability to learn a clustering strategy when facing continuous response as iide-nforrmatio. The Pacman-shaped data consists of two annuli and each point in the two annuli is associated with continuous response value (see the Appendix for detailed breakdown). These response values decrease linearly (from to 0) in one direction for the inner (yellow) annulus, and increase exponentially (from 0 to 1) in the opposite direction for the outer

---

Under review as a conference paper at ICLR 2021
(purple) annulus. Figure 3a contains 3D illustration of the dataset. We linear/exponential rates for the responses to not only test our model's 's ability to detect different trends, but also to test its ability to fit different rates.
Our goal is to separate the two annuli depicted in Fig. 3a This is challenging as the annuli were delib- erately chosen to be very close to each other. We ap- plied various traditional unsupervised learning meth- ods including K-means and hierarchical clustering to only the 2D nneman-shaper data (i.e., not using the responses, but only the 2D Cartesian coordinates). Besides hierarchical clustering with single linkage (and not other distance metric), none of the unsuper- vised methods managed to separate the two annuli.
Truth
Samples
10
1.0
-1.0
4.0
0.0
0,9
0.0
1.0
-1.5
0.0
-1.0
(a) Truth
(b) Samples
Moreover, these approaches also result in different clustering strategies as they are based on different distance metrics (see the Appendix for these results). This phenomenon echos a deep-rooted obstacle for clustering methods in general: the concept of clustering is inherently subjective, and different distance metrics can potentially produce different, but sometimes equally meaningful, clustering results.
Figure 3: (a) The ground truth 3D Pacman; (b) The generated samples from DGC.
Applying DGC with the input coordinates and the responses y as the response values described previously, we are able distinguish the two annulli wholly based on the discriminative information car-
the 2D) Cartesian point
Table 1: Test clustering accuracies on the Pacman dataset
Models
Test Clustering Accuracy
ried by the responses. We parameterize the task networks,
VaDE
50.4% + 0%
{p(ylz,c c k as Gaussian distributions where we learn the means and the covariance matrices. the gen-
NN-DGC
81.6% 1 5.3%
AUG-SS
82.3% = 4.6%
erated samples from Fig. shows, both the Pacman shape and its corresponding response values are captured.
DGC
93.5% = 3.9%
The generated samples from DGC substantiate the model's ability appropriately learn and use the side-information provided by the response values to obtain a sensible clustering strategy. Unlike most previously discussed methods, DGC work with continuous response values. This is highly attractive, as lends itself to any general regression setting in which one would believe the desired clustering strategy should be informed by the regression task.
Finally, we compare DGC to VaDE, its ablated version, and a baseline method to substantiate the efficacy of our proposed framework. First, although the solution to the convex programming in Eq. 4 provides an optimal choice of qcclz.y) y) from a theoretical standpoint, our proposed framework, specifically the proposed LELBO (Bq. (2. holds for any choice of q(clz,)). We thus ablate the convex programming component of our model and parameterize y())z,) using neural network (NN-DGC). Second, by choosing z ~ (O, I). the unsupervised part DGC recovers exactly the cupesuuee (SS) approach introduced by Kingma et al. (2014) in the case when all labels are missing. Since SS is not expected to perform well in a purely unsupervised setting, we include the probabilistic ensemble component as augmentation (AUG-SS). The results described in Tab Tare obtained from running each model 100 times, and demonstrate the following: 1) without the additional responses y. VaDE cannot distinguish between the two annuli at all, emphasizing the importance of exploiting the additional information; 2) the convex programming (Eq 4 is crucial to the success of DGC and is difficult for neural network to find the same optimal distribution; 3) the choice of the prior the latent code z is also paramount importance, and the Gaussian mixture distribution is more suitable for modeling clusters than isotropic Gaussian.
5.3 SVHN
We apply DGC to the Street View House Number (SVHN) dataset (Netzer et al. 2011) where the digit labels (10 digits in total) are used as the ground truth clustering labels. This dataset consists of 73,257 training images, 26,032 test images, and 531,131 additional training images. We train DGC using all the training and extra images. We parameterize the task networks, {p(ylz,c c = { as multinomial distributions over the 10 digits where we learn the parameters those distributions.
7

---

Under review as a conference paper at ICLR 2021
68
67
Curnuleivv Incidence
Clattra
Curnuletve Incidinnee of Rapurence
Clattra
of Rasurence
Clustor Chadler
Cluck'
Chateer
Ckullo
0014
Time (Years)
Time (Years)
(a) The Kaplan-Meier curves for DGC
(b) The Kaplan-Meier curves for VaDE
Figure 4: The Kaplan-Meier risk differences curves among clusters from DGC and VaDE.
The goal of this experiment is inves- tigate the impact the hyperparameter, the number of clusters desired, on the
Table 2: Clustering (and classification) accuracy variation over maximum possible clusters.
overall framework. Ideally one would hope that model would effectively ig-
Clusters
DGC
K-means
VaDE
AUG-SS
10
92.6 (94.7)
88.0
28.7
78,8 (80.2)
nore additional clusters when the num-
20
91.9 (92.2)
56.1
16.4
76.2 (71.3)
ber of chosen clusters is larger than the number of ground truth clusters. Instead, should automatically determine how many clusters are appropriate during learning. which is usually unknown a priori. To test if DGC has this property, we alter the desired number of clusters. We use the class labels as the responses to help the clustering. It is worth noting that while in general one will not use ground-trutt truth labels as iad-iinformation, we use them for this experiment as they provide a good idea of what a reasonable number of clusters in latent space should be.
50
87.9 (91.3)
27.3
14.7
74.3 (68,2)
100
84.1 (89.3)
17.0
9.3
71.3 (64.9)
Table demonstrates the clustering and classification accuracies obtained from DGC and various base- lines over varying number of underlying clusters. The results indicate that DGC most successfully groups the data into the proper classes and clusters regardless of the number of clusters available. Firstly, note that DGC outperforms VaDE and AUG-SS by significant margins, demonstrating the importance of the added side-information and the Gaussian mixture prior assumption, respectively. Additionally, although we use the ground truth labels as the side-information, DGC outperforms the K A-means baseline, where we perform the K- K-means clustering on the features obtained from the last hidden layer of a classification network that trained on the SVHN dataset (with a 95.7% classifi- cation accuracy). This demonstrates the advantages of the latent space sharing and the end-to-end nature of DGC. Secondly, even when the desired number of clusters is larger than the number of digits, network may still achieve clustering accuracy of 100% if it Iearns to group samples into a consistent set of clusters, with the cardinality of that set matching the number of digits. This is echoed by the clustering accuracies shown in Tab. 2 When the desired number of clusters exceeds 10, DGC is still able to achieve high clustering accuracies. By comparison, the clustering accuracy drops dramatically when the number of desired clusters increases for all baselines, demonstrating DGC's ability to choose an appropriate number of clusters.
5.4 CAROLINA BREAST CANCER STUDY (CBCS)
In this experiment we apply DGC to real-world breast cancer dataset collected as part of the Carolina Breast Cancer Study (CBCS). The dataset consists of 1,713 patients, each of which has 2-4 associated histopathological images and a list of biological markers. e.g., the Pam50 gene expressions (Troester ct al. 2018 and ER status.
As an exploratory investigation, we use the binary indicator for breast cancer recurrence as the response variable y. Applying deep learning techniques, supervised or unsupervised, to analyze histopathological images of breast cancer has gained traction in recent years (Xie et al. 2019). Distin- guished from those methods, our goal is inspect whether the discovered clusters, whose formation is influenced both by the supervised recurrence information and the unsupervised reconstruction signal, carry meaningful information in terms of survival rate or gene expression. We parameterize
8

---

Under review as a conference paper at ICLR 2021
the task networks, (plylz, - as Bernoulli distributions and learn the associated parameters. See the Appendix for experimental details.
To investigate whether the three clusters that we discovered were identifying meaningful differences in tumor biology, we examine the differences in rates cancer recurrence and features of tumor aggressiveness between the clusters. We also compare to the baseline clusters obtained from the purely unsupervised VaDE to corroborate the importance of the added indeinfformatio..
Using Kaplan-Meier estimator to estimate risk differ- ences for time to cancer recurrence within three years, we obtained p-value of 0.0024 and observed that Cluster had the lowest risk of recurrence and Cluster had the highest risk (see Tab. B). Even with the small sample size, we observed substantial differences in recurrence risk at three years of follow-up between the clusters, particularly Clusters 0 and 2 (see Fig. 4ap. By comparison, the differences in recurrence risk between the clusters from VaDE is much less significant, both visually (see Fig. 4b and in terms of p-value (0.073).
Table 3: The risk of recurrence differ- ence (RRD) between clusters for DGC.
Comparsion
RRD (95% CT) 16.3% (-6, 39)
Cluster VS Cluster I
Cluster O VS Cluster 2
30.0% (5, 55)
where two clusters almost overlap)
Comparing tumor characteristies, we observed that Cluster O contained more indolent tumors, characterized features such as estrogen-receptor (ER) positivity, low
Table 4: Tumor characteristics for each cluster. Features are color-coded as low intermediate or high risk.
by
Cluster 0 Cluster I Cluster 2 N(%)
N(%)
N(%)
grade, and Luminal A tumor subtype (see Tab. 4. In contrast, more aggres- sive tumor characteristics were fea- tured in Clusters 1 and 2, such as neg- ative ER status, high grade, and Basal- like tumor subtype, although Cluster I appeared to be intermediate between Cluster O and 2 in some characteris- ties. Coupled with the differences in cancer outcomes, these differences in tumor characteristics indicate that the method successfully distinguished between tumors with low-risk features (Cluster 0) and tumors with intermediate- and high-risk features (Clusters and 2). We include the same table that characterizes tumor characteristic for clusters obtained from VaDE in the Appendix. As one can see, cluster O from VaDE, which has the highest recurrence rate, should have the most negative ER subtype, the most high grade, and the most Basal-like tumor subtype. As for grade, it high grade patients. For ER status and tumor subtype, it does have the highest negative ER subtype and the most Basal-like tumor subtype, but the differences are much less significant compared to the clusters obtained from DGC. This indicates that using recurrence side-infornation, via our DGC approach, indeed resulted more meaningful clusters.
Positive Negative
15 (78.9)
53 (56.4)
42 (57.5)
ER Status
4 (21.1)
41 (43.6)
31 (42.5)
Low
7 (36.8)
14 (14.9)
4 (5.5)
Grade
Medium
5 (26.3)
25 (26.6)
25 (32.4)
High
7 (36.8)
55 (58.5)
44 (60.3)
Luminal A
10 (55.6)
27 (29.0)
16 (21.9)
Luminal
6 (33.3)
17 (18.3)
19 (26.0)
Tumor Subtype
ER-/HER2+
(5.5)
(8.6)
(4.1)
Basal-like
(5.5)
41 (44.1)
35 (47.9)
the cluster with the most
CONCLUSION
In this work, we introduced DGC, probabilistic framework that allows for the integration of both supervised and unsupervised information when searching for a congruous clustering in the latent space. This an extremely relevant, but daunting task, where previous attempts are either largely restricted to discrete, supervised, ground-truth, labels or rely heavily on the side-information being provided as manually tuned constraints. To the best of our knowledge, this is the first attempt to simultaneously learn from generally indirect, but informative inde-information and form sensible clustering strategy, all the while making minimal assumptions on either the form of the supervision or the relationship between the supervision and the clusters. This method is applicable to variety of fields where an instance's 's input and task are defined but its membership is important and unknown, e.g.. survival analysis. Training the model in end-to-end fashion, demonstrate that DGC is capable of capturing clustering that aligns with the provided information, while obtaining reasonable classification results various datasets.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Dzmitry Bahdanau. Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. CoRR, 1199.4494.77 2015.
Eric Bair. Semi-superrised clustering methods. Wiley interdisciplinary reviews. Computational stafistics, 5 5:349-361, 2013.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. In ECCV, 2018.
Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien. Semi-supervised learning. 2006.
Ines Färber, Stephan Günnemann, Hans-Peter Kriegel, Peer Kröger, Emmanuel Müller, Erich Schu- bert, Thomas Seidl, Arthur Zimek, and -aai-aaiiaillla-nnnniissiiiiii Muenchen. On using class-labels evaluation of clusterings. 2010.
Thomas Finley and Thorsten Joachims. Supervised clustering with support vector machines. In ICML '05, 2005.
Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embedding: An unsupervised and generative approach to clustering. In IJCAI, 2017.
Xin Jin, Jieho Luo, Jie Yu, Gang Wang, Dhiraj Joshi, and Jiawei Han. Reinforced similarity integration in mmage-rich information networks. IEEE Transactions on Knowledge and Data Engineering, 25: 448-4601 2013.
Daniel Khashabi, Jeffrey Yufei Liu, John Wieting, and Feng Liang, Clustering with side information: From a probabilistic model deterministic algorithm. ArXiv, abs/1508.066 06235, 2015.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. CoRR, abs/1312.6114,, 2014.
Diederik P. Kingma. Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling, Semi Semi-supervised learning with deep generative models. In NIPS, 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolu- tional neural networks. In NIPS, 2012.
Mare T. Law, Raquel Urtasun, and Richard S. Zemel. Deep spectral clustering learning. In ICML, 2017.
Lei Le, Andrew Patterson, and Martha White. Supervised autoencoders: Improving generalization performance with unsupervised regularizers. In S. Bengio, H. Wal- lach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances
in Neural Information Processing Systems, volume 31, pp. 107-117. Curran Asso- ciates, Inc., 2018. URL https://proceecing: neurips httppps.cccppppe ec/pappex/0.8/ELee 2a38a4a931 6c4 1ec495583335 o7w .ddf]
Arya Mazumdar and Barna Saha. Query complexity of clustering with side information, 2017.
Ali Bou Nassif, Ismail Shahin, Imtinan Attili, Mohammad Azzeh, and Khaled Shaalan. Speech recognition using deep neural networks: A systematic review. IEEE Access, 7:19143-19165, 2019.
Yuval Netzer, Tiejie Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, and Andrew Y. Ng.
Reading digits in natural images with unsupervised feature learning. 2011.
Peter Orbanz and Joachim M. Buhmann. Nonparametric bayesian image segmentation. International Journal of Computer Vision, 77:25-45, 2007.
Tom Ronan, Zhijic Qi, and Kristen M Naegle. Avoiding common pitfalls when clustering biological data. Science Signaling, 9-re6re66, 2016.
Emanuele Sansone, Andrea Passerini, and Francesco Natale. Classtering: Joint classification and clustering with mixture of factor analysers. In ECAI, 2016.
10

---

Under review as a conference paper at ICLR 2021
Uri Shaham, Kelly Stanton, Haochao Li, Boaz Nadler, Ronen Basri, and Yuval Kluger. Spectralnet: Spectral clustering using deep neural networks. ArXiv, abs/1801.01587, 2018.
M. Troester, Xuezheng Sun, Emma H. Allott, J. Geradts, S. Cohen, Chiu-Kit J. Tse, Erin L. Kirk, L. Thorne, M. Mathews, Y. Li, Z. Hu, W. Robinson, K. Hoadley, O. Olopade, K. Reeder-Hayes, H. S. Earp, A. Olshan, L. Carey, and C. Perou. Racial differences in pam50 subtypes in the carolina breast cancer study. JNCI: Journal of the National Cancer Institute. 110, 2018.
Viet- Vu Vu, Quan Do, Vu-Tuan Dang, and Do Toan. An efficient censity-baseed clustering with side information and active learning: A case study for facial expression recognition task. Intelligent Data Analysis, 23:227-240, 02 2019, doi: 10.3233/IDA- 173781.
Kiri Wagstaff and Claire Cardie. Clustering with instance-level constraints. In AAAI/AAI, 2000..
Xiang Wang and Ian Davidson. Flexible constrained spectral clustering. In KDD '10, 2010.
Mohammed Wasid and Rashid Ali. Fuzzy side information clustering- based framework for effective recommendations. Computing and Informatics, 38:597-620, 2019. doi: 10.31577/cai 2019.3. 597.
Juanying Xie, R. Liu, Joseph Luttrell, and Chaoyang Zhang. Deep learning based analysis of histopathological images of breast cancer. Frontiers in Genetics, 10, 2019.
Junyuan Xie, Ross B. Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In ICML, 2016.
Zhongbin Xie and Shuai Ma. Dual-view variational autoencoders for semi-supervised text matching. In Proceedings of the Eventy-Eightl International Joint Conference on Artificial Intelligence, IJCAI-19, pp. 5306-5312. International Joint Conferences on Artificial Intelligence Organization, 2019. doi: 10. 10.24963/jccii221177777 737
2019/
Eric P. Xing, Andrew Y. Ng. Michael I. Jordan, and Stuart J. Russell, Distance metric learning with application to clustering with side-information. In NIPS, 2002.


Reviews:
[{"review_id": "l_olhjYgRKq", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method called DGC (Deep Goal-Oriented Clustering) for clustering using side information in the form of a response veriable y. Therefore the objective is to cluster data objects x given a response variable y for each x, that may not be explicitely related to the cluster structure.\n\nThe approach is based on probabilistic modeling and constitutes an extension of VaDe with the addition of the response variable y. It performs end-to-end clustering while simultaneously learning from the available side information.\n\nThere are major issues to be addressed.\n1) It is a strict requirement that a response y should be available for each data point x. Typically in constrained clustering only a small set of constraints is availabe.\n2) If y is available for each x, then we could perform typical clustering using the augmented vectors (x,y). This is a trivial baseline for comparison.  \n3) In experiments with the SVHN dataset, ground truth labels are used as side information. In such a case the problem can be easily solved by training a neural classifier that for each x predicts the class y. If we wish to cluster x, the last hidden layer embeddings could be used for clustering.\n4) In eq. (5) and eq. (8) the formulas for computing q(c=k|x) do not involve x, but rather involve z. This inconcistency should be fixed or clarified.\n5) The discussion beyond eq. (5) is not clear.\n6) It is not clear how to compute eq. (8) since y is not available. \n7) A pseudocode of the proposed algorithm should be provided in order for several details to be clarified.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "hUcH1NQetSA", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary:**\n\nThis paper introduces Deep Goal Oriented Clustering, an approach for joint clustering and classification. The approach shares a latent embedding for the data between the two tasks. The latent embedding is parameterized by a mixture of Gaussians. The approach gives a probabilistic, VAE-based formulation and derives the variational lower bound for the model. The authors run experiments investigating the effectiveness of their approach and the impact of the clustering-component of their approach. \n\nSummary of Review: Clearly explained, intuitive approach for clustering+prediction, concerns over technical and experimental depth.\n\n**Strengths:** \n\nThe presentation of the paper is quite clear. The authors take care to explain their model intuitively and provide with the readers guidance In both high level formulation as well as details of the technical approach and the experiments performed. The model presented is simple and intuitive. The claims of the paper are generally well supported with the explained experiments. The authors provide experiments both on synthetic data as well as data from an ongoing breast cancer study. \n\n**Weaknesses:**\n\n**Technical Novelty**: Sharing latent space representations for clustering and classification in these VAE-based models is not, as I understand, particularly novel. There exist previous works that present quite similar approaches. For instance Le et al, (2018) uses a very similar architecture as do Xie and Ma (2019). The idea of exploring the \"cluster\" assumption and using a mixture model latent variable is quite interesting and distinct from previous work, but I feel that the current form of this contribution is somewhat lacking in terms of analytical and experimental depth. \n\n**Experimental Results**: I have some additional concerns over the level of competition compared to state-of-the-art in the empirical results of the paper. My understanding is that the approach is not state-of-the-art compared to other methods on the SVHN dataset (as compared to the error rates listed on this leader board [1]). My understanding is that noisy MNIST is rather limited in scope and in its scale (in number of classes). Unless I have missed something, DGC is not compared to other methods on the breast cancer data? \n\n**Comparison to Semi-Supervised VAE (M2)**: The set up of the proposed DGC graphical model is closely related to but distinct from that of the semi-supervised VAE. The authors discuss their relationship with respect to the cluster assumption, but I was eager to see additional insight as to why (or when) one model's dependencies for Y would be advantageous.  \n\n\n[Le et al, 2018] Lei Le, Andrew Patterson and Martha White. Supervised autoencoders: Improving generalization performance with unsupervised regularizers. NeurIPS 2018. https://papers.nips.cc/paper/7296-supervised-autoencoders-improving-generalization-performance-with-unsupervised-regularizers.pdf\n\n[Xie and Ma, 2019] Zhongbin Xie and Shuai Ma. Dual-View Variational Autoencoders for Semi-Supervised Text Matching https://www.ijcai.org/Proceedings/2019/0737.pdf \n\n[1] https://paperswithcode.com/sota/image-classification-on-svhn \n\n**Questions for the authors**\n\n* What is the hierarchical clustering method used on the Pacman data? It seems like something like Single Linkage may  be able to correctly recover this data? \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "eQn6BbOZ97W", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I find the problem setup a bit superficial and I was not convinced from the examples in the experiment section  that this is a meaningful real world situation.\n\nWhat is the parametric modeling of  p(y|z,c)?\n\nWhat is the parametric modeling of q(y|x)? Is it actually q(y|\\mu_z) where mu_z = E(z|x)?\n\nIn (6) do you have a tunable regularization coefficient for the regularization term?\n\nThe notation in (5) is odd. z appears on the right side of the equation but not on the left side.\n\nIn the last experiment (medical data) you show clustering results  but their is no reference in the results to the recurred binary indicator. We dont know whether  this information improved the clustering.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2qCNmn_Sj4R", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n \nIn traditional clustering algorithms, incorporation of “side-information”, or additional features only available during training time, typically assume some prior knowledge of the ground-truth clusters, or constraints on those clusters. However, this need not be the case, as training samples may contain arbitrary information which only indirectly corresponds to true cluster labels. This work proposes a novel method, called Deep Goal-Oriented Clustering (DGC), to incorporate such arbitrary “side-information” into a probabilistic auto-encoder based clustering algorithm.\n \nThe basis of this model is a traditional deep clustering algorithm, Variational Deep Embeddings (VaDE) - A deep auto-encoder that models the latent space with a Gaussian Mixture Model prior. The authors augment this fully unsupervised model to be able to additionally generate side information based on the latent gaussian class, and the latent code. This formulation allows the model to incorporate the information of $y$ into the posterior distribution over classes, but does not require any type of correspondence between $y$ and the clusters that will be generated.\n \nThe authors provide a description of a mean-field variational approximation to maximize the likelihood of the training samples containing $x$ and $y$. They show an analytical solution to find $q(c | x)$ given x and y, and provide intuitions backing this solution. Finally, they show a novel formulation to take advantage of the conditional likelihood $p(y | z, c)$ and incorporate it into the posterior $q(c | x)$, when $y$ is not available during test-time.\n \nThe authors corroborate their methods on 4 separate tasks, in total.\nThe first is an augmented MNIST dataset, which consists of 4 general classes of images: images of 2 and 7, with and without noisy background. During training, the ground-truth digit is provided as side-information, but no information is provided about the background information. The results demonstrate that DGC can leverage this additional side-information to vastly outperform VaDE, which cannot incorporate such additional information.\nThe second task is a synthetic PACMAN dataset, consisting of 2 very closely placed 2-D annuli with a 3rd response feature which is generated via either linear or exponential rates. Here, the input is the 2-D coordinates of a point in either annuli, and the side-information is the continuous response feature. The authors demonstrate that, without incorporating the side information, correctly clustering points from either annuli is difficult for any clustering algorithm. However, including the 3rd response feature into DCG allows the model to correctly separate and model both annuli.\nThe third dataset is the Street View House Number dataset, another digit recognition dataset. The authors use this dataset to explore the effect of changing the number of clusters, despite having a fixed number of ground-truth clusters, for which labels are provided as side-information.\nFinally, the authors examine the Carolina Breast Cancer Study, for which the demonstrate that DCG is able to utilize the side-information of whether or not a cancer recurred to separate instances of patients into 3 different, meaningful risk categories.\n \n### Quality\n\nThe motivation and methods are explained well and the experiments corroborate the usefulness of their method. However, some of the experiments could benefit from more salient baseline comparisons. For example, both the SVHN and CBCS experiments would benefit if the results from DCG were compared again to VaDE to demonstrate the effect of incorporating side-information. This is especially true for CBCS as the authors argue early in the work that DCG should improve the ambiguity of the learned clusters; without the VaDE baseline, improvement of ambiguity cannot be demonstrated. Additionally, for a dataset like SVHN where the side-information is the ground-truth labels, it would be useful to compare to a semi-supervised baseline, such as Kingma et. al, 2014, to demonstrate how DCG performs compared to models which make stronger assumptions about the side-information when those stronger assumptions are valid.\n \n### Clarity\n\nOverall the paper is clear and well-written. The motivation would benefit from a few real-world examples, to help distinguish the case where side-information fits under the cluster assumption and where it does not, but otherwise the motivation for the method is well-explained. The description and background of the model is precise and easy to understand. The only section I find lacking in terms of clarity is the section on SVHN - there is little description of the dataset itself, what the classes are, how classification is computed, or why the hyper-parameter of the number of clusters is an interesting hyper-parameter to ablate in this task. Also, some experimental details that are in the appendix should be moved to the main text, such as how significance is assessed and how validation was performed, which are important to interpret the results.\n \n### Originality \n\nOverall, the method is a somewhat simple extension of VaDE. However, the extension is a novel exploration of incorporating side-information into a probabilistic model, and the emphasis on making as few assumptions as possible about the side-informations’ relevance to the true cluster labels is interesting. The following works were not cited but perhaps should be:\n \n- Density-based clustering with side-information and active learning, Vu and Do, 2017: incorporate both constraint information and cluster labels jointly as side-information.\n \n- Query Complexity of  Clustering with Side-Information, Mazumda and Sahar, 2017: Prove some theoretical bounds on the complexity of clustering when you can ask an oracle for side-information (as constraints) on a pair of samples at a time.\n \n- Fuzzy Side Information Clustering-Based Framework for Effective Recommendations, Wasid and Ali, 2019. Here, incorporating side-information increases the complexity of the clustering model too much, so they instead represent this side-information with fuzzy sets, to make the algorithm more efficient. The side-information is essentially just extra features in this case, so we're not dealing with constraints or cluster labels.\n \n### Significance\n\nThe generality of the model is the strongest selling point, demonstrated in its ability to incorporate many different and flexible types of side-information. However, the significance of the paper would be improved if the authors could provide more salient baselines and comparisons in the tasks to show that the motivation for their model holds up in real-world tasks (most baseline comparisons and ablations are performed on the 2 synthetic tasks). Specifically, the paper could benefit from providing more evidence that (a) it’s generality and lack of assumptions do not significantly hurt it when stronger assumptions may be made and (b) that the incorporation of side-information in this way can actually lead to less ambiguous clusters than models which cannot incorporate such side-information.\n\n### Pros / Cons:\n\n* (+) Paper is well-written\n* (+) The main contribution, DGC, appears to be novel\n* (+) DGC is well motivated and technically principled\n* (+) Interesting set of experiments, with a variety of tasks\n* (-) Some relevant work not cited\n* (-) Meaningful baselines only provided for some of the tasks", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new clustering method that takes into account side information.  The paper was reviewed by four expert reviewers who expressed concerns for novelty, empirical and theoretical depth, and unclear parts of the paper. The authors are encouraged to continue research, taking into consideration the detailed comments provided by the reviewers.
Decision: Reject