Deep Conditional Gaussian Mixture Model for Constrained Clustering

Abstract: Constrained clustering has gained significant attention in the field of machine learning as it can leverage prior information on a growing amount of only partially labeled data. Following recent advances in deep generative models, we propose a novel framework for constrained clustering that is intuitive, interpretable, and can be trained efficiently in the framework of stochastic gradient variational inference. By explicitly integrating domain knowledge in the form of probabilistic relations, our proposed model (DC-GMM) uncovers the underlying distribution of data conditioned on prior clustering preferences, expressed as \textit{pairwise constraints}. These constraints guide the clustering process towards a desirable partition of the data by indicating which samples should or should not belong to the same cluster. We provide extensive experiments to demonstrate that DC-GMM shows superior clustering performances and robustness compared to state-of-the-art deep constrained clustering methods on a wide range of data sets. We further demonstrate the usefulness of our approach on two challenging real-world applications.


Reviews and author responses:

 We thank the reviewers for their positive feedback! 

We would like to summarise here the main strengths highlighted by the reviewers:
1) The proposed method is considered novel by all reviewers (2jYB, bU1a, 3xdu, JUWY).
2) The paper is well-written (2jYB, bU1a, 3xdu) and technically sound (3xdu, JUWY).
3) Comprehensive experiments have been conducted (2jYB, 3xdu, JUWY).

Few concerns about the general purpose of constrained clustering were raised by reviewer bU1a and some issues on the lack of clarity of the problem setting were raised by reviewer JUWY.
We addressed all concerns individually. Given that all reviewers are positive about the method, including reviewer bU1a, we hope our response could both foster insightful discussions and improve bU1a rating. 

We are thankful for the reviewers’ effort and their detailed feedback.

 I think authors' response generally improves my understanding of the paper. But my general impression of the paper remains unchanged. I will keep my score.

 We thank the reviewer for the detailed feedback. We apologize for the lack of clarity of the problem setting, we will make sure to improve it in the Camera-Ready version. Below we respond to each point raised by the reviewer:
1) δcicj is the Dirac delta function, in particular δcicj=1 if ci=cj and δcicj=0 otherwise.
2) We decided to not fine-tune W for each data set (even though that could result in even higher performance) because of the semi-supervised setting. Instead we set the same hyper-parameters for all data sets to prove the generalisation capabilities of the proposed approach. We observed empirically that a high value of the order of 10^4 is beneficial for the model performance for all datasets if the constraints have a high confidence. Decreasing the value of |Wij| decreases the importance of Eq. 12 in the Elbo (see paragraph “Constrained clustering with noisy labels”, Sect. 4). 
3) All terms of the objective are trained with mini-batches of size B. We will make sure to add this information in section B.3 of the Appendix after the derivation of the SGVB estimator.
4) The reasons we derived Eq. 11 are two-fold: a) all terms and their computational complexity are clearly defined and (b) the  SGVB estimator can be easily applied (see Eq. 29 of the Appendix) to optimise it with stochastic gradient descent.
5) We assumed K is pre-defined (following prior works on variational clustering), and we set it to the true number of clusters for each data set. However, it could also be treated as a hyper-parameter.
6) Clustering accuracy is evaluated by finding an optimal mapping between the predicted cluster assignments (c) and true cluster labels using the Hungarian algorithm, as implemented in sklearn.utils.linear_assignment.
7) The use of prior information W is motivated by the fact that often fully unsupervised clustering algorithms might naturally find a solution that is not consistent with the domain knowledge as explained in Section 1. Additionally, we validated it in Sect. 4 where we compared the performance of DC-GMM with and without prior information (see for example Fig. 5). We will happily include additional examples to clarify the motivations.


 We thank the reviewer for the detailed feedback. Below we respond to each point raised by the reviewer:
1) Why do we need domain expertise in deep learning? Because there are many situations in which purely data-driven approaches might lead to unsatisfactory results. The most common scenario is when not enough data is available to train well-performing and sufficiently generalized models [1]. 
One can also design features: First, feature engineering heavily relies on experts to select the most meaningful features for each data set. Second, even the selected features might contain unknown biases or could discard relevant information from the original data sets. Let’s take the Heart Echo experiments, where the same algorithm could cluster the data by either age or view, depending on the given prior information. How could a doctor select different features for the two problem settings? DC-GMM, on the other hand, automatically computes meaningful features in the latent space of the VAE according to the given prior information.
2) We agree with the reviewer that W scales quadratically with the number of labels, independently of the number of selected constraints. However, it is important to note that the complexity of the algorithm does not scale quadratically with the size of the data set, N, as explained in Section 2.4, but rather with the length of the batch size, which is independent of N. On the other hand, the smaller the ratio the less chance the algorithm will find pairwise constraints inside the considered batch. To prevent this scenario, we group together constrained samples at the end of each epoch and train them in batches. Empirically, we observed that the number of constraints required to reach a certain performance does not depend on the size of the data set, but rather on the complexity of the dataset. As an example, MNIST requires a smaller ratio of constraints than the Face data set. We also tested this hypothesis by training MNIST with different sizes of training points and with a fixed number of constraints. The results showed that increasing the training size actually slightly increased the performances.  
3) Conflict between experts is indeed an important real-world scenario. We derived an heuristic to set the values of |Wi,j| according to the noise level q of the entire set of constraints (see “Constrained clustering with noisy labels”, Section 4). However, the same heuristic could be used in the example provided by the reviewer. In that case the noise level qij will be 0.5 (50 percent probability of flipped sign for that specific constraint), and |Wij| is computed accordingly. In that specific case it will be equal to 0 as there is an equal probability of being both must-link and cannot-link. A less extreme scenario would be the following: 2 experts annotate xi and xj as must-link and one as cannot-link. The given annotation will be a must-link with probability pij= 2/3 and qij= 1-pij. 
4) W does not define a metric on data points as it could output negative values. 
5) It is important to observe that the MoGs assumption is applied on the latent space of the VAE, and not on the raw data. Hence, if the data set contains a certain separability, it is rather probable that there exists a highly non-linear function that could map the distribution of raw data into a MoGs in a reduced space. We proved this empirically in Fig.5.
6) Pairwise constraints have a major advantage over labels especially in the crowdsourcing setting, which is a way of annotating data sets from a large pool of people. Due to the lack of expertise of the crowd, it has been shown that comparing samples is easier than directly labeling them. For example, let’s assume the task is to classify many different species of trees. It might be easier to show a pair of samples to non-experts and ask them whether they look similar or not, rather than to ask the specific type. Additionally, it is always possible to automatically convert labels into pairwise annotations, hence the additional information could also come as a combination of labels and pairwise constraints, depending on the level of expertise of the annotators.  
7) The proposed method can easily scale to millions of data points. The reason is that we use stochastic gradient descent with mini-batches. Hence the computational complexity of an iteration is independent of the size of the dataset (see 2.4).
8) The proposed approach is a latent variable model, and the clustering is performed on the reduced latent space. It follows that it can scale to high-dimensional data sets. However, more computational resources might be needed as the neural network will naturally grow.

[1] Rueden, L. V. et al. “Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems.”

This paper proposes a deep generative model with latent variables drawn from a Gaussian mixture model.  Moreover, the prior over the components is conditioned on prior information that specifies clustering constraints, allowing the user to specify one of three cases: x_i and x_j (1) must be in the same cluster, (2) must not be in the same cluster, or (3) no information.  An ELBO is derived.  Experiments demonstrate clustering performance on a variety of data sets ranging from MNIST to STL-10 to heart ultrasounds.   Originality:  I find the paper novel, as I am aware of no other work that combines deep Gaussian mixture models with clustering constraints

Clarity:  I find the paper to be well-written, clear, and the results well-presented.

Significance:  Given that clustering is a pervasive methodology, there is potential for wide adoption.  However, I don't think that Gaussian mixture DGMs have found the popularity of other DGM types.  Moreover, while the combination is novel, there does not seem to be any outstanding originality or insight (i.e. this paper didn't make me view clustering or DGMs from a new perspective).

Quality:  I find the paper's execution to be well done.  The model is reasonable and practical.  The results use sensible benchmarks and baselines.

 Addressed in Section 5

Authors propose a probabilistic method for representation learning for clustering problem. Authors took a generative perspective and proposed an interesting algorithm for the problem at hand. Authors also evaluated their proposal on a set up problems.  Authors propose a probabilistic method for representation learning for clustering problem. Authors took a generative perspective and proposed an interesting algorithm for the problem at hand. Authors also evaluated their proposal on a set up problems.

The paper is well written and proposed method is novel. Intuitively integrating domain knowledge is making sense. However, I have several concerns about the method and problem setting. I have given my concerns below.
•	Why do we need domain expertise for deep learning? One can also design features.
•	Domain expertise is implemented as a pairwise relation matrix i.e., W \in R^{NxN} where N is the number of data points. To get full domain knowledge, one needs O(N^2) relations. Please note that this is quantity scale quadratically. I am aware that authors don’t claim the need of full information however this will not decrease the complexity. In other words, if one increases the data set size for a given number of relations, the ratio will decrease quadratically. How will authors address this point?
•	Conflict between experts: Authors assume that there exists a panel of experts which can create W \in R^{NxN}. In principle I agree that domain knowledge exits however one needs to address the conflicts between experts. Let’s assume we have two experts and each supplied relations on k points (the same points). By chance they give opposite responses. How will the algorithm handle these cases? 
•	Does W \in R^{NxN} define a metric on data points? If yes, would you please elaborate how to check this? If no, can one use any function for generating W? 
•	Authors assume the existence of Gaussian latent space. How realistic is this assumption for practical problems, i.e., Categorical distributions? 
•	In the experiments, there are 6000 constraints used in all datasets. One can consider labelling data as domain knowledge or constraint. One can ask experts to label 6000 data points and use a semi-supervised method. What is the benefit of current setup compared to semi-supervised methods?
•	Can the proposed algorithm scale to millions of points?
•	Can the proposed algorithm scale to high dimensional data e.g. large resolution images ?
In general, I am positive about the method however I would like to some of my concerns to be addressed.
 Please see above.

The authors propose a novel framework for constrained clustering via a Gaussian Mixture Model on the latent space of a Deep Variational Autoencoder. To impose the constraint information, the proposed paradigm incorporates instance-level clustering preferences via Bayesian prior probabilities with varying degrees of certainty. The authors have derived the Conditional Evidence Lower Bound in variational inference for their problem setup. They have also provided comprehensive experiments compared with state-of-the-art methods DEC, VaDE and IDEC on classic benchmark datasets of MNIST, FMNIST, Reuters and STL-10. They have also explored the performance of their method on a more challenging dataset, which is the Face image dataset.  The authors provide a novel framework for constrained clusterin via deep generative models by incorporating instance-level clustering preferences via Bayesian prior probabilities with varying degrees of certainty. The paper is technically sound and comprehensive experiments have been conducted for multiple classic benchmark datasets for clustering including MNIST, FMNIST, Reuters, STL-10 and more challenging cases such as the face image dataset. The proposed method has outperformed state-of-the-art clustering methods such as DEC, VaDE and IDEC. The paper is also well written and easy to follow.  Yes. The authors have adequately discussed the limitations of their method. 

This paper deals with a constrained clustering problem which differs from the vanilla clustering problem in the extra pre-existing knowledge. We can also regard this problem as clustering with some supervision. The prior information known here is the pairwise relationship among input samples (a small portion). The method introduces a latent categorical variable of corresponding group. Experimental results show superior performance over the other baselines.  *originality*

Designing a mixture prior based on the given pairwise sample relations is pretty novel. The experiments include several real-world datasets as well as standard benchmarks like MNIST. The claims are also supported by proofs or experiments.

*quality*

- some notions like $\delta_{c_ic_j}$ is not explained in the paper
- How $W$ is defined specifically is very important in this method but it is not clarified. I don't quite understand why "the values of |$W_{i,j}$| are set to $10^4$ for all data sets". $W$ is very important in the optimization. How it is defined directly affects the training.
- The objective function involves all the samples. Do you still use mini-batch training? I can see Eq. (12) is adopting the mini-batching. But what about other terms. 
- Eq. (11) is the final objective. I think the authors could give some insights or explanations why the Eq. (8) is decomposed to Eq. (11). The transition is not very smooth. Is it for the convenience of the computation?
- Is K pre-defined? With a fixed K, the problem would generally be easier.
- How is acc calculated? Do you directly use variable $c$?
- Some examples could be given to better motivated and validate the use of $W$.

*clarity*

Constrained clustering problem is not quite the same as the standard clustering problem. Some supervision is known a priori. I think the authors should explain the problem more clearly, stating explicitly which is know and which is not. Besides, the evaluation protocol should be more clear, e.g. which variable is used for prediction and how you obtain the accuracy.

*significance*

The real-world datasets are novel. So the technique might be useful in the healthcare domain. The method is also generally novel, taking advantage of the VAE framework. The authors mentioned some limitations in the "Limitations & Future Work" section. I agree that obtaining $W$ is complicated and might not be straightforward. The paper should improve the clarity of this part.

Besides, the explanation of the objective function should be elaborated. Some insights and training details should be given.

Meta-review: The paper describes a way of incorporating prior information in a deep Gaussian mixture model in the form of pairwise relation constraints.

The reviewers carefully considered the merits of the paper and the response of the authors. One of the primary concerns was around the implementation of domain expertise --- the method requires experts to express all pairwise relations leading to O(N^2) relations. The authors responded that the complexity of the algorithm does not scale quadratically, but instead with the batch size. It would be expected that the computational time of running a quadratic algorithm is dwarfed by the effort require to elucidate all pairwise relations from one or more experts. Therefore, the questions around the practicality of such an approach raised by the reviewer remains. It seems that there many be better data sets and problems for which the method is appropriate. The chosen evaluation data sets MNIST, etc, are very standard and the constrained clustering using expertise on pairwise relations may be better suited to other application domains.

In summary, the remaining concerns about practicality of the method and a demonstrable application domain outweigh the benefits of the novelty of the approach.


Decision: accept