Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a method for *constrained clustering* (a semi-supervised setting in which pairs of samples are labeled as belonging to the same or to different clusters). The method is based on learning a GMM prior for the latent space of a VAE where the cluster assignments depend on the pairwise constrains. The authors develop a conditional version of the ELBO for the constrained case and show how it can be optimized using SGVB and reparameterization, like in regular VAE. The proposed method (DC-GMM) is shown to achieve best clustering results for six different datasets (e.g. MNIST where 6000 random training pairs where used as pairwise constraints) compared to recent *constrained clustering* methods and to be more robust to label noise and to small amount of supervision. 
main_review: **Originality:** As the authors indicate, the idea of learning a GMM as the prior distribution of the VAE latent space is not by itself new (an additional example of deep clustering with a GMM prior is [1] below). The setting of a GMM latent prior conditional on the pairwise constraints matrix and the proposed optimization method seems to be novel and interesting.  

**Clarity:** The paper is very well written - the motivation is well explained, the setting is clearly described as a graphical model and the method is clearly and accurately explained. 

**Quality:** The proposed method seems to be technically sound. The experiments are thoroughly conducted and support the claims well. Keeping the number of constrains fixed at 6000 seemed a bit strange to me, but the authors have one result (figure 2) showing the affect of varying the number of constraints. 

**Significance:** Overall, the paper seems to be an important contribution to the constrained clustering research.

The fact the that mixture probability distribution $\pi$ is fixed at $1/K$ limits, in my opinion, the usability of the proposed method. The datasets used in the experiments section all have more or less equal cluster probabilities, but in real cases there might be a strong imbalance.

Although, as the authors describe, the size of the supervision is only a small fraction of all possible pairs, it is still significant both practically (manually labeling 6000 pairs) and theoretically - for example, if the pairwise constraints define chains of samples that must reside in the same clusters (e.g. $(x_1, x_2), (x_2, x_3), ...$), then $N$ constrains are sufficient to completely define the clusters. A large number of pairwise constrains, however, seems to be the typical case in the *constrained clustering* literature. 

Minor comments:
- It took me a while to understand why the cluster assignments random variable $c$ in the PGM in figure 1 is not part of the plate (I think because individual assignments are not iid). A short explanation can help.


[1] Multi-Modal Deep Clustering: Unsupervised Partitioning of Images, Shiran and Weinshall 2020

limitations_and_societal_impact: The authors discuss the main limitations of their method. I'm not aware of any potential negative social impact.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper developed a new generative clustering method. Compared with existing methods, this paper incorporates the pairwise prior knowledge, and then derived ELBO as the objective function to learn model parameters. The authors then conduct experiments to verify the performance of the proposed method. 
main_review: This paper developed a new generative clustering method. Compared with existing methods, this paper incorporates the pairwise prior knowledge, and then derived ELBO as the objective function to learn model parameters. The authors then conduct experiments to verify the performance of the proposed method. 

Pros:
1. This paper is well written and easy to follow. 

2. The authors conduct extensive experiments to verify the performance of the proposed paper. 

3. The problem that incorporates the prior knowledge is important.

Cons:
1. The novelty is limited. In fact, incorporating the pairwise prior knowledge is trivial here and it is easy to derive the lower bound by following existing works.  From Eq.(11-12), it can be seen that the difference between this method and existing methods is the weight W. If W=I, then it is exactly VAE.

The authors should clearly state the challenge when incorporating the pairwise prior knowledge. 

2. The prior knowledge in this paper is fixed. Hence, the contribution is incremental. A more practical choice is to assume that W is a random variable. But this paper didn't study this general scenario and therefore the contribution is limited. 

3. This paper missed an important related work [1], which also studied the pairwise prior knowledge.

[1]. Correlated Variational Auto-Encoders

Da Tang, Dawen Liang, Tony Jebara, Nicholas Ruozzi; Proceedings of the 36th International Conference on Machine Learning, PMLR 97:6135-6144 

2. 

limitations_and_societal_impact: Not applicable.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces a new method for constrained clustering with varying degree of certainty. The framework is based on the variational auto-encoder. The numerical experiments are extensive. 
main_review: Strengths. The paper is well and clearly written. The numerical experiments are well conducted. The technical details are provided. 

Weaknesses. Although I think that the contribution is interesting and important, my major concern is the setting of the considered problem. The matrix W that reflects the prior knowledge of the connections between data points and also degree of confidence, is in some sense, the direct indicator on how the points should be clustered. It seems that W turns, in some sense, the problem of clustering into a classification problem, since the information provided by W is very strong and rich. Probably this explains such a good empirical performance? 

My next question is very related to my previous concern. If a human expert who provides W, indicated which points are closely related and which are not, what is the principled difference if the expert provides directly class labels? I would appreciate some examples of real applications showing that the matrix W provides significantly different information from supervised learning.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper provides a new probabilistic constrained clustering algorithm, Deep Conditional Gaussian Mixture Model (DC-GMM).
DC-GMM is based on VaDE or GMM-VAE, but DC-GMM adopts a new prior to incorporate the prior information, constrained clustering. They provide the prior formulation in definition 2, and its corresponding evidence lower bound term in Eq. (11).
DC-GMM shows better performance on MNIST, Fashion, Reuters, STL-10 datasets. Furthermore, they compare their methods and baselines on realistic settings, constrained clustering with noisy labels.
main_review: < strength >
1. Constrained clustering is an important but not well-explored research area. This paper contributes an important research area, constrained clustering.
2. The proposed methods are so simple, and they can be adapted to many models.
3. They provide realistic experiments, realistic settings (noisy labels), and a realistic dataset (Heart Echo).

< weakness >
1. Table1 shows that the DC-GMM performs better than other methods. However, the intuitive description or reason is not well explained.
2. SCDC is one of the most important baselines, but there is no reported value for Fashion, Reuters, and STL-10 in Table 1
3. This paper focuses on probabilistic clustering to utilized the learned representation to the other tasks. However, this paper only performs clustering tasks.

< question >
1. There can be many candidate ways to construct prior p(c|W;\pi).
What are the intuition and the advantage of prior modeling in Eq. (4) instead of other candidate ways? 
2. The value of SCDC in Table1 is taken from the original paper. However, the experiment settings of this paper and the SCDC paper are something different. Is it a fair evaluation? 
3. There are other directions to incorporate relation information into the model [1,2,3,4]. I suggest that the authors include the discussion and experimental comparison. 
4. There are recent works for constrained clustering (even though the methods do not focus on deep learning).
It will be interesting that DC-GMM performs better than the combination of recent advanced VAE + recent advanced constrained clustering algorithm
 

< reference >
1. Xu, Hongteng, et al. "Learning autoencoders with relational regularization." International Conference on Machine Learning. PMLR, 2020.
2. Mylonas, Charilaos, Imad Abdallah, and Eleni Chatzi. "Relational VAE: A Continuous Latent Variable Model for Graph Structured Data." arXiv preprint arXiv:2106.16049 (2021).
3. Bai, Liang, JiYe Liang, and Fuyuan Cao. "Semi-supervised clustering with constraints of different types from multiple information sources." IEEE transactions on pattern analysis and machine intelligence (2020).
4. Tang, Da, et al. "Correlated variational auto-encoders." International Conference on Machine Learning. PMLR, 2019.
limitations_and_societal_impact: The authors adequately addressed the limitations of the paper.
review_title: null

Meta-review: Reviews have many positive comments about the proposed approach to incorporating pairwise constraints into prior work on VAE-based clustering, which allows for fairly simple inference but nevertheless has good empirical performance.  There are some concerns about significance of the results, but agreement that technical details are novel.  Some reviewers raised questions about conceptual comparisons to related work, that were reasonably addressed by the author response; please be sure to include these comparisons in future revisions. 
Decision: Accept (Poster)