Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Summary**
This work proposes CVaDE which is an extension of variational based deep clustering model (VaDE) with additional incorporation of prior clustering preferences as supervision. These priors guide the underlying clustering process towards a user-desirable partitioning of input data. The priors are provided in the form of pairwise constraints indicating which pair of samples belongs to same or different class. Clustering process is modelled using variational Bayes in which the clustering constraints are incorporated into prior probabilities with varying degree of uncertainty. The empirical results shows that in comparison to unconstrained clustering the small amount of pairwise constraints significantly improves clustering performance. Further, it demonstrates CVaDE's robustness to noise, generation capability as well as successful incorporation of different desirable preferences to drive clustering performance towards completely different partitioning.

**Quality**
The paper is well written albeit with numerous typographical error (some of which are listed at the end of this review). Experimental evaluation seems thorough. However, I would like to compare results on complex datasets as well as with large classes (> 10). Complex datasets includes STL-10, YouTube Faces, mini ImageNet etc. Please show efficacy on diverse sets of data covering large variation in number of classes, dimensionality, attributes.

Moreover, clustering being unsupervised (here semi-supervised) one should not (rather cannot) employ different hyper-parameters for different dataset. Under the context of zero ground truth data availability, they should rather be fixed. Table 7 says otherwise.

**Originality**
As mentioned above, CVaDE is extended from VaDE but with prior input constraints. Thus the conditional ELBO loss objective is thus a simple extension of VaDE objective. Apart from this, the prior distribution used for pairwise constraints is adapted from work of Lu & Leen (2004). In summary, the work carries very little novelty.

**Significance**
Constrained clustering has been around for some time in various forms. However, the subtle difference CVaDE brings to the table is how to incorporate them into prior probabilities.

Like VaDE, CVaDE is also clustering cum generative model. Once trained, model can be employed for sampling new data. Due to better training procedure using constraints, the generated samples is bound to be perceptually better. However, the samples are not better than the state of the art conditional generative models such as InfoGANs. 

**Clarity**
1. In eq(2), shouldn't it be $\mu_{z_i}$ instead of $\mu_{x_i}$. Is function $f(z_i; \theta)$ not deterministic ? My understanding is given fixed $\mu_{z_i}$ one can sample as many $x_i$. Same goes for $\sigma_{x_i}$.
2. Figure 5, axis labels are missing.
3. Under experiments, please make clear what are we solving for - $z$ and $c$ ? Have you tried k-means on extracted $z$ post training ? 
4. What is penalty weight ? I did not find any description.
5. Why C-IDEC cannot model different noise levels within same data set ?
6. Where is the derivation for Conditional ELBO formulation ? In appendix I only find solution to C-ELBO not how to derive Eq (5).
7. What is the impact of imbalanced dataset on CVaDE ? I presume apriori this imbalance is not known to the user.
8. Eq (19), is $\mathbb{E}$ different from $E$ ?
9. Eq (19), Eq (20) summation w.r.t. is pulled out. Typo in $W_{ij}$ component.
10. Eq (21), some of the terms are approximated by Monte carlo sampling while others are still taking expectation
11. In Eq (18), If 3rd term is marginalised w.r.t. $q(Z|X)$ then it is technically wrong to apply monte-carlo sample to central line in Eq (21). Remember $\frac{1}{L}$ approximates $q(z_i|x_i)$ which is applicable for 1st, 2nd and 4th terms. Not for all.
12. Eq(12) $\delta_{c_ic_j}$ is missing

review_title: Deep Constrained Clustering

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary.

This paper extends the variational deep embedding VaDE model (a VAE-based clustering method) to integrate pairwise constraints between objects, i.e., must-link and cannot-link. The constraints are integrated a priori as a condition. That is, the prior over the cluster labels is conditioned on the constraints. The whole model, referred to as Constrained VaDE (CVaDE), takes the form of a conditional VAE tailored for constrained clustering. Experiments are curried out on various real-world datasets, and the proposed method is compared to VaDE as well as to recent and classical constrained clustering methods. 

Strengths.

1. The different ideas used in this paper, such as adopting a mixture of Gaussians as a prior over the VAE latent space for clustering, or the specification of the conditional prior over the cluster labels to integrate pairwise constraints, are not new by themselves. However, combining them together within a VAE framework is interesting and has not been investigated before to my knowledge.  

2. The paper is well written, and the proposed method is clearly motivated and described.  

3. Experiments are conducted on various data types.

Weaknesses

1. The authors claim superior performance compared to recent constrained deep clustering models. However, looking at the results of Table 1, the proposed CVaDE and the C-IDEC baseline are tight, and the differences in performance do not appear to be statistically significant in most cases.

2. It does not seem like a lot of efforts have been spent for hyperparameters setting. 

    a. For instance the same encoder-decoder architecture is used for all datasets, including image and text ones, even though the latter exhibit very different characteristics. In particular, the retained 4-layers architecture (500-500-2000-10) is too complex (very prone to overfitting) for a text dataset such REUTERS, which is extremely sparse, i.e., with very few nonzero entries.

    b. There are important differences in the optimization-hyperparameters (e.g., batch size) used to train CVaDE and its building block VaDE. It would be useful to report the performance of VaDE when trained using the same settings as CVaDE.
 
3. Despite being a work on constrained clustering, no results regarding the number of satisfied constraints are reported.

4. The authors claim efficiency, but complexity analysis and training time comparisons are missing.

5. Comparisons with baselines when the number of constraints varies are not reported.

6. For the noisy labels experiment, integrating the noise level “q” in the specification of pairwise confidence level is not fair. In practice, we may not always have access to such information in a context of unsupervised learning.

Additional comments and questions. 

1. For tractability purposes, the cluster proportions are all set to be equal (1/K). It would be useful to investigate the impact of this assumption on datasets exhibiting very unbalanced cluster sizes. One possibility is to preprocess some of the considered datasets to create such case.

2. Performance is assessed using Normalized Mutual Information (NMI) and Accuracy.  I would suggest reporting the Adjusted Rand Index (ARI) as well. The latter metric is particularly suitable in a context of constrained clustering, as it measures the proportion of pairs of objects clustered similarly according to both the predicted and the ground truth partitions. 

3. Have you considered conditioning the variational posterior on the constraint information G?

4. Are you using a held-out test set for evaluation?   

review_title: An interesting VAE-based method for constrained clustering, however there are a number of concerns with experiments and some claims 

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper solves the constrained clustering from a probabilistic perspective in a deep learning framework. In general, this paper suffers from several major problems. I will illustrate my concerns point-by-point.
1. The authors mention that none of the existing work in the deep (constrained) clustering models the data generative process. First, this is not true. For example, Semi-crowdsourced Clustering with Deep Generative Models. Second, the authors should illustrate the benefits of data generative process for constrained clustering. That is the motivation of this paper. Unfortunately, the motivation is not strong and clear.
2. If I understand correctly, Eq. (9) and (10) are the core techniques for the proposed algorithm. Such a penalty is straightforward in   constrained clustering. 
3. In Section 3.4.2, there is another side information, named partition level constraint. The authors might want to explore this as well. This point is not a drawback. Just a suggestion.
4. Some traditional constrained clustering methods with deep VaDE features can be involved for comparisons. 
5. It is better to provide some insights on robustness with noisy side information.
6. How to set alpha? Is there some normalization to make alpha within a small range?
7. It is better to show the performance with different numbers of constraints.
8. I am thinking whether two applications in the experimental section are practical in real-world scenarios. I mean how to obtain the pairwise constraints? If I were the project manager in charge of annotation, I will directly label their categories, rather than providing the pairwise constraints.
review_title: Not good enough

Meta-review: We thank the authors for their detailed responses to reviewers, and for engaging in a constructive discussions.

As explained by the reviewers, the paper is clearly written and the method is novel. However, the novelty is to combine existing ideas and techniques to define an objective function that allows to incorporate cluster assignment constraints, which was considered incremental. Regarding quality, the discussion highlighted some possible improvements that the authors propose to do in a future version of the paper, and we encourage them to follow that direction. Regarding significance, although the experimental results are promising there were some concerns that the improvement over existing techniques is marginal, and that more experiments leading to a clearer message would be useful.

In summary, this is not a bad paper, but it is below the standards of ICLR in its current form.
Decision: Reject