Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary
 
In traditional clustering algorithms, incorporation of “side-information”, or additional features only available during training time, typically assume some prior knowledge of the ground-truth clusters, or constraints on those clusters. However, this need not be the case, as training samples may contain arbitrary information which only indirectly corresponds to true cluster labels. This work proposes a novel method, called Deep Goal-Oriented Clustering (DGC), to incorporate such arbitrary “side-information” into a probabilistic auto-encoder based clustering algorithm.
 
The basis of this model is a traditional deep clustering algorithm, Variational Deep Embeddings (VaDE) - A deep auto-encoder that models the latent space with a Gaussian Mixture Model prior. The authors augment this fully unsupervised model to be able to additionally generate side information based on the latent gaussian class, and the latent code. This formulation allows the model to incorporate the information of $y$ into the posterior distribution over classes, but does not require any type of correspondence between $y$ and the clusters that will be generated.
 
The authors provide a description of a mean-field variational approximation to maximize the likelihood of the training samples containing $x$ and $y$. They show an analytical solution to find $q(c | x)$ given x and y, and provide intuitions backing this solution. Finally, they show a novel formulation to take advantage of the conditional likelihood $p(y | z, c)$ and incorporate it into the posterior $q(c | x)$, when $y$ is not available during test-time.
 
The authors corroborate their methods on 4 separate tasks, in total.
The first is an augmented MNIST dataset, which consists of 4 general classes of images: images of 2 and 7, with and without noisy background. During training, the ground-truth digit is provided as side-information, but no information is provided about the background information. The results demonstrate that DGC can leverage this additional side-information to vastly outperform VaDE, which cannot incorporate such additional information.
The second task is a synthetic PACMAN dataset, consisting of 2 very closely placed 2-D annuli with a 3rd response feature which is generated via either linear or exponential rates. Here, the input is the 2-D coordinates of a point in either annuli, and the side-information is the continuous response feature. The authors demonstrate that, without incorporating the side information, correctly clustering points from either annuli is difficult for any clustering algorithm. However, including the 3rd response feature into DCG allows the model to correctly separate and model both annuli.
The third dataset is the Street View House Number dataset, another digit recognition dataset. The authors use this dataset to explore the effect of changing the number of clusters, despite having a fixed number of ground-truth clusters, for which labels are provided as side-information.
Finally, the authors examine the Carolina Breast Cancer Study, for which the demonstrate that DCG is able to utilize the side-information of whether or not a cancer recurred to separate instances of patients into 3 different, meaningful risk categories.
 
### Quality

The motivation and methods are explained well and the experiments corroborate the usefulness of their method. However, some of the experiments could benefit from more salient baseline comparisons. For example, both the SVHN and CBCS experiments would benefit if the results from DCG were compared again to VaDE to demonstrate the effect of incorporating side-information. This is especially true for CBCS as the authors argue early in the work that DCG should improve the ambiguity of the learned clusters; without the VaDE baseline, improvement of ambiguity cannot be demonstrated. Additionally, for a dataset like SVHN where the side-information is the ground-truth labels, it would be useful to compare to a semi-supervised baseline, such as Kingma et. al, 2014, to demonstrate how DCG performs compared to models which make stronger assumptions about the side-information when those stronger assumptions are valid.
 
### Clarity

Overall the paper is clear and well-written. The motivation would benefit from a few real-world examples, to help distinguish the case where side-information fits under the cluster assumption and where it does not, but otherwise the motivation for the method is well-explained. The description and background of the model is precise and easy to understand. The only section I find lacking in terms of clarity is the section on SVHN - there is little description of the dataset itself, what the classes are, how classification is computed, or why the hyper-parameter of the number of clusters is an interesting hyper-parameter to ablate in this task. Also, some experimental details that are in the appendix should be moved to the main text, such as how significance is assessed and how validation was performed, which are important to interpret the results.
 
### Originality 

Overall, the method is a somewhat simple extension of VaDE. However, the extension is a novel exploration of incorporating side-information into a probabilistic model, and the emphasis on making as few assumptions as possible about the side-informations’ relevance to the true cluster labels is interesting. The following works were not cited but perhaps should be:
 
- Density-based clustering with side-information and active learning, Vu and Do, 2017: incorporate both constraint information and cluster labels jointly as side-information.
 
- Query Complexity of  Clustering with Side-Information, Mazumda and Sahar, 2017: Prove some theoretical bounds on the complexity of clustering when you can ask an oracle for side-information (as constraints) on a pair of samples at a time.
 
- Fuzzy Side Information Clustering-Based Framework for Effective Recommendations, Wasid and Ali, 2019. Here, incorporating side-information increases the complexity of the clustering model too much, so they instead represent this side-information with fuzzy sets, to make the algorithm more efficient. The side-information is essentially just extra features in this case, so we're not dealing with constraints or cluster labels.
 
### Significance

The generality of the model is the strongest selling point, demonstrated in its ability to incorporate many different and flexible types of side-information. However, the significance of the paper would be improved if the authors could provide more salient baselines and comparisons in the tasks to show that the motivation for their model holds up in real-world tasks (most baseline comparisons and ablations are performed on the 2 synthetic tasks). Specifically, the paper could benefit from providing more evidence that (a) it’s generality and lack of assumptions do not significantly hurt it when stronger assumptions may be made and (b) that the incorporation of side-information in this way can actually lead to less ambiguous clusters than models which cannot incorporate such side-information.

### Pros / Cons:

* (+) Paper is well-written
* (+) The main contribution, DGC, appears to be novel
* (+) DGC is well motivated and technically principled
* (+) Interesting set of experiments, with a variety of tasks
* (-) Some relevant work not cited
* (-) Meaningful baselines only provided for some of the tasks
review_title: Nice ideas that would benefit from more experimental validation

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Summary:**

This paper introduces Deep Goal Oriented Clustering, an approach for joint clustering and classification. The approach shares a latent embedding for the data between the two tasks. The latent embedding is parameterized by a mixture of Gaussians. The approach gives a probabilistic, VAE-based formulation and derives the variational lower bound for the model. The authors run experiments investigating the effectiveness of their approach and the impact of the clustering-component of their approach. 

Summary of Review: Clearly explained, intuitive approach for clustering+prediction, concerns over technical and experimental depth.

**Strengths:** 

The presentation of the paper is quite clear. The authors take care to explain their model intuitively and provide with the readers guidance In both high level formulation as well as details of the technical approach and the experiments performed. The model presented is simple and intuitive. The claims of the paper are generally well supported with the explained experiments. The authors provide experiments both on synthetic data as well as data from an ongoing breast cancer study. 

**Weaknesses:**

**Technical Novelty**: Sharing latent space representations for clustering and classification in these VAE-based models is not, as I understand, particularly novel. There exist previous works that present quite similar approaches. For instance Le et al, (2018) uses a very similar architecture as do Xie and Ma (2019). The idea of exploring the "cluster" assumption and using a mixture model latent variable is quite interesting and distinct from previous work, but I feel that the current form of this contribution is somewhat lacking in terms of analytical and experimental depth. 

**Experimental Results**: I have some additional concerns over the level of competition compared to state-of-the-art in the empirical results of the paper. My understanding is that the approach is not state-of-the-art compared to other methods on the SVHN dataset (as compared to the error rates listed on this leader board [1]). My understanding is that noisy MNIST is rather limited in scope and in its scale (in number of classes). Unless I have missed something, DGC is not compared to other methods on the breast cancer data? 

**Comparison to Semi-Supervised VAE (M2)**: The set up of the proposed DGC graphical model is closely related to but distinct from that of the semi-supervised VAE. The authors discuss their relationship with respect to the cluster assumption, but I was eager to see additional insight as to why (or when) one model's dependencies for Y would be advantageous.  


[Le et al, 2018] Lei Le, Andrew Patterson and Martha White. Supervised autoencoders: Improving generalization performance with unsupervised regularizers. NeurIPS 2018. https://papers.nips.cc/paper/7296-supervised-autoencoders-improving-generalization-performance-with-unsupervised-regularizers.pdf

[Xie and Ma, 2019] Zhongbin Xie and Shuai Ma. Dual-View Variational Autoencoders for Semi-Supervised Text Matching https://www.ijcai.org/Proceedings/2019/0737.pdf 

[1] https://paperswithcode.com/sota/image-classification-on-svhn 

**Questions for the authors**

* What is the hierarchical clustering method used on the Pacman data? It seems like something like Single Linkage may  be able to correctly recover this data? 

review_title: Clearly explained, intuitive approach for clustering+prediction, concerns over technical and experimental depth.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes a method called DGC (Deep Goal-Oriented Clustering) for clustering using side information in the form of a response veriable y. Therefore the objective is to cluster data objects x given a response variable y for each x, that may not be explicitely related to the cluster structure.

The approach is based on probabilistic modeling and constitutes an extension of VaDe with the addition of the response variable y. It performs end-to-end clustering while simultaneously learning from the available side information.

There are major issues to be addressed.
1) It is a strict requirement that a response y should be available for each data point x. Typically in constrained clustering only a small set of constraints is availabe.
2) If y is available for each x, then we could perform typical clustering using the augmented vectors (x,y). This is a trivial baseline for comparison.  
3) In experiments with the SVHN dataset, ground truth labels are used as side information. In such a case the problem can be easily solved by training a neural classifier that for each x predicts the class y. If we wish to cluster x, the last hidden layer embeddings could be used for clustering.
4) In eq. (5) and eq. (8) the formulas for computing q(c=k|x) do not involve x, but rather involve z. This inconcistency should be fixed or clarified.
5) The discussion beyond eq. (5) is not clear.
6) It is not clear how to compute eq. (8) since y is not available. 
7) A pseudocode of the proposed algorithm should be provided in order for several details to be clarified.

review_title: A clustering method that uses an additional response variable as side information

final_score: rating: 3: Clear rejection
review_content: I find the problem setup a bit superficial and I was not convinced from the examples in the experiment section  that this is a meaningful real world situation.

What is the parametric modeling of  p(y|z,c)?

What is the parametric modeling of q(y|x)? Is it actually q(y|\mu_z) where mu_z = E(z|x)?

In (6) do you have a tunable regularization coefficient for the regularization term?

The notation in (5) is odd. z appears on the right side of the equation but not on the left side.

In the last experiment (medical data) you show clustering results  but their is no reference in the results to the recurred binary indicator. We dont know whether  this information improved the clustering.

review_title: Not a realistic modeling 

Meta-review: This paper proposes a new clustering method that takes into account side information.  The paper was reviewed by four expert reviewers who expressed concerns for novelty, empirical and theoretical depth, and unclear parts of the paper. The authors are encouraged to continue research, taking into consideration the detailed comments provided by the reviewers.
Decision: Reject