Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a contrastive hierarchical clustering model CoHiClust which attempts to learn a binary tree clustering structure rather than typical flat clustering. 
A pruning strategy is introduced to create a fixed number of leaves by removing a leaf and retraining the model iteratively.
 Experimental analysis performed on typical clustering benchmarks confirms that the produced partitions have high similarity with ground truth classes. 
strength_and_weaknesses: Strength: The paper is written clearly. The topic is somehow interesting.
The paper evaluates on various datasets.

Weakness:

1. The authors claim that the pruning strategy will "Assume that the importance of the cluster is related to the average number of assigned examples. We reduce a leaf with the lowest probability. After removing the leaf, we finetune the whole model using CoHiClust loss. If we want to reduce more leaves, we perform leaf pruning and model retraining iteratively until the requested number of leaves is obtained." This might just take a much longer time to re-train the model, which needs more discussion and comparison on the training efficiency.
2. CC and PIC often show better performance than the proposed CoHiClust with a large margin, which might suggest that the authors look into the deeper reason for the results and see if any suitable hierarchical clustering data can be utilized in the experiments that can justify the efficacy of the method.
clarity,_quality,_novelty_and_reproducibility: With both hierarchical clustering and contrastive learning techniques, it seems straightforward to get the extensions of previous approaches. However, the experiments indicate that they do not work better than baseline approaches. It would have been nice if the authors had also discussed ways in which one or more techniques could be combined to deal with the lower NMI though, or identified more suitable large-scale datasets to test and justify the efficacy of the proposed model. 
summary_of_the_review: Overall it is not difficult to parse and is well-organized. However, the paper’s claims recall more large-scale datasets in the experiment. The pruning stage takes a long time according to the tree depth, which in the worst case is very high.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a hierarchical clustering model with a designed contrastive objective. It decomposes the classification into a series of the decision process as a binary tree. To make the structure more flexible, a pruning strategy is developed. The results on CIFAR-10, STL-10, IN-10, and IN-Dogs show the proposed method still has some performance gap with the SOTA "flat" clustering methods. But, the results of MNIST and FMNIST show the proposed method beats other hierarchical clustering methods. 
strength_and_weaknesses: The reviewer believes the studied direction and problem are very important for deep learning and representation learning. This paper proposed an interesting perspective to model the classification as a binary tree decision process. A proper contrastive objective is proposed to make the framework trainable. Overall the paper is easy to follow.

Extensive results are conducted in MNIST, FMNIST, CIFAR-10, STL-10, IN-10, and IN-Dogs and show the strength of the proposed hierarchical method compared to other hierarchical clustering and the classification gap between the "flat" clustering method, which are good for readers to understand what the proposed method has achieved. However, the qualitative results are a bit limited for people to understand if the approach really can generate proper hierarchies. Not many ablation studies are included.

The reviewer is unsure if Mautz et al is the only related hierarchical clustering proposed in the deep learning era, as mentioned in the introduction. Some related methods could be developed in understanding or utilizing class hierarchy papers, such as [1] Large-Scale Few-Shot Learning: Knowledge Transfer With Class Hierarchy (***not need to cite***). The reviewer feels some application papers in CV and NLP may have developed related methods to hierarchical clustering, including agglomerative clustering. 

The proposed method assumes fixed height, which is a major limitation. Also, the proposed pruning strategy requires fine-tuning, which makes this step pretty ad-hoc because we can also set up a new structure and re-train from scratch. No ablation studies have been conducted to prove the pruning strategy is effective for improving performance or helping find tree hierarchy. 

According to Table 3, there are big gaps between the proposed clustering method and the previous ones, which is ok if the author can prove the proposed clustering alg can help find class hierarchy in large-scale datasets. However, the current experiments are limited to ImageNet-10 and ImageNet-Dogs. Training on ImageNet and comparing standard contrastive learning results can help here.
clarity,_quality,_novelty_and_reproducibility: The clarity is good, and the reproducibility is guaranteed due to the simplicity of the proposed method and the codes attached in the Appendix. The reviewer feels the novelty of the proposed binary tree + contrastive loss is ok for ICLR.
summary_of_the_review: In general, the reviewer likes the direction, and the proposed method can bring some fresh air to the community. However, the reviewer hopes the author can address the abovementioned concerns, especially about the new qualitative results on large-scale datasets and the ablation studies in the pruning strategy.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Please refer to Summary Of The Review.
strength_and_weaknesses: Please refer to Summary Of The Review.
clarity,_quality,_novelty_and_reproducibility: Please refer to Summary Of The Review.
summary_of_the_review: This paper applies the deep neural networks technology to clustering and establishes a novel clustering model which is dubbed CoHiClust. In this model, a contrastive learning method is employed to create the base network (i.e. a binary tree).so that the original data are represented on a higher-level. Then two regularization strategies are used to make the base neural network train with the arbitrary number of leaves. The cluster assignment of each sample is achieved by aggregating binary decisions. Besides, to create a tree with a fixed number of leaves, a pruning step is added. The experiments conducted on large-scale image datasets shows the CoHiClust better clustering performance compared to some existing deep and flat clustering algorithms.

Deep clustering is a research hotspot in machine learning in recent years, so the topic in this paper is up-to-date and meaningful. The work of this paper has a certain degree of innovation, but there are some issues or suggestions that I want to point out.

1) Section 1 lacks a sound analysis of research motivation. Why combine deep neural network with hierarchical clustering? A relevant algorithm DeepECT has been proposed, so why further research on this topic? Why is it important to examine the algorithm to large-scale image datasets?
2) The specific meaning of parameters \beta_1and \beta_2 in Eq. (1) and the basis for their values should be added. Further, in the experimental part there should be an analysis of the impact of their different values on CoHiClust performance.
3) In terms of experimental design, why only large-scale image datasets are used to test the performance of used algorithms? Can the algorithm proposed in this paper be applied to other types of datasets? The key point is that the proposed CoHiClust model does not seem to have a special design for large-scale image data. 
4) What does the abbreviation AE in Table 1 refer to? Classical hierarchical clustering algorithm?
5) When analyzing the clustering hierarchies on page 7, why not compare CoHiClust with the classical hierarchical clustering algorithms or the latest ones? To further evaluate the validity of this paper work, I strongly recommend adding this comparison.
6) The readability of the tables must be improved. For example, the best values can be in bold.
7) I highly recommend to the authors to add open issues and future directions of their work.

To sum up, this paper need more revisions for its presentation and more experimental work.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper focuses on hierarchical clustering which could build a hierarchy of clusters. The proposed method leverage some self-supervised learning techniques to obtain a high-level representation that is used for tree hierarchy building. As the proposed method is a deep model trained in an end-to-end manner, it could be applied to the large-scale dataset for hierarchical clustering. Some qualitative examples show that the proposed method can yield a good tree hierarchy.
strength_and_weaknesses: Strength:
1. By incorporating hierarchical clustering into the deep model, the proposed method could be easily applied to large-scale data.
2. The presentation is easy to follow.

Weaknesses:
1. My major concern is that the novelty is insufficient. It seems that the proposed method only uses some self-supervised methods for representation learning and combines them with some hierarchical clustering techniques. Although there are no specific models for performing hierarchical clustering under the contrastive learning framework, the existing techniques are somehow ready to apply like Contrastive Clustering. The novelty in terms of the solution technique is thus limited.
2. The authors claimed that the proposed method could achieve comparable performance to the SOTA clustering methods, but the results from Table 3 show a large gap between the proposed method and the SOTA baselines.
3. From Table 1 and Table 2, I notice that the DeepECT + Aug could achieve similar performance to the proposed method. So here comes the question: In the proposed method, is the augmentation technique from self-supervised learning the primary reason for boosting the performance? I wonder about the influence of different components of the proposed method on the final results. However, I didn't see any ablation studies about this. 
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly and easy to follow.
summary_of_the_review: This paper proposes a new deep hierarchical clustering model for large-scale data. The paper is well-written. However, the contribution and novelty of this paper are limited.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: # Summary
This paper focuses on hierarchical clustering with a designed contrastive objective. The proposed approach attempts to learn a binary tree clustering structure rather than typical flat clustering by decomposing the classification into a series of the decision process. Then two regularization strategies are used to make the base neural network train with the arbitrary number of leaves, and a pruning strategy is introduced to create a fixed number of leaves by removing a leaf and retraining the model iteratively. The experiments conducted on large-scale image datasets shows the CoHiClust better clustering performance compared to some existing deep and flat clustering algorithms.
# Strengths:
- The paper is written clearly and easy to follow.
- The problem is very interesting for representation learning.
- The proposed idea of using the classification as a binary tree decision process is interesting.
- Extensive results in MNIST, FMNIST, CIFAR-10, STL-10, IN-10, and IN-Dogs show the strength of the proposed hierarchical method
# Weaknesses:
The major weaknesses raised by the reviewer are the following:
- Training efficiency due to the training from scratch the model after pruning. 
- The proposed method assumes fixed height, which is a major limitation. 
- Current experiments are limited to ImageNet-10 and ImageNet-Dogs. Training on ImageNet and comparing standard contrastive learning results can help here.
- Issues with the novelty of proposed method that only uses some self-supervised methods for representation learning and combines them with some hierarchical clustering techniques Although there are no specific models for performing hierarchical clustering under the contrastive learning framework.


justification_for_why_not_higher_score: The paper proposes a very interesting approach with an extensive quantitative methods. There are some weaknesses that should be addressed as the reviewer mentioned before being accepted although they have been addressed in the author feedback. The first issue is to evaluate the training efficiency of the proposed method that should depend on the fixed height and the baselines, and compare the training time with the performance. The second issue is the height of the method and the number of classes of the datasets, why all the experiments are conducted with datasets of 10 classes, how does it change with different number of classes. Related with the previous comment, the paper should be trained on ImageNet as a reviewer mentioned. 

I encourage the author to keep working on the paper thanks to its potential, but needs some improvements before being accepted.
Decision: Reject