Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a framework to do a few-sample clustering evaluation. The few-sample clustering evaluation refers to the case that only with a few labeled samples, how to do the evaluation on the clustering quality. The framework proposed follows three steps: 1. an acquisition function with NMI (normalized mutual information) is designed; 2. semi-supervised learning method like FixMatch is used to train the surrogate model, which utilizes labeled data and unlabeled data together 3. pseudo-labeling the unlabeled data with the learned surrogate model before estimating the evaluation metric. 
strength_and_weaknesses: Strength:

1. The few-sample clustering evaluation is an interesting topic to explore, which also has practical importance. 
2. The paper is well-written and easy to follow. 
3. The proposed framework seems novel and achieves sound results. The acquisition function with NMI is newly proposed in the paper.

Weakness:

1. How to incorporate FixMatch in training the surrogate model is not clear. In FixMatch, there is a branch of strong augmentation on the input image x. How does this part work, specifically the strong-weak data augmentation branches, as the input of the surrogate model is a feature vector? 

2. The initialization of labeled data is uniformly sampling 50 data points, which doesn't consider the class balance in ground-truth labels. The random sampling could largely be class imbalanced which would affect the training of the surrogate model and consequently affect the final results. It would be helpful if the evaluations were separated into a class-balanced setting and a class-imbalanced setting. 

3. The effectiveness and robustness of the method are not fully evaluated. In table 1, Soft-NMI achieves the best result on CIFAR-10, better than utilizing FixMatch + Pseudo-labeling. Although the method achieves the best result on MNIST by quite a large margin.  on a more complicated CIFAR-10 dataset (same size, same number of classes), there is no advantage of utilizing FixMatch + Pseudo-labelling. Meanwhile, using FixMatch has the large risk of introducing wrong pseudo-labels for those unlabeled data and the training from scratch is always time-consuming, which would be emphasized as the dataset carries larger scope. This brings the question that: is it worth introducing the FixMatch+pseudo-labeling on larger evaluation datasets with more classes and complicated data distribution? Currently even on CIFAR-10, there is no guarantee that FixMatch or semi-supervised learning in this framework would benefit. 
clarity,_quality,_novelty_and_reproducibility: The overall clarity and quality of the paper are good, well, and clearly written; experiments are clearly discussed.
The proposed framework and acquisition function with NMI ensures the work having certain novelty. 
summary_of_the_review: My current decision on the paper is based on the weaknesses mentioned above, and it would be really helpful if the authors could provide further discussions.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes an evaluation method to estimate the clustering quality with a small number of labeled samples. The samples are selected by two novel acquisition functions.
strength_and_weaknesses: Strength
1. This paper studies an interesting and important problem, which is the efficient evaluation for unsupervised methods. 
2. The paper uses adequate experimental results to support the effectiveness of the method. 
Weakness:
1.The novelty of the paper is limited. The two proposed acquisition functions are simple and straightforward extensions of NMI. 
2. Section 4 which introduces the core methodology is very brief and it is not very clear how the proposed method addresses the clustering evaluation challenges.
3.The proposed approach is purely intuitive. It is hard to understand how and why it would work in general. 

clarity,_quality,_novelty_and_reproducibility: The paper is well written but the core methodology is unclear and very briefly described. \
The proposed method does not show much novelty.The proposed acquisition functions are small modifications from the previous work.The 
The experiment seems to show that the proposed method is effective compared with other baselines.

summary_of_the_review: The paper studies a meaningful problem. But the proposed method lacks novelty, a thorough discussion of the core methodology design, and theoretical underpinning.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a method to evaluate the clustering quality when in semi-supervised contexts where the number of labeled examples is limited. 
strength_and_weaknesses: The main strength of the paper is the simplicity of the approach. The proposed method (described in Algorithm 1) which consists in iteratively selecting labels to samples and assign pseudo-labels to unlabelled samples is simple to understand.

The paper contains lots of weaknesses. The first major weakness is the lack of motivation for the proposed method. In what real-world scenario would this evaluation approach be useful to the machine learning community? The evaluation approach might be interesting to the data mining or information retrieval communities but I do not know in what contexts. 

The second major weakness is the way the experiments are performed. The chosen datasets are toy datasets (MNIST, CIFAR-10, Newsgroup datasets) and the models are pretrained (on ImageNet) or via self-supervised learning. It is not surprising to see pseudo-label approaches work well on these datasets since classes are easily separable.
clarity,_quality,_novelty_and_reproducibility: The explanation of the method is clear, but there is a lack of motivation for the proposed evaluation metric and the choice of experiments on toy datasets is poor.
summary_of_the_review: See the paragraph above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers the problem of evaluating the quality of a proposed clustering. The traditional approach (computing normalized mutual information, NMI) requires every sample to have a ground-truth cluster label. The goal of this paper is to approximate NMI with a limited number of ground-truth cluster labels. The proposed approach is to adapt ideas from active learning and semi-supervised learning to the clustering context. The paper performs experiments on 3 datasets (MNIST, CIFAR-10, Newsgroup) and compares a number of baselines against variants of the proposed approach. 
strength_and_weaknesses: # Strengths
* The paper is well-written and polished. 
* To the best of my knowledge, the proposed method is novel and fills a gap in the literature (evaluating clustering methods with few labels). 
* The ablation studies / comparisons in Table 1 are quite thorough. 
* The proposed method seems to perform well in the scenarios where it has been tested. 

# Weaknesses
* On page 2 there's talk of "experiments across multiple real-world datasets" but the paper uses MNIST, CIFAR-10, and Newsgroup, which I wouldn't consider "real-world" datasets. I found myself wondering why these datasets were used in the paper? Is the reason because these methods don't scale up well? If so that's fine, but it should be clearly identified as a limitation in the text. 
* This is related to the previous point. As I reader, I had unanswered questions about the computational costs involved here. (1) How many GPU-hours does it take the run this algorithm from start to finish, and how does that scale with dataset size? (2) If we convert those GPU-hours to an amount of money, how does that cost compare the cost of just e.g. labeling the images on Mechanical Turk? The basic question is: "When is the algorithm worth it?" If the answer is "never, at least not with current methods" that's not a dealbreaker - it just needs to be spelled out clearly in the paper. 
* If I understand correctly, "ground truth" is defined to be the k-means results using K={5, 10, 15} for MNIST/CIFAR-10 and K={10, 20, 30} for Newsgroup. Final performance numbers are averaged across these clusterings. How were these values of K chosen? They seem both arbitrary and very important for the final results. 
* This AEC metric is a bit hard to interpret. Is this something being proposed by this paper? I didn't find references to it elsewhere. I think it would be better to compute error relative to the NMI value you're trying to estimate. i.e. are these estimates off by 1%? 10%? 50%? Hard to say from AEC. In addition, AEC also makes it hard to put the results in context because other papers seem to use different metrics. 
* Is the learning rate not tuned? How was it selected? My concern is that by using the same learning rate for all methods, some might be getting "lucky" (because that's a good learning rate for them) while others might be getting "unlucky" (and would do better with a different learning rate). This would invalidate the claim that CEREAL is better than the competition. 
* The "robustness" experiments were conducted on Newsgroup with 500 examples, which happens to be a setting where the proposed method does very well as we see in Figure 1(a). Is this a fair comparison? If not, please highlight in the text that this is cherrypicked.

# Minor Comments
* The text for the figures could be a little bigger. 
* It would be nice if all of the "NMI vs. Number of Labels" figures all had the same y-axis range. 
* I'm curious about the role of the initial embedding - I wonder how much things would change if we used e.g. ImageNet pretraining instead of CLIP.
* Since we're already using CLIP to embed the images, what if we pseudo-labeled the entire dataset with CLIP? Might that lead to a competitive baseline? 
* Does $\Phi$ return the actual points selected, or the acquisition function values? There seems to be a bit of ambiguity about this between the text and the algorithm box. 
clarity,_quality,_novelty_and_reproducibility: # Clarity and Quality
* The paper is generally well-written and readable. 

# Novelty
* To the best of my knowledge this is a novel approach to a novel problem. 

# Reproducibility
* Not all of the important details for reproducibility are in the paper, but code is provided. However, some questions related to reproducibility of tuning procedures remain unanswered - see "Weaknesses" above. 
summary_of_the_review: This is a well-written, interesting paper that addresses an important gap in the literature. I enjoyed reading it and found it very informative as someone who doesn't do too much with clustering. Still, I have some questions about experimental methodology and significance. If these can be satisfactorily addressed I will be glad to see this paper accepted. 

Dec 12, 2022 Update: I have revised my rating -- see comment below.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: # Summary
This paper considers the problem of evaluating the quality of a proposed clustering. The traditional approach (computing normalized mutual information, NMI) requires every sample to have a ground-truth cluster label. The goal of this paper is to approximate NMI with a limited number of ground-truth cluster labels. The proposed approach is to adapt ideas from active learning and semi-supervised learning to the clustering context. The paper performs experiments on 3 datasets (MNIST, CIFAR-10, Newsgroup) and compares a number of baselines against variants of the proposed approach.
# Strengths:
- The few-sample clustering evaluation is an interesting topic to explore, which also has practical importance.
- The paper is well-written and easy to follow.
-The acquisition function with NMI is newly proposed in the paper.
- To the best of my knowledge, the proposed method is novel and fills a gap in the literature (evaluating clustering methods with few labels).
- The proposed method seems to perform well in the scenarios where it has been tested.
# Weaknesses:
1.	How to incorporate FixMatch in training the surrogate model is not clear. In FixMatch, there is a branch of strong augmentation on the input image x. How does this part work, specifically the strong-weak data augmentation branches, as the input of the surrogate model is a feature vector?
2.	The initialization of labeled data is uniformly sampling 50 data points, which doesn't consider the class balance in ground-truth labels. The random sampling could largely be class imbalanced which would affect the training of the surrogate model and consequently affect the final results. It would be helpful if the evaluations were separated into a class-balanced setting and a class-imbalanced setting.
3.	The effectiveness and robustness of the method are not fully evaluated. In table 1, Soft-NMI achieves the best result on CIFAR-10, better than utilizing FixMatch + Pseudo-labeling. Although the method achieves the best result on MNIST by quite a large margin. on a more complicated CIFAR-10 dataset (same size, same number of classes), there is no advantage of utilizing FixMatch + Pseudo-labelling. Meanwhile, using FixMatch has the large risk of introducing wrong pseudo-labels for those unlabeled data and the training from scratch is always time-consuming, which would be emphasized as the dataset carries larger scope. This brings the question that: is it worth introducing the FixMatch+pseudo-labeling on larger evaluation datasets with more classes and complicated data distribution? Currently even on CIFAR-10, there is no guarantee that FixMatch or semi-supervised learning in this framework would benefit.
4.	On page 2 there's talk of "experiments across multiple real-world datasets" but the paper uses MNIST, CIFAR-10, and Newsgroup, which I wouldn't consider "real-world" datasets. Is the reason because these methods don't scale up well? If so that's fine, but it should be clearly identified as a limitation in the text.
5.	This AEC metric is a bit hard to interpret. Is this something being proposed by this paper? I didn't find references to it elsewhere. I think it would be better to compute error relative to the NMI value you're trying to estimate. i.e. are these estimates off by 1%? 10%? 50%? Hard to say from AEC. In addition, AEC also makes it hard to put the results in context because other papers seem to use different metrics.
6.	Is the learning rate not tuned? How was it selected? My concern is that by using the same learning rate for all methods, some might be getting "lucky" (because that's a good learning rate for them) while others might be getting "unlucky" (and would do better with a different learning rate). This would invalidate the claim that CEREAL is better than the competition.
7.	The "robustness" experiments were conducted on Newsgroup with 500 examples, which happens to be a setting where the proposed method does very well as we see in Figure 1(a). Is this a fair comparison? If not, please highlight in the text that this is cherrypicked.

justification_for_why_not_higher_score: First of all, I would like to apologize the lack of discussion among 2 reviewers and authors because the author provided an extensive author feedback. Most of the concerns have been addressed in the rebuttal phase, but the reviewers are not fully convinced with the answers. Moreover, the paper has been changed significantly; hence the paper should be reviewed again before being accepted.
Decision: Reject