Enhancing Cost Efficiency in Active Learning with Candidate Set Query

Authors: Yeho Gwon, Sehyun Hwang, Hoyoung Kim, Jungseul Ok, Suha Kwak
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Active Learning, Conformal Prediction, Label efficient learning

Abstract:
This paper introduces a cost-efficient active learning (AL) framework for classification, featuring a novel query design called candidate set query. Unlike traditional AL queries requiring the oracle to examine all possible classes, our method narrows down the set of candidate classes likely to include the ground-truth class, significantly reducing the search space and labeling cost. Moreover, we leverage conformal prediction to dynamically generate small yet reliable candidate sets, adapting to model enhancement over successive AL rounds. To this end, we introduce an acquisition function designed to prioritize data points that offer high information gain at lower cost. Empirical evaluations on CIFAR-10, CIFAR-100, and ImageNet64x64 demonstrate the effectiveness and scalability of our framework. Notably, it reduces labeling cost by 42% on ImageNet64x64.

Decision: reject
Reviewer scores: 5, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730207259869

thread_type:
official_review

<thread_object>
creator: reviewer_qtbh

<content>
summary:
In this paper, the authors proposed a cost-efficient active learning (AL) framework for classification, featuring a novel query design called candidate set query (CSQ). CSQ narrows down the set of candidate classes likely to include the ground-truth class and leverages conformal prediction to dynamically generate small yet reliable candidate sets. Experiments on several datasets demonstrate the proposed CSQ is effective.

strengths:
The proposed CSQ present the annotator with an image and a narrowed set of candidate classes that are likely to include the ground-truth class, which reduces labeling cost by minimizing the search space the annotator needs to explore. The various modules in the entire paradigm are relatively mature.

weaknesses:
1. The explanation of "empirical quantile" in Section 3.2, titled "Candidate Set Construction from Conformal Prediction," is not very clear. Could the authors provide further clarification, specifically regarding the insights that Conformal Prediction offers for method design and its role in the proposed approach?

2. Is Table 1 in the experiments derived from the cost-efficient acquisition function?

3. Does the Active Learning (AL) approach include corresponding real-world datasets? Would it be possible to include more experimental results using real-world datasets?

questions:
Please see the Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730356952516

thread_type:
official_review

<thread_object>
creator: reviewer_sjca

<content>
summary:
This paper introduces the Candidate Set Query (CSQ) framework which aims to improve the cost efficiency in active learning (AL) tasks. CSQ reduces the size of the set of candidate classes which reduces the search space, leveraging conformal prediction to dynamically adjust the optimal candidate set size across successive AL rounds. The paper also proposes an acquisition function that considers the ratio of information gain vs labeling cost to help prioritize samples to label. The author then benchmarked this framework across multiple datasets and empirically demonstrate the reduction in labeling costs.

strengths:
1. The motivation for this paper is clear, and the paper proposes a novel framework of high significance 
2. The paper presents a solid theoretical framework that is thoroughly explained and mostly straightforward to follow
3. The framework is benchmarked across 3 well-known datasets, empirically demonstrating the effectiveness of the method 
4. Thorough ablations studies were conducted to highlight the significance of each component of the framework

weaknesses:
1. The benchmarks are conducted on very similar datasets (CIFAR-10, CIFAR-100, and ImageNet64x64 are all image classification datasets), and also only compares against a small number of baseline AL methods. It is unclear if the results will generalize well across different datasets and domains, and if more advanced underlying AL acquisition methods are used
2. The paper does not consider the implication of real-world datasets, such as those containing label noise, imbalance classes etc might impact the performance of CSQ. More experiments could help identify CSQ’s robustness when it comes to noisy annotations, as it could lead to inefficient training and candidate sets with lower quality
3. CSQ relies on several hyperparameters, however there is limited justification on how to properly optimize the hyperparameter $d$, making it difficult to apply the method in new datasets

questions:
1. Have the authors considered any special handling of outlier or anomalous datapoints, which tend to have have high uncertainty, and could lead to inefficient construction of the candidate set, acquisition and labeling
2. Have the authors considered the risk of overfitting and drift of confidence scores across successive AL rounds and how CSQ could incorporate some steps to address it?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730698553745

thread_type:
official_review

<thread_object>
creator: reviewer_yk4x

<content>
summary:
The paper proposes a cost-efficient active learning strategy using conformal predictions. Instead of letting the annotator choose from all possible labels, a candidate set of fewer labels is given. The paper uses a log order expected cost function and shows the improved efficiency in terms of actual labeling cost.

strengths:
1. The content of the paper is well presented.
2. The paper studies the cost of AL query in a more realistic way and proposes a solution for reducing the cost by candidate set query. 
3. The candidate set is formed by conformal prediction and the candidate labels are related to the expected information gain with cost considerations.

weaknesses:
The proposed method still depends on the conformal prediction and the calibration set to determine the confidence level. It is a realistic solution however not guaranteed to be theoretically sound. The convergence can not be obtained in a proper label complexity analysis. Similarly, the labeling cost assumption in Theorem 3.1 is only a rough approximation.

questions:
1. Is random selection effective for the calibration set?
2. Is it possible that the candidate set obtained from eq(5) is empty?
3. Is there any guarantee that using the quantile from the previous round would work?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702378903

thread_type:
official_review

<thread_object>
creator: reviewer_6wyp

<content>
summary:
This paper proposes an active-learning framework called Candidate Set Query (CSQ) to reduce the size of possible classes during annotation process, minimizing the search space and lowering the labeling cost at the same time. Furthermore, the authors leverage conformal prediction to produce accurate candidate sets, and introduce an acquisition function that exploits data points with high information gain.

strengths:
1. This paper introduces a novel approach called Candidate Set Query (CSQ), which effectively reduces labeling costs by narrowing down the candidate classes presented to annotators, thereby minimizing annotation time.
2. The proposed method leverages conformal prediction to dynamically produce accurate candidate labels based on a cost-efficient data acquisition function. This function prioritizes samples with high information gain, leading to greater efficiency and reduced labeling costs.
3. The framework demonstrates strong performance across multiple image recognition datasets, consistently outperforming baseline methods by a significant margin.
4. The authors provide comprehensive ablation studies to thoroughly validate the effectiveness of all components.

weaknesses:
1. The rationale behind the cost-efficient acquisition function in Eq. (8) needs to be further explained. Additional motivation and explanation for this function are recommended.
2. As shown in Fig. 9a, the performance is sensitive to the hyperparameter d. Providing guidelines for setting this parameter to an appropriate range on different datasets would be beneficial.
3. In realistic scenarios, the samples with high uncertainty waiting to be annotated can be divided into two groups based on their probability distributions on categories, high confidence on several specific classes or low confidence on almost all classes. The proposed framework might only be suitable for the former case. As for the latter one, an intuitive solution is to sift out all candidate labels with low prediction probabilities, such as less than 0.1 * 1/C where C is the number of categories. So it’s suggested to conduct more experiments to evaluate the performance of this approach under this scenario.
4. A smaller candidate label set means model have higher certainty for samples, which seems to be against the motivation of selecting the most uncertain samples in active learning. Despite the proposed method minimizing the labeling cost by narrowing down the label space, the information gain is limited. It’s suggested to plot the graph of how performance varies with the number of queried samples.
5. There is a minor formatting issue in Fig. 5c, where certain data points from the top-1 prediction fall outside the scale range of the graph.
6. Line 274 contains a typo: “calcuclate” should be corrected to “calculate.”

questions:
Please refer to the above weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732092041488

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to Reviewers

comment:
We sincerely thank the reviewers for their valuable feedback and are encouraged by their recognition of the novelty (**6wYp**, **sJcA**) and soundness (**6wYp**, **YK4X**, **sJcA**, **QtBh**) of our work. We also appreciate their positive comments on our use of conformal prediction (**6wYp**, **YK4X**), the strong empirical performance of our method (**6wYp**, **sJcA**), and the thoroughness of our ablation studies (**6wYp**, **sJcA**).

We address each reviewer's comments and questions on each thread. All modifications made to the manuscript are highlighted in blue.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733304362562

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of reviewers’ overall comments and the corresponding responses

comment:
We thank the four reviewers and the Area Chair for their time and effort in reviewing our work and managing the review process. For your convenience, we have summarized the overall review comments and our responses during the rebuttal period below.

### **Summary of strength recognized by reviewers**

We are encouraged by the reviewers recognizing that our method is a novel (**6wYp**, **sJcA**), realistic (**YK4X**), and well-motivated (**sJcA**) solution for reducing labeling cost (**6wYp**, **YK4X**, **QtBh**). We also appreciate their positive comments on the use of conformal prediction in active learning (**6wYp**, **YK4X**), the strong empirical performance of our method (**6wYp**, **sJcA**), and the thorough ablation studies (**6wYp**, **sJcA**). They also recognized that the paper is well written (**YK4X**, **sJcA**) and theoretically sound (**sJcA**).

### **Summary of Rebuttal / Discussions**

We summarized the key comments and our corresponding responses below:

| Comment | Response |
| :--- | :--- |
| Clarification of our acquisition function (**6wYp**) and empirical quantile (**QtBh**) | We have added detailed explanations of the acquisition function and empirical quantile, and updated the manuscript accordingly. |
| Guidelines for setting hyperparameter $d$ (**6wYp**, **sJcA**) | We have conducted ablation studies on the hyperparameter $d$ on CIFAR-10, CIFAR-100, and ImageNet64x64, and provided a guideline for setting $d$ based on the observations. Reviewer **sJcA** expressed satisfaction with our responses. |
| Effectiveness of our method on uncertain samples (**6wYp**, **YK4X**) | We have shown that our method outperforms the sift-out baseline suggested by reviewer **6wYp** when using entropy sampling that favors uncertain samples. We have further justified that our method is compatible with diverse uncertainty measures. |
| Dependency on calibration set and quantile from previous round (**YK4X**) | We have clarified that the calibration set is also used for model training. We have also provided results showing that our method is robust to the size of the calibration set and remains highly effective without using the quantile from the previous round. |
| Realistic aspect of our cost model (**YK4X**) | We have explained the motivation behind our cost model, supported by the user study involving 40 participants. |
| Theoretical analysis on label complexity and its convergence (**YK4X**) | We have clarified that candidate set query is orthogonal to the acquisition function, ensuring compatibility with acquisition functions with label complexity analysis. We have also discussed the challenges of proving the label complexity under deep learning models. |
| Experiment on more realistic datasets (**sJcA**, **QtBh**) | We have conducted experiments on datasets with noisy labels and class imbalances and on a text classification dataset, demonstrating robustness on various realistic scenarios. Reviewer **sJcA** acknowledged the generalizability of our method across modalities and its robustness to real-world datasets. |
| Experiment on additional acquisition function (**sJcA**) | We have evaluated our method with the ProbCover (Yehuda et al., 2022) acquisition function, confirming its compatibility with advanced acquisition methods. |


### **Before/After Rebuttal**

- Reviewer **6wYp**: We believe our extensive experiments on the requested scenarios, along with the provided explanations, have addressed most of the concerns.
- Reviewer **YK4X**: The reviewer noted that several concerns, including confidence level settings, the cost model, and calibration set sampling, had been addressed but noted that a few concerns remained. To resolve these, we have provided additional experiments and clarifications.
- Reviewer **sJcA**: The reviewer stated that the responses satisfy most of their concerns and that they would **rate to accept** this paper.
- Reviewer **QtBh**: We believe our clarifications and extensive additional experiments have addressed the concerns raised by the reviewer, who was already positive with our work with a score of 6.


---
---

### **References**

Yehuda, O., Dekel, A., Hacohen, G., & Weinshall, D. (2022). Active learning through a covering lens. Advances in Neural Information Processing Systems, 35, 22354–22367.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper was reviewed by four experts in the field and received 5, 5, 6, 6 as the final ratings. The reviewers agreed that the paper proposes an interesting active learning (AL) technique to further reduce the labeling burden on the human annotators, the proposed solution utilizes the well-established theory of conformal predictions, the experimental results are encouraging, and that the content of the paper is well-presented. 


The authors have mentioned that their candidate set query design is independent of the sampling method, and it can be applied in conjunction with any acquisition function. However, in their paper, the authors have demonstrated their method with only three AL acquisition functions: random sampling, entropy sampling and BADGE sampling. Given the plethora of AL sampling functions (such as those based on uncertainty, diversity, representativeness and their combinations), demonstrating the performance with only three functions (out of which random and entropy are very naïve methods) is not sufficient. While the authors have conducted experiments in the rebuttal using the ProbCover acquisition function, in response to Reviewer sJcA's comment, the AC feels that a more thorough investigation of the proposed candidate set query method with more advanced acquisition functions is necessary to appropriately understand its merit and usefulness. 


The active learning setting proposed in this paper narrows down the set of candidate classes likely to contain the ground truth class, so that the oracle won’t have to examine all possible classes. This setting is very similar to that proposed in the following papers:

[1] Joshi et al. "Breaking the interactive bottleneck in multi-class classification with active selection and binary feedback". CVPR 2010

[2] Bhattacharya and Chakraborty. "Active Learning with n-ary Queries for Image Recognition". WACV 2019

These methods should be used as comparison baselines; it is difficult to assess the merit of the proposed method without a thorough comparison against related techniques. 


A concern was also raised about the convergence guarantee of the proposed acquisition function. While the authors have acknowledged this in the rebuttal, and have mentioned that this is due to the reliance on heuristic informativeness measures, establishing convergence guarantees will further strengthen the theoretical aspects of the proposed solution. The authors are encouraged to look into this aspect. 


We appreciate the authors' efforts in meticulously responding to each reviewer’s comments. We also greatly appreciate their efforts in conducting additional experiments to address some of the reviewers' concerns (such as the experiments on datasets containing imbalance classes, experiments on datasets containing label noise, experiments with the ProbCover acquisition function and experiments on text classification). However, in light of the above discussions, we conclude that the paper may not be ready for an ICLR publication in its current form. While the paper clearly has merit, the decision is not to recommend acceptance. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.