Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes generic active learning methods for any learning method.  The main method is motivated by the other method with theoretical guarantees. More precisely, the main proposed method is a practical heuristic one. The proposed method is compared with other previous active learning methods and naive ones and shows better performances on the benchmark data sets. 
strength_and_weaknesses: Strength:
The proposed method shows remarkable improvements over previous work in practice. Although the method does not have any theoretical guarantees, but it is motivated by the other theoretically guaranteed algorithm. 

Weaknesses:
The weakness of the paper is, as mentioned above, the lack of theoretical guarantees. However, other previous work does not have guarantees either. So, this is not a disadvantage over previous work. 


clarity,_quality,_novelty_and_reproducibility: The paper is written well in general. However, I have a minor concern about the presentation of the contribution. The method is presented as a subset selection method of the training sample. But, then, the standard stochastic gradient methods are in the same category and I wonder if those needs to be compared as well. As with other previous work, I feel more convinced if the proposed method is presented as an active learning method for querying labels. 

I think the methods are novel but not significantly innovative (motivated by several previous results). The experimental section seems to contain enough information to reproduce the results.
summary_of_the_review: The paper proposes a practical heuristic method for active learning by querying labels. The paper shows non-trivial practical improvements over previous work.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes an active learning method by utilizing importance reweighting technique. Authors derive two strategies, including entropy-based disagreement and entropy to sample examples. Then, authors reweight losses by the sampling probability of each example to achieve an unbiased estimator. Experiments are conducted on multiple benchmark datasets.
strength_and_weaknesses: Strength

The paper proposes the entropy-based disagreement sampling strategy, which is slightly different from the previous sampling strategies.

The paper conducts comparison experiments on a large-scale dataset OpenImage, which is relatively realistic.

Weakness

The main idea of the proposed two strategies is similar to the previous strategies, such as uncertainty sampling and largest margin sampling [1]. It seems that the paper does not propose some new techniques. I think its technical contribution is limited.

[1] Active learning literature survey

The theoretical analyses proposed in the paper are also trivial. The main results follow the previous works [1][2]. Furthermore, the main theoretical results are based on the unbiased estimator, which has been first proposed in [2].


[1] Active Learning for Convolutional Neural Networks: A Core-Set Approach
[2] Importance Weighted Active Learning 

The experiments are weak in some extent due to the lack of comparison with sota methods. In fact, besides two baselines uncertainty sampling and random sampling, the proposed method is only compared with two methods, BADGE (2019) and Coreset (2017). These methods have been proposed three years ago. The paper is suggested to compare with more recent methods [1]

[1] Influence selection for active learning, 2021
[2] Task-aware variational adversarial active learning, 2021

clarity,_quality,_novelty_and_reproducibility: The presentation of this paper is clear.

However, the contribution of this paper is insufficient for the following reasons: 1) The paper does not propose new techniques, e.g., novel sampling strategies. The main idea of the proposed two strategies is similar to the previous works. 2) The main theoretical results are applications of the prior works.
summary_of_the_review: The technical contribution is insufficient and the novelty is limited.
Although the paper reports the extensive experimental results, it suffers from the lack of comparisons with some sota methods.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed a method for subset selection, which is useful for applications such as active learning. Specifically, in the proposed method, when constructing the subset, data is selected by importance sampling, where the sampling probability relates to the model's entropy. In the subset, the selected data is weighted inversely proportional to its sampling probability for computing a weighted loss function. This method is empirically evaluated on six multi-class classification image datasets and a large-scale multi-label Open Images dataset, and shows good performance when compared with baselines. The theoretical motivation of the proposed method is also presented in the paper. 

The paper's contributions include a theoretically motivated method for subset selection that is suitable for general losses and hypothesis classes and extensive experimental results for evaluating the proposed method and baselines. 
strength_and_weaknesses: Strengths:
1. In this paper, the considered subset selection problem is promising and useful in many cases, such as efficient deep neural network training, limited budget for data storage, and active learning. The proposed method is applicable for deep models, and for both labeled and unlabeled data during the subset selection.
2. The proposed method is theoretically motivated and evaluated by extensive experiments on multi-class datasets and a large-scale multi-label dataset. 
3. The paper is well-written and presented. This paper's main messages are clear and easy to follow for readers.  
 
Weaknesses:
1. The core idea of the paper is largely based on prior work (i.e., Beygelzimer et al. (2009)), which may reduce the novelty of this paper. Compared with prior work, this paper contributes to using the model's entropy to define the sampling probability to make the method suitable for deep models.
2. I am wondering if the seed set's size or the quality of pertained models affects the performance of the proposed method. If the seed set is very small, then the model may not be well pretrained (in Line 6, Algorithm 1). In this case, perhaps the data cannot be effectively weighted, especially during the beginning of the training.

Other comments:
In the proposed method, it seems that once one data is selected, its weight would be fixed. However, as the model is trained to become better, the weight of one data may also change if recomputed. Then, for the proposed method, it may happen that the weights for one particular data may differ by the order of the data selection. Would it improves the proposed method if recomputing weights of all existing selected data at some specific training epoch?
clarity,_quality,_novelty_and_reproducibility: For clarity and quality, the paper is good and clear in presenting the core idea, related work, and baseline methods. 

For novelty, the paper's main idea is based on one prior work, but improving the sampling probability to make the method more suitable for deep models. The relationship between prior work and this work is well-explained and discussed in the paper. 

For reproducibility, it seems that the authors did not provide the code for reproducing the work, but they presented lots of details for reproducing the proposed method in the paper.
summary_of_the_review: The paper presents a subset selection method suitable for deep models with theoretical motivation and extensive experimental results. Although the paper's main idea is based on one prior work, this paper contributes to using model's entropy to define the sampling probability for the benefit of using deep models. This paper is also well-written and completed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on subset selection. Based on general loss functions and hypothesis classes, the author(s) proposed an importance-weighted subset selection algorithm called IWeS. Through Empirical experiments, the author(s) tried to show the advantage of the proposed algorithm.

---

strength_and_weaknesses: ### Strength:

1. Selecting an information subset from a large dataset can effectively reduce the computation time and storage costs. In this paper, the author(s) further considered the weights of selected samples.

2. Empirical evaluations are performed for validating the proposed algorithm.

---

### Weaknesses:

The theoretical analysis of the proposed algorithm, such as the generalization bound and sampling rate bound, may require further empirical experiments.

For more details, please see the section of "Clarity, Quality, Novelty And Reproducibility".

---

clarity,_quality,_novelty_and_reproducibility: I think the topic in this paper is interesting and this paper is generally well-written. I have the following comments/questions. I look forward to the response/clarification from the author(s). Thanks.

1. In the introduction, the author(s) mentioned another related method, IWAL. Maybe an empirical comparison with the algorithm IWAL is needed? Or is IWAL difficult to implement in practice?

2. One motivation of subset selection is to find the most informative subset from a large number of training samples to approximate training with the entire training set. So, in experiments, it is necessary to compare the performance on the whole training set, which can be regarded as a baseline to further evaluate the effectiveness of the proposed algorithm.

3. For a better understanding of the theoretical analyses, such as the generalization bound and sampling rate bound, some empirical experiments are needed for further verification.

4. If the input data matrix is transposed, can the work in this paper, i.e., data subset selection, be applied to feature selection?

In addition, the format of the references is quite inconsistent. Please check carefully and correct it.

---



summary_of_the_review: The work in this paper is interesting and it considers weighting the selected samples; However, there are some unclear/inadequate aspects in the description (including experiments) of this paper. Maybe it needs the author(s) to clarify them. Thanks.

In addition, in this paper, the author(s) performed empirical experiments. I am not sure the description would be enough to reproduce since no code seems to be provided (other than some pseudo-code descriptions). 

---
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents an approach for subset selection for training models in a batch setting.  The proposed approach, called IWeS, uses importance sampling to select examples for inclusion in the subset, where the sampling probability is based on the model’s entropy.  Experimental results show that IWeS provides moderate to substantial improvements in predictive performance compared to several competing baseline methods.  Furthermore, a theoretical analysis for a closely related approach, IWeS-V, including generalization and sampling rate bounds.  
strength_and_weaknesses: Strengths:
* IWeS is applicable to arbitrary model families.  It can also be used in an active learning where only unlabeled examples are available.
* IWeS matches or outperforms a number of competing baseline approaches for several datasets on multi-class and multi-label prediction tasks.
* Useful theoretical guarantees, in terms of generalization and sampling rate bounds, are provided for a closely related algorithm, called IWeS-V.

Weaknesses:
* The entropy-disagreement-based variant of IWeS (IWeS-dis) requires training two models, which can be expensive in terms of computation and memory.
* Theoretical guarantees are provided for the IWeS-V algorithm, which is closely related to the IWeS-dis algorithm.  However, theoretical guarantees are not provided for a variant of the IWeS-V algorithm that is closely related to the entropy-based IWeS algorithm (IWeS-ent), which is computationally less expensive since it requires training only one model.
* No detailed analysis of the computational costs associated with the proposed methods is presented in the paper.

clarity,_quality,_novelty_and_reproducibility: This paper is well written and clear.  The contributions appear to be novel, with good empirical and theoretical contributions.  Although the authors do not provide source code, enough detail appears to be present in the paper to allow the proposed methods to be reproduced.

summary_of_the_review: This is a strong paper, with good empirical and theoretical results.  The experimental results are convincing, showing that the proposed method matches or outperforms a number of baselines.  While there are some issues, as mentioned in the list of weaknesses above, they do not significantly detract from the value of the contributions.  Overall, this is a good paper worthy of acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
The paper proposes generic active learning methods for any learning method.

Strength:
The proposed algorithm does not require maintaining a version space, which has an advantage for general hypothesis classes, and it works well in experiments.

Weakness:
The proposed method lacks theoretical guarantees and technical novelty is rather limited.

justification_for_why_not_higher_score: Thanks for the detailed replies to the reviewers, which clarified several issues raised in the initial reviews. Overall the reviewers satisfied with the author responses and thus I recommend the acceptance of this paper. However, the authors' implementation is not clear from the paper and thus I explicitly request the authors to make their experiment code public so that readers can reproduce the results provided in the paper.
justification_for_why_not_lower_score: Even though there is no theoretical guarantee, the main idea is interesting and a good contribution to the active learning community.
Decision: Accept: poster