Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors review 19 methods for supervised and semi-supervised active learning and benchmark their performance on three small CV datasets (MNIST, CIFAR, GTSRB).   The paper shows that compared to supervised active learning, a semi-supervised active learning (SSAL) approach is consistently better (with RandomSampling being a baseline which many of the SAL methods underperform) - Further, they show that SSAL with uncertainty objective improves over SSLRandom and that sample uncertainty is more important than diversity in the SSAL setting. Further, the query size has limited role and higher unlabelled data helps.
strength_and_weaknesses: + The paper is clearly written. 
+ Several relevant methods have been compared and key insights clearly highlighted.  

- The datasets are toy datasets 
- I would have been very interested in seeing this evaluation in the context of NLP (where data augmentation has limited benefits compared to vision) and AL/HITL is a key component of modern ML systems. 

- Page 7 paragraph 3, "Second, sample uncertainty is more important than sample diversity during sample selection in
SSAL." I would have liked to see some empirical evaluation highlighting this - the explanation provided in the text is a bit unclear to me (or likely i misunderstood). 

Minor corrections: 

- Page 5, "Platform and Hardware". "Titian" -> "Titan"  
clarity,_quality,_novelty_and_reproducibility: - I did not find the link to code or zipped code in the submission. I missed it but if it is not there, this would considerably reduce the merit of the work. Especially, in the context of a survey, having the code well-documented, reproducible and publicly available to rerun experiments, and extend to other methods easily would be a strength. 
summary_of_the_review: Overall it is a nicely written survey which makes its point clearly. This is good since there is considerable value to this kind of evaluation work since it gleans insight from existing works each focussed on individual approaches.  However, in terms of contribution, I would have liked to see (1) real-world datasets if focussing only on vision; and ideally, (2) similar analysis in another application domain to highlight wider applicability of the lessons gleaned.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper conducts an expirical study of deep active learning, considering 19 approaches on 4 datasets. For the choice of deep AL approaches, they consider both supervised learning and semi-supervised learning based approaches.

strength_and_weaknesses: Strengths: this paper focus on the difference beween SAL and SSAL and conduct comparative experiments, it's quite interesting and meaningful. We can see from the comparative experiments that SSL does promote the AL performance.

Weakness: the evaluation of a empirical study is to see how many new things it brings to us and whether the experiments are reliable. However, this paper is somewhat limited:

1. About the comparison with existing empirical studies, the author should at least add: 1) [r1], conduct experiments on 19 DAL approaches across 10 datasets, the author said "Our study is the most comprehensive one, which covers 19 DAL methods,
including the state-of-the-art SAL and SSAL ones". But the experimental scale of this paper is smaller than [r1], [r1] also contain SSAL ones like WAAL; 2) [r2] DAL methods for NLP tasks; 3) [r3] empricial study of how AL affected by semi-supervised and self-supervised learning. Additionally, the author should provide a detailed analysis of comparison between the findings of their paper and related work like (Mittal et al., 2019) and [r3].

2. The author said "Another important observation for SAL methods is that: there is no SAL method that can outperform
others on every dataset". But in this empirical study, only 3 datasets are included, furthermore, MNIST is a very simple task and it can achieve more than 95% accuracy with a few data under strong basic classifiers, it's not enough representative. These datasets are just image classification tasks. For other tasks, e.g., in (Mittal et al., 2019), they found that random selection with SSL performs best under semantic segmentation tasks. The basic tasks should be enriched.

3. In Table 3, the performance of learningloss method is very strange. At least in [r1], the performance of LearningLoss is fairly good on CIFAR10. I cannot check the code since the author does not provide the code. Maybe there is because the learning rate is too high, as analysed in [r4] appendix E, high learning rate will lead poor performance.

4. In Table 3, why UncertainyGCN and CoreGCN differs so much?

5. The author said they run each method 5 runs, can author provide variance/standard deviation/error bars of the 5 runs?

6. GTSRB is a imbalanced dataset, but lack of analysis of how different AL methods perform on imbalance data. Does author have new findings?

[r1] Zhan X, Wang Q, Huang K, et al. A comparative survey of deep active learning[J]. arXiv preprint arXiv:2203.13450, 2022.

[r2] Dor L E, Halfon A, Gera A, et al. Active learning for BERT: an empirical study[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020: 7949-7962.

[r3] Chan Y C, Li M, Oymak S. On the Marginal Benefit of Active Learning: Does Self-Supervision Eat Its Cake?[C]//ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021: 3455-3459.

[r4] Mohamadi M A, Bae W, Sutherland D J. Making Look-Ahead Active Learning Strategies Feasible with Neural Tangent Kernels[J]. arXiv preprint arXiv:2206.12569, 2022.
clarity,_quality,_novelty_and_reproducibility: Clarity: this paper is easy to follow. 

Quality: the experimental analysis and the related work should be enriched as mentioned in previous part.

Novelty: it is an empirical study. The novelty analysis is not applicable here.

Reproducibility: the author does not provide code, it brings difficulty to check whether their implementations is right or not.
summary_of_the_review: This paper provides an empirical study of how SSL boosts AL performance. This paper still needs to be improved by enriching the comparisons with related work, especially (Mittal et al., 2019) and [r3].
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This is an empirical study of the efficacy of DAL on image classification tasks. It benchmarked 19 DAL algorithms, grouped into supervised and semi-supervised and tested on three simple different datasets (MNIST, CIFAR, GTSRB). After benchmarking, it concluded that the semi-supervised active learning techniques does a much better job than SAL and traditional SAL is not always better than even random sampling.
strength_and_weaknesses: Strength:

The drawing of figure 1 is really cute. The related work is well done with enough details and references. The clarity of the paper is strong and evaluation metrics are nicely introduced.

The scope of the paper is a strength with a focus on benchmarking DALs together in a similar setting. Directly exposing the weakness of current DAL approaches that they sometimes don't even beat random sampling (the majority of them from Table 3).


Weaknesses:

The title is broader than the study. Wil active learning is a more and more broad term used by more communities, since only image classification tasks are studied in this work, I suggest adding such limitation in the title.

The comparison feels unfair frankly speaking. As this work aims to evaluate the efficacy of "ACTIVE LEARNING", comparing semi-supervised AL with random sampling is not fair. Instead, authors should consider comparing SSAL with semi-supervised learning to show the effect of active learning piece.

The input dimension of the datasets are too small (28 x 28) for image classification tasks from my perspective. As majority of real-life applications would be carried out in larger image scales, whether the same conclusion holds for larger image sizes is a gap for me and impacts the significance of this work.

Claim 5.1 "START SSAL AS EARLY AS POSSIBLE": The smallest starting point in the experimental result is 0.5% instead of 0%. With minimal starting labels, the models would not even be appropriately trained, and therefore I highly doubt the efficacy of such a model providing good proposals for unlabeled points to be labeled. I understand the authors' suggestion, but the claim needs to be modified as this is not true (at least not proved if you start from 0.5%, not 0%).

What model architecture was used across the different tasks? This seems like a crucial detail that should be included in the experimental setting.

For the cost in the unit of seconds, I am wondering why some active learning methods can be faster than random sampling? 



Questions:
1. For such a comprehensive analysis, the exclusion of ImageNet seems odd to me, if the computational resource is an issue, smaller version of ImageNet can be considered?
2. Why is the step size of MNIST different from the other two methods?
3. May I request the proof of "SGD optimizer performs better than Adam" in the supplementary? This seems to be controversial that a single citation is not enough.
4. Did the learning rate have any decay scheduler? Active learning should mimic the real process of the machine learning cycle and therefore, the training process should be mimicking the same effort of a practitioner training for the best performance.
5. What does Figure 3 convey? I didn't see the contribution of supporting any claims from Figure 3.
clarity,_quality,_novelty_and_reproducibility: The paper is clear, with good novelty. The quality of the paper is overall high.

However, the paper would need to publish full code base details to have good reproducibility, also, as I mentioned in the weakness part, the model architecture detail is a key aspect of reproducibility as well.
summary_of_the_review: I think the paper definitely has its contribution to the field, pointing out the problem in the AL field that they might not work as advertised when benchmarked carefully together and that SSAL performs great. However, as I mentioned in the weakness part, there are several important misses in the current form of the argument that needs to be addressed until this gets published. Therefore I would like to recommend reject/marginally reject of its current form, with a open mindset to change my ratings once those concerns are addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper performs a benchmark of active learning for deep nets (aka DAL). 
The authors make the distinction between _supervised_ methods and _semi-supervised_ methods (respectively SAL and SSAL) by the ability to harness unlabeled data.
The proposed benchmark is comprehensive (as in the amount of methods tested) and puts an emphasis on both on usage of unlabeled data as well as measuring the cost of the *selection* process.

The supervised flavor of DAL is further divided into three types of data selection:
- Uncertainty based selection - uses the model's output to estimate the uncertainty a data sample, and select the ones with closest to the decision boundary. 
- Diversity/Representative-ness based selection - selection that maximizes the diversity of the data points, usually over a set of clusters or a metric space.
- Hybrid selection - a combination of the two former approaches

Semi-supervised approaches perform some of the learning process without the labels.
Three of the methods here use an approach termed in the literature as *self-supervised*, which are based on comparison of two forward runs through the net, one of which is of an augmented/corrupted version of the data. A fourth approach, WAAL, is base on a adversarial approach of training a discriminator-net to differentiate between the labeled an unlabeled data.

---

One of the main findings of this work is that unlabeled data should be mapped (as soon a possible), by and unsupervised approach. 
Methods that do not harness such an un/semi- supervised approach might not achieve performance better that random selection
strength_and_weaknesses: ## Strength 
This paper is well written and easy to follow. 
Most recent DAL methods are tested here, with the important emphasis both on (pre-) processing of unlabeled data, as well as the cost of such an approach.

The main insight of this work seems important - DAL struggles to beat the ransom baseline without usage of such unsupervised approaches. 

## Weakness 
- The one notion that weakens this work is the specific datasets on which this benchmark is based, in two aspects:
   1) There are substantial evidence that both MNIST and CIFAR are saturated (= contain non-negligible label noise)y, but mainly do not pose challenges like class-imbalance or long-tail distribution, which is the one of the main goals of DAL, IMHO.
   2) Why did the authors not tackle larger datasets - while there are some initial works using the (old but very large) ImageNet, I would argue that there are quite a few examples between the latter and (the too academic academic) MNIST/CIFAR 
- An important question here is regarding the **re-implementation** of all methods - some of the methods have made the code publically available, and such re-implementation might be unfair (both in performance and cost). I would expect, at the very least, to compare to the original implementation when available - or to state formally that there is little difference in performance or cost.
- The authors state that **5 runs were made** for each approach, why is the STD not reported?
- I understand that 2%/5% increments are from the previous benchmarks, but why not test smaller increments?
- SSLRandom is not properly defined in the text.
- I would strongly recommend using the term *self-supervised* whenever appropriate - that is whenever an augmented/corrupted version of the data is compared to the original (or another augmented version) 
clarity,_quality,_novelty_and_reproducibility: ### Clarity
The main idea of this paper is quite straightforward. The experimental setup and the key results are easy to understand. 
### Quality
This is the only caveat here - while the paper examines a good set of approaches using the correct metrics, the conclusions are still based on (too) academic datasets.
### Reproducibility
I think most of the approaches that are tested here are easy to implement, and I even found a few open source version for some. In addition, the authors promise to make the code publicly available.
### Novelty
The novelty of this work, as in most benchmarks, is very limited. The main goal here, as I see it, is to shed some light on the existing state of the art rather than pushing it forward
summary_of_the_review: This work provides an empirical evidence of how DAL should be addressed in practical sense. The notion that unlabeled data should *preferably* be consumed *earlier* via unsupervised methods is not surprising, but important - and was not addressed in most prior benchmarks. 

---
The main limitation of this work is that it is based on academic/toy datasets like MNIST and CIFAR, as I detail above.


---

Nevertheless, this work passes the bar IMHO, and lay a basis for future works to complete the gap(s) I mentioned
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: # Summary
The paper conducts a review of 19 methods for supervised and semi-supervised active learning and benchmark their performance on three small CV datasets (MNIST, CIFAR, GTSRB). The paper shows that compared to supervised active learning, a semi-supervised active learning (SSAL) approach is consistently better (with RandomSampling being a baseline which many of the SAL methods underperform).  After benchmarking, it concluded that the semi-supervised active learning techniques does a much better job than SAL and traditional SAL is not always better than even random sampling.
# Strengths:
- The paper is clearly written.
- Several relevant methods have been compared and key insights clearly highlighted. It's quite interesting and meaningful. We can see from the comparative experiments that SSL does promote the AL performance.
# Weaknesses:
- The datasets are toy datasets
- Limited to Computer Vision datasets; hence it would be very interested in seeing this evaluation in the context of NLP (where data augmentation has limited benefits compared to vision) and other type of datasets.
- An important question here is regarding the re-implementation of all methods - some of the methods have made the code publically available, and such re-implementation might be unfair (both in performance and cost)

justification_for_why_not_higher_score: The paper shows an interesting empirical study of active learning, but it should include a large number of datasets in terms of classes, size, examples, etc. Also the paper should be tackle other domains like NLP. Therefore, the paper should include a more extensive evaluation before being accepted.
Decision: Reject