Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper examines how active learning works with large pre-trained models, with a novel explanation with empirical evidence. 
main_review: Strengths:
 - I could learn various empirical observations of how active learning works in these large pre-trained models on various NLP tasks. I hope to see this paper at the conference and others can learn from it as well. 

Weakness:
 - It would be nice to see some macro analysis of data points showing the overall tendency of each setup. 
 - I hope to see more insights regarding how these two different AL methods (density-based and entropy-based) would work in a different way.
summary_of_the_review: Good empirical findings from extensive experiments, but limited to the individual setup in specific conditions without an overall picture of dataset dynamics.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work investigates Active Learning (AL) for fine-tuning large pre-trained transformer models (BERT, RoBERT) with applications to several NLP tasks.
They observe that depending on the task/dataset, AL actually does not improve over a random baseline.
Their ablation tests suggests that the instability of re-training the large models might be amplified by instances selected by AL. As a remedy, they suggest to train several times with different initializations, and then selecting the best model on the development data.
main_review: Strength:
- Experiments on many data sets.
- The idea of using convex hull of AUC is interesting, but it would it nice to confirm it experimentally. 

Weakness:
- Q seems to be too small, which might amplify the instability issue.
- Little novelty 

While their experiments, and findings are interesting, it leaves many questions that could (at least in principle) be answered by more experiments. 
Their main finding seems to be that larger transformer models lead to more instability for training with the instances selected by AL, Table 4 and Table 5.
However, Table 4 is only an approximation to results that could be achieved with training with different initializations. Though computationally expensive it would be very interesting to see whether the results from the approximation can be confirmed. 

Several questions:
- What exactly is meant by the Entropy method?
My understanding that is that Entropy is the same as Least Confidence, since high entropy <-> low confidence.
- Why is Entropy missing in Table 2?
- Why is Q set to 500? This seems to be too small. It would be interesting to learning curve graphs with larger Q.
summary_of_the_review: For an experimental paper, it is interesting, and might point to some new directions for AL.
However, more experiments to clarify some points (see above) should be added.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work evaluates active learning methods for pre-trained Transformer models, and diagnoses their inconsistent performance as a consequence of training examples which exacerbate variability in test-time accuracy. They propose re-training the model repeatedly to circumvent bad outcomes due to variability, and show this works particularly well on larger pre-trained models. The authors conduct a wide range of experiments, over many datasets, and also a series of ablation studies to reinforce their claim that these active learning methods are ineffective for pre-trained Transformers.
main_review: Strengths:
 
The authors connect active learning example selection to more severe training instability than random selection. Further, they show downside-variability (“greater instability on the losses than on the wins”) suggesting specific sets of examples are responsible for systemic failures in active learning, but not necessarily that other selections would greatly outperform the random baseline. Both of these observations are of interest to the NLP active learning community, adding depth to the characterization of the problem.
 
The wide range of datasets and active learning techniques they use (including BALD which prior works shows is very competitive) lends credence to the conclusions. It especially corroborates that certain tasks are innately more challenging for active learning than others. 
 
Their thorough ablation experiments and other analysis yield some interesting findings the authors could emphasize more. For instance, the greater instability of larger Transformers to active learning bodes poorly for practitioners leveraging ever increasing model sizes for low-resource datasets. Additionally, the authors conclude that pre-training is not a significant factor in the efficacy of active learning, but their numerical results suggest active learning methods (Entropy and Coreset) narrow the gap with the random baseline significantly from BERT-Base to RoBERTa-Base! The margin of change seems even larger than some results which are discussed in the paper as significant. More discussion could bolster this finding.
 
Weaknesses:
 
The primary weakness of the paper is the lack of convincing justification that the authors have discovered a phenomenon distinct from “collective outliers” (Karamcheti et al., 2021) -- or if it is distinct, how exactly is it distinct? 

1. For context, Karamcheti et al. (2021) introduce an "oracle" active learning method that prunes uninformative (outlier) examples from the unlabelled, selectable pool U, but it requires gold labels to filter these. This work tries a single pruning rate and finds it significantly degrades, rather than improves results: for both active learning and random selection. However, they only mention a single pruning rate of 50%, which could easily over-prune (all challenging as well as outlier examples) for the NLP datasets used here. Especially because of the surprising magnitude by which this pruning degrades absolute performance, it is unfortunately necessary to try more pruning rates for a fair comparison.

2. Even if pruning is ineffective (at multiple pruning rates), it’s never really explained why, despite this being a core contribution of the paper. 

A qualitative analysis was required in Karamcheti et al., (2021) to explain “collective outliers”. A similar analysis here could greatly demystify why these sets of examples cause instability, and whether they are indeed “informative”.
 
There are a couple decisions related to methodology that may need some further examination. 

1. The first is the total unlabelled pool size. It appears to vary by orders of magnitude according to Table 10 in the Appendix. While the batch selection L and total selection Q variables are constants, fixed at 25 and 500 respectively, the pool of examples to choose from varies wildly. This is very likely a confounding factor in the efficacy of active learning and pruning techniques. It deserves some discussion or control.

2. The second is the choice of L=25 examples chosen at each acquisition, and also as the seed set. I am concerned this is quite low and could yield exaggerated instability, especially for large Transformer models. If I read them correctly, Ein-Dor et al. (2020) use at least L=50 for simple classification tasks, and Karamcheti et al., (2021) use L>= 400 and larger seed sets. The authors investigate L=12 in the ablation, but in a real setting, it seems unlikely practitioners will label <50 examples before re-training.

Particularly because the reported accuracy margins are so slim, either of these variables could modify the empirical conclusions. A statistical test on the Average performances would also improve confidence in the conclusions.

Questions and minor suggestions for the authors:

1. Why is Entropy missing from Table 2? And BALD-MC from Tables 3 and 4? And why is BALD missing again from Table 6? Computational expense is used to justify the omission of BALD in one instance, but the others appear inconsistent?
2. It’s very hard to compare the absolute differences Tables 1 and 2 for ourselves. Perhaps there is some visual representation that could help demonstrate the comparisons you make in the text?
3. Margins are very small for the Average differences across all datasets as well -- have you considered confidence intervals on those as well?
4. Minor: Only half the datasets are shown in Tables 3 and 4, but it’s unclear how/why those were chosen?


summary_of_the_review: In summary, we would encourage the authors to more rigorously examine the heart of their work -- the source of the instability they uncover: is pruning really ineffective, or are we removing too many challenging examples along with the outliers? Can we qualitatively diagnose the problem and explain what makes an example “informative” but unstable? And as accuracy margins are so small, it may be unfortunately necessary to justify or relax some hyperparameter choices, and run additional experiments to confirm these findings generalize.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper test 4 deep active learning mehods across 10 datasets (including text classification and multi-choice commonsense reasoning) based on pre-trained LMs. It conduct many ablation studies to explore whether some inherent factors of like batch size in active learning, the model size of pre-training models would affect the effectiveness of active learning with pre-trained LMs. 
main_review: Strengths: The experimental scale of this work is undoubtedly large and comprehensive, involving many data sets, different types of active learning algorithms. It is constructive for the researchers who tried to use active learning methods for reducing the labeling cost in NLP tasks. 

Weaknesses: 
1) compare with [1], the research questions and the corresponding conclusions of this work are not enough novel and attractive. Actually, current studies have involved: i) marginal benefits of deep active learning methods; ii) employing active learning with pre-trained LMs like Bert on regular NLP tasks; iii) employing active learning on challenging tasks like VQA. In fact, these studies have covered almost all existing research topics that worth studying and revealed the problems existing in deep active learning. For now, it is more meaningful to address these issues than to continue to discuss whether deep active learning has similar problems in some subdivided tasks.

2) In the experiments, the authors only adopt 500 quota for each AL experiment, it's too small. For example, in Figure 2 (CODAH), the performance of AL methods are far from convergence, the author should adopt more quota or at least provide the performance trained on full dataset, like [2].
--------------------------------------- Minor Issues -------------------------------------------

3) Many writing problems could be found in this paper. For instance, in page 2, in contribution 2, the sentence is incompleted. In section 2.2, should add ';' or '.' at the end of each item when describing the AL algorithms.

4) It is inaesthetic to plot standard error bar, like Figure 1, can use plt.fill_between instead of plt.errorbar.
5) Should pay more attention to design the layout of tables, e.g., in Table 6, it should be Method | Dataset | AL methods.


References:
[1] Karamcheti S, Krishna R, Fei-Fei L, et al. Mind your outliers! investigating the negative impact of outliers on active learning for visual question answering[J]. arXiv preprint arXiv:2107.02331, 2021.

[2] Margatina K, Barrault L, Aletras N. Bayesian Active Learning with Pretrained Language Models[J]. arXiv preprint arXiv:2104.08320, 2021.
summary_of_the_review: This paper conducted a large amount of empirical experiments on AL with pre-trained LMs. However, the research problems they explored are somewhat covered by existing research outputs and their conclusions are less attractive compared with [1]. Additionally, the writing of this article should also be improved.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the usefulness of Active Learning (AL) on a variety of NLP tasks with pre-trained models. The authors show that with pre-trained models, AL can sometimes offer inconsistent and even poorer results than random baselines. In addition to outliers, they show that training instability is a major issue. There are also various further ablation studies on this issue.

main_review: Strengths:

The topic investigated in this work is interesting and would be useful for further works since pre-trained models have been shown effective at learning with small amounts of annotations and AL is also a method with similar spirit.

This paper provides a good and solid empirical analysis on this topic with various different models, datasets and AL methods.

There are several detailed ablation analyses which I find interesting, such as the influence of question format and adversarial filtering on the datasets.

Weakness:

One of my major concerns is on the point of training instability. I feel that this might not be the problem of AL, but at least some part of the problem is that we need a more robust way to train the model? Surely the random baseline is also treated in similar ways, so the comparisons are fair. Especially, fine-tuning might be sensitive to hyper-parameters, and it seems that there are no specific tunings? Surely I understand the cost of doing this, but I think it would be nice to select some of the datasets to carry out some hyper-parameter tuning.

As discussed in Section 3.4.4, annotation batch size could be important. I think there should be more experiments on this, for example, with larger sizes rather than smaller.

One specific question that I have but might not be fully answered in this work is that: why are the examples selected by AL more likely to cause the training instability? Some experiment results show that outliers might not tell the full story. Then why does this happen, is it related with instance diversity or even simpler label diversity (for example, maybe in some iterations, AL only chooses instances with the same label)? I'm quite interested to see more analysis on this.

Other comments:

For the hyper-parameter issue above, what are the differences of multiple runs in the multi-training (in Section 3.3), only changing seed or changing hyper-parameters? If only seed, I suggest doing similar experiments with some degree of tuning of hyper-parameters.

It would be nice to have results where all the datasets are available as in standard supervised settings as upper-bounds.

It would be interesting to explore tasks beyond simple classification.

summary_of_the_review: Generally I find that this work provides a reasonable study on the usefulness of AL with pre-trained models, there are lots of experiments and there are certain interesting findings. Nevertheless, I still have concerns with the training setup and there are still some analyses that I would like to see.
review_title: null

Meta-review: The paper proposes a novel explanation for the ineffectiveness of active learning (AL), namely, that AL will select unlearnable collective outliers. 

Reviewers generally find the finding is interesting, but the paper lacks in-depth analyses. There're additional concerns on the experimental setups.
Decision: Reject