Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper investigates active learning with pretrained models to achieve better performance when using uncertainty sampling. Empirical results show that active learning with pretrained models and uncertainty sampling outperform random sampling and un-pretrained models.
strengths_and_weaknesses: Strengths:
1. This paper is very interesting and well written.
Weaknesses:
1. From the Figure 2, the image results is much better than the text results. Is it necessary to conduct more experiment on more text datasets?
questions: 1. From the Figure 2, the image results is much better than the text results. Is it necessary to conduct more experiment on more text datasets?
limitations: yes
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper studies pre-trained models for active learning to address the problem of task ambiguity where limited training data cannot fully specify expected outputs for all possible inputs. The paper performs extensive experiments to evaluate uncertainty sampling approaches for active learning using different pre-trained models and datasets (in both vison and natural languages). To standardize the experiments, the paper proposes to remove validation set, leveraging training loss to terminate training and recommended hyperparameters from prior work. The considered datasets explore both combination of causal and spurious features and robustness under distribution shifts. Based on the experiments, the paper provides several findings for active learning with pre-trained models, i.e., uncertainty sampling is more better than random sampling for active learning, pre-trained models helps active learning to select atypical combinations of features or upsample raw types of examples, pre-training is much better than random initialization for active learning, and pre-training can learn better representations and encourage more informative selection for active learning.
strengths_and_weaknesses: **Strength

+Active learning with pre-trained large-scale models is still less explored in the literature.

+The experiments are extensive and controlled. The results in this paper might provide useful baselines for future research in this important area.

**Weakness

-Some experiments in the paper are quite conventional and the findings seem to be expected given the demonstrated effectiveness of pre-trained models in many previous work. For instance, it is pretty popular in active learning research that random sampling cannot perform as well as heuristic-based selection (e.g., uncertainty sampling). Also, it is pretty expected that pre-trained models would perform better than randomly initialized models due to better accumulation of knowledge. Pre-trained models are also considered to learn better representations in general. Although the paper does contribute to verify such general knowledge for active learning and pre-trained models, it might be better to study specific challenges of active learning when pre-trained models are employed for modeling, e.g., increased training time of the large-scale models that might drive up waiting time of annotators, trade-offs of different active learning approaches with pre-trained models (e.g., uncertainty-based vs disagreement-based).

-The paper only considers classification tasks that might not well represent different tasks in machine learning, e.g., structured prediction tasks in computer vision and natural language processing. The annotation costs in such tasks might be more significant and it is interesting to explore whether pre-trained models can help to reduce the costs. Also, it is unclear if the proposed heuristics to remove the validation set can generalize to such tasks/datasets.

-Minor: An interesting observation in lines 214-216 is that larger pre-trained models have minimal effect for the accuracy of active learning methods. It might be helpful to further explore this issue. Can larger models be more beneficial if more data is selected for annotation in active learning? 
questions: Please consider the weakness presented above.
limitations: The paper includes a section to describe potential limitations of current work, including increased costs for model training, human annotation, and limited contexts to unlabeled set.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors show that under active learning, pre-trained models benefit from fine-tuning on samples where they are less confident. In contrast, models trained from scratch perform similarly or worse when trained in this way - simply sampling random batches is optimal. They show that this performance gain is modulated by how ambiguous the task is, such as the presence of confounders. In particular, a pre-trained model learns a feature space in which the causal and confounding attributes are more linearly separable, suggesting that it can more easily clarify the intended task by identifying disambiguating instances.
strengths_and_weaknesses: Strengths
The paper is clearly written, and presents clear, thorough experiments to demonstrate and explore the behavior of pre-trained models in both vision and language tasks. The hypothesis is clear, and Fig. 1 is really nice. There are interesting findings - the pre-training dataset size matters more than model size, and the benefit of the least confidence heuristic disappears if there are no explicit confounders!
The figures and text are all very clear.

Weaknesses
I feel the findings are actually quite pessimistic - the least confidence heuristic requires quite a few conditions to hold in order to be effective (task ambiguity with distribution shift, and a model pre-trained on a sufficiently large dataset). So what can be done when the conditions don't hold? Is random labeling really the best option in other cases?
The overall scope and contributions seem a little narrow. It comes off as a careful examination of a very specific phenomenon.
The paper only performs experiments on classification, where active learning is less useful since labels tend to be much cheaper than e.g. video or volumetric segmentation.
questions: Would be helpful to briefly discuss when the least confidence heuristic might be more or less appropriate than entropy and margin sampling. Even if their empirical performance is similar, can you think of reasons to choose one over another?
It seems that there should be some way to extend or connect these findings outside of active learning, e.g. in curriculum learning, continual learning, or even in standard learning settings with distributional shift. Something to think about.
limitations: Limitations are well characterized.
review_title: null

Meta-review: metareview: The paper makes a simple but compelling claim, that using large pretrained models within an active learning loop improves the ability to collect labeled data efficiently for improved model training with limited labeling resources.  All revieweres agreed that the work is technically correct, that the empirical verification was thoroughly done.

The main point of contention among reviewers was whether this idea is novel or surprising enough to warrant publication at NeurIPS.  This is a tricky issue, and I spent considerable time digging into the paper itself after reading through all reviews and author responses.  In the end, I believe that it is important for the field to avoid creating a world in which only sensational-sounding ideas are published.  Yes, it is relatively intuitive that using a learned representation such as a large pretrained model will help put active learning in a lower dimensional space that will be easier to search through, making AL more efficient.  But the field on ML is filled with ideas that sound good in the abstract but fail in practice.  I believe that there is clear value to the field to having rigorous evaluation of obvious good ideas, both as a reality check and also to provide a clear documentation of current state of the art so that the field can build from there.

confidence: Less certain
Decision: Accept