Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
* This paper proposes a learning algorithm that interleaves data annotation and model updating to reduce the amount of labeled data needed.
* The algorithm requests labels for the most “informative” (or uncertain) instances with respect to the current model.
* The algorithm additionally proposes a loss that combines fitting data and controlling the degree of model update.
* Results are reported on SQuAD and NewsQA and the finding is that one needs 25% less samples for same or better performance.

Strengths:
* The work presents an interesting comparison of active learning algorithms on SQuAD and NewsQA. The results on SQuAD are promising, even though they are well below SOTA. It might have been more interesting to see this approach in action on a much more low resource dataset, perhaps publishing new labeled data to be used for the task.

Weaknesses:
* Novelty is limited. This paper seems to be a fairly straightforward application of active learning.
* Results are positive, but not exciting or surprising. Having to collect 25% less data is not so desirable, especially if there is suspicion that the active learning algorithm might introduce unwanted biases.
* On page 2, the authors discuss how their algorithm allows the model to only fit the new uncertain instances rather than retraining the model from scratch. However this improvement could be an unnecessary complication. If the focus of the work is on reducing the need for labeled data, wouldn’t it be better to simply retrain the model on all the labeled data every time a new annotated batch of data comes in? 
* Even though the authors experimented only on Machine Reading Comprehension, there is no modeling specific to this task. This is good for generality, but it calls into question the choice of the authors to make MRC a central topic of the paper. It would have maybe been more convincing to frame this work as focused on active learning and then investigate its effect in multiple applications.
* MRC will often have multiple candidate answers with high probability, but these candidates are not truly different. They will in many cases correspond to mostly overlapping spans, or to similar answers appearing in different parts of the passage. Intuitively it seems that this algorithm would focus annotators on manually labeling these cases, which could arguably be suboptimal use of annotator time.
* No real analysis of the results is presented. It might have been interesting to see what data is actually being selected by the active learning algorithm or what questions the model was learning to answer as active learning progressed.

review_title: A simulated application of active learning to SQuAD and NewsQA

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper is tackling the problem of labeling cost in Machine Reading comprehension.
The paper uses an uncertainty-based approach in the context of extractive machine reading to choose the point to annotate.
In addition, the paper uses an adaptive loss minimization schema that consists of penalizing large moves in the parameter space which strongly related trusted region types of approaches[1].
The authors compare to 6 baselines with mitigated results.
While the paper is well-written with reasonable experiments on two machine reading datasets, I can mention three limitations for acceptance which are 

(1) the lack of strong novelty beyond uncertainty-based active learning applied to MRC: In its current status, the paper is proposing to measure uncertainty as entropy over the probability distribution of each token to be the start and end token of the contiguous string constituting the answer. I find this proposition pretty interesting but trivial compare to uncertainty based active learning literature applied to text [2, 3]. 

(2) lack of significant improvement on the considered datasets: First, the results are difficult to compare to SoA results (https://rajpurkar.github.io/SQuAD-explorer/) as the amount of data-point is different (Table 1). Second, the difference between the baseline methods and the proposed approach doesn't seem significant (Figure 4). Furthermore, it is currently not possible to precisely assess the result significance as the variance of the results isn't reported.

(3) lack of numeral results justifying the use of the adaptive loss in this context: unfortunately, the authors proposed a second contribution related to the loss function (section 2.2) which is not evaluated with an ablation study to assess its possible utility.

Refs
[1] Trust Region Policy Optimization, Schulman and al, 2015
[2] Uncertainty-based active learning with instability estimation for text classification, Zhu and al, 2017
[3] Active Learning Using Uncertainty Information, Yang and al, 2017

review_title: Valuable study on active learning applied to machine reading with limited novely

final_score: rating: 3: Clear rejection
review_content: ### Summary

This paper proposes to apply uncertainty-based measures to guide the collection of training samples for reading comprehension. The paper describes a relatively simple metric to estimate model uncertainty of unlabeled examples, and develops an algorithm to sample examples that exhibit least model certainty. They describe a learning and regularization model for this scenario and evaluate their proposal on SQuAD and NewsQA datasets.

### Strong and weak points

Strong points:
- The use of uncertainty-based measures for active learning seem intuitive and elegant,
- The experimental evaluation contains a few interesting baselines and provide context to this work

### Weak points:

- The idea of using model uncertainty for sample efficiency is somewhat standard now. A few missing references on this are, for example [1], [2] and [3]. It is even described in blog articles online [4].  After reading the paper, it was not clear what part of the proposal is really novel, and which parts are well established in the literature.
Part of the work described seems to be dedicated to performing learning while sampling “in a loop” (Algorithm 1). Unless I’ve misunderstood the work, this assumption seems to be quite different than the premise of the work. After running an iteration of model training, new samples may be requested for annotation. In a real application, it would take hours, or most likely days, to receive new examples with labels. As such, I’m not sure it makes sense to design an algorithm that expects the new samples on every batch/iteration.  Given the time budgets, I would expect that models can be easily trained “from scratch” after new examples are available.

- I found the paper hard to follow at times. For example:
    - In Section 2.1.1 there seems to be use of informal notation: “We use a zero-one vector a to indicate the correct answer, ”. But at this point in the paper it’s unclear what the vector space represents (tokens in the input?). Later, in Section 3, a span-based representation is introduced.
    - The thresholding logic was hard to follow and understand the rationale. Are you simply sampling amongst the most uncertain samples? Can you not specify a $k$ to sample from? Why have an adaptive threshold?
    - The baselines models are not really explained clearly, with a single sentence per model,

- Overall, the results are quite inconclusive. Although there is modest improvement in SQuAD benchmark, it does not seem to be the case for NewsQA (the NewsQA results are only available as a chart, which makes comparison difficult). Also, in the SQuAD setup, the Random sampling method seems to generally be as good or better than the other baselines. There are no explanations as to why this may happen. Are the results being averaged over several runs? Do they represent the performance of a single run? Could the variations be attributed to natural variation in model training?
 
References used above:

[1] Heterogeneous uncertainty sampling for supervised learning
David D. Lewis and Jason Catlett
https://scholar.google.com/scholar?cluster=9211137857521772693&hl=en&as_sdt=0,33 

[2] Deep Bayesian Active Learning for Multiple Correct Outputs
https://arxiv.org/abs/1912.01119 

[3] You Need Only Uncertain Answers: Data Efficient Multilingual Question Answering
http://www.gatsby.ucl.ac.uk/~balaji/udl2020/accepted-papers/UDL2020-paper-113.pdf 

[4] Active learning — Uncertainty Sampling (P3)
https://medium.com/@duyanhnguyen_38925/active-learning-uncertainty-sampling-p3-edd1f5a655ac 

### Recommendation

I would recommend not to publish this work at ICLR at this time.  As described above, it is unclear what the novelty of this study is, besides applying an established technique (uncertainty-based sampling) to machine comprehension tasks. I have concerns regarding the premise of the learning setup (incremental learning assuming new examples can be sampled). Finally, the results seem inconclusive to me, and it is unclear whether statistical variations are dominating effects in SQuAD and NewsQA.

### Questions for authors

Please refer to the “weak points” described above.

### Additional feedback

- Besides the technical work, I think the paper could benefit from improved writing. Currently, some parts are a bit hard to read. 
- Please separate the notation, task definition and model design into separate (sub)sections, so readers can read the paper linearly.
- Overall, the writing could use some editorial help.
- Page 5, the text describes the best performance for 5k examples as “64.07%”, however,  Table 1 states it is “64.03%”.

review_title: Study has unclear novelty, and has some potentially important experimental flaws

final_score: rating: 4: Ok but not good enough - rejection
review_content: **Update after author response:** I appreciate the authors' efforts to address my concerns. Thanks for the correction on QBC, I appreciate it. I still believe that the paper needs to accompany a more comprehensive evaluation and qualitative insights to highlight the effectiveness of the proposed method. For instance, it is common practice in Active Learning to report mean accuracy over multiple runs of the same experiment as data is sampled based on a particular heuristic which isn't always deterministic. Furthermore, the choice of warmstart samples could also influence the results, which is why it is recommended to conduct multiple runs of the same experiments and report mean performance. In such a scenario, any claims that arise from only one run of the experiments (as in this paper) should be taken with a grain of salt. I also appreciate the pointer to Equation 4 but how does it translate in practice is another important piece that is missing. BERT based models produce highly confident predictions and to visualize the distribution from your empirical investigation would help bridge the divide between the equations and the empirical results (how they actually turn out in practice). I am also not convinced by the authors' response to why the difference in performance of the BERT model trained on randomly sampled data vs the model trained on data sampled via ALBUS stays within the 1-2% range, often increasing with more data. I believe there are critical questions regarding this paper that need to be addressed before the paper is published, hence, my score remains unchanged.

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Summary: The paper proposes a new method to actively sample data to train models in a limited labeled data regime. Focusing on passage based question answering, the authors employ an uncertainty based sampling strategy by selecting examples on which a model has little top-1 prediction confidence. In every training iteration, the model is trained with an adaptive regularizer in an attempt to keep the new model closer to the current model. The results presented in the paper show improvement over three active learning baselines as well as training over iid sampled data on the SQuAD dataset.

Pros:
- The paper looks at an important problem which is little studied in context of deep learning based NLP systems, particularly with pre-trained transformers. 
- The two components that together form ALBUS are simple and the authors show that they perform better than various active learning baselines as well as random sampling on the SQuAD dataset.
- I like the approach of ensuring that the model weights do not deviate too much as this also addresses the challenges faced in active learning when we end up sampling outliers.
- The paper is easy to follow.

Cons:
- All evaluations are conducted using a Bert base model. It is unclear whether this method generalizes to other models or not. Furthermore, the benefits of this approach on NewsQA are not easily interpretable, in fact the plots suggest that in many cases, baseline methods outperform this method. What happens when less than 20000 NewsQA examples are sampled for labeling is also not shown so it is hard to meaningfully interpret the results.
- Common and effective Active Learning baselines such as Query by Committee have not been compared against.
- The uncertainty based sampling method samples examples on which a model is less confident (determined by a threshold) on its top-1 prediction, however, the paper presents no qualitative analysis to show that this approach is indeed meaningful in context of models like BERT which, to the best of my knowledge, happen to have very confident predictions. It would help if the authors show how the distribution of the metric of informativeness over examples, and as more and more data is labeled.

Questions for authors for rebuttal:
- It would be great to share actual numbers for NewsQA besides the existing plots. It is clear from the table and the figure that your method consistently outperforms baselines on SQuAD but it is less clear as to what happens in NewsQA as the only figures present are less clear and show that other baselines often outperform your proposed method.
- Several experimental details are missing from the paper that make it hard to put the numbers into context. Are the results averaged over multiple runs? If yes, how many? And if not, then I’d recommend averaging results for each approach over at least 3 runs. 
- Since the authors have not shared their code with the submission, it is important to share how certain hyperparameter values were obtained. What influenced the choice to finetune BERT for only 2 epochs for both SQuAD and NewsQA or what influenced the choice of the number of datapoints for warm starting, etc.?
- On SQuAD, it appears that the difference in performance of the BERT model trained on randomly sampled data vs the model trained on data sampled via ALBUS stays within the 1-2% range, sometimes increasing with more data. Intuitively, this gap should be decreasing with more data (which does happen with other baselines that you’ve compared against). Why do you think this gap stays more or less the same with your method regardless of the amount of labeled data?

Typos and other suggestions:
- Page 2, second paragraph last line: “more faster” -> faster
- Page 2, first line: a more appropriate reference for Active Learning would be Cohn et al., 1996. Cohn, D. A., Ghahramani, Z., & Jordan, M. I. (1996). Active learning with statistical models. Journal of artificial intelligence research, 4, 129-145.
- Even though the paper is easy to follow, I believe the writing can be improved, some sections appear to be written in a casual manner, for instance (these are representative, not exhaustive): 
(i) line 4 of introduction “smooth-talking AI systems.” Consider using a better term to describe these systems.
(ii) lines 13 and 14 of abstract “demonstrate … that 25% less labeled samples suffice to guarantee.” Since the paper does not establish theoretical guarantees for this method, I would recommend not to use words like guarantee in this context.

Missing references:

- Siddhant, A., & Lipton, Z. C. (2018). Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Study. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.

Reasons for the score:
I think there are some outstanding questions regarding this paper that can be addressed in the author response, but until then, the proposed approach and associated results are less than convincing. I’m more than willing to increase my score following the author response.
review_title: New uncertainty-based adaptive learning method for passage based question answering but weak evaluation and analysis

Meta-review: All reviewers agree that the current approach is very similar to traditional uncertainty-based active learning, and that the empirical results are inconclusive, so at this point the paper is not ready for publication.
Decision: Reject