Uncertainty-Based Adaptive Learning for Reading Comprehension
Authors: Jing Wang
Jie Shen
Xiaofei Ma
Andrew Arnold
Venue:  (2021)
Keywords: machine reading comprehension
uncertainty-based sampling
adaptive loss minimization

Abstract: Recent years have witnessed a surge of successful applications of machine reading comprehension. Of central importance to the tasks is the availability of massive amount of labeled data, which facilitates the training of large-scale neural networks. However, in many real-world problems, annotated data are expensive to gather not only because of time cost and budget, but also of certain domain-specific restrictions such as privacy for healthcare data. In this regard, we propose an uncertainty-based adaptive learning algorithm for reading comprehension, which interleaves data annotation and model updating to mitigate the demand of labeling. Our key techniques are two-fold: 1) an unsupervised uncertainty-based sampling scheme that queries the labels of the most informative instances with respect to the currently learned model; and 2) an adaptive loss minimization paradigm that simultaneously fits the data and controls the degree of model updating. We demonstrate on the benchmark datasets that 25\% less labeled samples suffice to guarantee similar, or even improved performance. Our results demonstrate a strong evidence that for label-demanding scenarios, the proposed approach offers a practical guide on data collection and model training. 

Under review as a conference paper at ICLR 2021
UNCERTAINTY-BASED ADAPTIVE LEARNING FOR READING COMPREHENSION
Anonymous authors Paper under double-blind review
ABSTRACT
Recent years have witnessed a surge of successful applications of machine read- ing comprehension. Of central importance the tasks is the availability of mas- sive amount of labeled data, which facilitates the training of large-scale neural networks. However, many real- world problems, annotated data are expensive to gather not only because of time cost and budget, but also of certain domain- specific restrictions such as privacy for healthcare data. In this regard, we pro- pose an uncertainty-bast which interleaves data annotation and model updating to mitigate the demand of labeling. Our key techniques are two-fold: 1) unsupervised uncertainty- based sampling scheme that queries the labels of the most informative instances with respect to the currently learned model; and 2) an adaptive loss minimization paradigm that simultancously fits the data and controls the degree of model up- dating. We demonstrate on the benchmark dataset that 25% less labeled samples suffice to guarantee similar, or even improved performance. Our results demon- strate strong evidence that for laban-de scenarios, the proposed approach offers a practical guide on data collection and model training.
adaptive learning algorithm for reading comprehension,
INTRODUCTION
The goal of machine reading comprehension (MRC) is to train an Al model which is able to under- stand natural language text (e.g. a passage), and answer questions related it (Hirschman et al.] 1999); see Figure for an example. MRC has been one of the most important problems in nat- ural language processing thanks to its various successful applications, such as smooth-talking Al speaker assistants technology that was highlighted as among 10 breakthrough technologies by MIT Technology Review very recently (Karen 2019).
Of central importance to MRC is the availability of benchmarking question-answeriig datasets, where a larger dataset often enables training of a more informative neural networks. this regard, there have been number of benchmark datasets proposed in recent years, with the efforts of pushing forward the development of MRC. A partial list includes SQuAD (Rajpurkar et al.) 2016). NewsQA Triichher al. 2017), MSMARCO (Nguyen et al. 2016), and Natural Questions (kwiatkosski et al. 2019). While the emergence of these high-quality datasets have stimulated a surge of research and have witness a large volume of deployments of MRC, it is often challenging to go beyond the scale of the current architectures of neural networks, in that it is extremely expensive to obtain massive amount of labeled data. The barrier of data collection can be seen from SQuAD: the research group at Standford University spent 1,547 working hours for the annotation of SQuAD dataset, with the cost over $14.000. This issue was also set out and addressed by AI companies. However, even equipped with machine learning assisted labeling tools (e.g. Amazon SageMaker Ground Truth), is still expensive to hire and educate expert workers for annotation. What makes the issue more serious is that there is rise of security and privacy concerns in various problems, which prevents researchers from scaling their projects to diverse domains efficiently. For example, all annotators advised get series of training about privacy rules, such Health Insurance Portability & Accountability Act, before they can work on the medical records.
In this work, tackle the challenge by proposing computationally efficient learning algorithm that is amenable for debanddggg problems. Unlike prior MRC methods that separate data an- notation and model training, our algorithm interleaves these phases. Our algorithm, in spirit,

---

Under review as a conference paper at ICLR 2021
Question:
What causes precipitation to fall?
meteorology, precipitation is any product of the condensation of atmo- spheric water vapor that falls under gravity The main forms... intense periods of rain in scattered locations are called "shower".
Passage:
Answer: gravity
Figure 1: An illustrative example in the SQuAD dataset (Rajpurkar et al. 2016).
resembles the theme active learning (Balcan et al., 2007), where the promise of active learning is that always concentrate on fitting only the most informative examples without suffering a degraded performance. While there have been a considerable number of works showing that ac- tive learning often guarantees exponential savings of labels, the analysis holds typically for linear classification models /Awasthi et al.] (2017): Zhang (208);; Zhang et al.] (2020). In stark contrast, less is explored for the more practical neural network based models since it is nontrivial extend important concepts such as large margin of linear classifiers to neural networks. As remedy, we consider an unsupervised sampling scheme based on the uncertainty of the instances (Settles) 2009). Our sampling scheme is adaptive (i.e. active) in the sense that it chooses instances that the cur- rently learned model is most uncertain on. this end, we recall that the purpose of MRC is to take as input passage and question, and finds the most accurate answer from the passage. Roughly speaking. this can be thought of as weight assignment problem, where we need to calculate how likely each word span in the passage could be the correct answer. Ideally, we would hope that the algorithm assigns to the correct answer, and assigns to the remaining, leading to a large separa- tion between the correct and those incorrect. Alternatively, if the algorithm assigns, say 0.5 to two different answers and assigns to others, then it is very uncertain about its response this is a strong criterion that we need query the correct answer an expert, i.e. performing active labeling. Our mncertainty-based sampling scheme is essentially motivated by this observation: the uncertainty of an instance (i.e. pair of passage and question) is defined as the gap between the weight of the best candidate answer and the second best. We will present a more formal description in Section 2
After identifying these most uncertain, and hence most informative instances, we query their labels and use them to update the model. In this phase, in addition to minimize the widely used entropy- based loss function, we consider an adaptive regularizer which has two important properties. First, it enforces that the new model will not deviate far from the current model, since 1) with reasonable initialization we would expect that the initial model should perform not too bad; and 2) we do not want to overfit the data even if they are recognized as informative. Second, the regularizer has a coefficient that increasing with iterations. Namely, as the algorithm proceeds the stability of model updating outweighs loss minimization. In Section we elaborate on the concrete form of our objective function. is also worth mentioning that since in each iteration, the algorithm only fits the uncertain instances, the model updating is more faster than traditional methods.
The pipeline is illustrated in Figure 2 Given abundant unlabeled instances, our algorithm first evaluates their uncertainty and detects the most informative ones, marked red. Then send these instances to an expert to obtain the groundtruth answers, marked as yellow. With the newly added labeled samples, it is possible perform incremental updating of the MRC model.
Roadmap, We summarize our main technical contributions below, and discuss more related works in Section 5 In Section we present a detailed description of the core components of our algorithm, and in Section 3 we provide an end-to-end learning paradigm for MRC with implementation details. In Section 4 we demonstrate the efficacy of our algorithm in terms of exact match, F-1 score, and the savings of labels. Finally we conclude this paper in Section 6
1.1 SUMMARY OF CONTRIBUTIONS
We consider the problem learning an MRC model in the label-demanding context, and we pro- pose a novel algorithm that interleaves data annotation and model updating. In particular, there sampling scheme that only
two core components for this end: an unsupervised uncertainty-bast queries labels of the most informative instances with respect to the currently learned model, and
2

---

Under review as a conference paper at ICLR 2021
Passage
Question
Passage
Question
Passage
Question
Pessage
Question
expert
Question
Passage
Answer
MRC
MRC model
Passage
Question
Passage
Question
Passage
Answer
Passage
Question
Passage
Question
Figure 2: Illustration of our learning algorithm in each iteration. Given pool of unlabeled pairs of passage-uesstins, the algorithm first identifies the instances that it is most uncertain on, e.g. those marked in red. Then queries an expert to gather the answers (i.e. labels), and restricts to fit the newly labeled instances.
an adaptive loss minimization paradigm that simultancously fits the data and controls the degree of model updating. Moreover, our approach is modular nature, meaning that the community would benefit from this work by leveraging techniques into more real-world problems (e.g. image classification) where the availability of labels a major concern.
ALGORITHM
In this section, we formally introduce the problem setup and our main algorithm ALBUS (Algo- rithm DI. We use (p, q) to represent pair of passage p and question q. which is also called an unlabeled instance, or simply an instance. there are multiple questions, say 4i, 92. to a same passage p, will use two instances I1 (p. 91) and 22 (p. (2) Given an instance æ, goal is to predict an answer. We use a zero-one vector a to indicate the correct answer, and (z, a) called a labeled instance. The prediction made by the learner is denoted by â. We will always assume that all the coordinates of à are non-negative, and their sum equals one, which can be easily satisfied if the last layer of the neural network is softmax.
2.1 UNSUPERVISED UNCERTAINTY-BANR BASED RANDOM SAMPLING
Since data annotation is expensive, we treat the problem as such that all the instances are unlaheled before running the algorithm, and as the algorithm proceeds, it may adaptively detects the most informative instances to be labeled by experts or crowd workers. Thus, the central questions to learning are: 1) how measure the informativeness of the unlabeled instances in a computationally efficient manner, and 2) how to select manageable number of instances for annotation (since the algorithm might identify bunch of useful instances). We address both questions the following.
2.1.1 METRIC OF INFORMATIVENESS
Intuition. We first address the first question, i.e. design metric to evaluate the informativeness. To ease the discussion, suppose that for given instance æ, there are only two answers to choose from, iec. is two-dimensional vector, and that the algorithm has been initialized, e.g. via pre- pre-training. the current model takes as input æ, and predicts = (1, 0), then we think of this instance as less informative, in that the algorithm has extremely high confidence its prediction! On the other spectrum, the prediction à = (0.5, 0.5), then it indicates that the current model is not able distinguish the two answers. Thus, sending the correct answer together with the instance the algorithm will lead to significant progress.
We observe that underlying the intuition is notion of separation between the answer with highest confidence and that with second highest, denoted by (x), where w denotes the current model
'The algorithm may of course make mistake, but this will be treated by future model updating. Here are just giving an intuitive explanation following the idealized scenario.
3

---

Under review as a conference paper at ICLR 2021
Algorithm ALBUS:
Adaptive Learning By Uncertainty-Bast Sampling
Require:
a set of unlabeled instances = [.1,..... Fn initial MRC model wo, maximum itera-
tion number I", thresholds (Ti,..., 7T) number of instances be labeled no.
Ensure: A new MRC model wy-
1: U1 U.
2: for fort=1,... do
3:
Compute A , (a)) for all Uz.
4: B, < {xc E U, A (+)
5: Compute the sampling probability Pr(x) for all Br.
6: St < randomly choose no instances from Be by the distribution (Pr(z))eEB,, and query
their labels.
7: Update the model wi + arg min,, St):St).
8: U2+1 S.
9: end for
parameters. fact, let our algorithm be a function fm : x + à. Denote by a(1) and â(2) the highest and second highest value in à. Then
(x) (1)) (2)
(1)
Given the unlabeled training set {x1, ....... Fn and the currently learned model, we can evaluate where we write A; := (=.x;; to reduce notation clutter since most of the time, the model un clear from the context. This answers the first question
the degree of separation (Ai, ...... An)
proposed at the beginning of the section, i.e. how to measure the informativeness of the instances.
2.1.2 UNCERTAINTY-BAN BASED SAMPLING
remains to design a mechanism so that we can gather a manageable number of instances to be labeled. A natural approach will be specifying the maximum number no. so that in each iteration the algorithm chooses at most no instances with lowest degree of separation. Yet, we observe that for some marginal cases, many instances have very close A,, e.g. Å₁ 0.101 and A₃ 0.102. Using the above strategy may annotate while throwing away æz- From the practical perspective, however, we hope both instances have chance to be selected to increase diversity. Henceforth, we consider a "soft" approach based on random sampling in this paper.
Fix an iteration of the algorithm. First, we define threshold 7, (0, 1]. Based on the current Then we obtain sampling region
model 11,1, we calculate A An.
B
(x A S 7
which contains informative instances (recall that lower degree separation implies more infor- mative). Inspired by the probability selection scheme (Abe & Long 1999), define the sampling probability as
Vo E x,
Pr(x)
B. B.ltyA A. Ex' B.I+ A..)
(3)
when x"
In the above expression, x² is the instance in B. with lowest degree of separation, i.c. the most uncertain instance; 7 0 is tunable typer-parameters, Observe that if y 0, it becomes uniform sampling. addition, in view of the sampling probability in (3), the instance æ * x* will be sampled with probability less than B/) and x² is sampled with probability more than 1/p,
B B,
Pr(x*) 21
Σ s/ae"
= 1
(4)
B
|B|
Therefore, the sampling scheme always guarantees that x= will be selected with highest probability, and if needed, it is possible make this probability close to 1 by increasing 7. In our algorithm,
7 0( (B.) which works well in practice.
4

---

Under review as a conference paper at ICLR 2021
2.2 ADAPTIVE LOSS MINIMIZATION
Another crucial component in ALBUS is loss minimization. Here our novelty is an introduction of an adaptive regularizer that balances the progress of model updating and per-iteration data fitting.
Let St be the set of labeled instances determined by our random sampling method at the (-th itera- tion. For any (ax a) E St, since a is an indicator vector, the problem can be thought of as multi- classification. Therefore, a typical choice of sample-wise loss function is logistic loss, denoted by flw; æ, a), which can be easily implemented by using softmax layer in the neural network. On top of the logistic loss, we also consider an adaptive Eg-norm regularizer, which gives the following objective function:
Ae 1| 1 2
L(w; St) = |Sel
Σ (m,a)ES
L(w;; F. 0) +
(5)
Different from the broadly utilized fz-norm regularizer || we appeal to a localized form, in the sense that the objective function pushes the updated model to be close to the current model wt-1 under Euclidean distance. This motivated by the fact that in many cases, warm starting the algorithm with pre-training often exhibits favorable performance. Hence, though we want the model to be adapted to the new dataset, we carefully control the progress of model updating so that it does not deviate far from the current.
Regarding the coefficient λt, we increase it by a constant factor greater than one in each iteration. Therefore, as the algorithm proceeds, the localization property plays more important role than the logistic loss. Our treatment inspired by the literature of active learning, where similar localized fz-norm constraint is imposed (Balcan al., 2007 Zhang al., 2020). This can be viewed stability property of our algorithm, and we discover that it works very well on benchmark datasets.
3 IMPLEMENTATION DETAILS
Uncertainty-basee sampling. We introduce how to select the batch S. in each iteration with current MRC model w;-1- For given pair of (p. q). answer is of the form of word span from the i-th position to the j-th position the passage. Given the span (i,j) and the passage p, we use BERT (Devlin et al. 2019) as our embedding method, which produces a feature description denoted by (i. We then construct a probability matrix M whose jith entry Mc, is given by the following:
exp(w, 5))
Mr, E exp(w,,11 (i'
(6)
Observe that the matrix M forms a distribution over all possible word spans, i.c. all possible an- swers. It is then straightforward to convert M into the vector à, for example, by concatenating all the columns. Based on the obtained answer à, we are able to perform uncertainty-baser sampling as discussed in Section
Adaptive loss mininization. We already derived the probability matrix M in 66. During loss minimization, i.e. supervised fine-tuning, we aim to update w by minimizing [(u;; St). Since have clarified the regularizer, it suffices to give the detailed form of the loss (ww; ar, a.) where recall that æ (p, 4). Note that using the groundtruth answer a, we know the correct span (ia ja) for question q. Thus, the likelihood that we observe S, is
exp(w w Ep(la:aa)
Pr(S,) II
(p.q.a)Es, Exs exp(w (i,
The loss function (ww; is simply the negative log-likelihood.
4 EXPERIMENTS
Datasets. We focus on the span-based datasets, namely Stanford Question Answering Dataset (SQuAD) (Rajpurkar al. 2016) and NewsQA (Trischler al. .01A).QQuAD consists of over
5

---

Under review as a conference paper at ICLR 2021
Table 1: EM and Fl score on the SQuAD dataset.
EM
Fl score
#Labels queried
Badge
Cont
Entropy
Margin
Rand
Badge
Entropy
Margin
Rand
Ours
5000
60.94
59.62
60.52
62.71
62.58
64.03
72.20
72.29
72.78
74.28
73,97
75.30
15000
71.75
72.03
71.89
72.69
71.34
74.13
81.62
82.18
82.31
82.59
81.41
83.50
21000
73.88
74.38
74.67
74.31
73.75
75.48
83.54
83,90
84,25
83.77
83.08
84.53
41000
77.55
77.37
77.80
78.16
75.86
79.02
86.02
85,69
85.95
86.19
84.51
87.09
61000
77.90
77.98
77.75
78.06
77.98
80.44
86.15
85,86
85.43
86.32
86,
88.13
81000
76.08
76.08
75.66
76.34
78.03
81.14
84.75
84.08
83.74
84.64
86.98
88.33
100k questions posed by crowdworkers on a set of 536 Wikipedia articles. We use the original split 87,599 questions for training and test on the 10,570 questions. NewsQA is a machine comprehen- sion dataset of over 100k saman--genertte question-answer pairs from over 10k news articles from CNN. The dataset consists of 74, 160 questions for training and 4,212 questions for validation
Evaluation Metrics. We use two metries: Exact Match (EM) and Fl score. EM measures the percentage of predictions that matches any one of the annotated answers exactly. EM gives credit for predictions that exactly match (one of) the gold answers. Fl score measures the average overlap between the prediction and the annotated answer.
Baselines. We compare against the following baseline algorithms:
Badge (batched based sampling) (Ash et al. 2020y:
it learns the gradient embedding of samples and selects a set of samples by k MEEANS++ (Arthur & Vassilvitskii] 2007).
Conf (confidence sampling) (Wang & Shang that selects samples with lowest class probability.
2014 : it is an uncertainty-based algorithm
Entropy (Wang & Shang 2014): : it selects samples based on the entropy of the predicted
probability distribution.
Marg (margin based sampling) (Roth & Small 2006): it also checks the degree of separa- tion as our algorithm, but selects the no lowest rather than performing random sampling as we did.
Rand (Random sampling): is the naive baseline of uniformly randomly selecting samples from unlabeled set.
Other Settings. To ensure a comprehensive comparison among state-of-the-art approaches, we simulate the annotation process with human experts in the loop by selecting a fixed number of examples ng to query their labels from training set in each iteration (we set no = 2, 000 for SQuAD and no = 5, 000 for NewsQA). The labeled data is used to update the MRC model. We report the exact match and Fl score with the number of iterations. The BERT-base is used as the pretrained model and fine-tuned for 2 epochs with learning rate of 3e 5 and a batch size of 1 The MRC model is initialized with 1,000 labeled samples for SQuAD and 10,000 for NewsQA. The parameter To is tuned from the range of [0.01, 0.1] on the training set and decreases at the rate of 1.1.
Results. Figure 3 and Figure 4 present EM and Fl score with the increase of the number of labeled samples selected by various active learning algorithms. We show the results with all labeled data (Figure 3]a and Figure a)) and 20,000 labeled data (Figure Bib) and Figure b)). Our algorithm outperforms state-of-the-art active learning algorithms in almost all the cases.
Table I lists some detailed results with specific number of labeled samples. Our algorithm reaches the best performance in all cases and the advantage is significant specially with a small subset of labeled samples available. For example, in the case of 5,000 labeled examples, our algorithm reaches the EM of 64.07 % while the best of compared algorithms is 62.71 21.71% %. Figure 3 and Figure plot the trend EM and Fl score with the rise of labeled examples on SQuAD dataset. We observe that all active learning algorithms reach the best performance before accessing all labeled data compared with Rand. It demonstrates the active learning effectively reduces the number of required labeled data for learning process. Specifically, our algorithm reaches EM 80.44 % and F1 score 88.53 % with 61,000 queries which is close to the best result but with 25% less labeled samples. We can observe the same advantage our algorithm on the NewsQA dataset as shown in Figure

---

Under review as a conference paper at ICLR 2021
SQuAD Dataset
SQuAD Dataset
80
70
Methods
60
Methods
60
Badge
Badge
Conf
50
Conf
50
Entropy
Entropy
Margin
Margin
40
Rand
Rand
Ours
OurS
30
30
0
20000
40000
60000
80000
5000
10000
15000
20000
# #Labels queried
#Labels queried
Figure 3: Compared results of EM on SQuAD with over 80k (left) and 20k (right) labeled data.
SQuAD Dataset
SQUAD Dataset
90
80
Methods
Methods
70
70
Badge
Badge
Conf
Conf
60
Entropy
60
Entropy
Margin
Margin
50
Rand
Rand
Ours
Ours
20000
40000
60000
80000
5000
10000
15000
20000
# #Labels queried
# Labbels queried
Figure 4: Compared results of Fl score on SQuAD with over 80k (left) and 20k (right) labeled data.
RELATED WORKS
Machine Reading Comprehension. MRC the ability to read text and answer questions about it.
It is challenging task as it requires the abilities of understanding both the questions and the context.
data-driven approach to reading comprehension goes back to (Hirschman et al. 1999). There are
number of works have proposed to create datasets (Rajpurkar et al. 2016;] 2018 Kwiatkowski
et al., 2019, Reddy et al. 2019). For example, Stanford Question Answering Dataset (SQuAD) set of Wikipedia articles (Rajpurkar et al., 2016). Natural Questions dataset (K Ikwiatkowski et al. 2019) consists of queries issued to the Google search engine, the wikipedia page, long answers and short answers.
dataset consists of 100K questions
Recently, researchers are devoted to develop unsupervised deep learning frameworks to learn the word representation based on a batch of unlabeled data which could be simply fine-tuned for multiple downstream tasks. For example, ELMo (Peters et al. 2018) learned forward and backward language models: the forward one reads the text from left right, and the other encodes the text from right to left. GPT (Radford et al., 2018) used a left-to-right Transformer to predict a text sequence word- by-word. Devlin al. (2019) designed BERT to pre- traitrain deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. There are some following works aiming improve the framework of BERT for different language modeling tasks Yang et al. 2019 Dai et al. 2019; Dong et al. 2019 BERT significantly improves the performance of natural language understanding tasks. Our work is based on a pretrained BERT model. We fine- tuned BERT with additional output layer to create the model for the reading comprehension task following DDvlin et al. 2019).
Another direction in reading comprehension is to explore different real-world settings. For example, in open-domain reading comprehension. the passage that contains the answer is not provided but re- quires retrieval from the knowledge pool (Wang et al. 2019). Yue et al. 2020) considered language
7

---

Under review as a conference paper at ICLR 2021
NewsQA Dataset
NewsQA Dataset
67.5
50
65.0
Methods
Methods
45
Badge
Badge
Conf
60.0
Conf
Entropy
57.5
Entropy
40
Margin
Margin
Rand
Rand
Ours
52.5
Ours
20000 40000 60000 80000
1000001 20000000000000000 140000
20000 40000 60000 80000 100000 120000 140000
#Labels queried
#Laablls queried
Figure 5: Compared results EM and Fl score on NewsQA with increase labeled data.
understanding of clinical data. This work considers the general reading comprehension setting in which the question and related passage provided.
Active Learning. Active learning is a machine learning paradigm that mainly aims at reducing label requirements through interacting with the oracle (experts/annotatorr) (Settles) 2009).
Active learning has been well studied both theory and applications. One popular branch of the- oretical works follows the PAC active learning setting: researchers are engaged reduce the label complexity, i.e. the number of query requests, with the error bound guarantee for the produced halfspace with high probability Valiant] (1984;; Balcann al. (2009); Hanneke (2014). There are some exciting works focusing on active halispace learning, such as margin based active learning
(Balcan al., 2007 Balcan & Long 2013). These approaches learn a classifier from the class of homogeneous linear classifiers, to predict labels from instances.
Existing active learning approaches can be roughly divided to uncertainty- based sampling and rep- resentative sampling (Settles) 2009). Representative sampling based approaches select samples that are representative the whole unlaheled dataset. It can achieved by performing an optimization minimizing the difference between the selected subset and the global dataset (Sener & Savarese 2018 Gissin & Shalev-Shwartz 2019). The uncertainty sampling based algorithms select samples that maximally reduce the uncertainty the algorithm has target learning model, such as samples lying closest to the current decision boundary (Tür et al. 2005). The work in this paper belongs to nncertainty-basee sampling but using novel sampling scheme tailored to MRC.
Active learning has shown outstanding performance in real-world applications, such as computer vi- sion Joshi et al. 2009) and natural language processing (Culotta & McCallum 2005 Reichart al., 2008 For example, [Shen et al. (2004) combined multiple criteria of active learning for named entity recognition. Recent studies combining deep neural networks and active learning approaches have been proposed (Wang et al. 2 Zhang et al. 2017 Shen et al. 2018 Geifman & El- Yaniv 2019). not consider the correlation between adaptively learned models of se-
However, these approaches lected samples. There are some related work about active learning in visual question answering (Lin & Parikh 2017). However, little is known about active learning for machine reading comprehenston.
6 CONCLUSION AND FUTURE WORKS
In this work, we have proposed a novel adaptive learning algorithm for the reading comprehension task. There are two crucial components in our algorithm: an unsupervised uncertainty-base random sampling scheme, and localized loss minimization paradigm, both of which are adaptive to the currently learned model. We have described the strong motivation of using these techniques, and our empirical study serves as a clear evidence that our algorithm drastically mitigates the demand of labels on large-scale datasets. We highlight that approach is not essentially tied to MRC, and we expect that can be extended to other label- -ebeldddnnn problems in natural language processing and image classification.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Naoki Abe and Philip M Long. Associative reinforcement learning using linear prohabilistic con- cepts. In Proceedings of the Sixteenth International Conference on Machine Learning, pp. 3-11. Morgan Kaufmann Publishers Inc., 1999.
David Arthur and Sergei Vassilvitskii. k-means++: the advantages of careful seeding. In Nikhil Bansal, Kirk Pruhs, and Clifford Stein (eds.), Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 1027-1035, 2007.
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In Proceedings of the 8th International Conference Learning Representations, 2020.
Pranjal Awasthi, Maria-Floriin Florina Balcan, and Philip M. Long. The power of localization for efficiently learning linear separators with noise. In Symposium on Theory of Computing, pp. 449, 2017.
Maria-Florina Balcan and Philip M. Long. Active and passive learning of linear separators under log-concave distributions. Shai Shalev-Shwartz and Ingo Steinwart (eds.), The 26th Annual Conference on Learning Theory, volume 30, pp. 288-316, 2013.
tional Conference on Computational Learning Theory, pp. 35-50. Springer, 2007. Maria-Florina Balcan, Andrei Broder, and Tong Zhang. Margin based active learning. In Interna-
Maria-Florina Balcan, Alina Beygelzimer, and John Langford. Agnostic active learning. Comput. Syst. Sci., 75(1):78-85 88,, 2009.
Aron Culotta and Andrew McCallum. Reducing labeling effort for structured prediction tasks. In Manuela M. Veloso and Subbarao Kambhampati (eds.), Proceedings of the 20th AAAI Conference on Artificial Intelligence, pp. 746-751, 2005.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhut- dinov. Transformer-xl. Attentive language models beyond a fixed-length context. In Anna Ko- rhonen, David R. Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Conference of the Association for Compulational Linguistics, pp. 2978-2988, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171-4186, 2019.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang. Jianfeng Gao, Ming Zhou, and Hsiao Wuen Hon. Unified language model mre-triniig for natural language understand- ing and generation. In Annual Conference Neural Information Processing Systems 2019, pp- 13042-13054, 2019.
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. MRQA 2019 shared task: Evaluating generalization reading comprehension. In Proceedings of 2nd Machine Reading for Reading Comprehension (MRQA), 2019.
Snehal Gaikwad, Durim Morina, Rohit Nistala, Megha Agarwal, Alison Cossette, Radhika Bhanu, Saiph Savage, Vishwajeet Narwal, Karan Rajpal, Jeff Regino, et al. Daemo: self- self-governed crowdsourcing marketplace. In Adjunct proceedings of the 28th annual ACM symposium on user interface software & technology, pp. 101-102, 2015.
Yonatan Geifman and Ran Ylnva..v. Deep active learning with a neural architecture search. In Annual Conference Neural Information Processing Systems, pp. 5974-5984, 2019.
Daniel Gissin and Shai Shalev-Shwartz. Discriminative active learning. arXiv:1907.06344 2019.
arXiv preprint
Steve Hanneke. Theory of disagreement- based active learning. Found. Trends Mach. Learn., 7(2-3): 131-309, 2014.

---

Under review as a conference paper at ICLR 2021
Lynette Hirschman, Mare Light, Eric Breck, and John D. Burger. Deep read: A reading comprehen- sion system. In Robert Dale and Kenneth Ward Church (eds.), Proceedings of the 27th Anmual Meeting of the Association for Computational Linguistics, pp. 325-332, 1999.
Ajay Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. Multi-class active learning for image classification. In Proceedings of the 2009th IEEE Computer Society Conference on Computer Vision and Puttern, pp. 2372-2379, 2009.
Hao Karen. 10 breakthrough technologies 2019. MIT Technology Review, 2019.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Al- berti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming- Wei Chang. Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019.
Xiao Lin and Devi Parikh. Active learning for visual question answering: An empirical study. CoRR, http:/iv...rg////////////
abs/1711 11.1772,, 2017.
aataabbb/1111 .07322
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: A human generated machine reading comprehension dataset. In Proceed- ings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches, volume 1773, 2016.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Marilyn A. Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2227-2237, 2018.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing with unsupervised learning. Technical report, OpenAl, 2018.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. In Jian Su, Xavier Carreras, and Kevin Duh (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-22882 2392, 2016.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pp. 784-789, 2018
Siva Reddy, Danqi Chen, and Christopher D. Manning. Coqa: A conversational question answering challenge. Trans. Assoc. Comput. Linguisties, 7:249-2666 2019.
Roi Reichart, Katrin Tomanck, Udo Hahn, and Ari Rappoport. Multi-task active learning for lin- guistic annotations. In Kathleen R. McKeown, Johanna D. Moore, Simone Teufel, James Allan, and Sadaoki Furui (eds.), Proceedings of the 46th Annual Meeting of the Association for Compu- tational Linguistics, pp. 861-86616 2008
Dan Roth and Kevin Small. Margin-based active learning for structured output spaces. In Pro- ceedings of the 17th European Conference on Machine Learning, volume 4212, pp. 413-424, 2006.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In Proceedings of the 6th International Conference Learning Representations, 2018.
Burr Settles. Active learning literature survey. Technical report, University of Musconsin-Maddisee Department of Computer Sciences, 2009.
Dan Shen, Jie Zhang, Jian Su, Guodong Zhou, and Chew-Lim Tan. Multi-criteria-batt active learning for named entity recognition. Proceedings of the 42nd Anmual Meeting on Association for Computational Linguistics, pp. 589, Association for Computational Linguistics, 2004.
10

---

Under review as a conference paper at ICLR 2021
Yanyao Shen, Hyokun Yun, Zachary C. Lipton, Yakov Kronrod, and Animashree Anandkumar. Deep active learning for named entity recognition. In Proceedings of the 6th International Con- ference Learning Representations, 2018.
Adam Trischler, Tong Wang. Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. Newsqa: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP. pp. 191-200, 2017.
Gökhan Tür, Dilek Hakkani- and Robert E. Schapire. Combining active and semi-supervised learning for spoken language understanding. Speech Commun., 45(2):171-186, 2005.
Leslie G. Valiant. A theory of the learnable. In Richard A. DeMillo (ed.), Proceedings of the 16th Annual ACM Symposium on Theory of Computing, pp. 436 445, 1984.
Dan Wang and Yi Shang. A new active labeling method for deep learning. In Proceedings of the 2014 International Joint Conference on Neural Newworks, pp. 112-119, 2014.
Keze Wang, Dongyu Zhang. Li, Ruimao Zhang, and Liang Lin. Cost-effective active learning for deep image classification. IEEE Transactions Circuits and Systems for Video Technology, 27(12):2591 2016.
Zhiguo Wang, Patrick Ng. Xiaofei Ma, Ramesh Nallapati, and Bing Xiang. Multi colti-passage BERT: A globally normalized BERT model for open open-domain question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pp. 5877-5881, 2019.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoe V. Le. XInet: Generalized autoregressive pretraining for language understanding. Annual Conference on Neural Information Processing Systems 2019, pp. 5754-5764, 2019.
Xiang Yue, Bernal Jimenez Gutierrez, and Huan Sun. Clinical reading comprehension: A thorough analysis of the emrqa dataset. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4474-4486, 2020.
Chicheng Zhang. Efficient active learning of sparse halfspaces. In The 31th Annual Conference on Learning Theory, volume 75, pp. 1856-1880, 2018.
Chicheng Zhang, Jie Shen, and Pranjal Awasthi. Efficient active learning of sparse halfspaces with arbitrary bounded noise. CoRR, abs/2002.048400, 2020.
Ye Zhang, Matthew Lease, and Byron Wallace. Active discriminative text representation learning. In Proceedings of the st AAAI Conference on Artificial Intelligence, 2017.


Reviews:
[{"review_id": "YJscYpR5Psu", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is tackling the problem of labeling cost in Machine Reading comprehension.\nThe paper uses an uncertainty-based approach in the context of extractive machine reading to choose the point to annotate.\nIn addition, the paper uses an adaptive loss minimization schema that consists of penalizing large moves in the parameter space which strongly related trusted region types of approaches[1].\nThe authors compare to 6 baselines with mitigated results.\nWhile the paper is well-written with reasonable experiments on two machine reading datasets, I can mention three limitations for acceptance which are \n\n(1) the lack of strong novelty beyond uncertainty-based active learning applied to MRC: In its current status, the paper is proposing to measure uncertainty as entropy over the probability distribution of each token to be the start and end token of the contiguous string constituting the answer. I find this proposition pretty interesting but trivial compare to uncertainty based active learning literature applied to text [2, 3]. \n\n(2) lack of significant improvement on the considered datasets: First, the results are difficult to compare to SoA results (https://rajpurkar.github.io/SQuAD-explorer/) as the amount of data-point is different (Table 1). Second, the difference between the baseline methods and the proposed approach doesn't seem significant (Figure 4). Furthermore, it is currently not possible to precisely assess the result significance as the variance of the results isn't reported.\n\n(3) lack of numeral results justifying the use of the adaptive loss in this context: unfortunately, the authors proposed a second contribution related to the loss function (section 2.2) which is not evaluated with an ablation study to assess its possible utility.\n\nRefs\n[1] Trust Region Policy Optimization, Schulman and al, 2015\n[2] Uncertainty-based active learning with instability estimation for text classification, Zhu and al, 2017\n[3] Active Learning Using Uncertainty Information, Yang and al, 2017\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8jnKSTiiv9l", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n* This paper proposes a learning algorithm that interleaves data annotation and model updating to reduce the amount of labeled data needed.\n* The algorithm requests labels for the most “informative” (or uncertain) instances with respect to the current model.\n* The algorithm additionally proposes a loss that combines fitting data and controlling the degree of model update.\n* Results are reported on SQuAD and NewsQA and the finding is that one needs 25% less samples for same or better performance.\n\nStrengths:\n* The work presents an interesting comparison of active learning algorithms on SQuAD and NewsQA. The results on SQuAD are promising, even though they are well below SOTA. It might have been more interesting to see this approach in action on a much more low resource dataset, perhaps publishing new labeled data to be used for the task.\n\nWeaknesses:\n* Novelty is limited. This paper seems to be a fairly straightforward application of active learning.\n* Results are positive, but not exciting or surprising. Having to collect 25% less data is not so desirable, especially if there is suspicion that the active learning algorithm might introduce unwanted biases.\n* On page 2, the authors discuss how their algorithm allows the model to only fit the new uncertain instances rather than retraining the model from scratch. However this improvement could be an unnecessary complication. If the focus of the work is on reducing the need for labeled data, wouldn’t it be better to simply retrain the model on all the labeled data every time a new annotated batch of data comes in? \n* Even though the authors experimented only on Machine Reading Comprehension, there is no modeling specific to this task. This is good for generality, but it calls into question the choice of the authors to make MRC a central topic of the paper. It would have maybe been more convincing to frame this work as focused on active learning and then investigate its effect in multiple applications.\n* MRC will often have multiple candidate answers with high probability, but these candidates are not truly different. They will in many cases correspond to mostly overlapping spans, or to similar answers appearing in different parts of the passage. Intuitively it seems that this algorithm would focus annotators on manually labeling these cases, which could arguably be suboptimal use of annotator time.\n* No real analysis of the results is presented. It might have been interesting to see what data is actually being selected by the active learning algorithm or what questions the model was learning to answer as active learning progressed.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1Cf_kh3bmbT", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThis paper proposes to apply uncertainty-based measures to guide the collection of training samples for reading comprehension. The paper describes a relatively simple metric to estimate model uncertainty of unlabeled examples, and develops an algorithm to sample examples that exhibit least model certainty. They describe a learning and regularization model for this scenario and evaluate their proposal on SQuAD and NewsQA datasets.\n\n### Strong and weak points\n\nStrong points:\n- The use of uncertainty-based measures for active learning seem intuitive and elegant,\n- The experimental evaluation contains a few interesting baselines and provide context to this work\n\n### Weak points:\n\n- The idea of using model uncertainty for sample efficiency is somewhat standard now. A few missing references on this are, for example [1], [2] and [3]. It is even described in blog articles online [4].  After reading the paper, it was not clear what part of the proposal is really novel, and which parts are well established in the literature.\nPart of the work described seems to be dedicated to performing learning while sampling “in a loop” (Algorithm 1). Unless I’ve misunderstood the work, this assumption seems to be quite different than the premise of the work. After running an iteration of model training, new samples may be requested for annotation. In a real application, it would take hours, or most likely days, to receive new examples with labels. As such, I’m not sure it makes sense to design an algorithm that expects the new samples on every batch/iteration.  Given the time budgets, I would expect that models can be easily trained “from scratch” after new examples are available.\n\n- I found the paper hard to follow at times. For example:\n    - In Section 2.1.1 there seems to be use of informal notation: “We use a zero-one vector a to indicate the correct answer, ”. But at this point in the paper it’s unclear what the vector space represents (tokens in the input?). Later, in Section 3, a span-based representation is introduced.\n    - The thresholding logic was hard to follow and understand the rationale. Are you simply sampling amongst the most uncertain samples? Can you not specify a $k$ to sample from? Why have an adaptive threshold?\n    - The baselines models are not really explained clearly, with a single sentence per model,\n\n- Overall, the results are quite inconclusive. Although there is modest improvement in SQuAD benchmark, it does not seem to be the case for NewsQA (the NewsQA results are only available as a chart, which makes comparison difficult). Also, in the SQuAD setup, the Random sampling method seems to generally be as good or better than the other baselines. There are no explanations as to why this may happen. Are the results being averaged over several runs? Do they represent the performance of a single run? Could the variations be attributed to natural variation in model training?\n \nReferences used above:\n\n[1] Heterogeneous uncertainty sampling for supervised learning\nDavid D. Lewis and Jason Catlett\nhttps://scholar.google.com/scholar?cluster=9211137857521772693&hl=en&as_sdt=0,33 \n\n[2] Deep Bayesian Active Learning for Multiple Correct Outputs\nhttps://arxiv.org/abs/1912.01119 \n\n[3] You Need Only Uncertain Answers: Data Efficient Multilingual Question Answering\nhttp://www.gatsby.ucl.ac.uk/~balaji/udl2020/accepted-papers/UDL2020-paper-113.pdf \n\n[4] Active learning — Uncertainty Sampling (P3)\nhttps://medium.com/@duyanhnguyen_38925/active-learning-uncertainty-sampling-p3-edd1f5a655ac \n\n### Recommendation\n\nI would recommend not to publish this work at ICLR at this time.  As described above, it is unclear what the novelty of this study is, besides applying an established technique (uncertainty-based sampling) to machine comprehension tasks. I have concerns regarding the premise of the learning setup (incremental learning assuming new examples can be sampled). Finally, the results seem inconclusive to me, and it is unclear whether statistical variations are dominating effects in SQuAD and NewsQA.\n\n### Questions for authors\n\nPlease refer to the “weak points” described above.\n\n### Additional feedback\n\n- Besides the technical work, I think the paper could benefit from improved writing. Currently, some parts are a bit hard to read. \n- Please separate the notation, task definition and model design into separate (sub)sections, so readers can read the paper linearly.\n- Overall, the writing could use some editorial help.\n- Page 5, the text describes the best performance for 5k examples as “64.07%”, however,  Table 1 states it is “64.03%”.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "03NqELnhr22", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Update after author response:** I appreciate the authors' efforts to address my concerns. Thanks for the correction on QBC, I appreciate it. I still believe that the paper needs to accompany a more comprehensive evaluation and qualitative insights to highlight the effectiveness of the proposed method. For instance, it is common practice in Active Learning to report mean accuracy over multiple runs of the same experiment as data is sampled based on a particular heuristic which isn't always deterministic. Furthermore, the choice of warmstart samples could also influence the results, which is why it is recommended to conduct multiple runs of the same experiments and report mean performance. In such a scenario, any claims that arise from only one run of the experiments (as in this paper) should be taken with a grain of salt. I also appreciate the pointer to Equation 4 but how does it translate in practice is another important piece that is missing. BERT based models produce highly confident predictions and to visualize the distribution from your empirical investigation would help bridge the divide between the equations and the empirical results (how they actually turn out in practice). I am also not convinced by the authors' response to why the difference in performance of the BERT model trained on randomly sampled data vs the model trained on data sampled via ALBUS stays within the 1-2% range, often increasing with more data. I believe there are critical questions regarding this paper that need to be addressed before the paper is published, hence, my score remains unchanged.\n\n---------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n\nSummary: The paper proposes a new method to actively sample data to train models in a limited labeled data regime. Focusing on passage based question answering, the authors employ an uncertainty based sampling strategy by selecting examples on which a model has little top-1 prediction confidence. In every training iteration, the model is trained with an adaptive regularizer in an attempt to keep the new model closer to the current model. The results presented in the paper show improvement over three active learning baselines as well as training over iid sampled data on the SQuAD dataset.\n\nPros:\n- The paper looks at an important problem which is little studied in context of deep learning based NLP systems, particularly with pre-trained transformers. \n- The two components that together form ALBUS are simple and the authors show that they perform better than various active learning baselines as well as random sampling on the SQuAD dataset.\n- I like the approach of ensuring that the model weights do not deviate too much as this also addresses the challenges faced in active learning when we end up sampling outliers.\n- The paper is easy to follow.\n\nCons:\n- All evaluations are conducted using a Bert base model. It is unclear whether this method generalizes to other models or not. Furthermore, the benefits of this approach on NewsQA are not easily interpretable, in fact the plots suggest that in many cases, baseline methods outperform this method. What happens when less than 20000 NewsQA examples are sampled for labeling is also not shown so it is hard to meaningfully interpret the results.\n- Common and effective Active Learning baselines such as Query by Committee have not been compared against.\n- The uncertainty based sampling method samples examples on which a model is less confident (determined by a threshold) on its top-1 prediction, however, the paper presents no qualitative analysis to show that this approach is indeed meaningful in context of models like BERT which, to the best of my knowledge, happen to have very confident predictions. It would help if the authors show how the distribution of the metric of informativeness over examples, and as more and more data is labeled.\n\nQuestions for authors for rebuttal:\n- It would be great to share actual numbers for NewsQA besides the existing plots. It is clear from the table and the figure that your method consistently outperforms baselines on SQuAD but it is less clear as to what happens in NewsQA as the only figures present are less clear and show that other baselines often outperform your proposed method.\n- Several experimental details are missing from the paper that make it hard to put the numbers into context. Are the results averaged over multiple runs? If yes, how many? And if not, then I’d recommend averaging results for each approach over at least 3 runs. \n- Since the authors have not shared their code with the submission, it is important to share how certain hyperparameter values were obtained. What influenced the choice to finetune BERT for only 2 epochs for both SQuAD and NewsQA or what influenced the choice of the number of datapoints for warm starting, etc.?\n- On SQuAD, it appears that the difference in performance of the BERT model trained on randomly sampled data vs the model trained on data sampled via ALBUS stays within the 1-2% range, sometimes increasing with more data. Intuitively, this gap should be decreasing with more data (which does happen with other baselines that you’ve compared against). Why do you think this gap stays more or less the same with your method regardless of the amount of labeled data?\n\nTypos and other suggestions:\n- Page 2, second paragraph last line: “more faster” -> faster\n- Page 2, first line: a more appropriate reference for Active Learning would be Cohn et al., 1996. Cohn, D. A., Ghahramani, Z., & Jordan, M. I. (1996). Active learning with statistical models. Journal of artificial intelligence research, 4, 129-145.\n- Even though the paper is easy to follow, I believe the writing can be improved, some sections appear to be written in a casual manner, for instance (these are representative, not exhaustive): \n(i) line 4 of introduction “smooth-talking AI systems.” Consider using a better term to describe these systems.\n(ii) lines 13 and 14 of abstract “demonstrate … that 25% less labeled samples suffice to guarantee.” Since the paper does not establish theoretical guarantees for this method, I would recommend not to use words like guarantee in this context.\n\nMissing references:\n\n- Siddhant, A., & Lipton, Z. C. (2018). Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Study. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.\n\nReasons for the score:\nI think there are some outstanding questions regarding this paper that can be addressed in the author response, but until then, the proposed approach and associated results are less than convincing. I’m more than willing to increase my score following the author response.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that the current approach is very similar to traditional uncertainty-based active learning, and that the empirical results are inconclusive, so at this point the paper is not ready for publication.
Decision: Reject