Venue: EMNLP 2020

# Learning a Cost-Effective Annotation Policy for Question Answering

Bernhard Kratzwald\({}^{\diamondsuit}\)   Stefan Feuerriegel\({}^{\diamondsuit}\)   Huan Sun\({}^{\spadesuit}\)

\({}^{\diamondsuit}\) Chair of Management Information Systems, ETH Zurich

\({}^{\clubsuit}\) Department of Computer Science and Engineering, The Ohio State University

{bkratzwald, sfeuerriegel}@ethz.ch sun.397@osu.edu

###### Abstract

State-of-the-art question answering (QA) relies upon large amounts of training data for which labeling is time consuming and thus expensive. For this reason, customizing QA systems is challenging. As a remedy, we propose a novel framework for annotating QA datasets that entails learning a cost-effective annotation policy and a semi-supervised annotation scheme. The latter reduces the human effort: it leverages the underlying QA system to suggest potential candidate annotations. Human annotators then simply provide binary feedback on these candidates. Our system is designed such that past annotations continuously improve the future performance and thus overall annotation cost. To the best of our knowledge, this is the first paper to address the problem of annotating questions with minimal annotation cost. We compare our framework against traditional manual annotations in an extensive set of experiments. We find that our approach can reduce up to 21.1% of the annotation cost.

+
Footnote †: November 4, 2020. ©2020 Association for Computational Linguistics

## 1 Introduction

Question answering (QA) based on textual content has attracted a great deal of attention in recent years Chen et al. (2017); Lee et al. (2018, 2019); Xie et al. (2020). In order for state-of-the-art QA models to succeed in real applications (e.g., customer service), there is often a need for large amounts of training data. However, manually annotating such data can be extremely costly. For example, in many realistic scenarios, there exists a list of questions from real users (e.g., search logs, FAQs, service-desk interactions). Yet, annotating such questions is highly expensive Nguyen et al. (2016); He et al. (2018); Kwiatkowski et al. (2019): it requires the screening of a text corpus to find the relevant document(s) and subsequently screening the document(s) to identify the answering text span(s).

Motivated by the above scenarios, we study cost-effective annotation for question answering, whereby we aim to _accurately_1 annotate a given set of user questions _with as little cost as possible_. Generally speaking, there has been extensive research on how to reduce effort in the process of data labeling Haffari et al. (2009). For example, active learning for a variety of machine learning and NLP tasks Siddhant and Lipton (2018) aims to select a small, yet highly informative, subset of samples to be annotated. The selection of such samples is usually coupled with a particular model, and thus, the annotated samples may not necessarily help to improve a different model Lowell et al. (2019). In contrast, we aim to annotate _all_ given samples at low cost and in a manner that can subsequently be used to develop any advanced model. This is particularly relevant in the current era, where a dataset often outlives a particular model Lowell et al. (2019). Moreover, there has also been some research into learning from distant supervision Xie et al. (2020) or self-supervision Sun et al. (2019). Despite being economical, such approaches often produce inaccurate or noisy annotations. In this work, we seek to reduce annotation costs without compromising the resulting dataset quality.

Footnote 1: By “accurate,” we mean that the resulting annotations will be of a similar quality to those from conventional manual annotation.

We propose a novel annotation framework which learns a cost-effective policy for choosing between different annotation schemes, namely the conventional manual annotation scheme (Man) and a semi-supervised annotation scheme (Sem). Unlike the manual scheme, Sem does not require humans to screen a text corpus or document(s) in order to retrieve annotations. Instead, it leverages an initialized QA system, which can predict top-\(n\) candidate annotations for documents or answer spans and asks humans to provide

rect or incorrect) to the candidates. While this annotation scheme comes at a low cost, it fails when human annotators mark all candidates as incorrect. In such cases, the annotation cost has already been incurred and cannot be recouped. In order to produce an annotation, one must then draw upon the manual scheme (see Fig. 1), in which case the policy would have been more effective if it had chosen the manual annotation scheme instead. _Therefore, how to choose the best annotation scheme for each question is the challenge we must address for this task._

To tackle the above challenge, we propose a novel approach for learning a cost-effective policy. Here the policy receives several candidates and decides on this basis which annotation scheme to invoke. We train the policy with a supervised objective and learn a cost-sensitive decision threshold. The inherent advantage of this method is that our policy immediately reacts to changing costs (without re-optimizing model parameters) and does not exceed the cost of conventional manual annotation. Our policy is updated iteratively as more annotations are obtained.

We compare our framework against conventional, manual annotations in an extensive set of experiments. We simulate the annotation of NaturalQuestions Kwiatkowski et al. (2019), as it consists of real user questions from search logs. Models in our framework are initialized with an existing dataset (SQuAD, Rajpurkar et al. (2016) and, as more annotations on NaturalQuestions become available, the framework is continuously updated. We study the sensitivity of our framework to varying cost ratios between Sem and Man. Our framework outperforms traditional manual annotation, even under conservative cost estimates for Sem, and in general reduces annotation costs in the range of 4.1% to 21.1%.

All source code is publicly available from github.com/bernhard2202/qa-annotation.

## 2 Related Work

**Question answering:** In this paper, we study cost-effective annotation for question answering over textual content. There have been extensive efforts to create large-scale datasets for text-based QA, which have facilitated the development of state-of-the-art neural network based models (e.g., Chen et al. (2017); Min et al. (2018); Lee et al. (2018); Kratzwald and Feuerriegel (2018); Wang et al. (2018); Xie et al. (2020). Here we divide such datasets into two categories according to the way they were created: (1) Datasets whose questions were created by crowdsourcing during the annotation process. Prominent examples include the Stanford Question and Answer Dataset (SQuAD; Rajpurkar et al. (2016), HotPotQA Yang et al. (2018), or NewsQA Trischler et al. (2017). (2) "Natural" datasets in which real-world questions are a priori given. Here questions originate from, e.g., search logs or customer interactions. Prominent examples in this category include MS MARCO Nguyen

Figure 1: High-level overview of our framework: We leverage a QA model to predict candidate annotations for a given resource (e.g., \(x\) stands for a question or question-document pair, while \(y\) is the document or answer span). A policy model decides upon whether to invoke a Man or Sem scheme based on those predictions. In the event that the semi-supervised strategy fails, we switch back to a manual annotation scheme. Finally, we use the annotated sample to update both the QA model and the policy model.

 

et al., 2016), DuReader He et al. (2018), or NaturalQuestions Kwiatkowski et al. (2019). This paper focuses on the latter category, that is, annotating "natural" datasets in a more cost-effective fashion where a set of questions is given.

**Active Learning:** In the fields of machine learning and NLP, extensive research has been conducted on ways to reduce labeling effort (e.g., Zhu et al., 2008). For example, the objective of active learning is to select only a small subset that is highly informative (e.g., Haffari et al., 2009) for annotation. To this end, researchers have developed various techniques based on, e.g., model uncertainty (cf. Siddhant and Lipton, 2018), expected model change Cai et al. (2013), or functions learned directly from data (e.g., Fang et al., 2017). However, the success of active learning is often coupled with a particular model and domain Lowell et al. (2019). For instance, a dataset actively acquired with the help of an SVM model might underperform when used to develop an LSTM model. These problems become even more salient when complex black-box models are used in NLP tasks (cf. Chang et al., 2019). To summarize, active learning reduces annotation costs by deciding _which_ samples should be annotated. In our approach, we aim to annotate _all_ samples and study _how_ we should annotate them in order to reduce costs. Thus, the two approaches are orthogonal and can be combined.

**Learning from weak supervision and user feedback:** Another approach to reducing annotation costs is changing full supervision to some form of weak (but potentially noisier) supervision. This has been adopted for various tasks such as machine translation Saluja (2012); Petrushkov et al. (2018); Clark et al. (2018); Kreutzer and Riezler (2019), semantic parsing Clarke et al. (2010); Liang et al. (2017); Talmor and Berant (2018), or interactive systems that learn from user interactions Iyer et al. (2017); Gur et al. (2018); Yao et al. (2019, 2020). For instance, Iyer et al. (2017) used users to flag incorrect SQL queries. In contrast, similar approaches for text-based question answering are scarce. Joshi et al. (2017) used noisy distant supervision to annotate the answer span and document for given trivia questions and their answers. Kratzwald and Feuerriegel (2019) designed a QA system that continuously learns from noisy user feedback after deployment. In contrast to these works, this paper studies the problem of reducing labeling cost while maintaining accurate annotations.

**Quality estimation and answer triggering:** In a broader sense, this work is related to the literature on translation quality estimation (e.g., Martins et al., 2017; Specia et al., 2013). The goal in such works is to estimate (and possibly improve) the quality of translated text. Similarly, in question answering researchers use means of quality estimation for answer triggering Zhao et al. (2017); Kamath et al. (2020). Here, QA systems are given the additional option to abstain from answering a question when the best prediction is believed to be wrong. In our work, we estimate the quality of a set of suggested label candidates and, on the basis of these estimates we decide which annotation scheme to invoke.

## 3 Proposed Annotation Framework

We study the problem of reducing the overall cost for annotating every given question \([q_{1},\ldots,q_{m}]\). Specifically, our objective is to obtain the corresponding question-document-answer triples \(\langle q_{i},d_{i},s_{i}\rangle\). In this paper, the natural language question \(q_{i}\) is given, while we want to obtain the following annotations: the document from a text corpus \(d_{i}\in\mathcal{D}\) that contains the answer and the correct answer span \(s_{i}\) within the document \(d_{i}\).

### Framework Overview

Fig. 1 provides an overview of our framework for a cost-effective annotation of QA datasets. The framework comprises two main components: a **QA model** is used to suggest candidates for a resource to annotate while a **policy model** decides which annotation scheme to invoke (i.e., action). Our framework makes use of two annotation schemes: a traditional **manual annotation scheme (Man)** and our **semi-supervised annotation scheme (Sem)**. Both annotation schemes incur different costs and, hence, the learning task is to find and update a cost-effective policy \(\pi\) for making that decision.

**QA model:** We define \(\Omega\) as an arbitrary QA model over a text corpus \(\mathcal{D}\) with the following properties. First, the model can be trained from annotated data samples, e.g., \(\Omega\leftarrow\text{train}\left(\{\langle q_{i},d_{i},s_{i}\rangle\}_{0<i< \ldots}\right)\). Second, for a given question the model can predict a number of top-\(n\) documents likely to contain the answer, i.e., \(\Omega^{D}:q\rightarrow[d^{(1)},\ldots,d^{(n)}]\in\mathcal{D}\). Third, for a given question-document pair the model can predict a number of top-\(n\) answer spans, i.e., 

\(\Omega^{S}:\langle q,d\rangle\rightarrow[s^{(1)},\ldots,s^{(n)}]\). These properties are fulfilled by recent QA systems dealing with textual content (e.g., Chen et al., 2017; Wang et al., 2018).

**Policy model:** For every question, we distinguish two policy models: a policy model \(\pi^{D}\) responsible for annotating documents and \(\pi^{S}\) for answer spans. For brevity, we sometimes drop the superscripts \(S\) and \(D\) and simply refer to them as \(\pi\). The policy models decide whether a manual annotation scheme or rather our proposed semi-supervised annotation scheme is used, each of which is associated to different costs.

### Annotation Schemes

**Manual annotation (Man) scheme:** This scheme represents the status quo in which all annotations are determined manually. In order to annotate a question \(q_{i}\), a human annotator must first manually search through the text corpus \(\mathcal{D}\) in order to identify the document \(d_{i}\) that answers the question. In a second step, a human annotator manually reads through the document \(d_{i}\) and marks the answer span \(s_{i}\).

We assume separate costs, which are fixed over time, for every annotation-level. The price of annotating a document for a given question is defined as \(c_{0}^{D}\) and the price of annotating an answer span to a given question-document tuple as \(c_{0}^{S}\). We explicitly distinguish these costs as the tasks can be of differing difficulty.

**Semi-supervised annotation (Sem) scheme:** This scheme is supposed to reduce human effort by presenting candidates for annotation, so that only simple binary feedback is needed. In particular, human annotators no longer need to search through the entire document or corpus. Instead, we use the QA model \(\Omega\) to generate a set of candidates (e.g., top-ranked documents or answer spans) and ask human annotators to give binary feedback in response (e.g., accept the candidate or reject it). This replaces the complex search task with a simpler form of interaction. As an example, to annotate the answer span for a question-document pair, the human annotator would not be required to read the entire document \(d_{i}\), but only to determine which of the top-\(n\) answers provided by \(\Omega^{S}(\langle q_{i},d_{i}\rangle)\) are correct. We assume Sem costs \(c_{1}^{D}\) to annotate a document and \(c_{1}^{S}\) to annotate an answer span.

The Sem scheme should make annotations more straightforward, as providing binary feedback requires less time than reading through the texts. Hence, we assume that \(c_{1}^{S}<c_{0}^{S}\) and \(c_{1}^{D}<c_{0}^{D}\) hold. However, semi-supervised annotations might fail when none of the candidates is correct (i.e., the human annotators reject all candidates). In this case, our framework must revert to the Man procedure in order to obtain a valid annotation. As a consequence, the associated cost will increase to the accumulated cost for both the Sem and the Man schemes.

Note that, no matter which scheme is chosen in practice, all annotations are confirmed by human annotators and our resulting dataset will be equal in quality to those resulting from traditional annotation.

### Annotation Costs

Both annotation schemes, Man and Sem, incur different costs that further vary depending on whether annotation is provided at document level (\(c^{D}\)) or at answer span level (\(c^{S}\)). For annotating documents, the cost amounts to

\[c^{D}(a|q_{i},d_{i}^{*})=\begin{cases}c_{a}^{D},~{}\text{if}~{}a=0~{}\text{or }(a=1~{}\text{and}\\ &d_{i}^{*}\in\Omega^{D}(q_{i})),\\ c_{0}^{D}+c_{1}^{D},~{}~{}~{}\text{otherwise}\end{cases} \tag{1}\]

where \(a=\{0,1\}\) is the selected annotation scheme and \(d_{i}^{*}\) is the ground-truth document annotation. Hence, \(d_{i}^{*}\in\Omega^{D}(q_{i})\) indicates the candidate set contains the ground-truth annotation and Sem is successful.

For annotating answer spans, the cost is given by

\[c^{S}(a|\langle q_{i},d_{i}\rangle,s_{i}^{*})=\begin{cases}c_{a}^{S},~{}\text {if}~{}a=1~{}\text{or}~{}(a=0~{}\text{and}\\ &s_{i}^{*}\in\Omega^{S}(\langle q_{i},d_{i}\rangle),\\ c_{0}^{S}+c_{1}^{S},~{}~{}~{}\text{otherwise}.\end{cases} \tag{2}\]

Alternatively, we can write the cost function as a matrix of annotation costs (Tbl. 1). The diagonal entries reflect the costs paid for choosing the optimal scheme. The off-diagonals refer to the costs paid for a sub-optimal method (misclassification costs).

## 4 Learning a Cost-Effective Policy

### Objective

We aim to minimize overall annotation cost via

\[\sum_{i}\mathbb{E}_{\pi^{D},\pi^{S}}\left[c^{D}(a|q_{i},d_{i}^{*})+c^{S}(a| \langle q_{i},d_{i}\rangle,s_{i}^{*})\right].\] (3) 

It is important to see that the QA model and the policy model are intertwined, with both having an impact on Eq. 3. Updating the policy models learns the trade-off between Sem and Man annotations and, hence, directly minimizes the overall costs. Updating the QA model \(\Omega\) increases the number of times suggested candidates are correct and, therefore, the fraction of successful Sem annotations. For instance, when adapting to a new domain, only a small fraction of suggested candidate annotations are correct, limiting the effectiveness of the Sem annotation. However, as we annotate more samples, we improve \(\Omega\) and thus more suggested candidate annotations will be correct. For this, we later specify suitable updates for both the QA model and the policy model.

### Annotation Procedure and Learning

Our framework proceeds according to these nine steps when annotating a question \(q_{i}\) (see Alg. 1): First, we predict a number of top-\(n\) documents that would be shown to annotators in the case of Sem annotation (line 2). Next, we decide upon the annotation scheme conditional on the prediction from the QA model (line 3) and, based on the selected scheme, request the ground-truth document annotation (line 4). After receiving the ground-truth document and observing the annotation costs, we update our policy network in line 5 (see Sec. 4.3). Next, we predict a number of top-\(n\) answer span candidates for the question-document pair (line 6) and then decide upon the annotation scheme in line 7. After receiving the answer span annotation and observing a cost (line 8), we again update our policy model (line 9). Finally, we update the QA model with the newly annotated training sample in line 10 (see Sec. 4.4). In practice, both policy updates and QA model updates (lines 5, 9, and 10) are invoked after a batch of questions is annotated. Furthermore, we initialize all models with an existing dataset (e.g., SQuAD).

```
Input :list of questions \([q_{1},\dots,q_{m}]\) text corpus \(\mathcal{D}\); QA model \(\Omega\); policy models \(\pi^{D}\) and \(\pi^{S}\) Result:annotated dataset \(\{(q_{i},d_{i},s_{i})\}_{0<i<m}\) while\(i\leq m\)do
1\([d^{(1)},\dots,d^{(n)}]\leftarrow\Omega^{D}(q_{i})\); predict top-\(n\) documents
2\(a\gets\pi^{Q}(q_{i},[d^{(1)},\dots,d^{(n)}])\); decide upon annotation scheme
3\(d_{i}\getsannotate(q_{i}|a)\); annotate document
4update \(\pi^{D}\) w.r.t. the observed costs \(c^{D}(a|q_{i},d_{i})\);
5\([s^{(1)},\dots,s^{(n)}]\leftarrow\Omega^{S}(\langle q_{i},d_{i}\rangle)\); predict top-\(n\) answer candidates
6\(a\leftarrow\pi^{Q}(q_{i},[s^{(1)},\dots,s^{(n)}])\); decide uponannotation scheme
7\(s_{i}\getsannotate(\langle q_{i},d_{i}\rangle|a)\); annotate answer
8span
9update \(\pi^{S}\) w.r.t. the observed costs \(c^{S}(a|\langle q_{i},d_{i}\rangle,s_{i})\);
10update QA model \(\Omega\) with \(\langle q_{i},d_{i},s_{i}\rangle\)
11 end while
```

**Algorithm 1**High-Level Procedure of Annotation and Learning

### Policy Updates

Updating our policy model proceeds in three steps. (1) We calculate whether the chosen action for past annotations was cost-optimal (i.e., whether the policy should have chosen the other scheme for annotation or not). (2) We use this information to update the policy model with a supervised binary classification objective. This trains the policy to predict the probability of an annotation scheme given a new sample \(p(a|x)\) without taking costs into account. (3) We find a cost-sensitive decision threshold that chooses the optimal action with respect to the costs. All three steps are repeated after a full batch of samples has been annotated.

Separating the policy update and the cost-sensitive decision threshold has several benefits. First, we know from cost-sensitive classification that we can calculate an optimal threshold point for ground-truth probabilities \(p(a|x)\) (c.f. Elkan, 2001; Ting, 2000). Therefore, we can focus our effort on determining probabilities as accurate as possible. Second, the decision threshold is calculated only from the costs \(c_{a}^{S}\) and \(c_{a}^{D}\) and, hence, if costs change, we do not need to re-estimate parameters but can directly adjust our policy.

**(1) Finding the cost-optimal action:** In order to train the policy with a supervised update, we require labels for the cost-optimal annotation scheme

\begin{table}
\begin{tabular}{l|c c} \hline \multicolumn{3}{c}{**Annotation Costs for a Document \(d\)**} \\ \hline  & Cost-optimal: Man & Cost-optimal: Sem \\ Selected: Man & \(c_{0}^{D}\) & \(c_{1}^{D}\) \\ Selected: Sem & \(c_{0}^{D}+c_{1}^{D}\) & \(c_{1}^{D}\) \\ \hline \end{tabular}
\end{table}
Table 1: Costs for annotating documents (top) and answer spans (bottom). The costs depend on the selected annotation scheme (rows) and the scheme that would have been cost-optimal (columns).

 

for a given sample. If we choose the Sem annotation scheme, we immediately know whether the action was cost-optimal or not. This is due to the fact that, if the semi-supervised annotation fails, we have to switch to the Man scheme to receive an annotation and pay both costs. On the other hand, if we choose the Man scheme, we can observe the optimal action only after receiving the ground-truth annotation: We can then simply run the QA model and validate whether the annotation was contained within the top-\(n\) candidates. If so, the Sem action would have been the better choice; otherwise, choosing Man would have been cost-optimal.

**(2) Supervised model updates:** Our policy model is a neural network with parameters \(\theta\) that predicts an annotation scheme for a given sample \(x\), i.e.,

\[p(a|x)={\text{NN}_{\theta}}(x). \tag{4}\]

Note that we dropped the \(S\) and \(D\) indices here as both policies differ only in the neural network architecture used. We can then simply train the policy with a supervised binary cross-entropy loss given the cost-optimal action that we calculated beforehand. Since the Sem scheme is often sub-optimal in the beginning, the training data is highly imbalanced. Therefore, we down-sample past annotations with a sampling ratio of \(\alpha\) such that our training data is equally balanced.

**(3) Cost-sensitive decision threshold:** Choosing the annotation scheme with the highest probability does not take the actual costs into account. For instance, if Sem annotations are much cheaper than Man annotations, we want to choose the semi-supervised scheme even if its probability for success is low. More formally, we want to choose the annotation scheme \(a\) that has the lowest expected cost \(R(a|x)\), i.e.,

\[R(a\,|\,x)=\sum_{a^{\prime}}p(a^{\prime}\,|\,x)\,c(a,a^{\prime}) \tag{5}\]

where \(c(a,a^{\prime})\) is the annotation cost for choosing scheme \(a\) when the optimal scheme was \(a^{\prime}\) (see Tab. 1). Since we used down-sampling in our training, we have to calibrate the probabilities \(p(a|x)\) with the sampling ratio \(\alpha\)Pozzolo et al. (2015). Assuming the calibrated probabilities are accurate, there exists an optimal classification threshold \(\beta\)Elkan (2001) that minimizes Eq. 5 and Eq. 3. Therefore, we define our policy as follows:

\[\pi(a|x,\alpha,\beta)=\left\{\begin{array}{rl}&1,\,\mbox{if $\frac{\alpha\, p(a=|x)}{(\alpha-1)\,p(a=|x)+1}\geq\beta$}\\ &0,\,\mbox{otherwise}.\end{array}\right. \tag{6}\]

The optimal \(\beta\) can be derived from the classification cost matrix Elkan (2001) via

\[\beta=\frac{c(1,0)-c(0,0)}{c(1,0)-c(0,0)+c(0,1)-c(1,1)}, \tag{7}\]

again omitting superscripts \(D\) and \(S\) for brevity. Eq. 7 is then simplified as the fraction of Sem annotation costs to Man annotation costs. Therefore, we can derive \(\beta\) at document level

\[\beta^{D}=\nicefrac{{c_{1}^{D}}}{{c_{0}^{D}}}, \tag{8}\]

and answer span level

\[\beta^{S}=\nicefrac{{c_{1}^{S}}}{{c_{0}^{S}}}. \tag{9}\]

### Model Updates

Periodically updating the QA model \(\Omega\) allows the framework to adapt to the question style and domain at hand during the annotation process. Therefore, we improve the top-\(n\) accuracy and the success rate of the Sem scheme over time. For practical reasons, we refrain from updating \(\Omega\) after every annotation but periodically retrain the model after a batch of samples is annotated. In order to update the QA model, we can use the fully annotated QA samples in combination with a supervised objective.

## 5 Experimental Setup

In this section, we introduce our experimental setup and implementation details.

### Datasets

We base our experiments on the NaturalQuestion dataset Kwiatkowski et al. (2019). We choose this dataset as it is composed of about 300,000 real user questions posed to the Google search engine along with human-annotated documents and answer spans. Simulating the annotation of this dataset is similar to what would happen for domain customization of QA models in real practice (e.g., for search logs, FAQs, logs of past customer interactions). We focus on questions from the training-split that possess an answer span annotation and leave the handling of questions that do not have an answer for future work. The corpus for annotations 

is fixed to the English Wikipedia,2 containing more than 5 million text documents.

Footnote 2: We extracted articles from the Wikipedia dump collected in October 2019, as this is close to the time period in which the NaturalQuestions dataset was constructed.

_Simulation of annotations:_ Annotations in our experiments are simulated from the original dataset. If the framework chooses Man annotation, we simply use the original annotation from the dataset. If a Sem annotation is chosen, we simulate users that give positive feedback only to the ground-truth document and to the answer spans where the text matches3 the ground-truth annotation. We then construct the new annotation using the candidate with positive feedback. Since we simulate annotations, we conduct extensive experiments on how annotation costs influence the performance of our framework.

Footnote 3: Here we only count exact matches.

### Baselines

To the best of our knowledge, there is no comparable prior work. Owing to this fact, we evaluate our framework against several customized baselines. First, we compare our approach against a manual annotation baseline in which we always invoke the full Man method to annotate samples. This represents the traditional method of annotating QA datasets and thus our prime baseline. Second, we draw upon a clairvoyant oracle policy that always knows the optimal annotation method. We use this baseline to report an upper bound of the savings that our framework could theoretically achieve. Third, we use our framework without updates on the QA model \(\Omega\). This quantifies the cost-savings achieved by the interactive domain customization during annotation. Finally, we present a randomized baseline where the annotation scheme is decided by a randomized coin-toss.

### Implementation Details

The QA model \(\Omega\) is built as follows. We use a state-of-the-art BERT-based Devlin et al. (2018) implementation of RankQA Kratzwald et al. (2019). This combines a simple tf-idf-based information retrieval module with BERT as a module for machine comprehension. Both policy models \(\pi^{D}\) and \(\pi^{S}\) are implemented as three-layer feed-forward networks with dropout, ReLu activation, and a single output unit with sigmoid activation in the last layer. For the policy \(\pi^{D}\), we use the information retrieval scores as input. For \(\pi^{S}\), we use the statistical features of answer-span candidates as calculated by RankQA Kratzwald et al. (2019) as input. We also experimented with convolutional neural networks directly on top of the last layer of BERT, but without yielding improvements that justified the additional model complexity. We initialize all models with the SQuAD dataset (see our supplements).

_Hyperparameter setting:_ We set the number of candidates that are shown to annotators during a Sem annotation to \(n=5\). The policy networks decide upon the annotation method based on features of the \(2n\) highest-ranked candidates, i.e., the top-\(10\). The batch size for updates in Alg. 1 is set to 1,000 annotated questions. Details on hyperparameters of our QA and policy models are provided in the supplements.

## 6 Experimental Results

We group our experiments into three parts. First, we focus only on annotating the answer span for given question-document pairs, as this is the more challenging task.4 Second, we carry out a sensitivity analysis in order to demonstrate how our framework adapts to different costs of Sem annotations and to show that we never exceed the cost of traditional annotation. Third, we evaluate our framework based on the annotation of a full dataset, including both answer span and document annotations, in order to quantify savings in practice by using our framework.

Footnote 4: Manufactally finding an answer span involves reading a document in depth. Manual document annotation is easier, as it can be supported with tools such as search engines. In such a case, our framework could still be used for answer span annotation, as is shown in Section 6.1.

### Performance on Answer Span Annotations

The annotation framework was used to annotate 45 batches of question-document pairs with the corresponding answer spans. The annotation costs are set to one price-unit for each Man annotation and one third of the unit for each Sem annotation. (In the next section, we carry out an extensive sensitivity analysis where the ratio for annotation costs between Man and Sem is varied.)

In Fig. 2 (left), we plot the average annotation costs in every batch with a dashed line, together with a running mean depicted as a solid line. Compared to conventional, manual annotation, our framework successfully reduces annotation cost by around 15% after only 20 batches. We further 

compare it with an oracle policy that always picks the best annotation method. The latter provides a hypothetical upper bound according to which approximately \(40\)-\(45\%\) of annotation cost could be saved. Finally, we show the performance of our framework without updates of the QA model \(\Omega\). Here we can see that its improvement over time is lower, as the framework is not capable of adapting to the question style and domain used during annotation. In sum, our framework is highly effective in reducing annotation cost.

Fig. 2 (right) shows how many samples we could annotate (y-axis) with a restricted budget (x-axis). For instance, assume we have a budget of 40k price units available for annotation. Conventional, manual annotation would result in exactly 40k annotated samples as we fixed the cost for each Man annotation to one unit. With the same budget, our annotation framework with semi-supervised annotations succeeds in annotating an additional \(\sim\)9,000 samples.

### Cost-Performance Sensitivity Analysis

The advantage of our framework over manual annotations depends on the cost ratio between the Sem and Man schemes. In order to determine this, we identify the cost-range in which our framework is profitable as a function of Sem annotation costs. We study this via the following experiment: we repeatedly annotate 40k samples and keep the Man annotation costs fixed to one price unit, while we increase the costs of smart annotations from \(0.05\) to \(0.95\) in increments of \(0.05\). Finally, we measure the average annotation costs for a single sample; see Fig. 3 (left).

Fig. 3 (left) demonstrates that our framework effectively lowers annotation costs when the price for Sem annotations drops below \(0.6\) as compared to manual annotations, which are fixed to one price-unit. Most notably, even when Sem annotations become expensive and almost equal the costs of Man annotations, the average annotation costs do not exceed those of strictly manual annotation. This can be attributed to our cost-sensitive decision threshold, which does not require exploration as in reinforcement learning, but directly sets the threshold in Eq. 6 sufficiently high.

In Fig. 3 (right), we again show the number of samples that were annotated with a restricted budget of 40k price units. We marked the absolute gain in number of samples over traditional annotation in the plot. The benefit of our framework becomes evident once again when the ratio of Sem annotation costs to Man annotations costs falls below 0.6.

To summarize, our framework is highly cost-effective: it reduces overall annotation costs or, alternatively, increases the number of annotated samples under a restricted budget if annotation costs of Sem are approximately half those of Man. If the costs are less than half those of Man annotation, the benefits are especially pronounced. Even if this assumption does not hold, our framework never exceeds the costs of manual annotation and never results in fewer annotated samples.

### Performance on Full Dataset Annotation

In the last experiment, we simulate a complete annotation of the NaturalQuestions dataset, including annotations at both document level and answer span level. By annotating a complete dataset, we want to quantify the savings of our framework in practice. We again set the cost of each Man annotation to one price-unit and repeated the experiment three times by setting the Sem annotation cost (\(c_{1}\)

Figure 3: Left: average annotation costs when varying the ratio of Sem over Man annotation costs. Right: number of samples that were annotated with a restricted budget for a given Sem annotation cost.

Figure 2: Left: average annotation costs in every batch as a dashed line, together with a running mean as a solid line. Right: how many samples we could annotate (y-axis) with a restricted available budget (x-axis).

 

to one quarter, one third, and one half of the price unit. The results are shown in Tbl. 2. Depending on relative cost ratio \(c_{1}\), we are able to save between \(4.1\%\) and \(21.1\%\) percent of the overall annotation cost. This amounts to a total of 40,000 to 8,000 price units.5

Footnote 5: In these experiments, we obtain the overall cost by directly adding the costs of the two levels. Note that the cost can be different for document and answer span annotations, and that in such cases, our framework can still save costs at each level as shown in the table, although we cannot directly add up the costs as an overall sum.

## 7 Discussion and Future Work

We assume for the purposes of this study that questions have an answer span contained in a single document and leave an extension to multi-hop questions and unanswerable questions to future research. The robustness of our framework is demonstrated in an extensive set of simulations and experiments. We deliberately choose to leave experiments including real human annotators to future research for the following reason. Outcomes of such an experiment would be sensitive to the design of the user interface as well as the study design itself. In this paper, we want to put the emphasis on the methodological innovation of our framework and the novel annotation scheme itself.

On the other hand, experiments involving real users would provide valuable insights concerning the annotation costs and the quality of a dataset annotated with our method. Furthermore, it would be worth investigating how inter-annotator agreement or potential human biases manifest in traditional datasets as compared to those generated with our framework.

## 8 Conclusion

We presented a novel annotation framework for question answering based on textual content, which learns a cost-effective policy to combine a manual annotation scheme with a semi-supervised annotation scheme. Our framework annotates all given questions accurately while limiting costs as much as possible. We show that our framework never incurs higher costs than traditional manual annotation. On the contrary, it achieves substantial savings. For example, it reduces the overall costs by about \(4.1\%\) when Sem annotations cost about half of Man annotations. When that ratio is lowered to one fourth, our framework can reduce the total costs by up top \(21.1\%\). We think that our framework could contribute to more accessible annotation of datasets in the future and possibly even be extended to other fields and applications in natural language processing.

## Acknowledgments

We would like to thank the anonymous reviewers for their helpful comments. We are also deeply grateful to Ziyu Yao for her valuable feedback and help in brainstorming and formalizing this idea. This research was partly supported by the Army Research Office under cooperative agreements W911NF-17-1-0412, and by NSF Grant IIS1815674 and NSF CAREER #1942980. The views and conclusions contained herein are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notice herein.

## References

* Cai et al. [2013] W. Cai, Y. Zhang, and J. Zhou. 2013. Maximizing expected model change for active learning in regression. In _IEEE International Conference on Data Mining_, pages 51-60.
* Chang et al. [2019] Haw-Shiuan Chang, Shankar Vembu, Sunil Mohan, Rheeya Uppaal, and Andrew McCallum. 2019. Overcoming practical issues of deep active learning and its applications on named entity recognition. _arXiv preprint arXiv:1911.07335_.
* Chen et al. [2019]

\begin{table}
\begin{tabular}{l r r r r r r r r r} \hline \hline  & \multicolumn{3}{c}{Document-level} & \multicolumn{3}{c}{Answer span-level} & \multicolumn{3}{c}{Overall} \\ \cline{2-10}  & \(c_{1}=\nicefrac{{1}}{{4}}\) & \(c_{1}=\nicefrac{{1}}{{3}}\) & \(c_{1}=\nicefrac{{1}}{{2}}\) & \(c_{1}=\nicefrac{{1}}{{4}}\) & \(c_{1}=\nicefrac{{1}}{{3}}\) & \(c_{1}=\nicefrac{{1}}{{2}}\) & \(c_{1}=\nicefrac{{1}}{{4}}\) & \(c_{1}=\nicefrac{{1}}{{3}}\) & \(c_{1}=\nicefrac{{1}}{{2}}\) \\ \hline Traditional Annotation & 102.4 & 102.4 & 102.4 & 102.4 & 102.4 & 102.4 & 204.8 & 204.8 & 204.8 \\ Ours & 79.8 & 85.6 & 98.0 & 81.8 & 87.0 & 98.3 & 161.6 & 172.7 & 196.3 \\  & **(22.1\%)** & **(14.9\%)** & **(4.2\%)** & **(20.1\%)** & **(14.9\%)** & **(4.0\%)** & **(21.1\%)** & **(15.7\%)** & **(4.1\%)** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Overall cost (\(\times 10^{3}\) price unit) for annotating the NaturalQuestions dataset using our framework vs. conventional manual annotation for different Sem costs (\(c_{1}\)). Improvements are shown in parenthesis.

 

Daniqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In _Association for Computational Linguistics (ACL)_.
* Clark et al. [2018] Kevin Clark, Minh-Thang Luong, Christopher D. Manning, and Quoc Le. 2018. Semi-supervised sequence modeling with cross-view training. In _Empirical Methods in Natural Language Processing (EMNLP)_, pages 1914-1925.
* Clarke et al. [2010] James Clarke, Dan Goldwasser, Ming-Wei Chang, and Dan Roth. 2010. Driving semantic parsing from the world's response. In _Conference on Computational Natural Language Learning (CoNLL)_, pages 18-27.
* Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. In _Conference of the North American Chapter of the Association for Computational Linguistics (NACL)_.
* Volume 2_, IJCAI'01, page 973-978.
* Fang et al. [2017] Meng Fang, Yuan Li, and Trevor Cohn. 2017. Learning how to active learn: A deep reinforcement learning approach. In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 595-605.
* Gur et al. [2018] Izzeddin Gur, Semih Yavuz, Yu Su, and Xifeng Yan. 2018. DialSQL: Dialogue based structured query generation. In _Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 1339-1349.
* Haffari et al. [2009] Gholamreza Haffari, Maxim Roy, and Anoop Sarkar. 2009. Active learning for statistical phrase-based machine translation. In _Annual Conference of the North American Chapter of the Association for Computational Linguistics (NACL)_, pages 415-423.
* He et al. [2018] Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, Xuan Liu, Tian Wu, and Haifeng Wang. 2018. DuReader: a Chinese machine reading comprehension dataset from real-world applications. In _Workshop on Machine Reading for Question Answering_, pages 37-46.
* Iyer et al. [2017] Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. In _Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 963-973.
* Joshi et al. [2017] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In _Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 1601-1611.
* Kamath et al. [2020] Amita Kamath, Robin Jia, and Percy Liang. 2020. Selective question answering under domain shift. In _Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 5684-5696.
* Kratzwald et al. [2019] Bernhard Kratzwald, Anna Eigenmann, and Stefan Feuerriegel. 2019. Rankqa: Neural question answering with answer re-ranking. In _Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Kratzwald and Feuerriegel [2018] Bernhard Kratzwald and Stefan Feuerriegel. 2018. Adaptive document retrieval for deep question answering. In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 576-581.
* Kratzwald and Feuerriegel [2019] Bernhard Kratzwald and Stefan Feuerriegel. 2019. Learning from On-Line User Feedback in Neural Question Answering on the Web. In _The World Wide Web Conference (WWW)_, pages 906-916.
* Kreutzer and Riezler [2019] Julia Kreutzer and Stefan Riezler. 2019. Self-regulated interactive sequence-to-sequence learning. In _Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 303-315.
* Kwiatkowski et al. [2019] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: a benchmark for question answering research. _Transactions of the Association of Computational Linguistics (TACL)_.
* Lee et al. [2018] Jinhyuk Lee, Seongjun Yun, Hyunjae Kim, Miyoung Ko, and Jaewoo Kang. 2018. Ranking Paragraphs for Improving Answer Recall in Open-Domain Question Answering. In _Empirical Methods in Natural Language Processing (EMNLP)_, pages 565-569.
* Lee et al. [2019] Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In _Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 6086-6096.
* Liang et al. [2017] Chen Liang, Jonathan Berant, Quoc Le, Kenneth D. Forbus, and Ni Lao. 2017. Neural symbolic machines: Learning semantic parsers on Freebase with weak supervision. In _Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 23-33.
* Lowell et al. [2019] David Lowell, Zachary C. Lipton, and Byron C. Wallace. 2019. Practical obstacles to deploying active learning. In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 21-30.
* Martins et al. [2017] Andre F. T. Martins, Marcin Junczys-Downunt, Fabio N. Kepler, Ramon Astudillo, Chris Hokamp, and Roman Grundkiewicz. 2017. Pushing the limits of translation quality estimation. _Transactions of the 

Association for Computational Linguistics (TACL)_, 5:205-218.
* Min et al. (2018) Sewon Min, Victor Zhong, Richard Socher, and Caiming Xiong. 2018. Efficient and Robust Question Answering from Minimal Context over Documents. In _Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 1725-1735.
* Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In _Conference on Neural Information Processing Systems (NeurIPS)_.
* Petrushkov et al. (2018) Pavel Petrushkov, Shahram Khadivi, and Evgeny Matusov. 2018. Learning from chunk-based feedback in neural machine translation. In _Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 326-331.
* Pozzolo et al. (2015) A. D. Pozzolo, O. Caelen, R. A. Johnson, and G. Bontempi. 2015. Calibrating probability with undersampling for unbalanced classification. In _2015 IEEE Symposium Series on Computational Intelligence_, pages 159-166.
* Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In _Empirical Methods in Natural Language Processing (EMNLP) Language Processing_, pages 2383-2392.
* Saluja (2012) Avneesh Saluja. 2012. Machine Translation with Binary Feedback : a Large-Margin Approach. In _Biennal Conference of the Association for Machine Translation in the Americas_.
* Siddhant and Lipton (2018) Aditya Siddhant and Zachary C. Lipton. 2018. Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Study. _Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2904-2909.
* a translation quality estimation framework. In _Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 79-84.
* Sun et al. (2019) Yibo Sun, Duyu Tang, Nan Duan, Yeyun Gong, Xiaocheng Feng, Bing Qin, and Davin Jiang. 2019. Neural semantic parsing in low-resource settings with back-translation and meta-learning. In _Association for the Advancement of Artificial Intelligence (AAAI)_.
* Talmor and Berant (2018) Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In _Conference of the North American Chapter of the Association for Computational Linguistic (NACL)_, pages 641-651.
* Ting (2000) Kai Ming Ting. 2000. A comparative study of cost-sensitive boosting algorithms. In _International Conference on Machine Learning (ICML)_, pages 983-990.
* Trischler et al. (2017) Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A machine comprehension dataset. In _Workshop on Representation Learning for NLP_, pages 191-200.
* Wang et al. (2018) Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerald Tesauro, Bowen Zhou, and Jing Jiang. 2018. R^3: Reinforced Reader-Ranker for Open-Domain Question Answering. In _Association for the Advancement of Artificial Intelligence (AAAI)_.
* Xie et al. (2020) Yuqing Xie, Wei Yang, Luchen Tan, Kun Xiong, Nicholas Jing Yuan, Baoxing Huai, Ming Li, and Jimmy Lin. 2020. Distant supervision for multi-stage fine-tuning in retrieval-based question answering. In _The Web Conference (WWW)_, page 2934-2940.
* Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In _Empirical Methods in Natural Language Processing (EMNLP)_, pages 2369-2380.
* Yao et al. (2019) Ziyu Yao, Yu Su, Huan Sun, and Wen-tau Yih. 2019. Model-based interactive semantic parsing: A unified framework and a text-to-SQL case study. In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 5447-5458.
* Yao et al. (2020) Ziyu Yao, Yiqi Tang, Wen-tau Yih, Huan Sun, and Yu Su. 2020. An imitation game for learning semantic parsers from user interaction. _arXiv preprint arXiv:2005.00689_.
* Zhao et al. (2017) Jie Zhao, Yu Su, Ziyu Guan, and Huan Sun. 2017. An end-to-end deep framework for answer triggering with a novel group-level objective. In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1276-1282.
* Zhu et al. (2008) Jingbo Zhu, Huizhen Wang, Tianshun Yao, and Benjamin K Tsou. 2008. Active learning with sampling by uncertainty and density for word sense disambiguation and text classification. In _International Conference on Computational Linguistics (Coling)_, pages 1137-1144.

## Appendix A Source Code

All source code is available from github.com/berhard2202/qa-annotation.

## Appendix B Details on the QA Model

We use the same hyperparemter configuration as reported in Kratzwald et al. (2019) without further fine-tuning. The model was initialized by training on the training split of the SQuADv1.1. dataset Rajpurkar et al. (2016) 



## Appendix C Details on the Policy Model

The policy models \(\pi^{D}\) and \(\pi^{S}\) are implemented as feed forward networks composed of a dense layer with \(k\) output units and relu activation, a dropout layer with dropout probability \(z\), a second dense layer with \(k/2\) outputs and relu activation, a dropout layer with dropout probability \(z\), and a dense layer with a single output and sigmoid activation.

**Initialization:** for the first batch of annotations we initialize the policy models on SQuAD. After the first batch is annotated we only use the new data for policy updates.

**Hyperparameter search:** We tune hyperparamters on the SQuAD dataset using gridsearch with the values displayed in Tab.3 and Tab.4. Bold values mark final choices. We annotated the first 10 batches of SQuAD and choose the hyperparamters that had the lowest anntation cost. No hyperparemter tuning or architecture search was performed on the NaturalQuestions dataset which our experiments are based on.

## Appendix D Estimation of Real Annotation Costs

In order to provide additional insights on the actual annotation costs involving real users we conducted a pre-test on Amazon MTURK. For this we showed a textual explanation of the Man and Sem annotation scheme to workers and provided them with mockups for both inputs (answer-span annotation). Next, we asked \(40\) workers to report how much money they think would be a fair compensation for each of the tasks. Workers reported on average a compensation of SS5.9 for Man annotations and SS3.2 for Sem annotations. This ratio falls into the range where we make profits using our framework.

## Appendix E System

All experiments were conducted with a Nvidia Titan Xp GPU on a Server with 192GB DDR4 RAM and two 10 Core Intel Xeon Silver 4210 2.2GHz Processors.