SlowGuess/ACL_2020_OCR huggingface dataset:
Amita Kamath Robin Jia Percy Liang
{kamatha, robinjia, pliang}@cs.stanford.edu
To avoid giving wrong answers, question answering (QA) models need to know when to abstain from answering. Moreover, users often ask questions that diverge from the model's training data, making errors more likely and thus abstention more critical. In this work, we propose the setting of selective question answering under domain shift, in which a QA model is tested on a mixture of in-domain and out-of-domain data, and must answer (i.e., not abstain on) as many questions as possible while maintaining high accuracy. Abstention policies based solely on the model's softmax probabilities fare poorly, since models are overconfident on out-of-domain inputs. Instead, we train a calibrator to identify inputs on which the QA model errs, and abstain when it predicts an error is likely. Crucially, the calibrator benefits from observing the model's behavior on out-of-domain data, even if from a different domain than the test data. We combine this method with a SQuAD-trained QA model and evaluate on mixtures of SQuAD and five other QA datasets. Our method answers  $56\%$  of questions while maintaining  $80\%$  accuracy; in contrast, directly using the model's probabilities only answers  $48\%$  at  $80\%$  accuracy.
Question answering (QA) models have achieved impressive performance when trained and tested on examples from the same dataset, but tend to perform poorly on examples that are out-of-domain (OOD) (Jia and Liang, 2017; Chen et al., 2017; Yogatama et al., 2019; Talmor and Berant, 2019; Fisch et al., 2019). Deployed QA systems in search engines and personal assistants need to gracefully handle OOD inputs, as users often ask questions that fall outside of the system's training distribution. While the ideal system would correctly answer all
Figure 1: Selective question answering under domain shift with a trained calibrator. First, a QA model is trained only on source data. Then, a calibrator is trained to predict whether the QA model was correct on any given example. The calibrator's training data consists of both previously held-out source data and known OOD data. Finally, the combined selective QA system is tested on a mixture of test data from the source distribution and an unknown OOD distribution.
OOD questions, such perfection is not attainable given limited training data (Geiger et al., 2019). Instead, we aim for a more achievable yet still challenging goal: models should abstain when they are likely to err, thus avoiding showing wrong answers to users. This general goal motivates the setting of selective prediction, in which a model outputs both a prediction and a scalar confidence, and abstains on inputs where its confidence is low (El-Yaniv and Wiener, 2010; Geifman and El-Yaniv, 2017).
In this paper, we propose the setting of selective question answering under domain shift, which captures two important aspects of real-world QA: (i) test data often diverges from the training distribution, and (ii) systems must know when to abstain. We train a QA model on data from a source distribution, then evaluate selective prediction performance on a dataset that includes samples from both the source distribution and an unknown OOD distribution. This mixture simulates the likely scenario in which users only sometimes ask questions that are covered by the training distribution. While the sys
5684
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5684-5696 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
tem developer knows nothing about the unknown OOD data, we allow access to a small amount of data from a third known OOD distribution (e.g., OOD examples that they can foresee).
We first show that our setting is challenging because model softmax probabilities are unreliable estimates of confidence on out-of-domain data. Prior work has shown that a strong baseline for indomain selective prediction is MaxProb, a method that abstains based on the probability assigned by the model to its highest probability prediction (Hendrycks and Gimpel, 2017; Lakshminarayanan et al., 2017). We find that MaxProb gives good confidence estimates on in-domain data, but is overconfident on OOD data. Therefore, MaxProb performs poorly in mixed settings: it does not abstain enough on OOD examples, relative to in-domain examples.
We correct for MaxProb's overconfidence by using known OOD data to train a calibrator—a classifier trained to predict whether the original QA model is correct or incorrect on a given example (Platt, 1999; Zadrozny and Elkan, 2002). While prior work in NLP trains a calibrator on in-domain data (Dong et al., 2018), we show this does not generalize to unknown OOD data as well as training on a mixture of in-domain and known OOD data. Figure 1 illustrates the problem setup and how the calibrator uses known OOD data. We use a simple random forest calibrator over features derived from the input example and the model's softmax outputs.
We conduct extensive experiments using SQuAD (Rajpurkar et al., 2016) as the source distribution and five other QA datasets as different OOD distributions. We average across all 20 choices of using one as the unknown OOD dataset and another as the known OOD dataset, and test on a uniform mixture of SQuAD and unknown OOD data. On average, the trained calibrator achieves  $56.1\%$  coverage (i.e., the system answers  $56.1\%$  of test questions) while maintaining  $80\%$  accuracy on answered questions, outperforming MaxProb with the same QA model ( $48.2\%$  coverage at  $80\%$  accuracy), using MaxProb and training the QA model on both SQuAD and the known OOD data ( $51.8\%$  coverage), and training the calibrator only on SQuAD data ( $53.7\%$  coverage).
(1) We propose a novel setting, selective question answering under domain shift, that captures the practical necessity of knowing when to abstain on test data that differs from the training data.
(2) We show that QA models are overconfident on out-of-domain examples relative to indomain examples, which causes MaxProb to perform poorly in our setting.
(3) We show that out-of-domain data, even from a different distribution than the test data, can improve selective prediction under domain shift when used to train a calibrator.
Our setting combines extrapolation to out-of-domain data with selective prediction. We also distinguish our setting from the tasks of identifying unanswerable questions and outlier detection.
2.1 Extrapolation to out-of-domain data
Extrapolating from training data to test data from a different distribution is an important challenge for current NLP models (Yogatama et al., 2019). Models trained on many domains may still struggle to generalize to new domains, as these may involve new types of questions or require different reasoning skills (Talmor and Berant, 2019; Fisch et al., 2019). Related work on domain adaptation also tries to generalize to new distributions, but assumes some knowledge about the test distribution, such as unlabeled examples or a few labeled examples (Blitzer et al., 2006; Daume III, 2007); we assume no such access to the test distribution, but instead make the weaker assumption of access to samples from a different OOD distribution.
2.2 Selective prediction
Selective prediction, in which a model can either predict or abstain on each test example, is a longstanding research area in machine learning (Chow, 1957; El-Yaniv and Wiener, 2010; Geifman and El-Yaniv, 2017). In NLP, Dong et al. (2018) use a calibrator to obtain better confidence estimates for semantic parsing. Rodriguez et al. (2019) use a similar approach to decide when to answer QuizBowl questions. These works focus on training and testing models on the same distribution, whereas our training and test distributions differ.
Selective prediction under domain shift. Other fields have recognized the importance of selective prediction under domain shift. In medical applications, models may be trained and tested on different groups of patients, so selective prediction is needed to avoid costly errors (Feng et al., 2019). In computational chemistry, Toplak et al. (2014) use
5685
selective prediction techniques to estimate the set of (possibly out-of-domain) molecules for which a reactivity classifier is reliable. To the best of our knowledge, our work is the first to study selective prediction under domain shift in NLP.
Answer validation. Traditional pipelined systems for open-domain QA often have dedicated systems for answer validation—judging whether a proposed answer is correct. These systems often rely on external knowledge about entities (Magnini et al., 2002; Ko et al., 2007). Knowing when to abstain has been part of past QA shared tasks like RespubliQA (Peñas et al., 2009) and QA4MRE (Peñas et al., 2013). IBM's Watson system for Jeopardy also uses a pipelined approach for answer validation (Gondek et al., 2012). Our work differs by focusing on modern neural QA systems trained end-to-end, rather than pipelined systems, and by viewing the problem of abstention in QA through the lens of selective prediction.
2.3 Related goals and tasks
Calibration. Knowing when to abstain is closely related to calibration—having a model's output probability align with the true probability of its prediction (Platt, 1999). A key distinction is that selective prediction metrics generally depend only on relative confidence—systems are judged on their ability to rank correct predictions higher than incorrect predictions (El-Yaniv and Wiener, 2010). In contrast, calibration error depends on the absolute confidence scores. Nonetheless, we will find it useful to analyze calibration in Section 5.3, as miscalibration on some examples but not others does imply poor relative ordering, and therefore poor selective prediction. Ovadia et al. (2019) observe increases in calibration error under domain shift.
Identifying unanswerable questions. In SQuAD 2.0, models must recognize when a paragraph does not entail an answer to a question (Rajpurkar et al., 2018). Sentence selection systems must rank passages that answer a question higher than passages that do not (Wang et al., 2007; Yang et al., 2015). In these cases, the goal is to "abstain" when no system (or person) could infer an answer to the given question using the given passage. In contrast, in selective prediction, the model should abstain when it would give a wrong answer if forced to make a prediction.
Outlier detection. We distinguish selective prediction under domain shift from outlier detection, the task of detecting out-of-domain examples (Schölkopf et al., 1999; Hendrycks and Gimpel, 2017; Liang et al., 2018). While one could use an outlier detector for selective classification (e.g., by abstaining on all examples flagged as outliers), this would be too conservative, as QA models can often get a non-trivial fraction of OOD examples correct (Talmor and Berant, 2019; Fisch et al., 2019). Hendrycks et al. (2019b) use known OOD data for outlier detection by training models to have high entropy on OOD examples; in contrast, our setting rewards models for predicting correctly on OOD examples, not merely having high entropy.
We formally define the setting of selective prediction under domain shift, starting with some notation for selective prediction in general.
3.1 Selective Prediction
Given an input  $x$ , the selective prediction task is to output  $(\hat{y}, c)$  where  $\hat{y} \in Y(x)$ , the set of answer candidates, and  $c \in \mathbb{R}$  denotes the model's confidence. Given a threshold  $\gamma \in \mathbb{R}$ , the overall system predicts  $\hat{y}$  if  $c \geq \gamma$  and abstain otherwise.
The risk-coverage curve provides a standard way to evaluate selective prediction methods (El-Yaniv and Wiener, 2010). For a test dataset  $D_{\mathrm{test}}$ , any choice of  $\gamma$  has an associated coverage—the fraction of  $D_{\mathrm{test}}$  the model makes a prediction on—and risk—the error on that fraction of  $D_{\mathrm{test}}$ . As  $\gamma$  decreases, coverage increases, but risk will usually also increase. We plot risk versus coverage and evaluate on the area under this curve (AUC), as well as the maximum possible coverage for a desired risk level. The former metric averages over all  $\gamma$ , painting an overall picture of selective prediction performance, while the latter evaluates at a particular choice of  $\gamma$  corresponding to a specific level of risk tolerance.
3.2 Selective Prediction under Domain Shift
We deviate from prior work by considering the setting where the model's training data  $D_{\mathrm{train}}$  and test data  $D_{\mathrm{test}}$  are drawn from different distributions. As our experiments demonstrate, this setting is challenging because standard QA models are overconfident on out-of-domain inputs.
To formally define our setting, we specify three
5686
data distributions. First,  $p_{\mathrm{source}}$  is the source distribution, from which a large training dataset  $D_{\mathrm{train}}$  is sampled. Second,  $q_{\mathrm{unk}}$  is an unknown OOD distribution, representing out-of-domain data encountered at test time. The test dataset  $D_{\mathrm{test}}$  is sampled from  $p_{\mathrm{test}}$ , a mixture of  $p_{\mathrm{source}}$  and  $q_{\mathrm{unk}}$ :
$$
p _ {\text {t e s t}} = \alpha p _ {\text {s o u r c e}} + (1 - \alpha) q _ {\text {u n k}} \tag {1}
$$
for  $\alpha \in (0,1)$ . We choose  $\alpha = \frac{1}{2}$ , and examine the effect of changing this ratio in Section 5.8. Third,  $q_{\mathrm{known}}$  is a known OOD distribution, representing examples not in  $p_{\mathrm{source}}$  but from which the system developer has a small dataset  $D_{\mathrm{calib}}$ .
3.3 Selective Question Answering
While our framework is general, we focus on extractive question answering, as exemplified by SQuAD (Rajpurkar et al., 2016), due to its practical importance and the diverse array of available QA datasets in the same format. The input  $x$  is a passage-question pair  $(p,q)$ , and the set of answer candidates  $Y(x)$  is all spans of the passage  $p$ . A base model  $f$  defines a probability distribution  $f(y\mid x)$  over  $Y(x)$ . All selective prediction methods we consider choose  $\hat{y} = \arg \max_{y'\in Y(x)}f(y'\mid x)$ , but differ in their associated confidence  $c$ .
Recall that our setting differs from the standard selective prediction setting in two ways: unknown OOD data drawn from  $q_{\mathrm{unk}}$  appears at test time, and known OOD data drawn from  $q_{\mathrm{known}}$  is available to the system. Intuitively, we expect that systems must use the known OOD data to generalize to the unknown OOD data. In this section, we present three standard selective prediction methods for indomain data, and show how they can be adapted to use data from  $q_{\mathrm{known}}$ .
4.1 MaxProb
The first method, MaxProb, directly uses the probability assigned by the base model to  $\hat{y}$  as an estimate of confidence. Formally, MaxProb with model  $f$  estimates confidence on input  $x$  as:
$$
c _ {\text {M a x P r o b}} = f (\hat {y} \mid x) = \max  _ {y ^ {\prime} \in Y (x)} f \left(y ^ {\prime} \mid x\right). \tag {2}
$$
MaxProb is a strong baseline for our setting. Across many tasks, MaxProb has been shown to distinguish in-domain test examples that the model gets right from ones the model gets wrong
(Hendrycks and Gimpel, 2017). MaxProb is also a strong baseline for outlier detection, as it is lower for out-of-domain examples than in-domain examples (Lakshminarayanan et al., 2017; Liang et al., 2018; Hendrycks et al., 2019b). This is desirable for our setting: models make more mistakes on OOD examples, so they should abstain more on OOD examples than in-domain examples.
MaxProb can be used with any base model  $f$ . We consider two such choices: a model  $f_{\mathrm{src}}$  trained only on  $D_{\mathrm{train}}$ , or a model  $f_{\mathrm{src + known}}$  trained on the union of  $D_{\mathrm{train}}$  and  $D_{\mathrm{calib}}$ .
4.2 Test-time Dropout
For neural networks, another standard approach to estimate confidence is to use dropout at test time. Gal and Ghahramani (2016) showed that dropout gives good confidence estimates on OOD data.
Given an input  $x$  and model  $f$ , we compute  $f$  on  $x$  with  $K$  different dropout masks, obtaining prediction distributions  $\hat{p}_1, \dots, \hat{p}_K$ , where each  $\hat{p}_i$  is a probability distribution over  $Y(x)$ . We consider two statistics of these  $\hat{p}_i$ 's that are commonly used as confidence estimates. First, we take the mean of  $\hat{p}_i(\hat{y})$  across all  $i$  (Lakshminarayanan et al., 2017):
$$
c _ {\text {D r o p o u t M e a n}} = \frac {1}{K} \sum_ {i = 1} ^ {K} \hat {p} _ {i} (\hat {y}). \tag {3}
$$
This can be viewed as ensembling the predictions across all  $K$  dropout masks by averaging them.
Second, we take the negative variance of the  $\hat{p}_i(\hat{y})$  's (Feinman et al., 2017; Smith and Gal, 2018):
$$
c _ {\text {D r o p o u t V a r}} = - \operatorname {V a r} [ \hat {p} _ {1} (\hat {y}), \dots , \hat {p} _ {K} (\hat {y}) ]. \tag {4}
$$
Higher variance corresponds to greater uncertainty, and hence favors abstaining. Like MaxProb, dropout can be used either with  $f$  trained only on  $D_{\mathrm{train}}$ , or on both  $D_{\mathrm{train}}$  and the known OOD data.
Test-time dropout has practical disadvantages compared to MaxProb. It requires access to internal model representations, whereas MaxProb only requires black box access to the base model (e.g., API calls to a trained model). Dropout also requires  $K$  forward passes of the base model, leading to a  $K$ -fold increase in runtime.
4.3 Training a calibrator
Our final method trains a calibrator to predict when a base model (trained only on data from  $p_{\mathrm{source}}$ ) is
correct (Platt, 1999; Dong et al., 2018). We differ from prior work by training the calibrator on a mixture of data from  $p_{\mathrm{source}}$  and  $q_{\mathrm{known}}$ , anticipating the test-time mixture of  $p_{\mathrm{source}}$  and  $q_{\mathrm{unk}}$ . More specifically, we hold out a small number of  $p_{\mathrm{source}}$  examples from base model training, and train the calibrator on the union of these examples and the  $q_{\mathrm{known}}$  examples. We define  $c_{\mathrm{Calibrator}}$  to be the prediction probability of the calibrator.
The calibrator itself could be any binary classification model. We use a random forest classifier with seven features: passage length, the length of the predicted answer  $\hat{y}$ , and the top five softmax probabilities output by the model. These features require only a minimal amount of domain knowledge to define. Rodriguez et al. (2019) similarly used multiple softmax probabilities to decide when to answer questions. The simplicity of this model makes the calibrator fast to train when given new data from  $q_{\mathrm{known}}$ , especially compared to retraining the QA model on that data.
We experiment with four variants of the calibrator. First, to measure the impact of using known OOD data, we change the calibrator's training data: it can be trained either on data from  $p_{\mathrm{source}}$  only, or both  $p_{\mathrm{source}}$  and  $q_{\mathrm{known}}$  data as described. Second, we consider a modification where instead of the model's probabilities, we use probabilities from the mean ensemble over dropout masks, as described in Section 4.2, and also add  $c_{\mathrm{DropoutVar}}$  as a feature. As discussed above, dropout features are costly to compute and assume white-box access to the model, but may result in better confidence estimates. Both of these variables can be changed independently, leading to four configurations.
Data. We use SQuAD 1.1 (Rajpurkar et al., 2016) as the source dataset and five other datasets as OOD datasets: NewsQA (Trischler et al., 2017), TriviaQA (Joshi et al., 2017), SearchQA (Dunn et al., 2017), HotpotQA (Yang et al., 2018), and Natural Questions (Kwiatkowski et al., 2019). These are all extractive question answering datasets where all questions are answerable; however, they vary widely in the nature of passages (e.g., Wikipedia, news, web snippets), questions (e.g., Jeopardy and trivia questions), and relationship between pas
sages and questions (e.g., whether questions are written based on passages, or passages retrieved based on questions). We used the preprocessed data from the MRQA 2019 shared task (Fisch et al., 2019). For HotpotQA, we focused on multi-hop questions by selecting only "hard" examples, as defined by Yang et al. (2018). In each experiment, two different OOD datasets are chosen as  $q_{\mathrm{known}}$  and  $q_{\mathrm{unk}}$ . All results are averaged over all 20 such combinations, unless otherwise specified. We sample 2,000 examples from  $q_{\mathrm{known}}$  for  $D_{\mathrm{calib}}$ , and 4,000 SQuAD and 4,000  $q_{\mathrm{unk}}$  examples for  $D_{\mathrm{test}}$ . We evaluate using exact match (EM) accuracy, as defined by SQuAD (Rajpurkar et al., 2016). Additional details can be found in Appendix A.1.
QA model. For our QA model, we use the BERT-base SQuAD 1.1 model trained for 2 epochs (Devlin et al., 2019). We train six models total: one  $f_{\mathrm{src}}$  and five  $f_{\mathrm{src + known}}$ 's, one for each OOD dataset.
Selective prediction methods. For test-time dropout, we use  $K = 30$  different dropout masks, as in Dong et al. (2018). For our calibrator, we use the random forest implementation from Scikit-learn (Pedregosa et al., 2011). We train on 1,600 SQuAD examples and 1,600 known OOD examples, and use the remaining 400 SQuAD and 400 known OOD examples as a validation set to tune calibrator hyperparameters via grid search. We average our results over 10 random splits of this data. When training the calibrator only on  $p_{\mathrm{source}}$ , we use 3,200 SQuAD examples for training and 800 for validation, to ensure equal dataset sizes. Additional details can be found in Appendix A.2.
Training a calibrator with  $q_{\mathrm{known}}$  outperforms other methods. Table 1 compares all methods that do not use test-time dropout. Compared to MaxProb with  $f_{\mathrm{src + known}}$ , the calibrator has 4.3 points and 6.7 points higher coverage at  $80\%$  and  $90\%$  accuracy respectively, and 1.1 points lower AUC. This demonstrates that training a calibrator is a better use of known OOD data than training a QA model. The calibrator trained on both  $p_{\mathrm{source}}$  and  $q_{\mathrm{known}}$  also outperforms the calibrator trained on  $p_{\mathrm{source}}$  alone by  $2.4\%$  coverage at  $80\%$  accuracy. All methods perform far worse than the optimal selective predictor with the given base model, though
$295\%$  confidence interval is [1.01, 1.69], using the paired bootstrap test with 1000 bootstrap samples.
<sup>1</sup>We consider these different datasets to represent different domains, hence our usage of the term "domain shift."
5688
AUC↓	Cov @Acc=80%↑	Cov @Acc=90%↑
Train QA model on SQuAD
MaxProb	20.54	48.23	21.07
Calibrator (psource only)	19.27	53.67	26.68
Calibrator (psource and qknown)	18.47	56.06	29.42
Best possible	9.64	74.92	66.59
Train QA model on SQuAD + known OOD
MaxProb	19.61	51.75	22.76
Best possible	8.83	76.80	68.26
Table 1: Results for methods without test-time dropout. The calibrator with access to  $q_{\mathrm{known}}$  outperforms all other methods.  $\downarrow$ : lower is better.  $\uparrow$ : higher is better.
AUC↓	Cov @Acc=80%↑	Cov @Acc=90%↑
Train QA model on SQuAD
Test-time dropout (−var)	28.13	24.50	15.40
Test-time dropout (mean)	18.35	57.49	29.55
Calibrator (psource only)	17.84	58.35	34.27
Calibrator (psource and qknown)	17.31	59.99	34.99
Best possible	9.64	74.92	66.59
Train QA model on SQuAD + known OOD
Test-time dropout (−var)	26.67	26.74	15.95
Test-time dropout (mean)	17.72	59.60	30.40
Best possible	8.83	76.80	68.26
Table 2: Results for methods that use test-time dropout. Here again, the calibrator with access to  $q_{\mathrm{known}}$  outperforms all other methods.
achieving this bound may not be realistic.3