SlowGuess/ACL_2020_OCR huggingface dataset:
Test-time dropout improves results but is expensive. Table 2 shows results for methods that use test-time dropout, as described in Section 4.2. The negative variance of  $\hat{p}_i(\hat{y})$ 's across dropout masks serves poorly as an estimate of confidence, but the mean performs well. The best performance is attained by the calibrator using dropout features, which has  $3.9\%$  higher coverage at  $80\%$  accuracy than the calibrator with non-dropout features. Since test-time dropout introduces substantial (i.e.,  $K$ -fold) runtime overhead, our remaining analyses focus on methods without test-time dropout.
The QA model has lower non-trivial accuracy on OOD data. Next, we motivate our focus on selective prediction, as opposed to outlier detection, by showing that the QA model still gets a non-trivial fraction of OOD examples correct. Table 3 shows the (non-selective) exact match scores
Figure 2: Area under the risk-coverage curve as a function of how much data from  $q_{\mathrm{known}}$  is available. At all points, using data from  $q_{\mathrm{known}}$  to train the calibrator is more effective than using it for QA model training.
for all six QA models used in our experiments on all datasets. All models get around  $80\%$  accuracy on SQuAD, and around  $40\%$  to  $50\%$  accuracy on most OOD datasets. Since OOD accuracies are much higher than  $0\%$ , abstaining on all OOD examples would be overly conservative. At the same time, since OOD accuracy is worse than in-domain accuracy, a good selective predictor should answer more in-domain examples and fewer OOD examples. Training on 2,000  $q_{\mathrm{known}}$  examples does not significantly help the base model extrapolate to other  $q_{\mathrm{unk}}$  distributions.
Results hold across different amounts of known OOD data. As shown in Figure 2, across all amounts of known OOD data, using it to train and validate the calibrator (in an 80-20 split) performs better than adding all of it to the QA training data and using MaxProb.
5.3 Overconfidence of MaxProb
We now show why MaxProb performs worse in our setting compared to the in-domain setting: it is miscalibrated on out-of-domain examples. Figure 3a shows that MaxProb values are generally lower for OOD examples than in-domain examples, following previously reported trends (Hendrycks and Gimpel, 2017; Liang et al., 2018). However, the MaxProb values are still too high out-of-domain. Figure 3b shows that MaxProb is not well calibrated: it is underconfident in-domain, and overconfident out-of-domain. For example, for a Max-
4In Section A.3, we confirm that an outlier detector does not achieve good selective prediction performance.
5The in-domain underconfidence is because SQuAD (and some other datasets) provides only one answer at training time, but multiple answers are considered correct at test time. In Ap
As the QA model has fixed accuracy  $< 100\%$  on  $D_{\mathrm{test}}$ , it is impossible to achieve  $0\%$  risk at  $100\%$  coverage.
5689
Train Data ↓ / Test Data →	SQuAD	TriviaQA	HotpotQA	NewsQA	Natural Questions	SearchQA
SQuAD only	80.95	48.43	44.88	40.45	42.78	17.98
SQuAD + 2K TriviaQA	81.48	(50.50)	43.95	39.15	47.05	25.23
SQuAD + 2K HotpotQA	81.15	49.35	(53.60)	39.85	48.18	24.40
SQuAD + 2K NewsQA	81.50	50.18	42.88	(44.00)	47.08	20.40
SQuAD + 2K NaturalQuestions	81.48	51.43	44.38	40.90	(54.85)	25.95
SQuAD + 2K SearchQA	81.60	56.58	44.30	40.15	47.05	(59.80)
Table 3: Exact match accuracy for all six QA models on all six test QA datasets. Training on  $D_{\mathrm{calib}}$  improves accuracy on data from the same dataset (diagonal), but generally does not improve accuracy on data from  $q_{\mathrm{unk}}$ .
(a)
(c)
Figure 3: MaxProb is lower on average for OOD data than in-domain data (a), but it is still overconfident on OOD data: when plotting the true probability of correctness vs. MaxProb (b), the OOD curve is below the  $y = x$  line, indicating MaxProb overestimates the probability that the prediction is correct. The calibrator assigns lower confidence on OOD data (c) and has a smaller gap between in-domain and OOD curves (d), indicating improved calibration.
Prob of 0.6, the model is about  $80\%$  likely to get the question correct if it came from SQuAD (in-domain), and  $45\%$  likely to get the question correct if it was OOD. When in-domain and OOD examples are mixed at test time, MaxProb therefore does not abstain enough on the OOD examples. Figure 3d shows that the calibrator is better calibrated, even though it is not trained on any unknown OOD data. In Appendix A.5, we show that the calibrator abstains on more OOD examples than MaxProb.
Our finding that the BERT QA model is not overconfident in-domain aligns with Hendrycks et al. (2019a), who found that pre-trained computer vision models are better calibrated than models trained from scratch, as pre-trained models can be
trained for fewer epochs. Our QA model is only trained for two epochs, as is standard for BERT. Our findings also align with Ovadia et al. (2019), who find that computer vision and text classification models are poorly calibrated out-of-domain even when well-calibrated in-domain. Note that miscalibration out-of-domain does not imply poor selective prediction on OOD data, but does imply poor selective prediction in our mixture setting.
5.4 Extrapolation between datasets
We next investigated how choice of  $q_{\mathrm{known}}$  affects generalization of the calibrator to  $q_{\mathrm{unk}}$ . Figure 4 shows the percentage reduction between MaxProb and optimal AUC achieved by the trained calibrator. The calibrator outperforms MaxProb over all dataset combinations, with larger gains when  $q_{\mathrm{known}}$  and  $q_{\mathrm{unk}}$  are similar. For example, samples from TriviaQA help generalization to SearchQA and vice versa; both use web snippets as passages. Samples from NewsQA, the only other non-Wikipedia dataset, are also helpful for both. On the other hand, no other dataset significantly helps generalization to HotpotQA, likely due to HotpotQA's unique focus on multi-hop questions.
5.5 Calibrator feature ablations
We determine the importance of each feature of the calibrator by removing each of its features individually, leaving the rest. From Table 4, we see that the most important features are the softmax probabilities and the passage length. Intuitively, passage length is meaningful both because longer passages have more answer candidates, and because passage length differs greatly between different domains.
5.6 Error analysis
We examined calibrator errors on two pairs of  $q_{\mathrm{known}}$  and  $q_{\mathrm{unk}}$  one similar pair of datasets and one dissimilar. For each, we sampled 100 errors in which the system confidently gave a wrong answer (overconfident), and 100 errors in which the sys
pendix A.4, we show that removing multiple answers makes MaxProb well-calibrated in-domain; it stays overconfident out-of-domain.
5690
Figure 4: Results for different choices of  $q_{\mathrm{known}}$  (y-axis) and  $q_{\mathrm{unk}}$  (x-axis). For each pair, we report the percent AUC improvement of the trained calibrator over MaxProb, relative to the total possible improvement. Datasets that use similar passages (e.g., SearchQA and TriviaQA) help each other the most. Main diagonal elements (shaded) assume access to  $q_{\mathrm{unk}}$  (see Section 5.9).
AUC ↓	Cov @ Acc=80% ↑	Cov @ Acc=90% ↑
All features	18.47	56.06	29.42
-Top softmax probability	18.61	55.46	29.27
-2nd:5th highest softmax probabilities	19.11	54.29	26.67
-All softmax probabilities	26.41	24.57	0.08
-Context length	19.79	51.73	24.24
-Prediction length	18.6	55.67	29.30
Table 4: Performance of the calibrator as each of its features is removed individually, leaving the rest. The base model's softmax probabilities are important features, as is passage length.
tem abstained but would have gotten the question correct if it had answered (underconfident). These were sampled from the 1000 most overconfident or underconfident errors, respectively.
$q_{\mathrm{known}} = \mathrm{NewsQA}$ ,  $q_{\mathrm{unk}} = \mathrm{TriviaQA}$ . These two datasets are from different non-Wikipedia sources.  $62\%$  of overconfidence errors are due to the model predicting valid alternate answers, or span mismatches—the model predicts a slightly different span than the gold span, and should be considered correct; thus the calibrator was not truly overconfident. This points to the need to improve QA evaluation metrics (Chen et al., 2019).  $45\%$  of underconfidence errors are due to the passage requiring coreference resolution over long distances, including with the article title. Neither SQuAD nor NewsQA passages have coreference chains as long
or contain titles, so it is unsurprising that the calibrator struggles on these cases. Another  $25\%$  of underconfidence errors were cases in which there was insufficient evidence in the paragraph to answer the question (as TriviaQA was constructed via distant supervision), so the calibrator was not incorrect to assign low confidence.  $16\%$  of all underconfidence errors also included phrases that would not be common in SQuAD and NewsQA, such as using "said bye bye" for "banned."
$q_{\mathrm{known}} = \mathrm{NewsQA}$ ,  $q_{\mathrm{unk}} = \mathrm{HotpotQA}$ . These two datasets are dissimilar from each other in multiple ways. HotpotQA uses short Wikipedia passages and focuses on multi-hop questions; NewsQA has much longer passages from news articles and does not focus on multi-hop questions.  $34\%$  of the overconfidence errors are due to valid alternate answers or span mismatches. On  $65\%$  of the underconfidence errors, the correct answer was the only span in the passage that could plausibly answer the question, suggesting that the model arrived at the answer due to artifacts in HotpotQA that facilitate guesswork (Chen and Durrett, 2019; Min et al., 2019). In these situations, the calibrator's lack of confidence is therefore justifiable.
5.7 Relationship with Unanswerable Questions
We now study the relationship between selective prediction and identifying unanswerable questions.
Unanswerable questions do not aid selective prediction. We trained a QA model on SQuAD 2.0 (Rajpurkar et al., 2018), which augments SQuAD 1.1 with unanswerable questions. Our trained calibrator with this model gets 18.38 AUC, which is very close to the 18.47 for the model trained on SQuAD 1.1 alone. MaxProb also performed similarly with the SQuAD 2.0 model (20.81 AUC) and SQuAD 1.1 model (20.54 AUC).
Selective prediction methods do not identify unanswerable questions. For both MaxProb and our calibrator, we pick a threshold  $\gamma' \in \mathbb{R}$  and predict that a question is unanswerable if the confidence  $c < \gamma'$ . We choose  $\gamma'$  to maximize SQuAD 2.0 EM score. Both methods perform poorly: the calibrator (averaged over five choices of  $q_{\mathrm{known}}$ ) achieves 54.0 EM, while MaxProb achieves 53.1 EM. $^6$  These results only weakly outperform the
<sup>6</sup>We evaluate on 4000 questions randomly sampled from the SQuAD 2.0 development set.
Figure 5: Difference in AUC between calibrator and MaxProb, as a function of how much of  $D_{\mathrm{test}}$  comes from  $p_{\mathrm{source}}$  (i.e., SQuAD) instead of  $q_{\mathrm{unk}}$ , averaged over 5 OOD datasets. The calibrator outperforms MaxProb most when  $D_{\mathrm{test}}$  is a mixture of  $p_{\mathrm{source}}$  and  $q_{\mathrm{unk}}$ .
majority baseline of 48.9 EM.
Taken together, these results indicate that identifying unanswerable questions is a very different task from knowing when to abstain under distribution shift. Our setting focuses on test data that is dissimilar to the training data, but on which the original QA model can still correctly answer a nontrivial fraction of examples. In contrast, unanswerable questions in SQuAD 2.0 look very similar to answerable questions, but a model trained on SQuAD 1.1 gets all of them wrong.
5.8 Changing ratio of in-domain to OOD
Until now, we used  $\alpha = \frac{1}{2}$  both for  $D_{\mathrm{test}}$  and training the calibrator. Now we vary  $\alpha$  for both, ranging from using only SQuAD to only OOD data (sampled from  $q_{\mathrm{known}}$  for  $D_{\mathrm{calib}}$  and from  $q_{\mathrm{unk}}$  for  $D_{\mathrm{test}}$ ).
Figure 5 shows the difference in AUC between the trained calibrator and MaxProb. At both ends of the graph, the difference is close to 0, showing that MaxProb performs well in homogeneous settings. However, when the two data sources are mixed, the calibrator outperforms MaxProb significantly. This further supports our claim that MaxProb performs poorly in mixed settings.
5.9 Allowing access to  $q_{\mathrm{unk}}$
We note that our findings do not hold in the alternate setting where we have access to samples from  $q_{\mathrm{unk}}$  (instead of  $q_{\mathrm{known}}$ ). Training the QA model with this OOD data and using MaxProb achieves average AUC of 16.35, whereas training a calibrator achieves 17.87; unsurprisingly, training on examples similar to the test data is helpful. We do not focus on this setting, as our goal is to build
selective QA models for unknown distributions.
In this paper, we propose the setting of selective question answering under domain shift, in which systems must know when to abstain on a mixture of in-domain and unknown OOD examples. Our setting combines two important goals for real-world systems: knowing when to abstain, and handling distribution shift at test time. We show that models are overconfident on OOD examples, leading to poor performance in the our setting, but training a calibrator using other OOD data can help correct for this problem. While we focus on question answering, our framework is general and extends to any prediction task for which graceful handling of out-of-domain inputs is necessary.
Across many tasks, NLP models struggle on out-of-domain inputs. Models trained on standard natural language inference datasets (Bowman et al., 2015) generalize poorly to other distributions (Thorne et al., 2018; Naik et al., 2018). Achieving high accuracy on out-of-domain data may not even be possible if the test data requires abilities that are not learnable from the training data (Geiger et al., 2019). Adversarily chosen ungrammatical text can also cause catastrophic errors (Wallace et al., 2019; Cheng et al., 2020). In all these cases, a more intelligent model would recognize that it should abstain on these inputs.
Traditional NLU systems typically have a natural ability to abstain. SHRDLU recognizes statements that it cannot parse, or that it finds ambiguous (Winograd, 1972). QUALM answers reading comprehension questions by constructing reasoning chains, and abstains if it cannot find one that supports an answer (Lehnert, 1977).
NLP systems deployed in real-world settings inevitably encounter a mixture of familiar and unfamiliar inputs. Our work provides a framework to study how models can more judiciously abstain in these challenging environments.
Reproducibility. All code, data and experiments are available on the Codalab platform at https://bit.ly/35inCah.
Acknowledgments. This work was supported by the DARPA ASED program under FA8650-18-2-7882. We thank Ananya Kumar, John Hewitt, Dan Iter, and the anonymous reviewers for their helpful comments and insights.
J. Blitzer, R. McDonald, and F. Pereira. 2006. Domain adaptation with structural correspondence learning. In Empirical Methods in Natural Language Processing (EMNLP).
S. Bowman, G. Angeli, C. Potts, and C. D. Manning. 2015. A large annotated corpus for learning natural language inference. In Empirical Methods in Natural Language Processing (EMNLP).
A. Chen, G. Stanovsky, S. Singh, and M. Gardner. 2019. Evaluating question answering evaluation. In Workshop on Machine Reading for Question Answering (MRQA).
D. Chen, A. Fisch, J. Weston, and A. Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Association for Computational Linguistics (ACL).
J. Chen and G. Durrett. 2019. Understanding dataset design choices for multi-hop reasoning. In North American Association for Computational Linguistics (NAACL).
M. Cheng, J. Yi, H. Zhang, P. Chen, and C. Hsieh. 2020. Seq2Sick: Evaluating the robustness of sequence-to-sequence models with adversarial examples. In Association for the Advancement of Artificial Intelligence (AAAI).
C. K. Chow. 1957. An optimum character recognition system using decision functions. In IRE Transactions on Electronic Computers.
H. Daume III. 2007. Frustratingly easy domain adaptation. In Association for Computational Linguistics (ACL).
J. Devlin, M. Chang, K. Lee, and K. Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Association for Computational Linguistics (ACL), pages 4171-4186.
L. Dong, C. Quirk, and M. Lapata. 2018. Confidence modeling for neural semantic parsing. In *Association for Computational Linguistics (ACL)*.
M. Dunn, L. Sagun, M. Higgins, U. Guney, V. Cirik, and K. Cho. 2017. SearchQA: A new Q&A dataset augmented with context from a search engine. arXiv.
R. El-Yaniv and Y. Wiener. 2010. On the foundations of noise-free selective classification. Journal of Machine Learning Research (JMLR), 11.
R. Feinman, R. R. Curtin, S. Shintre, and A. B. Gardner. 2017. Detecting adversarial samples from artifacts. arXiv preprint arXiv:1703.00410.
J. Feng, A. Sondhi, J. Perry, and N. Simon. 2019. Selective prediction-set models with coverage guarantees. arXiv preprint arXiv:1906.05473.
A. Fisch, A. Talmor, R. Jia, M. Seo, E. Choi, and D. Chen. 2019. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Workshop on Machine Reading for Question Answering (MRQA).
Y. Gal and Z. Ghahramani. 2016. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning (ICML).
Y. Geifman and R. El-Yaniv. 2017. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems (NeurIPS).
A. Geiger, I. Cases, L. Karttunen, and C. Potts. 2019. Posing fair generalization tasks for natural language inference. In Empirical Methods in Natural Language Processing (EMNLP).
D. C. Gondek, A. Lally, A. Kalyanpur, J. W. Murdock, P. A. Duboue, L. Zhang, Y. Pan, Z. M. Qiu, and C. Welty. 2012. A framework for merging and ranking of answers in DeepQA. IBM Journal of Research and Development, 56.
D. Hendrycks and K. Gimpel. 2017. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations (ICLR).
D. Hendrycks, K. Lee, and M. Mazeika. 2019a. Using pre-training can improve model robustness and uncertainty. In International Conference on Machine Learning (ICML).
D. Hendrycks, M. Mazeika, and T. Dietterich. 2019b. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations (ICLR).
R. Jia and P. Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Empirical Methods in Natural Language Processing (EMNLP).
M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Association for Computational Linguistics (ACL).
J. Ko, L. Si, and E. Nyberg. 2007. A probabilistic framework for answer selection in question answering. In North American Association for Computational Linguistics (NAACL).
T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. 2019. Natural questions: a benchmark for question answering research. In Association for Computational Linguistics (ACL).
5693
B. Lakshminarayanan, A. Pritzel, and C. Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems (NeurIPS).
W. Lehnert. 1977. The Process of Question Answering. Ph.D. thesis, Yale University.
S. Liang, Y. Li, and R. Srikant. 2018. Enhancing the reliability of out-of-distribution image detection in neural networks. In International Conference on Learning Representations (ICLR).
B. Magnini, M. Negri, R. Prevete, and H. Tanev. 2002. Is it the right answer? exploiting web redundancy for answer validation. In Association for Computational Linguistics (ACL).
S. Min, E. Wallace, S. Singh, M. Gardner, H. Hajishirzi, and L. Zettlemoyer. 2019. Compositional questions do not necessitate multi-hop reasoning. In *Association for Computational Linguistics (ACL)*.
A. Naik, A. Ravichander, N. Sadeh, C. Rose, and G. Neubig. 2018. Stress test evaluation for natural language inference. In International Conference on Computational Linguistics (COLING), pages 2340-2353.
Y. Oren, S. Sagawa, T. Hashimoto, and P. Liang. 2019. Distributionally robust language modeling. In Empirical Methods in Natural Language Processing (EMNLP).
Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. V. Dillon, B. Lakshminarayanan, and J. Snoek. 2019. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems (NeurIPS).
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research (JMLR), 12.
A. Peñas, P. Forner, R. Sutcliffe, Álvaro Rodrigo, C. Forascu, I. Alegria, D. Giampiccolo, N. Moreau, and P. Osenova. 2009. Overview of ResPubliQA 2009: Question answering evaluation over european legislation. In *Cross Language Evaluation Forum*.
A. Peñas, E. Hovy, P. Forner, Álvaro Rodrigo, R. Sutcliffe, and R. Morante. 2013. QA4MRE 2011-2013: Overview of question answering for machine reading evaluation. In *Cross Language Evaluation Forum*.
J. Platt. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers, 10(3):61-74.