SlowGuess/ACL_2020_OCR huggingface dataset:
P. Rajpurkar, R. Jia, and P. Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In Association for Computational Linguistics (ACL).
P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP).
P. Rodriguez, S. Feng, M. Iyyer, H. He, and J. Boyd-Graber. 2019. Quizbowl: The case for incremental question answering. arXiv preprint arXiv:1904.04792.
B. Scholkopf, R. Williamson, A. Smola, J. Shawe-Taylor, and J. Platt. 1999. Support vector method for novelty detection. In Advances in Neural Information Processing Systems (NeurIPS).
L. Smith and Y. Gal. 2018. Understanding measures of uncertainty for adversarial example detection. In Uncertainty in Artificial Intelligence (UAI).
A. Talmor and J. Berant. 2019. MultiQA: An empirical investigation of generalization and transfer in reading comprehension. In Association for Computational Linguistics (ACL).
J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. In North American Association for Computational Linguistics (NAACL).
M. Toplak, R. Močnik, M. Polajnar, Z. Bosnic, L. Carlsson, C. Hasselgren, J. Demšar, S. Boyer, B. Zupan, and J. Stálring. 2014. Assessment of machine learning reliability methods for quantifying the applicability domain of QSAR regression models. Journal of Chemical Information and Modeling, 54.
A. Trischler, T. Wang, X. Yuan, J. Harris, A. Sordoni, P. Bachman, and K. Suleman. 2017. NewsQA: A machine comprehension dataset. In Workshop on Representation Learning for NLP.
E. Wallace, S. Feng, N. Kandpal, M. Gardner, and S. Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. In Empirical Methods in Natural Language Processing (EMNLP).
M. Wang, N. A. Smith, and T. Mitamura. 2007. What is the jeopardy model? a quasi-synchronous grammar for QA. In Empirical Methods in Natural Language Processing (EMNLP).
T. Winograd. 1972. Understanding Natural Language. Academic Press.
Y. Yang, W. Yih, and C. Meek. 2015. WikiQA: A challenge dataset for open-domain question answering. In Empirical Methods in Natural Language Processing (EMNLP), pages 2013-2018.
5694
Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Empirical Methods in Natural Language Processing (EMNLP).
D. Yogatama, C. de M. d'Autume, J. Connor, T. Kocisky, M. Chrzanowski, L. Kong, A. Lazaridou, W. Ling, L. Yu, C. Dyer, et al. 2019. Learning and evaluating general linguistic intelligence. arXiv preprint arXiv:1901.11373.
B. Zadrozny and C. Elkan. 2002. Transforming classifier scores into accurate multiclass probability estimates. In International Conference on Knowledge Discovery and Data Mining (KDD), pages 694-699.
A.1 Dataset Sources
The OOD data used in calibrator training and validation was sampled from MRQA training data, and the SQuAD data for the same was sampled from MRQA validation data, to prevent train/test mismatch for the QA model (Fisch et al., 2019). The test data was sampled from a disjoint subset of the MRQA validation data.
A.2 Calibrator Features and Model
We ran experiments including question length and word overlap between the passage and question as calibrator features. However, these features did not improve the validation performance of the calibrator. We hypothesize that they may provide misleading information about a given example, e.g., a long question in SQuAD may provide more opportunities for alignment with the paragraph, making it more likely to be answered correctly, but a long question in HotpotQA may contain a conjunction, which is difficult for the SQuAD-trained model to extrapolate to.
For the calibrator model, we experimented using an MLP and logistic regression. Both were slightly worse than Random Forest.
A.3 Outlier Detection for Selective Prediction
In this section, we study whether outlier detection can be used to perform selective prediction. We train an outlier detector to detect whether or not a given input came from the in-domain dataset (i.e., SQuAD) or is out-of-domain, and use its probability of an example being in-domain for selective prediction. The outlier detection model, training data (a mixture of  $p_{\mathrm{source}}$  and  $q_{\mathrm{known}}$ ), and features are the same as those of the calibrator. We find
Figure 6: When considering only one answer option as correct, MaxProb is well-calibrated in-domain, but is still overconfident out-of-domain.
that this method does poorly, achieving an AUC of 24.23, Coverage at  $80\%$  Accuracy of  $37.91\%$ , and Coverage at  $90\%$  Accuracy of  $14.26\%$ . This shows that, as discussed in Section 2.3 and Section 5.2, this approach is unable to correctly identify the OOD examples that the QA model would get correct.
A.4 Underconfidence of MaxProb on SQuAD
As noted in Section 5.3, MaxProb is underconfident on SQuAD examples due to the additional correct answer options given at test time but not at train time. When the test time evaluation is restricted to allow only one correct answer, we find that MaxProb is well-calibrated on SQuAD examples (Figure 6). The calibration of the calibrator improves as well (Figure 7). However, we do not retain this restriction for the experiments, as it diverges from standard practice on SQuAD, and EM over multiple spans is a better evaluation metric since there are often multiple answer spans that are equally correct.
A.5 Accuracy and Coverage per Domain
Table 1 in Section 5.2 shows the coverage of MaxProb and the calibrator over the mixed dataset  $D_{\mathrm{test}}$  while maintaining  $80\%$  accuracy and  $90\%$  accuracy. In Table 5, we report the fraction of these answered questions that are in-domain or OOD. We also show the accuracy of the QA model on each portion.
Our analysis in Section 5.3 indicated that Max-Prob was overconfident on OOD examples, which we expect would make it answer too many OOD questions and too few in-domain questions. Indeed,
Figure 7: When considering only one answer option as correct, the calibrator is almost perfectly calibrated on both in-domain and out-of-domain examples.
at  $80\%$  accuracy,  $62\%$  of the examples MaxProb answers are in-domain, compared to  $68\%$  for the calibrator. This demonstrates that the calibrator improves over MaxProb by answering more indomain questions, which it can do because it is less overconfident on the OOD questions.
MaxProb Accuracy	MaxProb Coverage	Calibrator Accuracy	Calibrator Coverage
At 80% Accuracy
in-domain	92.45	61.59	89.09	67.57
OOD	58.00	38.41	59.55	32.43
At 90% Accuracy
in-domain	97.42	67.85	94.35	78.72
OOD	71.20	32.15	72.30	21.28
Table 5: Per-domain accuracy and coverage values of MaxProb and the calibrator  $\left( {{p}_{\text{source }}\text{and}{q}_{\text{known }}}\right)$  at  ${80}\%$  and  ${90}\%$  Accuracy on  ${D}_{\text{test }}$  .
5696