SlowGuess/NAACL_2021_OCR huggingface dataset:
Shayne Longpre*
slongpre@apple.com
Yi Lu*
ylu7@apple.com
Christopher DuBois
cdubois@apple.com
Recent work (Feng et al., 2018) establishes the presence of short, uninterpretable input fragments that yield high confidence and accuracy in neural models. We refer to these as Minimal Prediction Preserving Inputs (MPPIs). In the context of question answering, we investigate competing hypotheses for the existence of MPPIs, including poor posterior calibration of neural models, lack of pretraining, and "dataset bias" (where a model learns to attend to spurious, non-generalizable cues in the training data). We discover a perplexing invariance of MPPIs to random training seed, model architecture, pretraining, and training domain. MPPIs demonstrate remarkable transferability across domains — achieving significantly higher performance than comparably short queries. Additionally, penalizing over-confidence on MPPIs fails to improve either generalization or adversarial robustness. These results suggest the interpretability of MPPIs is insufficient to characterize generalization capacity of these models. We hope this focused investigation encourages more systematic analysis of model behavior outside of the human interpretable distribution of examples.
Feng et al. (2018) establish the presence of shortened input sequences that yield high confidence and accuracy for non-pretrained neural models. These Minimal Prediction Preserving Inputs (MPPIs) are constructed by iteratively removing the least important word from the query to obtain the shortest sequence for which the model's prediction remains unchanged (example shown in Figure 1).<sup>1</sup> Humans are unable to make either confident or accurate predictions on these inputs. Follow up work treats
... The site currently houses three sinemas, including the restored Classic the United Kingdom's last surviving news cinema still in full-time operation—alongside two new screens ...
What's the name of the United Kingdom 's sole remaining news cinema ?
$0.57\rightarrow 0.51$
Figure 1: A SQUAD dev set example. Given the original Context, the model makes the same correct prediction ("Classic") on the Reduced question (MPPI) as the Original, with almost the same score. For humans, the reduced question, "news", is nonsensical.
strong model performance on such partial-inputs as equivalent with models improperly learning the task (Feng et al., 2019; Kaushik and Lipton, 2018; He et al., 2019). Accordingly, we evaluate this proposition in question answering (QA), investigating the properties of MPPIs and how their existence relates to "dataset bias", out-of-domain generalization, and adversarial robustness.
First we examine the hypothesis that MPPIs are a symptom of poor neural calibration. Feng et al. (2018) propose we can "attribute [these neural] pathologies primarily to the lack of accurate uncertainty estimates in neural models." As neural models tend to overfit the log-likelihood objective by predicting low-entropy distributions (Guo et al., 2017) this can manifest in over-confidence on gibberish examples outside of the training distribution (Goodfellow et al., 2014). We test this hypothesis using pretrained models, shown to have better posterior calibration and out-of-distribution robustness (Hendrycks et al., 2020; Desai and Durrett, 2020). Contrary to expectations, we find large-scale pretraining does not produce more human interpretable MPPIs.
Second we examine the hypothesis that MPPIs are the symptom of "dataset bias" — where a
<sup>1</sup>For question answering we construct MPPIs by only removing words from the query. Modifying the context paragraph is poorly defined in MPPI generation as it perturbs the output space, rendering an answer impossible or trivial.
Association for Computational Linguistics: Human Language Technologies, pages 1288-1300
flawed annotation procedure results in hidden linguistic cues or "annotation artifacts" (Gururangan et al., 2018; Niven and Kao, 2019). Models trained on such data distribution can rely on simple heuristics rather than learning the task. As such, input fragments or "partial inputs" are often sufficient for a model to achieve strong performance on flawed datasets. This explanation has been considered for both Natural Language Inference tasks (the "hypothesis-only" input for Poliak et al. (2018); Gururangan et al. (2018)) and for Visual Question Answering (the "question-only" model for Goyal et al. (2017)). We expect models which rely on these spurious cues would fail to generalize well to other "domains" (datasets with different collection and annotation procedures). We discover even models trained in different domains perform nearly as well on MPPIs as on full inputs, contradicting this hypothesis. Further, we test their transferability across a number of other factors, including random training seed, model size, and pretraining strategy, and confirm their invariance to each of these.
Third we examine the hypothesis that MPPIs inhibit generalization. This intuition is based on MPPI's poor human interpretability, which could suggest models should not attend to these signals. To test this hypothesis, we regularize this phenomenon directly to promote more human understandable MPPIs, and measure the impact on outdomain generalization and adversarial robustness. Interestingly, out-domain generalization and robustness on Adversarial SQUAD (Jia and Liang, 2017) vary significantly by domain, with both declining slightly on average due to regularization.
In conjunction, these results suggest MPPIs may represent an unique phenomenon from what previous work has observed and analyzed. The performance of these inputs is not well explained by domain-specific biases, or posterior overconfidence on out-of-distribution inputs. Instead, this behavior may correspond to relevant signals as the impact of their partial mitigation suggests. We hope these results encourage researchers to not assume MPPIs, or other uninterpretable model behaviour, are dataset artifacts that require mitigation a priori. Before presenting mitigation solutions, we propose they follow a more systematic analysis proposed by our actionable framework by (a) rigorously testing the alleged causes of the observed behaviour, (b) confirming the bias does not generalize/transfer, and (c) ensuring the solution provides
Dataset	ORIGINAL	BERT-B	XLNET-L
SQUAD (Rajpurkar et al., 2016)	11.54	2.32	2.65
HOTPOTQA (Yang et al., 2018)	18.96	2.07	2.55
NEWSQA (Trischler et al., 2016)	7.59	2.08	1.80
NATURALQ (Kwiatkowski et al., 2019)	9.17	1.22	1.26
TRIVIAQA (Joshi et al., 2017)	15.68	2.33	1.80
SEARCHQA (Dunn et al., 2017)	17.43	1.81	1.05
Table 1: Number of MPPI query tokens, for different datasets and models.
consistent improvements across domains within a task.
2 Experimental Methodology
All models trained, including DRQA (Chen et al., 2017), BERT (Devlin et al., 2019), and XLNET (Yang et al., 2019), employ setup and parameter choices from Longpre et al. (2019). We generate MPPIs by iteratively removing the least important word from the question, while keeping the original prediction unchanged. The least important word is given as that for which the model's confidence in its prediction remains highest in its absence.
To examine how MPPIs transfer across Question Answering domains we employ 6 diverse QA training sets and 12 evaluation sets. The datasets were selected for annotation variety, differing on: question type, document source, annotation instructions, whether the question was collected independently of the passage, and skills required to answer the question. This set represents a realistic spectrum of domains for evaluating generalization.
We set aside  $2k$  examples from each domain's validation sets in order to generate MPPIs for model evaluation. For each experiment we also generate a set of randomly shortened queries to compare against the MPPIs — we refer to this as the "Random MPPI" baseline. For each of the original examples, we generate this baseline by randomly removing words until the length matches that of the corresponding MPPI.
3.1 Invariance of MPPIs
Feng et al. (2018) establish the "human-insufficiency" property of MPPIs for non-pretrained, LSTM and attention-based models, in
2For DRQA, we borrowed the hyper-parameters from hitvoice (https://github.com/hitvoice/DrQA))
3Details of model training and examples of MPPI generation are described in Appendix A.
4Refer to Appendix A.3 for details, or the MRQA 2019 workshop: https://mrqa.github.io/shared. Fisch et al. (2019) normalized these datasets into purely answerable, extractive format.
DRQA	BERT-B	XLNET-L
BERT-B	32.1 / 9.9	- / -	29.8 / 9.9
XLNET-L	26.0 / 7.2	29.8 / 9.9	- / -
RANDOM	13.0 / 1.8	12.6 / 0.9	14.2 / 1.3
cluding DRQA, and BIMPM (Wang et al., 2017). We extend this investigation for modern, pretrained Transformers, and assess the "invariance" of MPPIs: measuring whether they are random, or are affected by model architecture, pretraining strategy, or training dataset (domain).
In subsequent experiments we compare sets of MPPIs using the mean Exact Match or Generalized Jaccard Similarity (GJS), a variant of Jaccard Similarity, which accounts for the possibility of repeated tokens in either of the sequences being compared. Generalized Jaccard Similarity is defined between two token sequences  $X$  and  $Y$  in Equation 1. Here,  $n$  is the index of every element that appears in  $X \cup Y$ .
$$
G J S (X, Y) = \frac {\sum_ {i = 1} ^ {n} \operatorname* {m i n} \left(X _ {i} , Y _ {i}\right)}{\sum_ {i = 1} ^ {n} \operatorname* {m a x} \left(X _ {i} , Y _ {i}\right)} \tag {1}
$$
We will refer to this as "Jaccard Similarity" for simplicity.
3.1.1 Random Seed
First, we investigate whether MPPIs are "random", or influenced by weight initialization and training data order. Measuring the Jaccard Similarity between MPPI sequences produced by models with different training seeds we find  $\mathrm{JS}_{\mathrm{MPPI}} = 57.1\% \pm 1.2$  as compared to  $\mathrm{JS}_R = 13.8\% \pm 0.8$  on the Random MPPI baseline. This suggests MPPIs are not simply the side-effect of randomness in the training procedure.
3.1.2 Pretraining and Architecture
One hypothesis is that traditional LSTM-based models, such as DRQA, do not have sufficient pretraining or "world knowledge" to rely on the entire sequence, and overfit to subsets of the input. If this were the primary source of MPPIs, we might expect models that are better calibrated and more robust to out-of-distribution examples to have longer and more interpretable MPPIs. Accordingly, we test this hypothesis with large pretrained transformers, which recent work demonstrates have
Table 2: The mean similarity, measured in Jaccard Similarity / Exact Match (\%), between the MPPIs from different model types and the random baseline.
Train Dataset	Reduction Dataset
SQUAD	HOTPOTQA	NEWSQA	NATRUALQ
SQUAD	- (-)	31.4 (8.8)	41.0 (21.6)	29.2 (12.5)
HOTPOTQA	39.7 (12.8)	- (-)	39.6 (18.8)	33.8 (13.5)
NEWSQA	41.1 (13.0)	31.6 (7.2)	- (-)	35.2 (12.5)
NATRUALQ	37.5 (12.7)	28.7 (7.1)	40.2 (17.9)	- (-)
Average	39.4 (12.8)	30.6 (7.7)	40.3 (19.4)	32.7 (12.8)
Table 3: The Jaccard Similarity  $(\%)$  between BERT generated MPPIs, across domains. In parentheses are the Jaccard Similarity scores between the Random MPPI baseline and Train Dataset MPPIs.
better posterior calibration and robustness to out-of-distribution inputs.
Specifically, Desai and Durrett (2020) examine 3 separate NLP tasks, using "challenging out-of-domain settings, where models face more examples they should be uncertain about", and find that "when used out-of-the-box, pretrained models are calibrated in-domain, and compared to baselines, their calibration error out-of-domain can be as much as  $3.5 \times$  lower". Similarly Hendrycks et al. (2020) systematically show "Pretrained transformers are also more effective at detecting anomalous or [out-of-distribution] examples". These findings suggest pretrained transformers should produce more interpretable MPPIs than non-pretrained models.
However, in Table 1 we show MPPIs remain incomprehensively short for all 6 domains and even for pretrained transformer models (DRQA produces MPPIs on SQUAD of mean length 2.04). In Table 2 we show MPPIs produced by different model architectures and pretraining strategies are similar, significantly exceeding the Jaccard Similarity of the Random MPPI baseline  $(\mathrm{JS}_{\mathrm{R}} = 13.8\%)$ . This would not be problematic if pretrained models produced lower confidence for MPPIs than the original examples (demonstrating some form of calibration). However, we find the opposite is true. Taking SQuAD for instance we see in  $85\%$  of cases the BERT model is more confident on the MPPI than the original example.
Lastly, we verify with manual grading tasks that the MPPIs for BERT and XLNet are no more interpretable to humans than DrQA's MPPIs, as shown in Table 5. This suggests that short, uninterpretable MPPIs are ubiquitous in modern neural question answering models and unmitigated by large scale pretraining, or improved out-of-distribution robustness.
3.1.3 Cross-Domain Similarity
Next, we investigate the extent to which MPPIs are domain-specific. We do this by measuring their similarity when produced by models trained in different domains. If MPPIs are the product of bias in the training data, such as annotation artifacts, we would expect them to be relatively domain specific, as different datasets carry different biases. In Table 3 a model trained from each domain (Train Dataset) generates MPPIs for each other domain (Reduction Dataset). For each Reduction Dataset, we measure the mean Jaccard Similarity between MPPIs produced by the Train Dataset model and MPPIs produced by the Reduction Dataset (domain) model. In parentheses we show the mean Jaccard Similarity between the Random MPPIs and the Train Dataset MPPIs. In all cases, MPPIs demonstrate higher similarity than the random baseline, indicating that they are not domain specific.
3.2 Cross-Domain Transferability of MPPIs
Even when models generate different MPPIs, they may still transfer to the other domain. We would like to measure MPPI transferability, independent of their similarity between models. If QA models perform well on MPPIs generated from a range of domains then this would suggest they are not a product of bias in the training data. Instead, they may retain information important to question answering, rather than annotation artifacts. To better measure the extent of MPPI transferability, we (a) train one model on SQuAD (Train Dataset), and another on NewsQA (Reduction Dataset), (b) use the NewsQA-model to generate  $2k$  MPPIs on the NewsQA evaluation set, and (c) measure the F1 performance of the SQuAD-model evaluated on both the original NewsQA evaluation set and the MPPI queries as generated in part (b).
Figure 2 shows performance on out-domain MPPIs are  $46.6\%$  closer to original performance than on Random MPPIs. This evidence suggests MPPIs are highly transferable across domains. Consequently, MPPIs may relate to generalization, despite their poor human interpretability.
3.3 Human-Sufficient MPPIs do not Improve Generalization
Even though MPPIs are highly transferable between domains, their presence may be associated with poor generalization. To evaluate this possibility, we examine whether the penalization of MPPIs improves generalization, or adversarial robustness. While penalizing over-confidence on MPPIs has
Train Dataset	F1 Score (%)
△ID	OD	△OD	AR	△AR
SQUAD	-0.8	52.9	-1.5 ± 2.3	72.1	+3.1
HOTPOTQA	+0.6	48.5	-0.6 ± 1.2	45.5	+1.0
NEWSQA	-0.9	53.0	-0.9 ± 0.6	62.9	-1.8
NATURALQ	+0.9	51.6	-2.9 ± 3.5	54.9	-0.9
TRIVIAQA	-0.6	42.3	-4.1 ± 2.8	38.9	-1.1
SEARCHQA	-0.5	38.0	-5.9 ± 2.9	32.3	-4.0
OVERALL AVG	-0.2	47.7	-2.7 ± 1.1	51.1	-0.6
Table 4: The impact of MPPI regularization on indomain (ID) performance, macro-average out-domain (OD) generalization over 12 evaluation datasets, and adversarial robustness (AR) on Adversarial SQUAD.  $\triangle X = F1$  of MPPI regularized model minus F1 of regular model on target X (any of ID, OD, or AR).
been shown to maintain equivalent in-domain performance, and yield subsequently longer and more human interpretable MPPI queries (Feng et al., 2018), its impact on generalization or robustness has not yet been examined.
Figure 2: Cross-Domain Transferability: BERT question answering performance (F1) with different training sets (y-axis), and  $2k$  evaluation sets (x-axis). Bars are colored by input type. On average, MPPI queries close the gap between models' performance on Random MPPIs and original queries by  $46.6\%$ .
We employ a simplified version of the MPPI penalization used by Feng et al. (2018), training a model with equal quantities of regular and MPPI examples — maintaining normal QA loss terms for the regular examples, and applying an entropy penalty to MPPI examples. When penalizing overconfidence on MPPIs, we confirm the new MPPI length is significantly longer (Appendix sections B), and more human interpretable (Table 5).
In Table 4 we show the difference in F1 scores  $(\triangle)$  between the regularized and original models. Results demonstrate that in-domain F1 (ID), macro-average out-domain F1 over 12 datasets (OD), and
See Appendix section A.4 for details.
HUMAN F1 (EM)
ORIGINAL QUERY	91.2 (82.3)
DRQA† MPPI	- (31.7)
BERT-B MPPI	41.6 (32.0)
XLNET-L MPPI	37.6 (26.0)
BERT-B* MPPI	60.7 (43.5)
RANDOM MPPI	26.5 (17.0)
Table 5: The mean human performance (in F1 and Exact Match over 100 examples) on different variants of MPPIs for SQUAD.
† Human performance cited from Feng et al. (2018)
* Indicates a model trained with MPPI regularization.
adversarial robustness F1 on Adversarial SQUAD (AR) all decline slightly on average with MPPI regularization — by  $0.2\%$ ,  $2.7\%$ , and  $0.6\%$  respectively. These results suggest a model's ability to make predictions on MPPIs is not strongly correlated with either generalization or robustness across 13 total QA datasets. However, the relative stability of in-domain performance as compared to outdomain performance suggests mitigating MPPIs is more harmful when crossing domain boundaries.
Certain train datasets exhibit greater sensitivity to MPPI regularization than others. For instance SearchQA is drastically affected in all measures, HotpotQA hardly at all, and SQuAD actually improves by  $3.1\%$  in adversarial robustness. Additionally, Table 4 shows the  $95\%$  confidence intervals for out-domain generalization are often as large as the mean change in performance. Empirically, this demonstrates the effect of MPPI regularization is not consistent, having both positive and negative impacts on performance, depending on which of the 12 out-domain datasets is in question.[6]
In SQUAD, the most common MPPI is the empty string (40%). Among non-empty strings, the most common MPPI tokens are: "what", "?", "who", "how", "when". Despite the pattern of interrogative words, these tokens are already among the most frequent in SQUAD questions, so it's challenging to measure the unique information they convey.
A more direct approach to understand the informative signal of MPPIs is to measure their "human insufficiency" property directly. We conduct a grading task, comparing human ability to answer real, MPPI, and random MPPI queries. Table 5 shows that humans could only correctly
answer BERT and XLNet MPPIs slightly more often than random MPPIs (32% and 26% exact match compared to 17%), but could answer 43.5% of MPPIs produced by MPPI-regularized BERT. Although this confirms MPPI-regularization partially resolves over-confident behaviour for these human non-interpretable inputs, we've observed the resulting model fares slightly worse in domain generalization and robustness.
We find no evidence that MPPIs are explained by poorly calibrated neural models, lack of pretraining knowledge, or dataset-specific bias. Alternatively they may relate, at least in part, to useful and transferable signals. While practitioners, especially in model debiasing tasks, have focused on human understandable and generalizable features, this work would encourage them to also consider the presence of generalizable features which are not human interpretable. This observation closely relates to prior work in computer vision suggesting human uninterpretable, adversarial examples can be the result of "features", not "bugs", in which Ilyas et al. (2019) observe "a misalignment between the (human-specified) notion of robustness and the inherent geometry of the data." We hope this work provides a framework to rigorously evaluate the impact of bias mitigation methods on robustness and generalization, and encourages ML practitioners to examine assumptions regarding unexpected model behaviour on out-of-distribution inputs.
We empirically verify the surprising invariance of MPPIs to random seed, model architecture, and pretraining, as well as their wide transferability across domains. These results suggest that MPPIs may not be best explained by poorly calibrated neural estimates of confidence or dataset-specific bias. Examining their relationship to generalization and adversarial robustness, we highlight the ability to maintain in-domain performance but significantly alter out-domain performance and robustness. We hope our results encourage a more systematic analysis of hypotheses regarding model behavior outside the human interpretable distribution of examples.
We would like to acknowledge Eric Wallace, Shi Feng, Jordan Boyd-Graber, Christopher Clark, Drew Frank, Kanit Wongsuphasawat, Ni Lao, and Charlie Maalouf for their guiding insights and helpful discussion.
See Figure 9 in Appendix A.4 for details.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870-1879, Vancouver, Canada. Association for Computational Linguistics.
Shrey Desai and Greg Durrett. 2020. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 295-302, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368-2378, Minneapolis, Minnesota. Association for Computational Linguistics.
Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179.
Shi Feng, Eric Wallace, and Jordan Boyd-Graber. 2019. Misleading failures of partial-input baselines. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5533-5538, Florence, Italy. Association for Computational Linguistics.
Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018. Pathologies of neural models make interpretations difficult. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3719-3728.
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 1-13, Hong Kong, China. Association for Computational Linguistics.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples.