SlowGuess/NAACL_2021_OCR huggingface dataset:
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6325-6334. IEEE.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1321-1330. JMLR.org.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107-112, New Orleans, Louisiana. Association for Computational Linguistics.
He He, Sheng Zha, and Haohan Wang. 2019. Unlearn dataset bias in natural language inference by fitting the residual. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 132-142, Hong Kong, China. Association for Computational Linguistics.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. 2020. Pretrained transformers improve out-of-distribution robustness. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2744-2751, Online. Association for Computational Linguistics.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. 2019. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems, pages 125-136.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021-2031, Copenhagen, Denmark. Association for Computational Linguistics.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551.
Divyansh Kaushik and Zachary C. Lipton. 2018. How much reading does reading comprehension require? a critical investigation of popular benchmarks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5010-5015, Brussels, Belgium. Association for Computational Linguistics.
Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4999-5007.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785-794, Copenhagen, Denmark. Association for Computational Linguistics.
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 333-342, Vancouver, Canada. Association for Computational Linguistics.
Shayne Longpre, Yi Lu, Zhucheng Tu, and Chris DuBois. 2019. An exploration of data augmentation and sampling techniques for domain-agnostic question answering. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 220–227, Hong Kong, China. Association for Computational Linguistics.
Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658-4664, Florence, Italy. Association for Computational Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018.
Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180-191.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Amrita Saha, Rahul Aralikatte, Mitesh M. Khapra, and Karthik Sankaranarayanan. 2018. DuoRC: Towards complex language understanding with paraphrased reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1683-1693, Melbourne, Australia. Association for Computational Linguistics.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016. Newsqa: A machine comprehension dataset. arXiv preprint arXiv:1611.09830.
George Tsatsaronis, Michael Schroeder, Georgios Paliouras, Yannis Almirantis, Ion Androutsopoulos, Eric Gaussier, Patrick Gallinari, Thierry Artieres, Michael R Alvers, Matthias Zschunke, et al. 2012. Bioasq: A challenge on large-scale biomedical semantic indexing and question answering. In 2012 AAAI Fall Symposium Series.
Zhiguo Wang, Wael Hamza, and Radu Florian. 2017. Bilateral multi-perspective matching for natural language sentences.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5754-5764.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600.
A.1 Question Answering Models
For reproducibility, we share our hyper-parameter selection in Table 6. We borrow our hyperparameters from Longpre et al. (2019) for training all Question Answering (QA) models. Their parameters are tuned for the same datasets in the MRQA Shared Task. We found these choices to provide stable and strong results across all datasets.
Our BERT and XLNet question answering modules build upon the standard PyTorch (Paszke et al., 2019) implementations from HuggingFace, and are trained on 8 NVIDIA Tesla V100 GPUs. For DRQA, by Chen et al. (2017), we borrowed the implementation and hyper-parameters from hitvoice (https://github.com/hitvoice/DrQA) and train on 1 NVIDIA Tesla V100 GPU.
We employ 6 diverse QA training sets and 12 evaluation sets from the MRQA 2019 workshop (https://github.com/mrqa/MRQA-Shared-Task-2019) (Fisch et al., 2019). These datasets have been normalized into purely extractive format and all questions are answerable. The 6 training datasets are SQuAD (Rajpurkar et al., 2016), NewsQA (Trischler et al., 2016), TriviaQA (Joshi et al., 2017), SearchQA (Dunn et al., 2017), HotpotQA (Yang et al., 2018), and Natural Questions (Kwiatkowski et al., 2019). Six other evaluation datasets are included: BioASQ (Tsatsaronis et al., 2012), DROP (Dua et al., 2019), DuoRC (Saha et al., 2018), RACE (Lai et al., 2017), RelationsExtraction (Levy et al., 2017), and TextbookQA (Kembhavi et al., 2017). Table 7 shows their statistics.
We use the hyperparameters described in Table 6 for training on each dataset. We use all the training data provided for each by MRQA.
A.3 Generating MPPIs
The process for generating MPPIs closely follows the procedure described by Feng et al. (2018). We operate with a beam size of  $k = 3$ , finding that larger beam sizes exhibit diminishing returns, and
$^{7}$ https://github.com/huggingface/ transformers  
 $^{8}$ We used the open source version available at https://github.com/bitvoice/drqa.
Model Parameter	Value
DRQA
Model Size (# params)	31.5M
Avg. Train Time	10h 30m
Learning Rate	0.1
Optimizer	Adamax
Num Epochs	35
Batch size	32
Dropout	0.4
Hidden size	128
BERT-BASE
Model Size (# params)	108.3M
Avg. Train Time	2h 20m
Learning Rate	5e-5
Optimizer	Adam
Num Epochs	2
Batch Size	25
Gradient Accumulation	1
Dropout	0.1
Lower Case	False
Max Query Length	64
Max Sequence Length	512
XLNet-LARGE
Model Size (# params)	364.5M
Avg. Train Time	4h 45m
Learning Rate	2e-5
Optimizer	Adam
Num Epochs	2
Batch Size	6
Gradient Accumulation	3
Dropout	0.1
Lower Case	False
Max Query Length	64
Max Sequence Length	512
Table 6: Hyperparameter selection for each model type.
rarely produce different results. The procedure involves iteratively removing the token which is "least important" to the model. The least important token is defined as the one that when removed provides the smallest decrease in confidence in the originally predicted span. Note that in some cases confidence in the originally predicted span can even increase with the removal of a token. In any case, the least important token is always designated by the lowest confidence in the original prediction. The stop condition is when removing any additional token would change the model's prediction.
Note that we follow previous work in only removing words from the query in extractive question answering. The reason for this is the MPPI can be poorly defined when context tokens are removed. Since the output predictions are over the context tokens for extractive question answering, its possible to warp the answer space, or remove the answer
Dataset	Question (Q)	Context (C)	Avg. Q Len	Avg. C Len	Train	Dev
SQuAD	Crowdsourced	Wikipedia	11	137	86,588	10,507
NewsQA	Crowdsourced	News articles	8	599	74,160	4,212
TriviaQA	Trivia	Web snippets	16	784	61,688	7,785
SearchQA	Jeopardy	Web snippets	17	749	117,384	16,980
HotpotQA	Crowdsourced	Wikipedia	22	232	72,928	5,904
Natural Questions	Search logs	Wikipedia	9	153	104,071	12,836
BioASQ	Domain experts	Science articles	11	248	-	1,504
DROP	Crowdsourced	Wikipedia	11	243	-	1,503
DuoRC	Crowdsourced	Movie plots	9	681	-	1,501
RACE	Domain experts	Examinations	12	349	-	674
RelationExtraction	Synthetic	Wikipedia	9	30	-	2,948
TextbookQA	Domain experts	Textbook	11	657	-	1,503
Table 7: Statistics about datasets used: The first block presents six domains used for training, the second block presents six additional domains used for model evaluation and generating MPPIs.
altogether. Additionally, if we do not permit any alterations to the original prediction tokens, then there exists a trivial solution: remove all tokens except for the predicted answer. In this case an extractive question answering model is forced to predict that answer, with no alternative options. Consequently, MPPIs that allow modifications to the context, or output space, can be poorly defined. Since in question answering the query is an essential input to provide confident answers, we believe this is the most reasonable setup for the task.
$$
p _ {i j} = \max  (\operatorname {s o f t m a x} \left(S _ {i} + E _ {j}\right)) \tag {2}
$$
For completeness, we describe our method of computing span confidence for question answering, given that there are many variations. Let  $S \in \mathbb{R}^N$  be the vector of start logits and  $E \in \mathbb{R}^N$  be the vector of end logits, both of sequence length  $N$ . For every combination of  $i$ ,  $j \in [0, N]$  where  $j \geq i \leq \min(j + C, N)$ , and  $C = 30$  is the maximum answer span length, we compute the confidence for that span of answer text as the sum of their respective logits  $S_i + E_j$ . The final confidence probability  $p_{ij}$  for a given span is as shown in Equation 2.
The model, on the other hand, can still make the same prediction as it did on the full input, and with a similar degree of confidence.
A.4 Regularizing MPPIs
There are a couple differences between the MPPI entropy-regularization strategy employed in this work and in Feng et al. (2018). While Feng et al. (2018) fine-tune an a model already trained for the question answering task, we regularize MPPIs in the initial fine-tuning stage (starting from BERT and XLNet's pre-trained weights). Secondly, they
alternate updates between two optimizers, one batch of maximum likelihood, two for MPPI entropy maximization, whereas we use the same optimizer and shuffle together equal numbers of MPPI and regular inputs. We find our method (without rigorous comparison) to be slightly more effective on BERT at mitigating the MPPI phenomenon (measured by subsequent MPPI length). We suspect, if there is an advantage, it is due to the regularization beginning with the start of fine-tuning, rather than on a subsequent stage of fine-tuning.
$$
\mathcal {L} _ {\mathcal {M P P I}} = C - \lambda \sum_ {\tilde {x} \in \tilde {\mathcal {X}}} \mathbb {H} (f (y | \tilde {x})) \tag {3}
$$
$$
\mathcal {L} = \mathcal {L} _ {\mathcal {Q A}} + \mathcal {L} _ {\mathcal {M P P I}} \tag {4}
$$
For completeness, we provide our entropy regularization loss term in Equation 3. Let  $\tilde{\mathcal{X}}$  denote the set of inputs that have been reduced to their MPPI,  $\mathbb{H}(\cdot)$  denote the entropy and  $f(y|x)$  denote the predicted confidence for  $y$  given  $x$ . We then represent the loss term for MPPIs as  $\mathcal{L}_{\mathcal{MPPI}}$ , where the constant  $C = 10$  is chosen such that maximizing the entropy will minimize the loss. We use  $\lambda = 0.1$  as the most effective choice in our limited set of trials. The full loss term, for all inter-mixed regular question answering, and MPPI examples is the sum of standard QA loss  $\mathcal{L}_{\mathcal{QA}}$ , and the MPPI loss term  $\mathcal{L}_{\mathcal{MPPI}}$ , as shown in Equation 4.
In Figure 3 we display the full comparison between the performances of the MPPI regularized models and the regular models on 13 QA datasets, including Adversarial SQuAD (Jia and Liang, 2017).
Figure 3: The generalization and robustness of BERT models evaluated on 12 datasets, as well as Adversarial SQuAD. The  $(*)$  indicates MPPI-regularization during training.
B How do MPPI Lengths Compare?
In the main paper we describe the differences in length distributions between original and MPPI queries. To provide more detail into the length distributions we plot histograms of the query word lengths, for the original queries, MPPI queries, and MPPI queries after the MPPI regularization procedure. These lengths are plotted below for SQuAD (Figure 4), HotpotQA (Figure 5), NewsQA (Figure 6), Natural Questions (Figure 7), TriviaQA (Figure 8), and SearchQA (Figure 9).
Figure 6: NEWSQA question length generated by different MPPI reduction methods
Figure 4: SQUAD question length generated by different MPPI reduction methods
Figure 7: NATRUALQUESTION question length generated by different MPPI reduction methods
Figure 5: HOTPOTQA question length generated by different MPPI reduction methods
Figure 8: TRIVIAQA question length generated by different MPPI reduction methods
Figure 9: SEARCHQA question length generated by different MPPI reduction methods
The query length distributions show that MPPIs are significantly shorter than original queries, with the MPPIs of regularized models somewhere in between. These length distributions may be sufficient to explain why humans find the non-regularized MPPIs completely uninterpretable, and the regularized MPPIs somewhat more interpretable.
C Are MPPIs Invariant to Random Seed?
One of the preliminary questions in our investigation was whether changing the random training seed significantly altered the MPPI produced by a model. If it were the case that this had a drastic effect, we might suspect MPPIs were somewhat random, or the product of meangingless overconfidence on out-of-distribution inputs. Table 8 illustrates the random seed experiment in full. Training 10 SQuAD models, each with different random seeds, we generate MPPIs on the  $2k$  SQuAD evaluation set, and compare 5 pairs. We measure the mean Generalized Jaccard Similarity of MPPIs produced by 2 models trained with different seeds.
We see the similarity between MPPIs trained with different seeds far exceed those of Rand-A, and Rand-B, which are akin to a "random" simulation of MPPIs. As with our previous random baselines these are generated by randomly sampling tokens from the original query, preserving word order, and ensuring that the length distribution matches that of the actual MPPIs to which they are being compared.
D Are MPPIs Invariant to Training Domain?
We discussed the invariance of MPPIs to training domain at length in the paper for BERT. For completeness, we provide the raw results for BERT in Table 9 and for XLNet in Table 10. These results show that MPPIs are far more similar to one an
Seed A	Seed B	JS / EM
0	1	55.0 / 31.7
2	3	56.8 / 33.2
4	5	58.3 / 34.7
6	7	57.4 / 33.2
8	9	58.1 / 35.2
Overall	57.1 / 33.6
Rand-A	Rand-B	13.8 / 0.9
Table 8: Observing the Jaccard Similarity and Exact Match between MPPIs on the SQuAD  $2k$  evaluation set, we see significant token overlap despite seed differences. In contrast, the randomly generated sequences, preserving the length distribution of MPPIs, produces far less similar token sequences.
other, even when training domain is different. The random baseline, in parenthesis, once again shows the Jaccard Similarity we would expect if MPPIs were purely random.
E Do QA Models Generalize to different MPPI Domains?
Expanding on the MPPI generalization analysis in Section 3.2, we provide the raw results. The cross-domain generalization of BERT and XLNet models on MPPIs sourced from different training domains is available in Table 11 and Table 12 respectively. Figure 10 visualizes how well XLNet generalizes to different MPPI domains. The results mirror those of BERT shown in the main paper.
Figure 10: XLNET performance with different training sets (y-axis), and evaluation sets (x-axis). Bars measure the F1 score on the  $2k$  evaluation set, colored by input type.
Train Dataset
Reduction Dataset
SQUAD	HOTPOTQA	NEWSQA	NATRUALQ	TRIVIAQA	SEARCHQA
SQUAD	- (-)	31.4 (8.8)	41.0 (21.6)	29.2 (12.5)	24.9 (10.9)	11.9 (9.6)
HOTPOTQA	39.7 (12.8)	- (-)	39.6 (18.8)	33.8 (13.5)	25.8 (10.7)	16.6 (12.6)
NEWSQA	41.1 (13.0)	31.6 (7.2)	- (-)	35.2 (12.5)	25.8 (10.8)	13.3 (9.4)
NATRUALQ	37.5 (12.7)	28.7 (7.1)	40.2 (17.9)	- (-)	25 (10.7)	15.0 (11.0)
TRIVIAQA	33.3 (13.0)	27.7 (8.0)	34.8 (18.1)	29.2 (15.4)	- (-)	23.1 (15.2)
SEARCHQA	23.4 (12.3)	16.8 (7.8)	24.7 (17.2)	24.4 (14.6)	23.4 (11.9)	- (-)
Average	35.0 (12.8)	27.2 (7.5)	36.1 (18.7)	30.4 (13.7)	25.0 (11.0)	16.0 (11.6)
Table 9: The Jaccard Similarity (%) between BERT generated MPPIs, across domains. The Random baseline MPPIs are in parentheses.
Train Dataset
Reduction Dataset
SQUAD	HOTPOTQA	NEWSQA	NATRUALQ	TRIVIAQA	SEARCHQA
SQUAD	- (-)	25.8 (9.0)	37.7 (19.7)	30.9 (11.1)	18.1 (10.5)	22.7 (26.3)
HOTPOTQA	28.4 (15.3)	- (-)	31.2 (17.6)	31.5 (12.4)	17.8 (12.0)	27.1 (25.5)
NEWSQA	31.8 (13.1)	25.3 (8.2)	- (-)	36.6 (11.9)	20.6 (9.0)	13.3 (11.7)
NATRUALQ	29.9 (12.9)	24.2 (8.4)	40.2 (16.8)	- (-)	22.3 (11.0)	19.2 (16.3)
TRIVIAQA	25.6 (14.8)	19.0 (8.0)	29.8 (17.4)	29.2 (13.7)	- (-)	31.2 (20.6)
SEARCHQA	21.6 (13.8)	15.5 (7.7)	25.2 (15.1)	24.6 (14.1)	28.3 (13.4)	- (-)
Average	27.5 (14.0)	22.0 (8.3)	32.8 (17.3)	30.6 (12.6)	21.4 (11.2)	22.7 (20.1)
Table 10: The Jaccard Similarity (%) between XLNET generated MPPIs, across domains. The Random baseline MPPIs are in parentheses.
Train Dataset	Query Type	SQuAD	HotpotQA	NewsQA	NaturalQ	TriviaQA	SearchQA
SQuAD	Original	87.74	56.31	48.81	21.53	56.74	52.62
SQuAD	MPPI	87.74	28.84	31.68	13.52	43.02	30.93
SQuAD	Random MPPI	26.42	16.19	19.69	9.55	13.01	17.46
TriviaQA	Original	54.64	71.04	42.27	47.53	51.85	34.45
TriviaQA	MPPI	34.25	71.04	24.67	32.28	33.85	18.91
TriviaQA	Random MPPI	15.21	32.23	15.92	25.49	14.12	12.67
NaturalQ	Original	75.28	58.18	77.78	37.84	54.08	51.16
NaturalQ	MPPI	55.15	40.43	77.78	24.32	44.52	34.25
NaturalQ	Random MPPI	23.28	23.94	38.43	18.39	16.32	17.34
SearchQA	Original	40.25	59.44	32.58	78.11	35.93	20.61
SearchQA	MPPI	24.84	41.67	21.24	78.11	24.96	15.9
SearchQA	Random MPPI	11.92	24.69	14.73	45.7	12.34	9.7
HotpotQA	Original	71.52	53.4	52.51	38.9	75.09	47.03
HotpotQA	MPPI	49.52	34.56	37.23	20.66	75.09	30.38
HotpotQA	Random MPPI	21.09	19.28	24.92	15.2	17.76	17.88
NewsQA	Original	78.16	60.91	59.94	33.79	56.53	68.19
NewsQA	MPPI	61.32	39.17	42.48	19.44	48.58	68.19
NewsQA	Random MPPI	22.78	20.4	22.74	15.05	14.49	24.83
Table 11: Cross-Domain Generalization of BERT Base models on different types of inputs. Values correspond to F1 scores on the question answering  $2k$  evaluation set specified by the column.
Train Dataset	Query Type	SQuAD	HotpotQA	NewsQA	NaturalQ	TriviaQA	SearchQA
SQuAD	Original	93.92	64.97	66.62	15.13	70.51	65.06
SQuAD	MPPI	93.95	17.75	39.95	6.37	41.36	35.27
SQuAD	Random MPPI	31.0	12.86	21.12	7.3	16.19	17.85
TriviaQA	Original	67.55	78.15	51.7	67.69	57.85	44.97
TriviaQA	MPPI	31.77	78.18	27.8	43.63	32.34	22.24
TriviaQA	Random MPPI	17.01	34.03	16.03	33.18	14.54	13.27
NaturalQ	Original	85.61	67.84	82.06	42.92	67.43	60.82
NaturalQ	MPPI	63.02	35.73	82.06	20.44	46.19	42.95
NaturalQ	Random MPPI	31.74	23.31	36.07	18.05	19.98	22.45
SearchQA	Original	55.25	74.37	45.43	84.08	45.33	32.82
SearchQA	MPPI	25.6	47.5	26.33	84.08	29.41	18.57
SearchQA	Random MPPI	15.92	33.29	16.81	53.58	15.57	12.13
HotpotQA	Original	82.85	61.03	61.93	23.98	80.28	54.19
HotpotQA	MPPI	51.11	14.57	40.95	9.08	80.3	23.66
HotpotQA	Random MPPI	27.83	14.0	23.89	14.03	21.33	15.25
NewsQA	Original	88.56	69.32	67.61	30.74	69.14	73.17
NewsQA	MPPI	65.64	29.66	48.56	11.0	45.15	73.12
NewsQA	Random MPPI	30.47	16.29	20.27	6.64	15.71	25.35
Table 12: Cross-Domain Generalization of XLNET Large models on different types of inputs. Values correspond to F1 scores on the question answering  ${2k}$  evaluation set specified by the column.