SlowGuess/NAACL_2021_OCR huggingface dataset:
Krithara, Sergios Petridis, Dimitris Polychronopoulos, Yannis Almirantis, John Pavlopoulos, Nicolas Baskiotis, Patrick Gallinari, Thierry Artieres, Axel Ngonga, Norman Heino, Eric Gaussier, Liliana Barrio-Alvers, Michael Schroeder, Ion Androutsopoulos, and Georgios Paliouras. 2015. An overview of the bioasq large-scale biomedical semantic indexing and question answering competition. BMC Bioinformatics, 16.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. In Empirical Methods in Natural Language Processing.
Chao Wang and Hui Jiang. 2018. Explicit utilization of general knowledge in machine reading comprehension. arXiv preprint arXiv:1809.03449.
Yicheng Wang and Mohit Bansal. 2018. Robust machine comprehension models via adversarial training. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 575-581.
Johannes Welbl, Pasquale Minervini, Max Bartolo, Pontus Stenetorp, and Sebastian Riedel. 2020. Undersensitivity in neural reading comprehension. arXiv preprint arXiv:2003.04808.
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. 2020. Break it down: A question understanding benchmark. Transactions of the Association for Computational Linguistics, 8:183-198.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. EMNLP.
Ziqing Yang, Yiming Cui, Wanxiang Che, Ting Liu, Shijin Wang, and Guoping Hu. 2019. Improving machine reading comprehension via adversarial training. arXiv preprint arXiv:1911.03614.
Mantong Zhou, Minlie Huang, and Xiaoyan Zhu. 2019. Robust reading comprehension with linguistic constraints via posterior regularization. arXiv preprint arXiv:1911.06948.
A Adversarial Datasets
Added sentences Jia and Liang (2017) propose to append an adversarial distracting sentence to the normal SQuAD development set to test the robustness of a QA model. In this paper, we use the two main test sets they introduced: addSent and addOneSent. Both of the two sets augment the normal test set with adversarial samples annotated by Turkers that are designed to look similar to question sentences. In this work, we mainly focus on the adversarial examples.
Universal Triggers Wallace et al. (2019) use a gradient based method to find a short trigger sequence. When they insert the short sequence to the original text, it will trigger the target prediction in the sequence independent of the rest of the passage content or the exact nature of the question. For QA, they generate different triggers for different types of questions including "who", "when", "where" and "why".
Datasets from MRQA For Natural Questions (Kwiatkowski et al., 2019), NewsQA (Trischler et al., 2017), BioASQ (Tsatsaronis et al., 2015) and TextbookQA (Kembhavi et al., 2017), we use the pre-processed datasets from MRQA (Fisch et al., 2019). They differ from the original datasets in that only the paragraph containing the answer is picked as the context and the maximum length of the context is cut to 800 tokens.
B Results on SQuAD addOneSent
The results of our model compared to BERT QA on SQuAD addOneSent is shown in Table 4. Here we see the results on addOneSent and addSent generally have the same trend. The global train+inf helps more on the more difficult addSent.
SQuAD normal	SQuAD addSent	SQuAD addOneSent
ans in wh	F1	ans in wh	F1	ans in wh	F1
Sub-part Alignment	84.7	84.5	49.5	50.5	61.9	62.8
- global train+inf	85.8	85.2	45.0	46.8	58.9	59.6
- ans from full sent	84.7	81.8	49.5	46.7	61.9	59.2
BERT QA	-	87.8	-	39.2	-	52.6
Table 4: The performance and ablations of our proposed model on the development set of SQuAD normal, SQuAD addSent, and SQuAD addOneSent. - global train+inf denotes the locally trained and evaluated model. - ans from full sent denotes extracting the answer using only the wh-aligned node. ans in wh denotes the percentage of answers found in the span aligned to the wh-span, and F1 denotes the standard QA performance measure.