SlowGuess/ACL_2020_OCR huggingface dataset:
Table 1: Results on the MultiRC development and test sets. The first column specifies the runtime overhead required for selection of evidence sentences, where  $N$  is the total number of sentences in the passage, and  $K$  is the selected number of sentences. The second column specifies if the retrieval system is a supervised method or not. The last three columns indicate evidence selection performance, whereas the previous three indicate overall QA performance. Only the last block of results report performance on the test set. The bold italic font highlights the best performance without using parallel evidences.  $\star$  denotes usage of external labeled data for pretraining.
MultiRC dataset, which includes the gold annotations for evidence text, unlike the version available on SuperGlue (Wang et al., 2019a).
Question Answering using Sentence Composition (QASC), a large KB-based multiple-choice QA dataset (Khot et al., 2019a). Each question is provided with 8 answer candidates, out of which 4 candidates are hard adversarial choices. Every question is annotated with a fixed set of two justification sentences for answering the question. The
justification sentences are to be retrieved from a KB having 17.2 million facts. As shown in the example of fig. 1 and also highlighted by (Khot et al., 2019a), multiple evidence text are possible for a given question in QASC where the annotated gold justification sentences explain it more precisely.
We report overall question answering performance as well as evidence selection performance in table 1 for MultiRC, and table 2 for  $\mathrm{QASC^{10}}$ .
$^{9}$ https://cogcomp.seas.upenn.edu/multirc/
10https://leaderboard.allenai.org/qasc/submissions/public
#	Number of steps used?	Method	Accuracy	Recall@10 both found	Recall@10 atleast one found
Baselines
0	Single	Naive Lucene BM25	35.6	17.2	68.1
1	Two	Naive Lucene BM25	36.3	27.8	65.7
2	Two	Heuristics+IR (Khot et al., 2019a)	32.4	41.6	64.4
3	-	ESIM Q2Choice (Khot et al., 2019a)	21.1	41.6	64.4
Previous work
4	Single	BERT-LC (Khot et al., 2019a)	59.8	11.7	54.7
5	Two	BERT-LC (Khot et al., 2019a)	71.0	41.6	64.4
6	Two	BERT-LC[WM]* (Khot et al., 2019a)	78.0	41.6	64.4
Alignment + RoBERTa baselines
7	-	No Justification + RoBERTa	20.5	0	0
8	Single	Alignment (K=1 sentences) + RoBERTa	54.4	-	-
9	Two	Alignment (K=2 sentences) + RoBERTa	71.5	-	-
10	Two	Alignment (K=3 sentences) + RoBERTa	73.3	-	-
11	Two	Alignment (K=4 sentences) + RoBERTa	73.5	-	-
12	Two	Alignment (K=5 sentences) + RoBERTa	74.1	-	-
AIR+ RoBERTa
13	Two	AIR (lexical) top chain + RoBERTa	75.8	-	-
14	Two	AIR top chain + RoBERTa	76.2	-	-
15	Two	AIR Parallel evidence chains (p=2) + RoBERTa	79.8	-	-
16	Two	AIR Parallel evidence chains (p=3) + RoBERTa	80.9	-	-
17	Two	AIR Parallel evidence chains (p=4) + RoBERTa	79.7	-	-
18	Two	AIR Parallel evidence chains (p=5) + RoBERTa	81.4	44.8	68.6
TEST DATASET
19	Two	BERT-LC (Khot et al., 2019a)	68.5	-	-
20	Two	BERT-LC[WM]* (Khot et al., 2019a)	73.2	-	-
21	Two	AIR Parallel evidence chains (p=5) + RoBERTa	81.4	-	-
Table 2: QA and evidence selection performance on QASC. We also report recall@10 similar to Khot et al. (2019a). both found reports the recall scores when both the gold justifications are found in top 10 ranked sentences and similarly atleast one found reports the recall scores when either one or both the gold justifications are found in the top 10 ranked sentences. Recall@10 are not reported (row 8-17) when number of retrieved sentences are lesser than 10. Other notations are same as table 1.
In addition to previously-reported results, we include in the tables several in-house baselines. For MultiRC, we considered three baselines. The first baseline is where we feed all passage sentences to the RoBERTa classifier (row 11 in table 1). The second baseline uses the alignment method of (Kim et al., 2017) to retrieve the top  $k$  sentences  $(k = 2,5)$ . Since AIR uses the same alignment approach for retrieving justifications in each iteration, the comparison to this second baseline highlights the gains from our iterative process with query reformulation. The third baseline uses a supervised RoBERTa classifier trained to select the gold justifications for every query (rows 16-21 in table 1). Lastly, we also developed a RoBERTa-based iterative retriever by concatenating the query with the retrieved justification in the previous step. We retrain the RoBERTa iterative retriever in every step, using the new query in each step.
We considered two baselines for QASC. The first baseline does not include any justifications (row 7 in table 2). The second baseline uses the top  $k$  sentences retrieved by the alignment method
(row (8-12 in table 2).
4.2 Evidence Selection Results
For evidence selection, we report precision, recall, and F1 scores on MultiRC (similar to (Wang et al., 2019b; Yadav et al., 2019b)). For QASC, we report Recall@10, similar to the dataset authors (Khot et al., 2019a). We draw several observation from the evidence selection results:
(1) AIR vs. unsupervised methods - AIR outperforms all the unsupervised baselines and previous works in both MultiRC (row 9-15 vs. row 23 in table 1) and QASC(row 0-6 vs. row 18). Thus, highlighting strengths of AIR over the standard IR baselines. AIR achieves  $5.4\%$  better F1 score compared to the best parametric alignment baseline (row 12 in table 1), which highlights the importance of the iterative approach over the vanilla alignment in AIR. Similarly, rows (4 and 5) of table 2 also highlight this importance in QASC.
(2) AIR vs. supervised methods - Surprisingly, AIR also outperforms the supervised RoBERTa-retriever in every settingrows 16-21 in table 1). Note that the performance of this supervised re
trieval method drops considerably when trained on passages from a specific domain (row 19 in table 1), which highlights the domain sensitivity of supervised retrieval methods. In contrast, AIR is unsupervised and generalize better as it is not tuned to any specific domain. AIR also achieves better performance than supervised RoBERTa-iterative-retriever (row 21 in table 1) which simply concatenates the retrieved justification to the query after every iteration and further trains to retrieve the next justification. The RoBERTa-iterative-retriever achieves similar performance as that of the simple RoBERTa-retriever (row 16 vs. 21) which suggests that supervised iterative retrievers marginally exploit the information from query expansion. On the other hand, controlled query reformulation of AIR leads to  $5.4\%$  improvement as explained in the previous point. All in all, AIR achieves state-of-the-art results for evidence retrieval on both MultiRC (row 23 in table 1) and QASC (row 18 of table 2).
(3) Soft-matching of AIR - the alignment-based AIR is  $10.7\%$  F1 better than AIR that relies on lexical matching (rather than the soft matching) on MultiRC (row 22 vs. 23), which emphasizes the advantage of alignment methods over conventional lexical match approaches.
4.3 Question Answering Results
For overall QA performance, we report the standard performance measures  $(F1_{a}, F1_{m}$ , and  $EM0)$  in MultiRC (Khashabi et al., 2018a), and accuracy for QASC (Khot et al., 2019a).
The results in tables 1 and 2 highlight:
(1) State-of-the-art performance:
Development set - On both MultiRC and QASC, RoBERTa fine-tuned using the AIR retrieved evidence chains (row 23 in table 1 and row 14 in table 2) outperforms all the previous approaches and the baseline methods. This indicates that the evidence texts retrieved by AIR not only provide better explanations, but also contribute considerably in achieving the best QA performance.
Test set - On the official hidden test set, RoBERTa fine-tuned on 5 parallel evidences from AIR achieves new state-of-the-art QA results, outperforming previous state-of-the-art methods by  $7.8\%$  accuracy on QASC (row 21 vs. 20), and  $10.2\%$  EM0 on MultiRC (row 35 vs. 34).
(2) Knowledge aggregation - The knowledge aggregation from multiple justification sentences
# of hops	BM25	AIR (Lexical) uncontrolled	Alignment	AIR uncontrolled
1	38.8	38.8	46.5	46.5
2	48.4	45.9	58.8	54.1
3	48.4	45.8	56.1	52.2
4	47.0	44.0	52.3	49.1
5	44.8	41.1	48.4	46.0
Table 3: Impact of semantic drift across consecutive hops on justification selection F1-performance of MultiRC development set. The uncontrolled configuration indicates that the justification sentences retrieved in each hop were appended to the query in each step. Here, AIR is forced to retrieve the same number of justifications as indicated by the # of hops.
leads to substantial improvements, particularly in QASC (single justification (row 4 and 8) vs. evidence chains (row 5 and row 9) in table table 2). Overall, the chain of evidence text retrieved by AIR enables knowledge aggregation resulting in the improvement of QA performances.
(3) Gains from parallel evidences - Further, knowledge aggregation from parallel evidence chains lead to another  $3.7\%$  EM0 improvement on MultiRC (row 27), and  $5.6\%$  on QASC over the single AIR evidence chain (row 18). To our knowledge, these are new state-of-the-art results in both the datasets.
To further understand the retrieval process of AIR we implemented several analyses.
5.1 Semantic Drift Analysis
To understand the importance of modeling missing information in query reformulation, we analyzed a simple variant of AIR in which, rather than focusing on missing information, we simply concatenate the complete justification sentence to the query after each hop. To expose semantic drift, we retrieve a specified number of justification sentences. As seen in table 3, now the AIR(lexical)-uncontrolled and AIR-uncontrolled perform worse than both BM25 and the alignment method. This highlights that the focus on missing information during query reformulation is an important deterrent of semantic drift. We repeated the same experiment with the supervised RoBERTa retriever (trained iteratively for 2 steps) and the original parameter-free AIR, which decides its number of hops using the stopping conditions. Again, we observe similar performance drops in both: the RoBERTa retriever drops from  $62.3\%$  to  $57.6\%$  and AIR drops to  $55.4\%$ .
\( {Q}_{r} \)	MultiRC F1 score	QASC Both Found One Found
1	64.2	41.7	67.7
2	62.7	42.7	67.7
3	61.8	43.1	68.6
4	60.63	40.6	68.4
5	59.8	39.0	67.5
Table 4: Impact on justification selection F1-performance from the hyper parameter  ${Q}_{r}$  of  $\mathbf{{AIR}}$  (eq. (5)).
M	MultiRC
F1 score	QASC
Both Found	One Found
0.95	64.2	43.1	68.6
0.85	63.7	42.3	67.9
0.75	63.4	42.5	68.0
5.2 Robustness to Hyper Parameters
We evaluate the sensitivity of AIR to the 2 hyper parameters: the threshold  $(Q_r)$  for query expansion, and the cosine similarity threshold  $M$  in computation of alignment. As shown in table 5, evidence selection performance of AIR drops with the lower values of  $M$  but the drops are small, suggesting that AIR is robust to different  $M$  values.
Similarly, there is a drop in performance for MultiRC with the increase in the  $Q_{r}$  threshold used for query expansion, hinting to the occurrence of semantic drift for higher values of  $Q_{r}$  (table 4). This is because the candidate justifications are coming from a relatively small numbers of paragraphs in MultiRC; thus even shorter queries (=2 words) can retrieve relevant justifications. On the other hand, the number of candidate justifications in QASC is much higher, which requires longer queries for disambiguation ( $> = 4$  words).
5.3 Saturation of Supervised Learning
To verify if the MultiRC training data is sufficient to train a supervised justification retrieval method, we trained justification selection classifiers based on BERT, XLNet, and RoBERTa on increasing proportions of the MultiRC training data (table 6). This analysis indicates that all three classifiers approach their best performance at around  $5\%$  of the training data. This indicates that, while these supervised methods converge quickly, they are unlikely to outperform AIR, an unsupervised method, even if more training data were available.
Table 5: Impact on justification selection F1 score from the hyper parameter  $M$  in the alignment step (section 3.1).
% of training data	BERT	XLnet	RoBERTa	AIR
2	55.2	54.6	62.3
5	60.0	59.6	60.8
10	59.9	57.0	59.8
15	58.3	59.9	59.1
20	58.5	60.2	60.0	64.2
40	58.5	58.7	58.8
60	59.1	61.4	59.8
80	59.3	61.0	60.5
100	60.9	61.1	62.3
Table 6: Comparison of AIR with XLNet, RoBERTa, and BERT on justification selection task, trained on increasing proportion of the training data on MultiRC.
We introduced a simple, unsupervised approach for evidence retrieval for question answering. Our approach combines three ideas: (a) an unsupervised alignment approach to soft-align questions and answers with justification sentences using GloVe embeddings, (b) an iterative process that reformulates queries focusing on terms that are not covered by existing justifications, and (c) a simple stopping condition that concludes the iterative process when all terms in the given question and candidate answers are covered by the retrieved justifications. Overall, despite its simplicity, unsupervised nature, and its sole reliance on GloVe embeddings, our approach outperforms all previous methods (including supervised ones) on the evidence selection task on two datasets: MultiRC and QASC. When these evidence sentences are fed into a RoBERTa answer classification component, we achieve the best QA performance on these two datasets. Further, we show that considerable improvements can be obtained by aggregating knowledge from parallel evidence chains retrieved by our method.
In addition of improving QA, we hypothesize that these simple unsupervised components of AIR will benefit future work on supervised neural iterative retrieval approaches by improving their query reformulation algorithms and termination criteria.
We thank Tushar Khot (AI2) and Daniel Khashabhi (AI2) for helping us with the dataset and evaluation resources. This work was supported by the Defense Advanced Research Projects Agency (DARPA) under the World Modelers program, grant number W911NF1810014. Mihai Surdeanu declares a financial interest in lum.ai. This interest has been properly disclosed to the University of Arizona Institutional Review Committee and is managed in accordance with its conflict of interest policies.
David Alvarez-Melis and Tommi Jaakkola. 2017. A causal framework for explaining the predictions of black-box sequence-to-sequence models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 412-421.
Leila Arras, Franziska Horn, Gregoire Montavon, Klaus-Robert Müller, and Wojciech Samek. 2017. "what is relevant in a text document?": An interpretable machine learning approach. PloS one, 12(8):e0181142.
Pratyay Banerjee. 2019. Asu at textgraphs 2019 shared task: Explanation regeneration using language models and iterative re-ranking. In Proceedings of the Thirteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-13), pages 78-84.
Lisa Bauer, Yicheng Wang, and Mohit Bansal. 2018. Commonsense for generative multi-hop question answering tasks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4220-4230.
Or Biran and Courtenay Cotton. 2017. Explanation and justification in machine learning: A survey. In *IJCAI-17 workshop on explainable AI (XAI)*, volume 8.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870-1879.
Jifan Chen and Greg Durrett. 2019. Understanding dataset design choices for multi-hop reasoning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4026-4032.
Jifan Chen, Shih-ting Lin, and Greg Durrett. 2019. Multi-hop question answering via reasoning chains. arXiv preprint arXiv:1910.02610.
Eunsol Choi, Daniel Hewlett, Jakob Uszkoreit, Illia Polosukhin, Alexandre Lacoste, and Jonathan Berant. 2017. Coarse-to-fine question answering for long documents. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 209-220.
Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, and Andrew McCallum. 2018. Multi-step retriever-reader interaction for scalable open-domain question answering.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of
deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368-2378.
Yair Feldman and Ran El-Yaniv. 2019. Multi-hop paragraph retrieval for open-domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2296-2309.
Mor Geva and Jonathan Berant. 2018. Learning to search in long documents using document structure. In Proceedings of the 27th International Conference on Computational Linguistics, pages 161-176.
Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. 2018. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pages 80-89. IEEE.
Alessio Gravina, Federico Rossetto, Silvia Severini, and Giuseppe Attardi. 2018. Cross attention for selection-based question answering. In 2nd Workshop on Natural Language for Artificial Intelligence. Aachen: R. Piskac.
Peter Jansen and Dmitry Ustalov. 2019. Textgraphs 2019 shared task on multi-hop inference for explanation regeneration. In Proceedings of the Thirteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-13), pages 63-77.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018a. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 252-262.
Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Dan Roth. 2018b. Question answering as global reasoning over semantic abstractions. In Thirty-Second AAAI Conference on Artificial Intelligence.
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2019a. Qasc: A dataset for question answering via sentence composition. arXiv preprint arXiv:1910.11473.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2019b. What's missing: A knowledge gap guided approach for multi-hop question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2807-2821.
Sun Kim, Nicolas Fiorini, W John Wilbur, and Zhiyong Lu. 2017. Bridging the gap: Incorporating a semantic similarity measure for effectively mapping pubmed queries to documents. Journal of biomedical informatics, 75:122-127.
Yankai Lin, Haozhe Ji, Zhiyuan Liu, and Maosong Sun. 2018. Denoising distantly supervised open-domain question answering. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1736-1745.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Sewon Min, Victor Zhong, Richard Socher, and Caiming Xiong. 2018. Efficient and robust question answering from minimal context over documents. arXiv preprint arXiv:1805.08092.
Yixin Nie, Songhe Wang, and Mohit Bansal. 2019. Revealing the importance of semantic retrieval for machine reading at scale. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2553-2566.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Rajkumar Pujari and Dan Goldwasser. 2019. Using natural language relations between answer choices for machine comprehension. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4010-4015.
Peng Qi, Xiaowen Lin, Leo Mehr, Zijian Wang, and Christopher D Manning. 2019. Answering complex open-domain questions through iterative query generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2590-2602.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. In Proceedings of the 56th Annual
Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784-789.
Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333-389.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In AAAI, pages 4444-4451.
Haitian Sun, Tania Bedrax-Weiss, and William Cohen. 2019a. Pullnet: Open domain question answering with iterative retrieval on knowledge bases and text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2380-2390.
Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019b. Dream: A challenge data set and models for dialogue-based reading comprehension. Transactions of the Association for Computational Linguistics, 7:217-231.
Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2019c. Improving machine reading comprehension with general reading strategies. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2633-2643.
Mihai Surdeanu, Massimiliano Ciaramita, and Hugo Zaragoza. 2008. Learning to rank answers on large online qa collections. In Proceedings of ACL-08: HLT, pages 719-727.
Alona Sydorova, Nina Poerner, and Benjamin Roth. 2019. Interpretable question answering on knowledge bases and text. arXiv preprint arXiv:1906.10924.
Harsh Trivedi, Heeyoung Kwon, Tushar Khot, Ashish Sabharwal, and Niranjan Balasubramanian. 2019. Repurposing entailment for multi-hop question answering tasks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2948-2958.
Ming Tu, Kevin Huang, Guangtao Wang, Jing Huang, Xiaodong He, and Bowen Zhou. 2019. Select, answer and explain: Interpretable multi-hop reading comprehension over multiple documents. arXiv preprint arXiv:1911.00484.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537.