SlowGuess/NAACL_2021_OCR huggingface dataset:
(1) Unsupervised retrieval: Indicating initial benefits of retrieving evidence chains, our alignment-based evidence retrieval approach (WAIR) outperforms the other IR benchmarks (BM25 and alignment) as shown in rows 10-11 vs. 12-13 in table 4 and rows  $\{1,9,10\}$  vs. 11 in table 2. WAIR also outperforms the two-step IR-based methods for evidence retrieval (row (9, 10 vs. 11) in table 2), highlighting the importance of query reweighing in iterative retrieval methods.
(2) Supervised reranking: Reranking WAIR candidate evidence chains (JointRR) leads to absolute  $10.4\%$  on QASC (row 12 vs row 13 in table 2) and  $3.6\%$  F1 improvement on MultiRC (row 14 vs row 15 in table 4) over the case where the same reranker is fed with individual sentences (SingleRR). This highlights the importance of feeding candidate evi
dence chains to the supervised reranker.
(3) Recall comparison: As shown in table 3, just feeding WAIR candidate chains result in higher performance for retrieving complete evidence (the "Both found" columns) than SingleRR, especially for low recall scenarios. Notably, SingleRR achieves marginally better performance on finding at least 1 evidence sentence but performs poorly on retrieving both the evidence sentences indicating absence of compositional multi-hop reasoning. We observe similar gains on MultiRC i.e., JointRR achieves  $6\%$  higher recall compared to SingleRR (row 14, row 15 in table 4).
(4) (Pseudo) oracle JointRR: To investigate the ceiling of JointRR, we inserted the gold justification sentences within the WAIR retrieved sentences and then created candidate evidence chains. These chains were then reranked by the same RoBERTa reranker. As shown in row 18a of table 4 and row 14 of table 2, the performance of JointRR approach is substantially improved when gold evidence sentences are retrieved in the initial WAIR pool. The ceiling performance of JointRR is much higher than the current actual method (row 13 in table 2 and row 15 in table 4), which suggests there is
<sup>11</sup>We found similar trends for MultiRC but present analysis only on QASC (large KB based QA) because of space constraints.
Recall@N	SingleRR	JointRR
Evidence Both found	Evidence Atleast 1 found	QA Accuracy	Evidence Both found	Evidence Atleast 1 found	QA Accuracy
Recall@2	20.1	65.3	73.8	30.5	65.1	78.6
Recall@4	35.0	67.9	74.7	40.5	66.7	80.7
Recall@6	40.2	69.0	77.9	44.1	68.2	80.0
Recall@8	43.3	69.4	76.8	45.2	69.0	79.6
Recall@10	44.4	69.6	79.7	45.3	69.4	81.7
Table 3: Evidence retrieval and QA performance comparison of SingleRR and JointRR at different recall levels on the QASC development dataset. "Both found" and "Atleast 1 found" notations are same as in table 2 but at top  $N$  sentences. Recall@N of "Both found" means when both the gold justifications are found in top  $N$  sentences. All the  $N$  sentences are concatenated to feed into the answer classifier for QA task.
#	Other-resources /Ensembling	Method	F1m	F1a	EM0	Evidence retrieval
P	R	F1
DEVELOPMENT DATASET
Baselines
1	No	IR(paragraphs) (Khashabi et al., 2018)	64.3	60.0	1.4	-
2	No	SurfaceLR (Khashabi et al., 2018)	66.5	63.2	11.8	-
3	No	RoBERTa+ Full passage (Yadav et al., 2020b)	73.9	71.7	28.7	17.4	100.0	29.6
Previous work
5	No	EERDPL + FT (Wang et al., 2019b)	70.5	67.8	13.3	-
6	Yes	Multee (ELMo)* (Trivedi et al., 2019)	73.0	69.6	22.8	-
7	Yes	RS* (Sun et al., 2019)	73.1	70.5	21.8	-	-	60.8
8	No	AutoROCC (Yadav et al., 2019b)	72.9	69.6	24.7	48.2	68.2	56.4
9	Yes	AIR + RoBERTa (Yadav et al., 2020b)	74.7	72.3	29.3	66.2	63.1	64.2
Our work
10	No	3 Evidence sents(BM25) + RoBERTa	70.5	68.0	24.9	42.6	56.1	48.4
11	No	3 Evidence sents(Alignment) + RoBERTa	72.4	69.8	25.1	49.3	65.1	56.1
12	No	3 WAIR sents + RoBERTa	74.3	71.5	24.6	50.9	67.6	58.1
13	No	WAIR max-coverage + RoBERTa	74.2	72.2	27.0	55.0	67.2	60.5
14	No	SingleRR + RoBERTa	74.9	72.4	25.9	63.9	64.0	64.0
15	No	JointRR + RoBERTa	75.2	72.7	28.2	65.4	69.9	67.6
15a	No	JointRR (±1 neighboring sentence) + RoBERTa	77.0	74.5	32.9	65.4	69.9	67.6
Reranking checkpoints transferred to QA task
16	No	SingleRR transferred	71.7	68.8	21.6	63.9	64.0	64.0
17	No	JointRR transferred	75.9	73.1	28.2	65.4	69.9	67.6
Ceiling systems with gold justifications
18	No	Oracle knowledge + RoBERTa	81.4	80	39	100.0	100.0	100.0
18a	No	Pseudo oracle + JointRR + RoBERTa	77.9	74.8	32.9	87.8	82.9	85.3
19	No	Human	86.4	83.8	56.6	-
TEST DATASET
20	No	SurfaceLR (Khashabi et al., 2018)	66.9	63.5	12.8
21	Yes	Multee (ELMo)* (Trivedi et al., 2019)	73.8	70.4	24.5	-
22	No	AutoROCC (Yadav et al., 2019b)	73.8	70.6	26.1
23	Yes	RoBERTa + AIR (Yadav et al., 2020b)	79.0	76.4	36.3
24	No	JointRR (±1 neighboring sentence) + RoBERTa	79.5	76.5	35.4
Table 4: Answer selection (column 4-6) and evidence retrieval results (column 7-10) on the MultiRC development and test sets. The second column specifies if any external labeled or ensembling resources were used in the approach.  $\pm 1$  neighboring sentence (row 15a) indicates concatenation of neighboring sentences with the predicted evidence sentences to utilize coreferences in the context.
potential for progress from future works.
(5) State-of-the-art evidence retrieval performance: The top reranked WAIR chain achieves  $30.5\%$  Recall@2 on QASC (row 13, table 2) and  $67.6\%$  F1 on MultiRC (row 15, table 4). Thus, establishing the new state-of-the-art evidence retrieval performance on both the datasets.
4.2 Answer Selection Results
(1) Impact of two-step evidence retrieval: Unsurprisingly, the two-step evidence retrieval process substantially impacts QA performance (e.g., row 1 vs. row 9 in table 2), which is consistent with the observations of previous works (Khot et al., 2019a; Yadav et al., 2020b). The top reranked WAIR chain
leads to higher QA performance  $(+5.2\%)$  on QASC (row 12 vs. 13, table 2), and  $2.3\%$  F1 on MultiRC (row 14 vs. 15, table 4)).
(2) Impact of retrieval recall: As shown in table 3, JointRR always achieves higher Recall@N score for finding both (or complete) evidence. As a result, it also achieves better QA accuracy when compared to SingleRR. On the other hand, SingleRR always achieves marginally better performance on finding at least 1 evidence sentence indicating that retrieval of incomplete information leads to lower QA performance. Further, the best QA performance is also achieved at higher recalls (last row of table 3 and row 15 in table 4).
(3) Ceiling performance: When coupled with the (pseudo) oracle retriever, the QA scores of JointRR approaches human performance (row 18, table 4). This emphasizes the importance of evidence retrieval for the QA performance.
(4) Top QA performance: RoBERTa answer classifier that just the uses top reranked evidence of WAIR achieves state-of-the-art QA performance on MultiRC development and test sets. It also achieves the second and third best results on QASC development and test sets. Notably, the approaches that score higher than JointRR use ensembling or additional labeled data.
5 Representational Analysis
5.1 Attention Analysis
To better understand the differences in learned features of RoBERTa reranker from WAIR chains (JointRR) and individual candidate evidence sentences (SingleRR), we performed several analyses of their attention weights. We focus on the attention score on the [CLS] token, whose representation is fed into the decision layer of the RoBERTa classifier (Wolf et al., 2019). We compute the attention score from a given token to [CLS] by summing up the attention scores from all the 12 heads in each layer (Clark et al., 2019). Similar to Clark et al. (2019); Rogers et al. (2020), we remove the attention scores from  $< s >, < / s>$ , punctuation and stopwords tokens in our analysis.
Attention from semantically matching tokens in query and evidence: Retrieval tasks are often driven by the lexically matching query tokens in the retrieved document(Robertson et al., 2009; Manning et al., 2008). Thus, to understand the fo
Token type	QASC	MultiRC
SingleRR	JointRR	SingleRR	JointRR
SMA	50.3	56.0	60.0	64.0
Linking	50.6	54.8	55.7	64.4
Table 5: Various attention scores of the SingleRR and the JointRR approaches. These normalized attention values are reported from the average of last 3 layers (10th, 11th and 12th layer) of RoBERTa-base. We observed similar trends with few exceptions in the lower layers as well which are farthest away from the decision layer that uses representation of [CLS].
cus of the reranker on semantic matching, we compute the attention on [CLS] from all the tokens that are not lexically matched between the given question+candidate answer text and the retrieved evidence text (Yadav et al., 2020a). We refer it to as Semantic Matching Attention (SMA) score. As shown in table 5, reranker fed with WAIR chain (JointRR approach) attends more on the tokens requiring semantic matching when compared to SingleRR (50.3% vs 56% on QASC and 60.0 vs. 64.0% on MultiRC) suggesting that it learns how to "bridge the lexical chasm" between question and answers (Berger et al., 2000)
Attention from linking tokens of evidence: Here, we focus only on the terms that are shared between sentences in the gold evidence texts (referred to as Linking terms). As shown in fig. 1, {nuclear, membrane} are examples of linking terms that compose the two justification sentences into a complete explanation. The remaining terms in the evidence text, i.e., terms that are uniquely present in any one of the evidence sentences are referred to as Non linking terms. As shown in table 5, JointRR attends considerably more to the Linking terms (50.6 vs. 54.8 and 55.7 vs. 64.4), which suggests that it focuses more on the relevant compositional pieces after the retrieval training.
5.2 Learned Embedding Analysis
We also analyzed the embedding representations of the reranking model (Ethayarajh, 2019). In particular, we computed the embedding based cosinesimilarity scores (or alignment scores (Yadav et al., 2019a)) between the two gold evidence sentences to determine their similarity in embedding space. As shown in fig. 3, the inter-justification alignment similarity score of JointRR is substantially lower across the majority of the layers after layer
Figure 3: Layer-wise embedding based alignment similarity scores between the two gold justification sentences. In QASC, every question is annotated with just two gold justification sentences; for simplicity, we consider only the subset of MultiRC questions which have two gold justifications (  $65\%$  of dev set).
3. This indicates that the RoBERTa reranker fed with WAIR chains has learned to differentiate the individual justification sentences (in embedding space) enabling complementary and compositional knowledge aggregation. As shown in table 4 (row 17 vs. row 15), this compositionality information is useful when the evidence reranking RoBERTa is transferred to the answer selection component i.e., we see a (small) QA performance improvement. On the other hand, SingleRR learns to consider both sentences similar, and this hurts the QA performance by  $4.3\%$  EM0 (row 16 vs. row 14, table 4).
Recent works have shown importance of vector normalization (Kobayashi et al., 2020) for analyzing the transformer embeddings. In future works, normalized embedding analysis can be added to further study the behavior of trained retriever's across different layers.
We introduced a simple unsupervised approach for retrieving candidate evidence chains that after reranking achieves state-of-the-art evidence retrieval performance on two multi-hop QA datasets: QASC and MultiRC. We highlight the importance of generating and feeding candidate evidence chains by showing several benefits over the widely followed approach that retrieves evidence
sentences individually. Further, we introduced few attention and embedding analyses demonstrating that jointly retrieving and reranking chains assist in learning compositional information, which is also beneficial to the downstream QA task. Overall, our work highlights the strengths and potential of joint retrieval+reranking approaches for future works.
This work was supported by the Defense Advanced Research Projects Agency (DARPA) under the World Modelers program, grant number W911NF1810014. Mihai Surdeanu declares a financial interest in lum.ai. This interest has been properly disclosed to the University of Arizona Institutional Review Committee and is managed in accordance with its conflict of interest policies.
Pratyay Banerjee. 2019. Asu at textgraphs 2019 shared task: Explanation regeneration using language models and iterative re-ranking. In Proceedings of the Thirteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-13), pages 78-84.
Pratyay Banerjee and Chitta Baral. 2020. Knowledge fusion and semantic knowledge ranking for open domain question answering. arXiv preprint arXiv:2004.03101.
Adam Berger, Rich Caruana, David Cohn, Dayne Freytag, and Vibhu Mittal. 2000. Bridging the lexical chasm: Statistical approaches to answer finding. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research & Development on Information Retrieval, Athens, Greece.
Delphine Bernhard. 2010. Query expansion based on pseudo relevance feedback from definition clusters. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, pages 54-62. Association for Computational Linguistics.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870-1879.
Jifan Chen and Greg Durrett. 2019. Understanding dataset design choices for multi-hop reasoning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4026-4032.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019. What does bert look at? an analysis of bert's attention. arXiv preprint arXiv:1906.04341.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C Wallace. 2019. Eraser: A benchmark to evaluate rationalized nlp models. arXiv preprint arXiv:1911.03429.
Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179.
Kawin Ethayarajh. 2019. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 55–65.
Yair Feldman and Ran El-Yaniv. 2019. Multi-hop paragraph retrieval for open-domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2296-2309.
Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1161-1166.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. arXiv preprint arXiv:1803.02324.
Peter Jansen. 2018. Multi-hop inference for sentence-level textgraphs: How challenging is meaningfully combining information for science question answering? In Proceedings of the Twelfth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-12), pages 12-17.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems.
In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021-2031.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 252-262.
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2019a. Qasc: A dataset for question answering via sentence composition. arXiv preprint arXiv:1910.11473.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2019b. What's missing: A knowledge gap guided approach for multi-hop question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2807-2821.
Sun Kim, Nicolas Fiorini, W John Wilbur, and Zhiyong Lu. 2017. Bridging the gap: Incorporating a semantic similarity measure for effectively mapping pubmed queries to documents. Journal of biomedical informatics, 75:122-127.
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020. Attention is not only a weight: Analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7057-7075, Online. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Christopher D Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to information retrieval. Cambridge university press.
Yixin Nie, Songhe Wang, and Mohit Bansal. 2019. Revealing the importance of semantic retrieval for machine reading at scale. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International
Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2553-2566.
Xiaoman Pan, Kai Sun, Dian Yu, Heng Ji, and Dong Yu. 2019. Improving question answering with external knowledge. arXiv preprint arXiv:1902.00993.
Peng Qi, Xiaowen Lin, Leo Mehr, Zijian Wang, and Christopher D Manning. 2019. Answering complex open-domain questions through iterative query generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2590-2602.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
Juan Ramos et al. 2003. Using tfidf to determine word relevance in document queries. In Proceedings of the first instructional conference on machine learning, volume 242, pages 29-48. CiteSeer.
Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333-389.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in bertology: What we know about how bert works. arXiv preprint arXiv:2002.12327.
Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2019. Improving machine reading comprehension with general reading strategies. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2633-2643.
Harsh Trivedi, Heeyoung Kwon, Tushar Khot, Ashish Sabharwal, and Niranjan Balasubramanian. 2019. Repurposing entailment for multi-hop question answering tasks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2948-2958.
Ming Tu, Kevin Huang, Guangtao Wang, Jing Huang, Xiaodong He, and Bowen Zhou. 2019. Select, answer and explain: Interpretable multi-hop reading comprehension over multiple documents. arXiv preprint arXiv:1911.00484.
Betty van Aken, Benjamin Winter, Alexander Löser, and Felix A Gers. 2019. How does bert answer questions? a layer-wise analysis of transformer representations. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 1823-1832.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537.
Hai Wang, Dian Yu, Kai Sun, Jianshu Chen, Dong Yu, David McAllester, and Dan Roth. 2019b. Evidence sentence extraction for machine reading comprehension. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 696-707.
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association of Computational Linguistics, 6:287-302.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtopicz, et al. 2019. Transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
Vikas Yadav, Steven Bethard, and Mihai Surdeanu. 2019a. Alignment over heterogeneous embeddings for question answering. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (Long Papers), Minneapolis, USA. Association for Computational Linguistics.
Vikas Yadav, Steven Bethard, and Mihai Surdeanu. 2019b. Quick and (not so) dirty: Unsupervised selection of justification sentences for multi-hop question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2578-2589.
Vikas Yadav, Steven Bethard, and Mihai Surdeanu. 2020a. Having your cake and eating it too: Training neural retrieval for language inference without losing lexical match. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1625-1628.
Vikas Yadav, Steven Bethard, and Mihai Surdeanu. 2020b. Unsupervised alignment-based iterative evidence retrieval for multi-hop question answering. arXiv preprint arXiv:2005.01218.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380.