Venue: NAACL 2021

# Multi-Step Reasoning Over Unstructured Text with Beam Dense Retrieval

Chen Zhao

University of Maryland

chenz@cs.umd.edu Chenyan Xiong

Microsoft Research

chenyan.xiong@microsoft.com Jordan Boyd-Graber

University of Maryland

jbg@umiacs.umd.edu Hal Daume III

Microsoft Research & University of Maryland

me@hal3.name Question: Ralph Heffleine was a psychology professor at a university that is located in what city? Evidence Chain: Ralph Heffleine > Columbia University Pi: Ralph Heffleine P2: Columbia University is a private was a psychology professor at Voyage research university in Columbia University, Upper Manhattan, New York City.

###### Abstract

Complex question answering often requires finding a reasoning chain that consists of multiple evidence pieces. Current approaches incorporate the strengths of structured knowledge and unstructured text, assuming text corpora is semi-structured. Building on dense retrieval methods, we propose a new multi-step retrieval approach (BeamDR) that iteratively forms an evidence chain through beam search in dense representations. When evaluated on multi-hop question answering, BeamDR is competitive to state-of-the-art systems, _without_ using any semi-structured information. Through query composition in dense space, BeamDR captures the implicit relationships between evidence in the reasoning chain. The code is available at [https://github.com/henryzhao5852/BeamDR](https://github.com/henryzhao5852/BeamDR).

## 1 Introduction

Answering complex questions requires combining knowledge pieces through multiple steps into an evidence chain (Ralph Hefferline\(\to\) Columbia University in Figure 1). When the available knowledge sources are graphs or databases, constructing chains can use the sources' inherent structure. However, when the information needs to be pulled from unstructured text (which often has better coverage), standard information retrieval (ir) approaches only go "one hop": from a query to a single passage.

Recent approaches Dhingra et al. (2020); Zhao et al. (2020a, b); Asai et al. (2020), _inter alia_) try to achieve the best of both worlds: use the unstructured text of Wikipedia with its structured hyperlinks. While they show promise on benchmarks, it's difficult to extend them beyond academic testbeds because real-world datasets often lack this structure. For example, medical records lack links between reports.

Dense retrieval Lee et al. (2019); Guu et al. (2020); Karpukhin et al. (2020), _inter alia_) provides a promising path to overcome this limitation. It encodes the query and evidence (passage) into dense vectors and matches them in the embedding space. In addition to its efficiency--thanks to maximum inner-product search (mips)--Xiong et al. (2021) show that dense retrieval rivals bert Devlin et al. (2019)-based (sparse) retrieve-then-rerank ir pipelines on single step retrieval. Unlike traditional term-based retrieval, fully learnable dense encodings provide flexibility for different tasks.

This paper investigates a natural question: can we build a retrieval system to find an evidence chain on unstructured text corpora? We propose a new multi-step dense retrieval method to model the implicit relationships between evidence pieces. We use beam search (Section 2) in the dense space to find and cache the most relevant candidate chains and iteratively compose the query by appending the retrieval history. We improve the retrieval by encouraging the representation to discriminate hard negative evidence chains from the correct chains, which are refreshed by the model.

We evaluate **Beam**Dense **Re**trieval (BeamDR) on HotPotQA Yang et al. (2018), a multi-

Figure 1: Top: A complex question example from HotPotQA that requires finding an evidence chain. Bottom: BeamDR iteratively composes the new query and retrieves evidence in dense space without the need for linked documents.



hop question answering benchmark. When retrieving evidence chains directly from the corpus (full retrieval), BeamDR is competitive to the state-of-the-art cascade reranking systems that use Wikipedia links. Combined with standard reranking and answer span extraction modules, the gain from full retrieval propagates to findings answers (Section 3). By iteratively composing the query representation, BeamDR captures the hidden "semantic" relationships in the evidence (Section 4).

## 2 BeamDR: Beam Dense Retriever

This section first discusses preliminaries for dense retrieval, then introduces our method, BeamDR.

### Preliminaries

Unlike classic retrieval techniques, dense retrieval methods match distributed text representations Bengio et al. (2013) rather than sparse vectors Salton (1968). With encoders (e.g., bert) to embed query \(q\) and passage \(p\) into dense vectors \(E_{Q}(q)\) and \(E_{P}(p)\), the relevance score \(f\) is computed by a similarity function \(sim(\cdot)\) (e.g., dot product) over two vector representations:

\[f(q,p)=sim(E_{Q}(q),E_{P}(p)). \tag{1}\]

After encoding passage vectors offline, we can efficiently retrieve passage through approximate nearest neighbor search over the maximum inner product with the query, i.e., mipsShivastava Li (2014); Johnson et al. (2017).

### Finding Evidence Chains with BeamDR

We focus on finding an evidence chain from an unstructured text corpus for a given question, often the hardest part of complex question answering. We formulate it as multi-step retrieval problem. Formally, given a question \(q\) and a corpus \(C\), the task is to form an ordered evidence chain \(p_{1}...p_{n}\) from \(C\), with each evidence a passage. We focus on the supervised setting, where the labeled evidence set is given during training (but not during testing).

Finding an evidence chain from the corpus is challenging because: 1) passages that do not share enough words are hard to retrieve (e.g., in Figure 1, the evidence Columbia University); 2) if you miss one evidence, you may err on all that come after.

We first introduce scoring a single evidence chain, then finding the top \(k\) chains with beam search, and finally training BeamDR.

Evidence Chain ScoringThe score \(S_{n}\) of evidence chain \(p_{1},\ldots,p_{n}\) is the product of the (normalized) relevance scores of individual evidence pieces. At each retrieval step \(t\), to incorporate the information from both the question and retrieval history, we compose a new query \(q_{t}\) by appending the tokens of retrieved chains \(p_{1},\ldots,p_{t-1}\) to query \(q\) (\(q_{t}=[q;p_{1};\ldots;p_{t-1}]\)), we use mips to find relevant evidence piece \(p_{t}\) from the corpus and update the evidence chain score \(S_{t}\) by multiplying the current step \(t\)'s relevance score \(f(q_{t},p_{t})*S_{t-1}\).

Beam Search in Dense SpaceSince enumerating all evidence chains is computationally impossible, we instead maintain an evidence cache. In the structured search literature this is called a _beam_: the \(k\)-best scoring candidate chains we have found thus far. We select evidence chains with beam search in dense space. At step \(t\), we enumerate each candidate chain \(j\) in the beam \(p_{j,1}...p_{j,t-1}\), score the top \(k\) chains and update the beam. After \(n\) steps, the \(k\) highest-scored evidence chains with length \(n\) are finally retrieved.

Training BeamDRThe goal of training is to learn embedding functions that differentiate positive (relevant) and negative evidence chains. Since the evidence pieces are unordered, we use heuristics to infer the order of evidence chains. A negative chain has at least one evidence piece that is not in the gold evidence set. For each step \(t\), the input is the query \(q\), a positive chain \(P^{+}_{t}=p^{+}_{1},\ldots,p^{+}_{t}\) and \(m\) sampled negative chains \(P^{-}_{j,t}=p^{-}_{1},\ldots,p^{-}_{t}\). We update the negative log likelihood (nll) loss:

\[L(q,P^{+},P^{-}_{1},...,P^{-}_{m}) \tag{2}\] \[=\sum_{t}\frac{e^{f([q;P^{+}_{t-1}],p^{+}_{t})}}{e^{f([q;P^{+}_{ t-1}],p^{+}_{t})}+\sum_{j=1}^{m}e^{f([q;P_{j,t-1}],p^{-}_{j,t})}}.\]

Rather than using local in-batch or term matching negative samples, like Guu et al. (2020) we select negatives from the whole corpus, which can be more effective for single-step retrieval Xiong et al. (2021a). In multi-step retrieval, we select negative evidence chains from the corpus. Beam search on the training data finds the top \(k\) highest scored negative chains for each retrieval step. Since the model parameters are dynamically updated, we asynchronously refresh the negative chains with the up-to-date model checkpointGuu et al. (2020); Xiong et al. (2021a) 



## 3 Experiments: Retrieval and Answering

Our experiments are on HotPotQA fullwiki setting Yang et al. (2018), the multi-hop question answering benchmark. We mainly evaluate on retrieval that extracts evidence chains (passages) from the corpus; we further add a downstream evaluation on whether it finds the right answer.

### Experimental Setup

MetricsFollowing Asai et al. (2020), we report four metrics on retrieval: answer recall (ar), if answer span is in the retrieved passages; passage recall (pr), if at least one gold passage is in the retrieved passages; Passage Exact Match (p em), if both gold passages are included in the retrieved passages; and Exact Match (em), whether both gold passages are included in the top two retrieved passages (top one chain). We report exact match (em) and \(F_{1}\) on answer spans.

ImplementationWe use a bert-base encoder for retrieval and report both bert base and large for span extraction. We warm up BeamDR with tf-idf negative chains. The retrieval is evaluated on ten passage chains (each chain has two passages). To compare with existing retrieve-then-rerank cascade systems, we train a standard bert passage reranker Nogueira and Cho (2019), and evaluate on ten chains reranked from the top 100 retrieval outputs. We train BeamDR on six 2080Ti GPUs, three for training, three for refreshing negative chains. We do not search hyper-parameters and use suggested ones from Xiong et al. (2021a).

### Passage Chain Retrieval Evaluation

BaselinesWe compare BeamDR with tf-idf, Semantic Retrieval Nie et al. (2019), sr), which uses a cascade bert pipeline, and the Graph recurrent retriever Asai et al. (2020), grr), our main baseline, which iteratively retrieves passages following the Wikipedia hyperlink structure, and is state-of-the-art on the leaderboard. We also compare against a contemporaneous model, multi-hop dense retrieval Xiong et al. (2021b), mdr).

Results: Robust Evidence Retrieval without Document LinksTable 1 presents retrieval results. On full retrieval, BeamDR is competitive to grr, state-of-the-art _reranker_ using Wikipedia hyperlinks. BeamDR also has better retrieval than the contemporaneous mdr. Although both approaches build on dense retrieval, mdr is close to BeamDR with tf-idf negatives. We instead refresh negative chains with intermediate representations, which help the model better discover evidence chains. Our ablation study (Greedy search) indicates the importance of maintaining the beam during inference. With the help of cross-attention between the question and the passage, using bert to rerank BeamDR outperforms all baselines.

Varying the Beam sizeFigure 2 plots the Passage cm with different beam sizes. While initially increasing the beam size improves Passage Exact Match, the marginal improvement decreases after a beam size of forty.

### Answer Extraction Evaluation

BaselinesWe compare BeamDR with tkh Zhao et al. (2020), grr Asai et al. (2020) and the contemporaneous mdrXiong et al. (2021b). We use released code from grrAsai et al. (2020) following its settings on bert base and large. We use four 2080Ti GPUs.

ResultsUsing the same implementation but on our reranked chains, BeamDR outperforms grr

\begin{table}
\begin{tabular}{l c c c c} \hline Models & AR & PR & P EM & EM \\ \hline \multicolumn{5}{l}{_Full Retrieval_} \\ tf-idf & 39.7 & 66.9 & 10.0 & 18.2 \\ mdr * & 75.4 & - & 65.9 & - \\ BeamDR (ir Neg) & 76.8 & 86.4 & 64.1 & 40.4 \\ BeamDR (Greedy) & 83.6 & 90.7 & 72.7 & 34.1 \\ BeamDR (Ours) & 87.0 & 92.9 & 79.2 & 60.7 \\ \hline \multicolumn{5}{l}{_Reranking from Retrieval Outputs_} \\ sr & 77.9 & 93.2 & 63.9 & 46.5 \\ grr & 87.8 & 93.3 & 77.9 & 61.1 \\ mdr * & 88.2 & - & 81.2 & - \\ BeamDR (Ours) & 90.7 & 94.7 & 83.7 & 70.7 \\ \hline \end{tabular}
\end{table}
Table 1: Compare BeamDR with other retrieval systems. Top: Retrieval from the whole corpus, bottom: Reranking from top 100 full retrieval outputs. * indicates parallel work.

Figure 2: Passage retrieval accuracy on different beam size. Our system is robust to the increase of beam size.



(Table 2), suggesting gains from retrieval could propagate to answer span extraction. BeamDR is competitive with mdr but slightly lower; we speculate different reader implementations might be the cause.

## 4 Exploring How we Hop

In this section, we explore how BeamDR constructs evidence chains.

### Qualitative Analysis

Figure 3 shows query and passage representations with t-sne Maaten and Hinton (2008). Unsurprisingly, in the dense space, the first hop query (question) is close to its retrieved passages but far from second hop passages (with some negative passages in between). After composing the question and first hop passages, the second hop queries indeed land closer to the second hop passages. Our quantitative analysis (Table 3) further shows BeamDR has little overlap between retrieved passages in two hops. BeamDR mimics multi-step reasoning by hopping in the learned representation space.

### Hop Analysis

To study model behaviors under different hops, we use heuristics1 to infer the order of evidence passages. In Table 3, BeamDR slightly wins on first hop passages, with the help of hyperlinks, grr outperforms BeamDR on second hop retrieval. Only \(21.9\%\) of the top-10 BeamDR chains are connected by links. BeamDR wins after using links to filter candidates.

Footnote 1: We label the passage that contains the answer as the second hop passage, while the other one as the first hop passage. If both passages include the answer, passage title mentioned in the question is the first hop passage.

### Human Evaluation on Model Errors and Case Study

To understand the strengths and weaknesses of BeamDR compared with grr, we manually analyze \(100\) bridge questions from the HotPotQA development set. BeamDR predicts fifty of them correctly and grr predicts the other fifty correctly (Tables 4 and 5).

Strengths of BeamDRCompared to grr, the largest gain of BeamDR is to identify question entity passages. As there is often little context overlap besides the entity surface form, a term-based approach (tf-idf used by grr) falters. Some of the grr errors also come from using reverse links to find second hop passages (i.e., the second hop passage links to the first hop passage).

\begin{table}
\begin{tabular}{c c c} \hline \hline
**Errors** & **Type** & \(\%\) \\ \hline  & Question entities & 62 \\ grr & Connect with reverse links & 16 \\  & Text matching & 14 \\  & Others & 8 \\ \hline  & Text matching & 46 \\ BeamDR & No links between passages & 39 \\  & Question entities & 15 \\ \hline \hline \end{tabular}
\end{table}
Table 4: We manually analyze 100 bridge questions and categorize model errors.

\begin{table}
\begin{tabular}{l r r r r r} \hline \hline \multirow{2}{*}{Retriever} & \multirow{2}{*}{Reader} & \multicolumn{2}{c}{Dev} & \multicolumn{2}{c}{Test} \\  & & EM & F1 & EM & F1 \\ \hline \multicolumn{7}{l}{_BERT base Reader_} \\ txh & txh & 54.0 & 66.2 & 51.6 & 64.1 \\ grr & grr & 52.7 & 65.8 & - & - \\ BeamDR & grr & 54.9 & 68.0 & - & - \\ \hline \multicolumn{7}{l}{_BERT large wwm Reader_} \\ grr & grr & 60.5 & 73.3 & 60.0 & 73.0 \\ BeamDR & grr & 61.3 & 74.1 & 60.4 & 73.2 \\ mdr\({}^{*}\) & mdr\({}^{*}\) & 61.5 & 74.7 & - & - \\ \hline \multicolumn{7}{l}{_ELECTRA large Reader_} \\ mdr\({}^{*}\) & mdr\({}^{*}\) & 63.4 & 76.2 & 62.3 & 75.3 \\ \hline \hline \end{tabular}
\end{table}
Table 2: HotPotQA dev and test set answer exact match (EM) and F1 results. \({}^{*}\) indicates parallel work.

Figure 3: t-sne visualization of query (Q) and passage (P) embeddings over different retrieval steps. BeamDR conducts multi-step reasoning by hopping in the learned representation space.

 

Weaknesses of BeamDR.Like Karpukhin et al. (2020), many of BeamDR's errors could be avoided by simple term matching. For example, matching _"What screenwriter with credits for Evolution co-wrote a film starring Nicolas Cage and Tea Leoni?"_ to the context _"The Family Man is a 2000 American film written by David Diamond and David Weissman, and starring Nicolas Cage and Tea Leoni."_.

## 5 Related Work

Extracting multiple pieces of evidence automatically has applications from solving crossword puzzles Littman et al. (2002), graph database construction De Melo and Weikum (2009), and understanding relationships Chang et al. (2009); Iyyer et al. (2016) to question answering Ferrucci et al. (2010), which is the focus of this work.

Given a complex question, researchers have investigated multi-step retrieval techniques to find an evidence chain. Knowledge graph question answering approaches Talmor and Berant (2018); Zhang et al. (2018), _inter alia_ directly search the evidence chain from the knowledge graph, but falter when kg coverage is sparse. With the release of large-scale datasets Yang et al. (2018), recent systems Nie et al. (2019); Zhao et al. (2020); Asai et al. (2020); Dhingra et al. (2020), _inter alia_) use Wikipedia abstracts (the first paragraph of a Wikipedia page) as the corpus to retrieve the evidence chain. Dhingra et al. (2020) treat Wikipedia as a knowledge graph, where each entity is identified by its textual span mentions, while other approaches Nie et al. (2019); Zhao et al. (2020) directly retrieve passages. They first adopt a single-step retrieval to select the first hop passages (or entity mentions), then find the next hop candidates directly from Wikipedia links and rerank them. Like BeamDR, Asai et al. (2020) use beam search to find the chains but still rely on a graph neural network over Wikipedia links. BeamDR retrieves evidence chains through dense representations without relying on the corpus semi-structure. Qi et al. (2019, 2020) iteratively generate the query from the question and retrieved history, and use traditional sparse ir systems to select the passage, which complements BeamDR's approach.

## 6 Conclusion

We introduce a simple yet effective multi-step dense retrieval method, BeamDR. By conducting beam search and globally refreshing negative chains during training, BeamDR finds reasoning chains in dense space. BeamDR is competitive to more complex sota systems albeit not using semi-structured information.

While BeamDR can uncover relationship embedded within a single question, future work should investigate how to use these connections to resolve ambiguity in the question Elgohary et al. (2019); Min et al. (2020), resolve entity mentions Guha et al. (2015), connect concepts across modalities Lei et al. (2018), or to connect related questions to each other Elgohary et al. (2018).

## Acknowledgments

We thank the anonymous reviewers and metareviewer for their suggestions and comments. Zhao is supported by the Office of the Director of National Intelligence (odni), Intelligence Advanced Research Projects Activity (iarpa), via the better Program contract 2019-19051600005. Boyd-Graber is supported by NSF Grant IIS-1822494. Any opinions, findings, conclusions, or recommendations expressed here are those of the authors and do not necessarily reflect the view of the sponsors.

\begin{table}
\begin{tabular}{p{370.0pt}} \hline \multicolumn{2}{p{370.0pt}}{**Q: Chris Williams last played for which football club from the National League North?**} \\
**Passage 1: Christopher Jonathan ”Chris”Williams is an English semi–professional footballer who last played for Salford City as a forward.** \\
**Passage 2: Salford City Football Club is a professional football club in the Kersal area of Safford, Greater Manchester, England.** \\
**BeamDR: Chris Williams (English Footballer)**\(\rightarrow\) Salford City F.C. ✓ \\
**grr: Chris Williams (Wide Receiver)**\(\rightarrow\) Miami Dolphins ✗ \\ \hline \end{tabular}
\end{table}
Table 5: Case study of BeamDR and grrr retrieval. Term-based retrieval approaches (tf-idf used by grr) is unable to distinguish two players with same name. BeamDR correctly identifies the question entity.

 



## References

* Asai et al. (2020) Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. 2020. Learning to retrieve reasoning paths over wikipedia graph for question answering. In _Proceedings of the International Conference on Learning Representations_.
* Bengio et al. (2013) Y. Bengio, A. Courville, and P. Vincent. 2013. Representation learning: A review and new perspectives. _IEEE Transactions on Pattern Analysis and Machine Intelligence_.
* Chang et al. (2009) Jonathan Chang, Jordan Boyd-Graber, and David M. Blei. 2009. Connections between the lines: Augmenting social networks with text. In _Knowledge Discovery and Data Mining_.
* De Melo and Weikum (2009) Gerard De Melo and Gerhard Weikum. 2009. Towards a universal WordNet by learning from combined evidence. In _Proceedings of the ACM International Conference on Information and Knowledge Management_.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Conference of the North American Chapter of the Association for Computational Linguistics_.
* Dhingra et al. (2020) Bhuwan Dhingra, Manzil Zaheer, Vidhisha Balachandran, Graham Neubig, Ruslan Salakhutdinov, and William W. Cohen. 2020. Differentiable reasoning over a virtual knowledge base. In _International Conference on Learning Representations_.
* Elgohary et al. (2019) Ahmed Elgohary, Denis Peskov, and Jordan Boyd-Graber. 2019. Can you unpack that? learning to rewrite questions-in-context. In _Proceedings of Empirical Methods in Natural Language Processing_.
* Elgohary et al. (2018) Ahmed Elgohary, Chen Zhao, and Jordan Boyd-Graber. 2018. Dataset and baselines for sequential open-domain question answering. In _Proceedings of Empirical Methods in Natural Language Processing_.
* Ferrucci et al. (2010) David Ferrucci, Eric Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya A Kalyanpur, Adam Lally, J William Murdock, Eric Nyberg, John Prager, et al. 2010. Building watson: An overview of the deepqa project. _AI magazine_.
* Guha et al. (2015) Anupam Guha, Mohit Iyyer, Danny Bouman, and Jordan Boyd-Graber. 2015. Removing the training wheels: A coreference dataset that entertains humans and challenges computers. In _Conference of the North American Chapter of the Association for Computational Linguistics_.
* Gou et al. (2020) Kelvin Gau, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. REALM: Retrieval-augmented language model pre-training. In _Proceedings of the International Conference of Machine Learning_.
* fyyer et al. (2016) Mohit Iyyer, Anupam Guha, Sniglha Chaturvedi, Jordan Boyd-Graber, and Hal Daume III. 2016. Feuding families and former friends: Unsupervised learning for dynamic fictional relationships. In _North American Association for Computational Linguistics_.
* Johnson et al. (2017) Jeff Johnson, Matthijs Douze, and Herve Jegou. 2017. Billion-scale similarity search with GPUs. _arXiv preprint arXiv:1702.08734_.
* Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In _Proceedings of Empirical Methods in Natural Language Processing_.
* Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In _Proceedings of the Association for Computational Linguistics_.
* Lei et al. (2018) Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. 2018. TVQA: Localized, compositional video question answering. In _Proceedings of Empirical Methods in Natural Language Processing_.
* Litttman et al. (2002) Michael L Littman, Greg A Keim, and Noam Shazeer. 2002. A probabilistic approach to solving crossword puzzles. _Artificial Intelligence_, 134(1).
* van der Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. _Journal of machine learning research_, 9(11).
* Min et al. (2020) Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering ambiguous open-domain questions. In _Proceedings of Empirical Methods in Natural Language Processing_.
* Nie et al. (2019) Yixin Nie, Songhe Wang, and Mohit Bansal. 2019. Revealing the importance of semantic retrieval for machine reading at scale. In _Proceedings of Empirical Methods in Natural Language Processing_.
* Nogueira and Cho (2019) Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with BERT. _arXiv preprint arXiv:1901.04085_.
* Qi et al. (2020) Peng Qi, Haejun Lee, Ogbenetegiri "TG" Sido, and Christopher D. Manning. 2020. Retrieve, rerank, read, then iterate: Answering open-domain questions of arbitrary complexity from text. _arXiv preprint arXiv:2010.12527_.
* Qi et al. (2019) Peng Qi, Xiaowen Lin, Leo Mehr, Zijian Wang, and Christopher D. Manning. 2019. Answering complex open-domain questions through iterative query generation. In _Proceedings of Empirical Methods in Natural Language Processing_.
* Salton (1968) Gerard. Salton. 1968. _Automatic Information Organization and Retrieval_. McGraw Hill Text.



Anshumali Shrivastava and Ping Li. 2014. Asymmetric lsh (ALSH) for sublinear time maximum inner product search (MIPS). In _Proceedings of Advances in Neural Information Processing Systems_.
* Talmor and Berant [2018] Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In _Conference of the North American Chapter of the Association for Computational Linguistics_.
* Xiong et al. [2021a] Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2021a. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In _Proceedings of the International Conference on Learning Representations_.
* Xiong et al. [2021b] Wenhan Xiong, Xiang Lorraine Li, Srini Iyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Wen tau Yih, Sebastian Riedel, Douwe Kiela, and Barlas Oguz. 2021b. Answering complex open-domain questions with multi-hop dense retrieval. In _Proceedings of the International Conference on Learning Representations_.
* Yang et al. [2018] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In _Proceedings of Empirical Methods in Natural Language Processing_.
* Zhang et al. [2018] Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander Smola, and Le Song. 2018. Variational reasoning for question answering with knowledge graph. In _Proceedings of the Association for the Advancement of Artificial Intelligence_.
* Zhao et al. [2020a] Chen Zhao, Chenyan Xiong, Xin Qian, and Jordan Boyd-Graber. 2020a. Complex factoid question answering with a free-text knowledge graph. In _Proceedings of the World Wide Web Conference_.
* Zhao et al. [2020b] Chen Zhao, Chenyan Xiong, Corby Rosset, Xia Song, Paul Bennett, and Saurabh Tiwary. 2020b. Transformer-xh: Multi-evidence reasoning with extra hop attention. In _Proceedings of the International Conference on Learning Representations_.