Hybrid Hierarchical Retrieval for Open-Domain Question Answering
Year: 2023

Abstract:
Retrieval accuracy is crucial to the performance of open-domain question answering (ODQA) systems. Recent work has demonstrated that dense hierarchical retrieval (DHR), which retrieves document candidates first and then relevant passages from the refined document set, can significantly outperform the single stage dense passage retriever (DPR). While effective, this approach requires document structure information to learn document representation and is hard to adopt to other domains without this information. Additionally, the dense retrievers tend to generalize poorly on out-of-domain data comparing with sparse retrievers such as BM25. In this paper, we propose Hybrid Hierarchical Retrieval (HHR) to address the existing limitations. Instead of relying solely on dense retrievers, we can apply sparse retriever, dense retriever, and a combination of them in both stages of document and passage retrieval. We perform extensive experiments on ODQA benchmarks and observe that our framework not only brings in-domain gains, but also generalizes better to zero-shot TriviaQA and Web Questions datasets with an average of 4.69% improvement on recall@100 over DHR. We also offer practical insights to trade off between retrieval accuracy, latency, and storage cost. The code is available on github∗.

heading: 1 Introduction
text: Open-domain question answering (ODQA) (Voorhees, 1999) aims to answer questions based on a large corpus without pre-specified context, and enjoys a broad scope of real-world applications such as chatbots, virtual assistants, search engines, etc. Recent ODQA systems often follow a two stage retrieve-then-read architecture (Zhu et al., 2021; Chen et al., 2017; Lee et al., 2019). Given a question, a retriever module first selects a
+The work was done during an internship at AWS AI Lab. †These authors contributed equally to this work. ∗https://github.com/ghuhan17/HybridHierarchicalRetrieval.
candidate set of relevant contexts from a diversified large corpus such as Wikipedia; afterwards, a reader module consumes the retrieved evidence to predict an answer. Here, retrieval performance is crucial to the accuracy of the QA system as it determines whether the correct context to answer the question can be presented to the reader.
While most work in information retrieval focus on document retrieval (Nguyen et al., 2016; Thakur et al., 2021), existing work in ODQA often splits documents into short passages and directly retrieve passages for the reader (Karpukhin et al., 2020; Izacard and Grave, 2020) to accommodate reader models that handle shorter sequences most effectively. A drawback of such single-stage passage retrieval approaches is that they tend to be susceptible to distracting passages that contain seemingly relevant local context but not the correct answer, since they cannot incorporate information from other parts of the document (see Figure 1). Further, the large number of passage candidates also contributes negatively to system throughput. To mitigate these issues, Liu et al. (2021) recently proposed a two-stage hierarchical retrieval framework where the retriever first retrieves relevant documents, then discern relevant passages within re-
10680
trieved documents. This helps prune passages that look relevant but are from irrelevant documents to improve answer accuracy, meanwhile greatly reducing the candidate set for passage retrieval and improve the inference speed of ODQA systems.
Despite its success, Liu et al.’s (2021) approach (dense hierarchical retrieval, DHR) relies on dense neural retrievers (Lee et al., 2019; Karpukhin et al., 2020) for both document retrieval and passage retrieval, which suffers from two key weaknesses. First, neural encoders used in retrieval are often limited in context length for effectiveness and efficiency, which is too short to encompass most documents. As a result, DHR needs to make use of the structure of Wikipedia documents, and represents the documents succinctly with title, abstract and table of contents, which is not always available for non-Wikipedia text. Second, dense retrievers have been shown to suffer from poor generalization on out-of-domain data (Thakur et al., 2021), whereas sparse retrievers like BM25 (Robertson et al., 2009) excel with lexical matches (Sciavolino et al., 2021).
In this work, we propose a hybrid hierarchical retrieval (HHR) framework to alleviate these issues. Specifically, we investigate the tradeoffs and complementary strengths of sparse retrievers and dense retrievers at both the document retrieval and passage retrieval stages for ODQA (see Figure 2). We find, among other things, that sparse retrievers can complement dense retrievers at both retrieval stages with a simple approach to aggregate results from both. Besides in-domain evaluation on the dataset that neural models are trained on, we also perform zero-shot evaluation on unseen datasets to compare the generalization of these retriever architectures. We find that sparse retrievers can help HHR generalize better to unseen data and potentially replace dense retrievers in document retrieval. In addition, we also study the accuracy, storage
cost, and latency tradeoff for these architectures under the HHR framework, and offer practical insights to real-world ODQA systems that often need to factor these into consideration.
Our main contributions are as follows. First, we propose a hybrid hierarchical retrieval framework on ODQA, and extensively study tradeoffs and complementary strengths of sparse and dense retrievers in both document and passage retrieval. Second, we perform both in-domain and out-ofdomain evaluation to provide insight into the generalization performance of different model choices. Finally, we present the accuracy-storage-latency landscape for HHR architectures and offer practical insights to real-world applications.

heading: 2 Background & Related Work
text: Open-domain question answering (ODQA). ODQA is a task that takes a question, such as “Who got the first Nobel prize in physics?”, and aims to find the answer from a large corpus. ODQA systems often rely on efficient and accurate retrievers to find relevant context to answer questions (Chen et al., 2017), where retrieval performance is usually critical to QA accuracy (Karpukhin et al., 2020).
Passage retrieval. Since most reader models in ODQA systems struggle to effectively handle long contexts, ODQA retrieval is often performed at the passage level (usually around 100 words long). Earlier work (Chen et al., 2017; Yang et al., 2019) relied on bag-of-words-based sparse retrievers such as BM25 (Robertson et al., 2009). More recent work showed that neural retrievers can generate effective dense representations for retrieval when trained on ODQA (Lee et al., 2019; Karpukhin et al., 2020; Liu et al., 2021). Sciavolino et al. (2021) showed, however, that these dense retrievers tend to generalize worse to unseen entities during training since they lack the capacity for lexical matching, which is a strong suit for sparse retrievers and important for out-of-domain generalization.
Hierarchical retrieval. Passage retrievers are limited by the context available in each passage, and can retrieve spurious passages to hurt answer performance. A remedy is to incorporate documentlevel relevancy during passage retrieval. Qi et al. (2021) explored combining document and passage relevancy scores in a BM25 retriever for ODQA. Liu et al. (2021) applied this idea to dense retrievers with a hierarchical retrieval framework (DHR),
where a document retriever first retrieves documents of high relevancy, followed by a passage retriever to rerank passages within those documents, and our work extends this approach.

heading: 3 Methodology
text: Our hybrid hierarchical retrieval (HHR) framework extends DHR, a hierarchical retriever built on dense retrievers that first retrieves top-kd documents and then top-kp passages from those documents. We follow DHR to build the dense retrievers in HHR, and expand both document and passage retrievers to work with sparse retrievers to address limitations of the DHR approach (Figure 2). Specifically, in DHR, to make documents amenable to neural encoders with limited context length, the authors proposed to leverage the document structure of Wikipedia articles to construct a document summary that contains the document abstract and table of contents. While effective, this also potentially limits the applicability of this approach to corpora where this information is not available. In contrast, a sparse retriever can easily handle documents of arbitrary lengths efficiently without the need of structure information. Besides, dense retrievers tend to generalize poorly to out of domain data. We extend each of the document retrieval and passage retrieval stages with the option to use sparse retrievers to help alleviate this issue, and to help us understand the tradeoff between the two.
Besides switching between sparse and dense retrievers in HHR, we also introduce a simple heuristic to combine results from both retrievers at the same stage by simply interleaving their top-k/2 results for top-k retrieval to better understand the complementary strengths of sparse and dense retrievers. This yields a total of 9 possible configurations for HHR for our extensive studies. Finally, for both sparse and dense passage retrievers in HHR, we implement on-the-fly passage reranking for all passages in the top retrieved documents with pre-computed passage representations. This helps reduce the latency of the passage retrievers in our implementation and provide more realistic insights into the accuracy-storage-latency tradeoff of different HHR settings in real-world systems.

heading: 4 Experimental Setup
text: Data Three commonly used ODQA datasets are considered: Natural Questions (NQ) (Kwiatkowski et al., 2019), Web Questions (WebQ) (Berant et al.,
2013), and TriviaQA (Joshi et al., 2017). Following Liu et al. (2021), we use the Wikipedia dump from Dec. 20, 2018 as the source corpus. When splitting documents into passages, the text under same sections are split into non-overlapping passages with a maximum of 100 words.
Training and evaluation We followed the same configuration as DHR to train the document and passage encoders up to 40 epochs on 8 V100 Tensor Core GPUs. In order to measure the retriever framework in both in-domain and zero-shot settings, the dense encoders are trained only on NQ dataset and tested on all three datasets.

heading: 5 Experiments and Results

heading: 5.1 Main Results
text: We present the in-domain and zero-shot evaluation results for all datasets in Table 1. Following DHR, we retrieve 100, 500, 500 documents in the first stage for NQ, WebQ, and TriviaQA, respectively. We compare HHR against DHR and single stage sparse and dense passage retrievers. In addition, we also study the effect of retrieving varying numbers of documents in HHR (see Figure 3). We find that:
First, dense retrievers are crucial for indomain performance, yet sparse retrievers bring gains with complementary strengths. We see that replacing either or both components in the DHR baseline (Dense+Dense) with sparse retriever leads to noticeable drops in recall@100 on NQ, with 1.7% for replacing the document retriever, 2.3% for the passage retriever, and 7.2% for both. However, adding sparse retriever to document, passage or both retrievers brings 2.0%, 0.9% and 2.4% gain.
Second, sparse retrievers significantly improve HHR’s generalization on zero-shot datasets. We corroborate previous work’s (Sciavolino et al., 2021) finding that dense retrievers struggle to generalize in zero-shot settings. Likewise, DHR underperforms the optimal setting by 3.79% on WebQ, and leads to the worst performance on TriviaQA per recall@100. Adding sparse retrievers to both stages, Combined+Combined brings an average of 4.69% recall@100 improvement on WebQ and TriviaQA.
Third, sparse document retriever can completely replace dense document retriever. While this leads to performance drop in-domain, we see that dense passage retrievers can help make up for the performance gap, and that the gap diminishes
as more documents are retrieved in the first stage (Figure 3). Further, in the zero-shot setting, replacing dense document retrievers with sparse ones can actually lead to gains (of 1.26% and 3.15% recall@100 on WebQ and TriviaQA, respectively). This helps HHR generalize better to out-of-domain data not only lexically, but also removes the requirement of document structure information needed by the dense document retriever.

heading: 5.2 Accuracy-Storage-Latency Landscape
text: We report the trade off between retrieval accuracy, latency and storage cost for different HHR configurations based on inference on a single CPU core. Figure 4, presents the accuracy-storage-latency plot of different retriever configurations for the NQ, WebQ and TriviaQA datasets. We find that in the in-domain case of NQ, all settings involving dense document retrievers are Pareto-efficient on
accuracy and latency, while the passage retriever presents a tradeoff among accuracy, latency, and storage. In contrast, in zero-shot settings the sparse document retrievers can be Pareto-efficient when complemented with dense passage retrievers. Table 2 presents the storage cost and retrieval latency for the four essential components in HHR framework, namely, sparse and dense retrievers in document and passage retriever stages. Sparse retrievers are storage efficient compared to the dense retrievers at the same level as they use inverted index to represent the text whereas dense retrievers use the dense embeddings. We also observe that the PyLucene’s sparse document retriever is slower than the FAISS dense document retriever whereas our implementation of the on-the-fly sparse passage retriever is faster than its dense counterpart. Dense passage retrieval takes the most storage cost to store the embedding dictionary among others.

heading: 6 Conclusion
text: In this paper, we study a Hybrid Hierarchical Retrieval (HHR) framework for ODQA that integrates sparse and dense retrievers through a simple aggregation strategy in document and passage retrievers. We demonstrate that sparse retrievers complement dense retrievers in-domain and greatly improve the poor generalization from dense encoders to zeroshot data. Our framework addresses the limitation of DHR by achieving better zero-shot performance without relying on document structure information. We also study the accuracy-storage-latency land-
scape. We believe these findings are critical to the real-world adoption of HHR to ODQA systems.

heading: 7 Limitations
text: Our HHR framework uses a simple combination strategy to take top-k/2 documents (or passages) from dense and sparse retrievers for top-k retrieval in the combined setting. The overlap between dense and sparse results can lead to less than k retrieved results to be consumed by next stage. Future work might improve the aggregation strategy by: 1) developing a more advanced combination strategy that is not solely based on rank but also retrieval scores, and 2) accounting document retriever scores in the passage retrieval stage to rerank the passages according to both the global document relevancy and local passage relevancy. Futhermore, we only evaluate against Wikipedia ODQA datasets due to its presence of document structure information to be used in the dense document retriever. However, future work can extend the evaluation to other ODQA corpus.

heading: B Similarities and Differences between different HHR configurations
text: In this section, we analyze pairwise HHR configurations to understand their similarities and differences. Table 3 shows the percentage of queries in the NQ, WebQ and TriviaQA datasets respectively for which the correct evidences are retrieved exclusively by the row HHR configuration but not by the column HHR configuration. Higher percentage for a given pair of HHR configuration indicates that the row configuration is more exclusive and is able to answer a different set of queries when compared to the column configuration.
We can observe the following high-level takeaways: 1) Observing the Sparse+Sparse row in all the 3 tables indicate that Sparse retrievers are more effective and exclusively retrieve correct evidence for more queries in zero-shot settings as the percentages are more for WebQ and TriviaQA compared to NQ dataset. 2) Observing the Combined+Combined column shows that there are only few percentage of queries that
∗https://lucene.apache.org/pylucene/
other HHR configurations are able to answer over Combined+Combined. This is expected as Combined+Combined leverages both sparse and dense retrievers in both the stages. Out of other HHR configurations, Sparse+Dense seems to have the highest percentage over Combined+Combined. Ensembling the results of these two configuration might result in an improved performance over Combined+Combined and can be an interesting future work.
Table 4 shows for a given pair of HHR configurations, the percentage of of queries, out of the total correctly retrieved queries by the row configuration, for which the column configuration also retrieves the correct evidence. Higher percentage indicates that the column HHR configuration is also able to correctly retrieve evidences for the queries as the row configuration. In case of NQ dataset, we observe that configurations that have dense retrievers in both the stages are able to correctly retrieve evidences for most of the queries of the other configurations. However, this percentage decreases in the case of WebQ and TriviaQA datasets.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 7
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3, Section 4 and Section 5
3 B1. Did you cite the creators of artifacts you used? Section 3, Section 4 and Section 5
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 4
7 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. We follow standard dataset splits
C 3 Did you run computational experiments? Section 4 and Section 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 3, Section 4, and Section 5
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4 and Appendix A
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Ensembling the results of these two configuration might result in an improved performance over Combined+Combined and can be an interesting future work. Table 4 shows for a given pair of HHR configurations, the percentage of of queries, out of the total correctly retrieved queries by the row configuration, for which the column configuration also retrieves the correct evidence. Higher percentage indicates that the column HHR configuration is also able to correctly retrieve evidences for the queries as the row configuration. In case of NQ dataset, we observe that configurations that have dense retrievers in both the stages are able to correctly retrieve evidences for most of the queries of the other configurations. However, this percentage decreases in the case of WebQ and TriviaQA datasets. ACL 2023 Responsible NLP Checklist