Expand, Rerank, and Retrieve: Query Reranking for Open-Domain Question Answering
Year: 2023

Abstract:
We propose EAR, a query Expansion And Reranking approach for improving passage retrieval, with the application to open-domain question answering. EAR first applies a query expansion model to generate a diverse set of queries, and then uses a query reranker to select the ones that could lead to better retrieval results. Motivated by the observation that the best query expansion often is not picked by greedy decoding, EAR trains its reranker to predict the rank orders of the gold passages when issuing the expanded queries to a given retriever. By connecting better the query expansion model and retriever, EAR significantly enhances a traditional sparse retrieval method, BM25. Empirically, EAR improves top-5/20 accuracy by 3-8 and 5-10 points in in-domain and out-of-domain settings, respectively, when compared to a vanilla query expansion model, GAR, and a dense retrieval model, DPR.1

heading: 1 Introduction
text: Open-domain question answering (QA) (Chen and Yih, 2020), a task of answering a wide range of factoid questions of diversified domains, is often used to benchmark machine intelligence (Kwiatkowski et al., 2019) and has a direct application to fulfilling user’s information need (Voorhees et al., 1999). To provide faithful answers with provenance, and to easily update knowledge from new documents, passage retrieval, which finds relevant text chunks to given questions, is critical to the success of a QA system. Retrieval in early opendomain QA systems (Chen et al., 2017) is typically based on term-matching methods, such as BM25 (Robertson et al., 2009) or TF-IDF (Salton et al., 1975). Such methods are sometimes called sparse retrievers, as they represent documents and queries with high-dimensional sparse vectors, and can efficiently match keywords with an inverted
1Source code: https://github.com/voidism/EAR.
index and find relevant passages. Despite their simplicity, sparse retrievers are limited by their inability to perform semantic matching for relevant passages that have low lexical overlap with the query. Lately, dense retrievers (Karpukhin et al., 2020), which represent documents and queries with dense, continuous semantic vectors, have been adopted by modern QA systems. Dense retrievers usually outperform their sparse counterparts, especially when there exists enough in-domain training data.
However, dense retrievers have certain weaknesses compared to sparse ones, including: 1) being computationally expensive in training and inference, 2) potential information loss when compressing long passages into fixed-dimensional vectors (Luan et al., 2021), which makes it hard to match rare entities exactly (Sciavolino et al., 2021), and 3) difficulty in generalizing to new domains (Reddy et al., 2021). As a result, dense retrievers and sparse ones are usually complementary to each other and can be combined to boost performance. Recent studies on query expansion, such as GAR (Mao et al., 2021a), have attempted to improve sparse retrievers by adding relevant contexts to the query using pre-trained language models (PLMs), which has been shown effective in closing the gap between sparse and dense retrievers.
In this paper, we introduce a novel query Expansion And Reranking approach, EAR, which enhances generative query expansion with query reranking. EAR first generates a diverse set of expanded queries with query expansion models, and then trains a query reranker to estimate the quality of these queries by directly predicting the rank order of a gold passage, when issuing these queries to a given retriever, such as BM25. At inference time, EAR selects the most promising query expansion as predicted by the query reranker and issues it to the same retriever to find relevant documents. EAR is motivated by a simple observation—while the greedy decoding output of a query expansion
12131
model, such as GAR, could be suboptimal, some randomly sampled query expansions achieve superior performance with BM25 (see Section 2.2). EAR better connects the query expansion model and the underlying retrieval method, and thus can select a more suitable query.
We empirically evaluated EAR in both in-domain and cross-domain settings. Our in-domain experimental results on Natural Questions and TriviaQA show that EAR significantly improves the top-5/20 accuracy by 3-8 points. For the cross-domain setting, while the query expansion model suffers from substantial performance degradation when applied to new domains, EAR seems to be more domainagnostic, and can still find useful queries from a diverse set of query expansions, which leads to a significant improvement over GAR and DPR by 5-10 points for top-5/20 accuracy. Our contributions can be summarized as follows:
• We proposed EAR to select the best query from a diverse set of query expansions, by predicting which query can achieve the best BM25 result. This improves the connection of query expansion models and BM25, resulting in enhanced performance that surpasses DPR. • EAR not only performs well on in-domain data, but also shows strong generalization abil-
ities on out-of-domain data, outperforming GAR and DPR by a large margin. • End-to-end evaluation with a generative reader demonstrates the benefits of EAR in improving the exact match score. • Lastly, we show that the improvements provided by EAR and passage reranking are complementary, allowing for effective aggregation of performance gains from both methods.

heading: 2 Background

heading: 2.1 Generation-Augmented Retrieval
text: Generation-Augmented Retrieval (GAR) (Mao et al., 2021a) aims to enhance sparse retrievers by query expansion with text generation from PLMs. Given the initial query, GAR generates relevant contexts including the answer, answer sentence, and title of answer passages, and then concatenates them to the initial query before performing retrieval with BM25. GAR achieves decent performance close to that of DPR while using the lightweight BM25 retriever. However, a limitation is that GAR is not aware of the existence of BM25, potentially generating suboptimal queries for retrieval. Additionally, GAR is only trained on in-domain data, limiting their ability to transfer to out-of-domain data.

heading: 2.2 Preliminary Experiments
text: Let us first take a look at some preliminary experimental results to better understand the motivation of this paper. In Table 1, we present the top-k retrieval results on Natural Questions (Kwiatkowski et al., 2019) for BM25, DPR, and GAR (greedy/beam search) in rows 1-4. To investigate the potential of GAR, we randomly sampled 50 query expansions from GAR, ran BM25 separately for these queries, and chose the best one by looking at the BM25 results, which requires ground truth labels. The resulting scores are shown in row 5 (GAR best query).
From the results, we see that GAR best query can lead to a significant improvement of up to 20 points compared to DPR. Since we do not have access to labels for selecting the best query in reality, a naive solution is to concatenate all 50 query expansions together as a single, long query, which will definitely include high-quality expansions if they exist. However, as shown in row 6, the performance of GAR concat is even worse than that of GAR alone with greedy decoding outputs. This indicates that the single long query may include too much distracting information, negatively impacting the performance of the BM25 retriever.
From these preliminary results, we reach two conclusions: 1) GAR does have the ability to generate very useful query expansions; 2) however, the useful query expansions may not always be included in the GAR greedy decoding outputs. It is non-trivial to extract these useful query expansions from GAR. Motivated by these findings, we leverage a query reranker to estimate if a query will be beneficial to BM25 retrieval results, so as to unlock the potential of GAR and sparse retrievers.

heading: 3 Proposed Method
text: We illustrate our proposed method, EAR, in Figure 1, along with a comparison with the BM25 and GAR pipelines. Given the original query q, EAR first generates a set of query expansions E = {e1, e2, ..., en} using random sampling. We believe that among these n queries, some may achieve very good retrieval performance. Thus, we train a reranker model M to re-score all the queries. Here we propose two kinds of rerankers: 1) retrieval-independent (RI) reranker, and 2) retrievaldependent (RD) reranker. Both rerankers can estimate the quality of a query expansion without using information from answer annotations.

heading: 3.1 Retrieval-Independent (RI) Reranker
text: The inputs to the RI reranker are quite simple: (q, ei), which consists of the original query q and one of the query expansions ei. When training this reranker, we first obtain the minimum answer passage ranking among all retrieved passages for each query, when issued to a BM25 retriever. We denote this minimum answer passage ranking as R = {r1, r2, ..., rn}, which corresponds to each of the expanded queries {(q, e1), (q, e2), ..., (q, en)}.
To clarify the concept, let us consider an example with two query expansions, e1 and e2. Say the ex-
panded query (q, e1) retrieves the answer passage as the top result (first position), we assign r1 = 1. Similarly, we assign r2 = 15 if the expanded query (q, e2) retrieves the answer passage in the 15th position. In this case, we conclude that e1 is a better query expansion than e2 since its corresponding ranking value, r1, is lower than r2. ri can be seen as the score that can be obtained by the query of (q, ei), with smaller ri corresponding to better quality of (q, ei). We now train a scoring model to estimate the rank ri for given inputs (q, ei). However, considering that the scoring model will be used as a reranker, we only need to ensure the model’s relative accuracy of estimating ri, rather than its absolute value. Thus, we employ a “contrastive” loss rather than a “regression” loss, which is inspired by the contrastive method in summarization re-scoring (Liu and Liu, 2021).
For all pairs of query expansions (ei, ej) such that ri < rj (which means ei is a better expansion than ej), the ranking loss is calculated as follows:
LRank =∑
i,j∈[1,n] ri<rj
max(0,M(q, ei)−M(q, ej)+(rj−ri)·α)
Here, M is a model that estimates the rank ri for a given query expansion ei. Instead of predicting the absolute rank of ri, the model M is trained to predict the difference between ri and rj for each pair of expansion (ei, ej).
The ranking loss LRank forces the model to estimate a lower rank for ei and a higher rank for ej , such that the difference between M(q, ei) and M(q, ej) is greater than the threshold (rj − ri) ·α, where α is a scalar. If some of the expansions do not retrieve the answer passages within the top-k results (e.g. within the top-100 results), we assign a constant value, MAX_RANK, to these expansions.

heading: 3.2 Retrieval-Dependent (RD) Reranker
text: The input to the RI Reranker only contains the original query q and the expansion ei, which may not be sufficient to distinguish good expansions from bad expansions. For example, in Figure 1 (c), for the original query Where do they grow hops in the US?, it is easy to tell that Central and South America is a bad expansion because the US is not in Central and South America. However, for these two expansions: 1) Colorado, Arizona 2) Oregon, Idaho, Washington, it is very hard to tell which one is better without any external knowledge. To alleviate
this problem, we propose the Retrieval-Dependent (RD) Reranker, which is able to see the top-1 passages D = {d1, d2, ..., dn} retrieved by each query expansion. 2 The inputs of RD reranker will contain the original query q, the query expansions ei, and the top-1 passage di. We train RD reranker with the same ranking loss LRank, but replace the model with M(q, ei, di).

heading: 3.3 Training Examples Construction
text: To construct training examples, we generate diverse query expansions, run BM25 retrieval on them, and train the rerankers based on the results. However, using the GAR generators directly may not yield diverse sequences and limit the rerankers’ learning, since the GAR generators are trained with supervision and may have already overfit on the training set, which would lead to almost identical generation samples. To address this, we propose two alternatives: 1) Split the training set into K subsets, train K different GAR generators on (K-1) subsets and randomly sample from the remaining subset; and 2) Use a large language model (LLM) such as T0 (Sanh et al., 2021) to randomly sample query expansions directly without fine-tuning. Both options performed equally well in our experiments and will be further discussed in Section 6.1.

heading: 4 Experiments

heading: 4.1 Data
text: For in-domain experiments, we use two public datasets for training and evaluation: Natural Questions (NQ) (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017). For out-of-domain (cross-dataset) experiments, we directly evaluate our in-domain models on three additional public datasets without using their training sets: WebQuestions (WebQ) (Berant et al., 2013), CuratedTREC (TREC) (Baudiš and Šedivỳ, 2015), and EntityQuestions (EntityQs) (Sciavolino et al., 2021). (See dataset statistics in Appendix A.) All experiments are performed with Wikipedia passages used in DPR (Karpukhin et al., 2020), consisting of 21M 100-word passages from the English Wikipedia dump of Dec. 20, 2018 (Lee et al., 2019).
2For RD reranker, we need additional computational costs to retrieve the top-1 passage. However, this process can be efficiently parallelized for all queries using a lightweight BM25 retriever, so the required time is not excessive. We will discuss the latency of EAR further in Section 7.

heading: 4.2 Setup
text: Model For sparse retrieval, we use Pyserini (Lin et al., 2021) for BM25 with its default parameters. For query rerankers, we use the DeBERTa V3 base (He et al., 2021) model from Huggingface Transformers (Wolf et al., 2020). For RI reranker, the input format is: [CLS] <question> ? <expansion> [SEP]; for RD reranker, the input format is [CLS] <question> ? <expansion> [SEP] <top-1 retrieved passage> [SEP]. Training details can be found in Appendix B.
Context Generator At training time, we use T03B (Sanh et al., 2021) to randomly sample 50 query expansions per question, as we mentioned in Section 3.3. We add a short prompt, To answer this question, we need to know, to the end of the original question, and let T0-3B complete the sentence. During inference, we still use the GAR generators to randomly sample 50 query expansions per question on the testing set, since the examples are not seen during GAR training and the generations are diverse enough. To speed up the inference process, we de-duplicate the query expansions that appear more than once. The average number of query expansions we use is 25 for Natural Questions and 34 for TriviaQA, respectively.

heading: 4.3 Baselines
text: We compare EAR with 1) DPR (Karpukhin et al., 2020): a standard BERT-based dense retriever; 2) BM25 (Robertson et al., 2009): a standard sparse retriever based on term matching; 3) GAR (Mao et al., 2021a): generation-augmented retrieval with BM25; 4) Liu et al. (2022): a concurrent work that uses a GAR-like generative model to perform beam search decoding, followed by filtering to obtain multiple expanded queries for performing multiple retrievals with BM25, and then fusion of the results; and 5) SEAL (Bevilacqua et al., 2022): an autoregressive search engine, proposing constrained decoding with the FM-index data structure that enables autoregressive models to retrieve passages.

heading: 4.4 Result: In-Domain Dataset
text: We first train and evaluate EAR on NQ and TriviaQA. In Table 2, we see that both EAR-RI and EARRD improve the performance of GAR significantly. EAR-RI improves the top-5/20/100 accuracy of GAR by 1-2 points, while EAR-RD improves the top-5 accuracy of GAR by 6-8 points, and the top20 accuracy by 3-5 points on both datasets. More-
over, EAR-RD is significantly better than DPR except for the top-20 accuracy on NQ. These results show that it is possible for BM25 to beat dense retrieval with the help of an optimized process to generate high-quality query expansions. Additional qualitative studies in Appendix E provide further insight into how EAR works. We also report the results of the best query from GAR, which presents the potential performance upper bound that could be achieved by query reranking. It suggests that there is still room for EAR to improve if mechanisms for more effective query selection are developed. At the bottom of Table 2, we present the fusion retrieval results of combining EAR and DPR. EAR-RD+DPR outperforms the fusion results of BM25/GAR/Liu et al. (2022), showing the complementarity between EAR-RD and DPR.

heading: 4.5 Result: Cross-Dataset Generalization
text: To better evaluate the robustness of these models for out-of-domain examples, we train our models only on NQ or TriviaQA, and then test them on WebQ, TREC, and EntityQs in a zero-shot manner. The results are shown in Table 3. We observe that when transferring from NQ or TriviaQA, DPR experiences a decline in performance compared to in-domain supervised training on WebQ.3 GAR performs even worse than DPR on both WebQ and
3The in-domain DPR performs poorly on TREC since it only has 1,125 training examples.
TREC. However, GAR performs better than DPR on EntityQs, which is designed to challenge dense retrieval by including many rare entities. Here we also present the performance of GAR best query. We see that although GAR transfers poorly on crossdomain datasets, it still has the ability to generate high-quality query expansions by random sampling. This provides an opportunity for EAR to improve performance. After adopting EAR, we see that EAR-RI improves the performance of GAR by 2-4 points for top-5/20 accuracy, and EAR-RD further boosts the performance of GAR by 5-10 points for top-5/20 accuracy. Overall, EAR-RD outperforms DPR except when transferring from TriviaQA to WebQ.
These results suggest that query reranking is a general technique that can work well even on outof-domain examples, showing that generating relevant contexts (GAR) is largely dependent on the domains, while judging which contexts may be more beneficial to retriever is a more domain-agnostic skill.

heading: 4.6 Result: End-to-end QA with FiD
text: To fully understand whether EAR can benefit endto-end QA systems, we further evaluate the exact match scores with Fusion-in-Decoder (FiD) (Izacard and Grave, 2021), a generative reader model trained from T5-large (Raffel et al., 2020). We take the FiD models that were pre-trained on
NQ/TriviaQA and directly test on our retrieval results without further fine-tuning. The exact match scores using the top-100 retrieved passages as input to FiD is shown at the top of Table 4. We observe that EAR consistently outperforms previous work, including DPR, GAR, SEAL, and Liu et al. (2022), on both NQ and TriviaQA. Although these gains may appear relatively small, however, this is primarily due to FiD’s ability to take the top-100 retrieved passages as input and generate answers using cross-attention across all passages. Thus, even with low-ranked answer passages (say
the answer is in the 99th passage), it is still possible that FiD could produce correct answers.
As there are many methods where relatively smaller context windows compared to FiD are used, especially when models are scaled up and crossattention becomes much more expensive, improving retrieval accuracy for smaller k may be beneficial. For example, GPT-3 (Brown et al., 2020) only has a context window size of 2048, which can only support 10-20 passages as input. We explore this setting by selecting only the top-10 retrieved passages as input to FiD, and show the results at the bottom of Table 4. EAR achieve significant improvement over GAR, roughly 10% in exact match on both datasets, showing potential benefits for methods with limited context window size.

heading: 5 Query Reranking vs Passage Reranking
text: EAR shares similarities with passage reranking (PR). EAR reranks the queries before retrieving the passages, while PR reranks the retrieved list of passages after the retrieval process is completed. To better understand the relationship between EAR and PR, we implement a BERT-based passage reranker, following the method outlined in Nogueira and Cho (2019), to rerank the retrieval results of GAR. The implementation details can be found in Appendix C. From the experiments we aim to answer three questions: 1) Is EAR better than PR? 2) Are the contributions of EAR and PR
complementary? Can their performance gains be aggregated if we apply both? 3) What are the extra advantages of EAR compared to PR?
Is EAR better than PR? We focus on comparing EAR-RD with PR, as EAR-RI is limited by its input, being able to see only the short expanded queries. On the other hand, EAR-RD has access to the top-1 passage retrieved by each query candidate, providing it with the same level of information as PR. In Table 5, we first present the performance of PR when reranking the same number of passages as the average number of query candidates considered by EAR (25 for NQ; 32 for TriviaQA), which can be found in row 3. The result of EAR-RD (shown in row 5) is better than row 3, indicating that when considering the same amount of information as inputs, EAR-RD outperforms PR. However, when PR is able to rerank a larger number of passages, such as the top-100 passages shown in row 4, it achieves better performance than EAR-RD. This implies that EAR-RD is more effective when PR can only access to the same level of information.
Are EAR and PR complementary? We found that the effects of EAR-RD and PR can be effectively combined for even better performance. When applying PR on the retrieval results of EARRD (shown in row 6), we see a significant improvement compared to both row 4 and row 5. This suggests that the contributions of EAR-RD and PR are complementary: EAR strengthens first-pass retrieval by selecting good queries, while PR rescores all the retrieved passages and generates an entirely new order for these passages. The distinction between these two mechanisms makes improvements accumulative and leads to superior results.
Extra advantages of EAR? An advantage of EAR is that it improves retrieval results beyond the top-k passages. In row 4, the top-100 accu-
racy cannot be improved by PR as it reranks within the top-100 passages. In contrast, the improvements provided by EAR are not limited to top-100 passages. As long as EAR selects good query expansions, it can improve the whole list of retrieved passages; we can see EAR-RD improves the top100 accuracy of GAR from 84.7 to 86.5.

heading: 6 Discussions

heading: 6.1 Generating Training Examples with GAR
text: In Section 3.3, we discussed two methods to construct training examples for EAR. In our main experiments, we used T0-3B to randomly sample diverse query expansions. An alternative method was also explored, where we trained K = 5 different GAR models separately on (K − 1) training subsets, then randomly sampled from the hold-out sets. The performance of this method, as shown in Table 6 (EAR-RI holdout), is slightly better than using T0-3B, but the difference is less than 0.5 points on Top-5/20/100 accuracy. Therefore, we continue to use T0-3B to generate training data in our main experiments as it eliminates the need to train K different GAR models separately.

heading: 6.2 EAR with Dense Retrievers
text: EAR is specifically optimized to work well with the BM25 retriever and hence its performance may be impacted when changing the retriever to DPR. As shown at the bottom of Table 6, when coupled with DPR, the top-5 accuracy of EAR-RD decreases,
while the top-20/100 accuracy remains relatively unchanged. This suggests that EAR is heavily reliant on the retriever, and thus changing the retriever negatively impacts its performance. Making EAR work with DPR would require retraining with DPR retrieval results and significantly more compute. We leave this direction for future work.

heading: 6.3 Reducing the Query Candidate Size
text: In our experiments, we generate 50 query expansions per question and then de-duplicate the repeated ones. However, we can also limit the maximum query expansions considered by our reranker to trade off between efficiency and performance. In Table 7 we show the top-5 accuracy of lowering the maximum candidate size N from 50 to 30/20/10/5. We observe that the performance drops gradually as N decreases. However, we still see improvement over GAR even when N = 5, showing that EAR still works with a small candidate size. We also show the curves of the top-k accuracy in Figure 2, where we observe a big gap between DPR (solid line) and GAR (dotted line with x mark). EAR-RI gradually eliminates the gap as N increase, while EAR-RD even matches DPR for k < 50 and outperforms DPR for k ≥ 50 with a small N = 5.

heading: 7 Computational Cost and Latency
text: We report the latency of DPR, GAR, and EAR in Table 8. Inference details can be found in Appendix D.
Dense Retrieval We first generate DPR document embeddings on 4 GPUs for ∼3.5 hours on 21M documents. Standard indexing takes ∼10 minutes with a 64GB index size. Indexing with the more advanced Hierarchical Navigable Small World (HNSW) (Malkov and Yashunin, 2018) takes ∼5 hours and results in a huge index size of 142GB. For retrieval, standard indexing takes 22.3s per query, while the highly optimized HNSW can shorten it to 0.04s per query.
Sparse Retrieval For BM25 with Pyserini, indexing only takes 0.5 hours, with a very small index size of 2.4GB. Retrieval for BM25 takes 0.15s per query. For GAR, it needs an extra 0.58s to generate the query expansions, and retrieval time is 0.56s. For EAR, it needs 1.29s to batch sample 50 query expansions. EAR-RI only takes 0.04s to rerank queries. EAR-RD needs extra time to retrieve the top-1 passages for each expansion, which takes an extra 0.70s, and then run the actual reranking process, taking 0.14s, giving a total of 0.84s for query reranking. For retrieval, the time needed for both EAR-RI and EAR-RD is similar to GAR.
To conclude, EAR inherits the advantage of
BM25: fast indexing time and small index size. This makes it possible to index large collections of documents in a relatively short amount of time, which is important for tasks where documents are frequently added or updated. The main cost for EAR is the time for sampling query expansions. However, this can potentially be reduced by speedup toolkits that optimize the inference time of transformers, such as FasterTransformer 4 (3.8∼13× speedup for decoding) or FastSeq (Yan et al., 2021; 7.7× speedup for BART decoding). Moreover, we can leverage model distillation (Shleifer and Rush, 2020) and quantization (Li et al., 2022) for transformers. We leave these directions for future work.

heading: 8 Related Work
text: Query Expansion and Reformulation Traditionally, query expansion methods based on pseudo relevance feedback utilize relevant context without external resources to expand queries (Rocchio, 1971; Jaleel et al., 2004; Lv and Zhai, 2010; Yu et al., 2021). Recent studies attempt to reformulate queries using generative models, relying on external resources such as search sessions (Yu et al., 2020) or conversational contexts (Lin et al., 2020; Vakulenko et al., 2021), or involve sample-inefficient reinforcement learning training (Nogueira and Cho, 2017). More recently, GAR (Mao et al., 2021a) explored the use PLMs for query expansion instead of external resources. A concurrent study (Liu et al., 2022) generates multiple expansions with beam search and filters and fuses the results, but EAR is aware of the BM25 retriever and could select more promising query expansions and run fewer BM25 retrievals.
Retrieval for OpenQA Sparse retrieval with lexical features such as BM25 was first explored for OpenQA (Chen et al., 2017). Dense retrieval methods were shown to outperform sparse methods (Karpukhin et al., 2020; Guu et al., 2020), while requiring large amounts of annotated data and much more compute. Although powerful, dense retrievers often fall short in the scenarios of 1) requiring lexically exact matching for rare entities (Sciavolino et al., 2021) and 2) out-of-domain generalization (Reddy et al., 2021). For 1), Luan et al. (2021) proposed a sparse-dense hybrid model, and Chen et al. (2021) trained a dense retriever to imitate a sparse one. For 2), Ram et al. (2022) created
4https://github.com/nvidia/fastertransformer
a pre-training task for dense retrievers to improve zero-shot retrieval and out-of-domain generalization. Another recent line of research explores passage reranking with PLMs to improve performance for both sparse and dense methods. Nogueira and Cho (2019) first explored BERT-based supervised rerankers for standard retrieval tasks and Mao et al. (2021b) proposed reranking by reader predictions without any training. Sachan et al. (2022) attempt to use an LLM directly as the reranker, but it requires huge amounts of computation at inference time and underperforms fine-tuned rerankers.

heading: 9 Conclusion
text: We propose EAR, which couples GAR and BM25 together with a query reranker to unlock the potential of sparse retrievers. EAR significantly outperforms DPR while inheriting the advantage of BM25: fast indexing time and small index size compared to the compute-heavy DPR. Cross-dataset evaluation also shows that EAR is very good at generalizing to out-of-domain examples. Furthermore, we demonstrate that contributions of EAR and passage reranking are complementary, and using both methods together leads to superior results. Overall, EAR is a promising alternative to existing dense retrieval models, providing a new way to achieve high performance with less computing resources.
Limitations
First, as EAR largely relies on GAR generators, the performance of the method is closely tied to the quality of the generator used. We have attempted to use large language models such as T0-3B without fine-tuning as a replacement for the GAR generator during testing, but the performance becomes worse. The main reason is that the quality of query expansions generated by T0-3B is too diverse, which makes EAR has a higher chance to select from terrible expansions. In contrast, the output quality of GAR is more stable. We may need a more complex mechanism that can exclude terrible query expansion if we want to directly use the query expansions generated by T0-3B during inference. Second, EAR has demonstrated a strong generalization ability to out-of-domain data, but the method may still face challenges when transferring to other languages without any supervised QA data, which GAR and EAR are trained on. Although challenging, we are still trying to train the EAR system without supervised QA data.
Ethics Statement
In this research, we used publicly available datasets and we did not collect any personal information. Our method is designed to improve the performance of information retrieval systems, which can have a positive impact on various applications, such as search engines, QA systems, and other applications that rely on text retrieval. When deployed, however, our approach also poses the ethical risk typical of pre-trained language models, for instance, producing retrieval results that contain human biases which could potentially exacerbate discrimination. Therefore, caution should be exercised before implementing our approach in realworld situations and thorough audit of training data and testing of model outputs should be conducted.

heading: Acknowledgements
text: We thank Yuning Mao for his helpful feedback. We thank Ori Ram for providing detailed results of the experiments from Ram et al. (2022). This research was sponsored by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2-1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the United States Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.

heading: A Dataset Statistics
text: We show the number of train/dev/test examples in each dataset in Table 9.

heading: B Training Details
text: For the training set, we use T0-3B 5 to randomly sample 50 query expansions per query. For the dev set and test set, we use the three GAR generators (answer/sentence/title), which are BART-large seq2seq models (Lewis et al., 2020) to generate 50 query expansions per query. We use the DeBERTa V3 base model6, which has 86M parameters that are the same as BERT-base (Devlin et al., 2019), as EAR-RI and EAR-RD rerankers. For the implementation of rerankers, we reference the implementation of SimCLS (Liu and Liu, 2021)7, which also does reranking for sequences. We start from the code of SimCLS and change the loss function to our ranking loss LRank. During training, we use the dev set generated from three GAR generators to pick the best checkpoints, resulting in three different reranker models corresponding to the answer/sentence/title generators.
The ranges we search for our hyperparameters are shown in Table 10. Each training example in our dataset contains 50 sequences (generated by T0-3B). To prevent memory issues of GPU, we used gradient accumulation to simulate a batch size of 4 or 8, which effectively consists of 200 or 400 sequences, respectively.
The training time on a single NVIDIA V100 GPU is around 12 hours for EAR-RI and 1 to 2 days for EAR-RD. The best hyperparameters according to the dev set are shown in Table 11. However, in our experiments, the variance between different hyperparameters is actually quite small.
5https://huggingface.co/bigscience/T0_3B 6https://huggingface.co/microsoft/
deberta-v3-base 7https://github.com/yixinL7/SimCLS

heading: C Passage Reranking
text: For the implementation of a BERT-based passage reranker, we generally follow the setting of (Nogueira and Cho, 2019) for training. We separately fine-tuned two bert-base-uncased models on the NQ training set and the TriviaQA training set. Each pre-trained BERT model is fine-tuned for reranking using cross-entropy loss on the binary classification head on top of the hidden state corresponding to the [CLS] token. We use the top-10 outputs of BM25 ran on the training sets as the training examples, which contains both positive and negative examples. We fine-tune the models using 2 GPUs with mixed precision (fp16) with a batch size of 128 for 3 epochs. AdamW (Loshchilov and Hutter, 2018) is used for optimization with a learning rate of 5e-5, linear warmup over the first 10k steps and linear decay afterwards, and a weight decay of 0.01.
D Inference Details
For inference of GAR retrieval results, we follow GAR to retrieve with three queries generated by three context generators (answer/sentence/title), and then fuse the three retrieved passages lists in the order of sentence, answer, title. In other words, given the three retrieved lists of passages: (a1, a2, ..., a100), (s1, s2, ..., s100), (t1, t2, ..., t100), we fuse the results as (s1, a1, t1, s2, a2, t2, ..., s33, a33, t33, s34). We skip all the duplicated passages that exist twice during the fusion process.
For EAR, we use the same pipeline of GAR, while the only difference is that instead of greedy decoding, now the three generators of GAR can do random sampling, and three different query rerankers (answer/sentence/title) are applied to select the best queries. After that, the pipeline to obtain retrieval results is exactly the same as GAR.
To fairly compare the latency of these methods, we run the 3610 queries in NQ test set one-byone without batching (batch size = 1) and compute the average the latency per query, where document encoding, query expansion and reranking are run on NVIDIA RTX A5000 GPUs and indexing and retrieval are run on fifty Intel Xeon Gold 5318Y CPUs @ 2.10GHz, for both FAISS (Johnson et al., 2019) (DPR) and Pyserini (Lin et al., 2021) (BM25).
DPR document indexing We used 4 GPUs to encode 21M Wikipedia passages in parallel with mixed precision (fp16), which takes around 3.5 hours.
GAR and EAR For inference of GAR and EAR, answer/sentence/title generators/rerankers are run in parallel on three GPUs.
FiD We take the public checkpoints of FiD8, which are trained from T5-Large (?) with NQ/TriviaQA, to directly evaluate the end-to-end QA performance.

heading: E Qualitative Study
text: In this section, we aim to investigate the differences between queries generated by GAR and EAR. We first look at the lengths of the expanded queries for GAR, EAR-RI, EAR-RD. In general, the lengths of queries from EAR are slightly shorter than that of GAR, but the trends are not very obvious. Thus,
8https://github.com/facebookresearch/FiD
we conduct a qualitative study to see what is the difference between these queries.
As shown in Table 13, we provide three examples to demonstrate how our method EAR works. In the first example, the initial query only includes two keywords, "Deadpool" and "released," that can match the answer passage. As a result, the BM25 algorithm is unable to retrieve the correct passage within the top results until the 77th passage. The greedy decoding output for GAR also fails to retrieve the correct passage, as it includes many irrelevant name entities. However, both EAR-RI and EAR-RD are able to select useful outputs from GAR, which contain keywords such as “scheduled,” “2018,” “Leitch,” and “in the United States.” Although none of these keywords contains the real answer May 18, 2018, these keywords already provide enough lexical overlap with the answer passage, allowing BM25 to correctly retrieve the answer passage in the top-1 result.
For the second example, the original query only contains three keywords “India’s,” “next,” and “star” that can match the answer passage, so BM25 with the original query cannot retrieve the correct passage within the top retrieved results until the 96th passage. For GAR, the greedy decoding output for GAR is also not effective, as it is a misleading answer and only includes one useful keyword “winner” and thus cannot retrieve the correct passage within the top-100 results. For EAR-RI and EARRD, they are able to select a sentence that, while not containing the correct answer “Natasha Bharadwaj” or “Aman Gandotra,” does include useful keywords such as “winner,” “Superstar,” “Season,” and “2018.” These keywords provide enough lexical overlap with the answer passage, allowing EARRI and EAR-RD to correctly retrieve the answer passage in the top-1 result.
The third example presents a challenging scenario. The initial query only includes two common keywords, “method” and “writer,” which is difficult to match the answer passage. While BM25 is able to correctly retrieve the answer at the 92nd
Model Query [Answer = May 18, 2018] Answer Rank
BM25 When is the next Deadpool movie being released? 77
GAR When is the next Deadpool movie being released? Miller Brianna Hildebrand Jack >100Kesy Music by Tyler Bates Cinematography Jonathan Sela Edited by Dirk Westervelt...
sions generated by GAR. Bold words are useful keywords from the original query. Words highlighted in green are useful keywords generated by GAR. Answer Rank shows the ranking of the answer passage in the retrieval results.
passage, the generated query expansions are not helpful and instead are misleading, resulting in GAR and EAR-RI/EAR-RD all unable to retrieve the correct passage within the top-100 results due to the distracting query expansions. This example
illustrates the importance of the GAR generators. If all of the generated query expansions are not useful, EAR is unable to improve the results.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations before Reference
3 A2. Did you discuss any potential risks of your work? Ethics Statement before Reference
3 A3. Do the abstract and introduction summarize the paper’s main claims? In the Abstract
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix B
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix B
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix B
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: We leave these directions for future work