Analyze, Generate and Refine: Query Expansion with LLMs for Zero-Shot Open-Domain QA (2024)

Abstract: Query expansion (QE) is a critical component in the open-domain question answering (OpenQA) pipeline, enhancing the retrieval performance by broadening the scope of queries with additional relevant texts. However, existing methods like GAR and EAR rely heavily on supervised training and often struggle to maintain effectiveness across domains and datasets. Meanwhile, although large language models (LLMs) have demonstrated QE capability for information retrieval (IR) tasks, their application in OpenQA is hindered by the inadequate analysis of query’s informational needs and the lack of quality control for generated QEs, failing to meet the unique requirements of OpenQA. To bridge this gap, we propose a novel LLM-based QE approach named AGR for the OpenQA task, leveraging a threestep prompting strategy. AGR begins with an analysis of the query, followed by the generation of answer-oriented expansions, and culminates with a refinement process for better query formulation. Extensive experiments on four OpenQA datasets reveal that AGR not only rivals in-domain supervised methods in retrieval accuracy, but also outperforms state-of-the-art baselines in out-of-domain zero-shot scenarios. Moreover, it exhibits enhanced performance in end-to-end QA evaluations, underscoring the superiority of AGR for OpenQA.1

heading: 1 Introduction
text: Open-domain question answering (OpenQA) is a key task in Natural Language Processing, aiming to provide accurate answers to a wide range of factual questions across different domains (Chen and Yih, 2020; Kwiatkowski et al., 2019). The challenge in OpenQA is to retrieve relevant information from large text corpora without specific contexts (Zhu et al., 2021). Retrieval methods are therefore es-
1Our code and data are publicly available at https:// github.com/process-cxr/AGR.
sential, with two primary approaches being lexicalbased sparse retrieval, like BM25 (Robertson and Zaragoza, 2009), and embedding-based dense retrieval (Karpukhin et al., 2020; Guu et al., 2020). Dense retrieval models (Luan et al., 2021; Xiong et al., 2021; Qu et al., 2021), while effective with ample domain-specific training data, are computationally demanding and risk omitting crucial information due to their reliance on fixed-length embedding that may not capture all the textual nuances, leading to potential exclusion of relevant details. Conversely, sparse retrieval combined with query expansion techniques (Lavrenko and Croft, 2017; Chuang et al., 2023) can address these semantic challenges and achieve competitive performance.
11908
The recent advent of generation-augmented retrieval has shown promise in providing more precise information for OpenQA, as evidenced by methods such as GAR (Mao et al., 2021) and EAR (Chuang et al., 2023). These approaches utilize seq2seq models like BART (Lewis et al., 2020) to generate contexts that are tailored to the query, incorporating elements such as the answer, the sentence containing the answer, and the title of the passage where the answer is located. Nevertheless, these methods often require extensive supervised training data, which can lead to the generation of subpar QEs, particularly in out-of-domain zeroshot scenarios. For instance, as depicted in Figure 1, the EAR-RD+ model inadequately addresses the information needs about the phrase "war and order," resulting in the generation of incorrect content. Concurrently, the rise of large language models (LLMs) like GPT-3 (Brown et al., 2020) and Flan-T5 (Raffel et al., 2020) has underscored their potential as effective QE tools in information retrieval (IR) tasks (Li et al., 2023), operating without the need for training data or external corpora, such as Q2D (Wang et al., 2023) and Q2D+PRF (Jagerman et al., 2023). However, while these models excel in directly bolstering queries with expansions derived from a LLM, they often lack mechanisms for ensuring the quality of the generated QEs. This issue is exemplified in Figure 1, where the QE produced by Q2D+PRF includes a mix of relevant and irrelevant information, signaling the necessity of effective quality control measures.
To this end, we propose a novel LLM-based QE method specially designed for the OpenQA task, coined as AGR. As shown in Figure 2, AGR employs a three-step progressive prompting strategy, namely Analyze, Generate, and Refine, to leverage the extensive capabilities of LLMs, facilitating cross-domain query expansion for OpenQA. Specifically, to address the informational needs of a given query, the Analyze phase utilizes the question understanding capabilities of LLMs to generate an analysis. Subsequently, in Generate phase, AGR utilizes the knowledge retrieval and integration capabilities of LLMs to generate various answer-oriented query expansions as candidates. These expansions are then used to retrieve reference texts from the corpus to filter out erroneous and irrelevant generated information for generating new candidates closer to potential answers. Finally, for the purpose of quality control over QEs, the Refine phase conducts a self-review of all candi-
dates, refining them to achieve a high-quality query expansion. Obviously, AGR is explicitly oriented towards QA tasks with a series of progressively advancing sub-task steps, making it more suitable for high-quality QE generation in out-of-domain zero-shot scenarios.
Extensive experiments are carried out on four widely-used OpenQA datasets, including Natural Questions (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), WebQuestions (Berant et al., 2013), and CuratedTREC (Baudis and Sedivý, 2015). The results of retrieval evaluation demonstrate that AGR not only achieves comparable performance to SOTA supervised QE methods such as EAR (Chuang et al., 2023) on their trained in-domain datasets but also surpasses them in the context of zero-shot scenarios on out-of-domain datasets, which emphasizes the zero-shot capabilities of our AGR method. Furthermore, in comparison to LLM-based QE methods tailored for IR tasks, AGR exhibits the ability to generate more answer-oriented information so that sparse retrievers leveraging AGR can identify more accurate passages containing the answers, which contributes to the improved end-to-end QA quality.
Our contributions are three-fold: 1) We propose a LLM-based QE method AGR, which adopts a novel reasoning chain generation process suited for OpenQA. 2) Extensive experiments show that AGR achieves SOTA out-of-domain zero-shot performance in boosting the retrieval quality across diverse datasets. 3) End-to-end QA evaluation indicates that AGR enhances the exact match scores of answers, emphasizing its practical utility in realworld applications.

heading: 2 Related Work

heading: 2.1 Query Expansion
text: Query expansion (QE) has received widespread attention in the early literature of information retrieval (Efthimiadis, 1996), fundamentally boosts retrieval systems by enriching queries with additional, conceptually similar terms (Carpineto and Romano, 2012). Early QE methods in IR mainly augment queries with additional terms based on user relevance feedback (Rocchio Jr, 1971), which is often unavailable. Then, the Pseudo-Relevance Feedback (PRF) mechanisms (Croft et al., 2009) are developed, wherein the top-ranking results of an original query are utilized for expansions, but this is also constrained by the quality of the top
documents (Carpineto and Romano, 2012). More recent QE studies for IR tasks have started to leverage pre-trained language models in the process of query expansion (Zheng et al., 2020; Naseri et al., 2021), commonly achieved through training or finetuning models.
The capability of QE in enhancing retrieval has also been utilized in studies on OpenQA, such as GAR(Mao et al., 2021) and EAR(Chuang et al., 2023). GAR utilizes a trained BART model to generate answer-oriented QEs, effectively bridging the information gap between the original query and potential answers. Building upon by GAR, EAR employs the generator from GAR for sampling and generating a various set of QEs, and uses a query ranking model to select the best QE. Although these methods enhance the effectiveness of retrievers, they heavily depend on supervised indomain training of the generator or reranker, which constrains their adaptability across domains and datasets under zero-shot OpenQA.

heading: 2.2 Large Language Models
text: The rapid advancements in generative modeling have led to the development of large language models (LLMs), like ChatGPT (Ouyang et al., 2022) and GPT-4 (OpenAI, 2023), which demonstrate exceptional multitasking capabilities, particularly exemplifying outstanding zero-shot learning abilities (Liu et al., 2022; Dong et al., 2023). Moreover, LLMs are suitable for a wide array of tasks (Brown et al., 2020; Alayrac et al., 2022), ranging from language translation and question answering to more intricate challenges like sentiment analysis and di-
alogue generation (Kaddour et al., 2023). Recent studies have highlighted the emergence of prominent open-source LLMs like LLama2 (Touvron et al., 2023) and Mistral (Jiang et al., 2023), which excel in various LLM benchmarks and display robust multitasking capabilities, garnering substantial academic interest due to their accessibility and manageable parameter sizes.
In the domain of query expansion, LLMs have also shown promising capabilities. For instance, WebCPM (Qin et al., 2023) and Q2E (Jagerman et al., 2023) employ LLMs to generate topic-related terms as query expansion, Query2Doc (Wang et al., 2023) focuses on employing LLMs to generate passages related to the potential answers, aiming to alleviate the issue of word mismatch between query and documents. While these LLM-based methods have notably enhanced QE in IR, underscoring the revolutionary impact of LLMs, they are not specifically tailored for OpenQA. As a result, these methods may not fully align with the informational nuances required for queries in OpenQA, and lack quality control for QEs. In contrast, our proposed AGR introduces a multi-step generation framework designed for QA tasks, which is designed to meet the informational needs of query and implements quality control to obtain a refined QE.

heading: 3 Methodology
text: As shown in Figure 2, the proposed AGR leverages the concept of multi-step task decomposition, encompassing a progressive sub-task workflow to analyze, generate, and refine the query. This approach
thoroughly exploits the abilities of LLMs (Zhao et al., 2023), like semantic comprehension, knowledge retrieval, and integration, as well as contextual understanding and reasoning. The result is an expanded query that is anticipated to be more aligned with the answer, thereby improving the performance of sparse retrieval mechanisms within the OpenQA task.

heading: 3.1 Overview
text: Given an original query q, AGR first makes use of an LLM (like Mistral) to analyze this query and generate an analytic text A on it. Subsequently, building on this analysis, a set of answer-oriented query expansions Qe = {qe1, qe2, ..., qen} is generated by the LLM through random sampling strategy, which ensures the diversity of generated expansion content. However, empirical observations indicate that a portion of query expansions align with the correct answer, while others may result from model hallucinations, manifesting as either irrelevant to the correct answer or erroneous. Thus, similar to pseudo relevance feedback (PRF), all top-k documents from BM25 for all query expansions Qe can be integrated to rectify and output a series of new query expansion Qe = {qe1, qe2, . . . , qem}, which combines the knowledge exist in LLM with the knowledge retrieved from the corpus. After that, a final review by the LLM on these all enriched QE candidates Qe is further conducted to identify relevant answers and erroneous information, ultimately generating the refined query expansion qeo, which is appended to the original query q to execute the following retrieval.

heading: 3.2 Analysis of Original Query
text: In the initial phase, we first leverage the semantic comprehension abilities of LLM to extract key phrases K from the original query q, and then conduct an analysis A based on K.
K = LLM(q) (1)
A = LLM(q,K) (2) The generation of key phrases aims to identify the core elements of the original query, which turns back to help the LLM grasp the essential theme and contextual cues within the query, and output a more targeted and relevant analysis. The ablation study conducted on the generation of K and A in Section 4.3 highlights the significance of this particular step.

heading: 3.3 Generation of Candidate QEs
text: Building on the insights gained from the initial analysis A of the query q, the LLM’s knowledge retrieval and integration capabilities are employed to produce multiple candidate expansions Qe = {qe1, qe2, . . . , qen}.
Qe = LLM(q,A) (3) Subsequently, to enhance the reliability of expansions, we turn to use the relevant documents from the corpus to rectify them. Specifically, the BM25 top-k documents Dqei for each candidate expansion qei ∈ Qe are all collected to form contextual references CRs. Then, the texts CRs in conjunction with the original query q are re-introduced into the LLM to conduct a second round of sampling generation. This process blends the LLM’s intrinsic knowledge with information retrieved from the corpus, yielding more enriched query expansions Qe = {qe1, qe2, ..., qem}. CRs = ∪{Dqei} = ∪{BM25(qei)}, qei ∈ Qe
(4) Qe = LLM(q, CRs) (5)
In both the generation steps in Eq. 3 and Eq. 5, a random sampling strategy is adopted, which ensures that LLMs produce a range of potential expansions, each reflecting different facets and interpretations of the initial query q. The breadth of generated expansions provides a comprehensive pool for the following refinement step.

heading: 3.4 Refinement for Optimal QE
text: Due to their stochastic and diverse nature, the documents provided from the corpus to the LLM facilitate the generation of enriched QE candidates, Qe. While many of these candidates are highly relevant or closely aligned with the correct answer for the query q, some may still be irrelevant or erroneous, likely due to the hallucinatory tendencies of LLMs. Consequently, in this phase, we leverage the LLMs’ capabilities in contextual understanding and reasoning to conduct a thorough review of all QE candidates:
qeo = LLM(q,Qe) (6) This process involves discerning between valuable and superfluous information. In other words, we refine and distill these expansions into an optimized query expansion qeo, which is then appended to the original query q, enhancing the quality of sparse retrieval outcomes.
Throughout these stages, AGR capitalizes on the LLMs’ analytical and generative capabilities, offering a novel paradigm for answer-oriented QE, enhancing the overall performance in OpenQA scenarios. Overall, AGR is a three-step prompting method based on LLMs, and note that the whole process of AGR does not require additional training or fine-tuning of any models, it only relies on the inherent capabilities of LLMs to generate enhanced QEs in a zero-shot context.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Datasets For the evaluation datasets, we select four diverse datasets pertinent to OpenQA task, including Natural Questions (NQ) (Kwiatkowski et al., 2019), TriviaQA (Trivia) (Joshi et al., 2017), WebQuestions (WebQ) (Berant et al., 2013) and CuratedTREC (TREC) (Baudis and Sedivý, 2015). The former two datasets are utilized for baselines involving supervised approaches and the latter are employed to assess all baseline methods under an unsupervised zero-shot evaluation setting. Detailed introduction of datasets refers to Appendix A. Given that datasets relevant to OpenQA frequently encompass questions annotated with gold-standard answers as benchmarks for evaluation (Rajpurkar et al., 2016), we have consequently established the task paradigm as Retriever-Reader.
Details of AGR We choose Mistral-7B (Jiang et al., 2023) as the backbone model of AGR, and more analysis on different models, including LLama-3B, LLama2-7B and LLama2-13B (Touvron et al., 2023), are available in Appendix B.1. Besides, we utilize the SamplingParams class from vLLM (Kwon et al., 2023), allowing for precise and optimized settings across different phases. Specifically, temperature is configured at 0.2 for the Analyze and Refine phases, and adjusted to 0.8 for the Generate phase. Considering the constraint of the LLMs’ token input capacity limitation, in the Generate phase, the number of generated query expansion n is fixed at 15, and the number of top-ranking documents retrieved for contextual reference k is set at 3. The impacts of these two hyper-parameters are detailed in Section 4.3. In the Refine phase, the number of generated query expansion m is determined to be 10, aiming for a balance between accuracy and practicality. As for inference resources, our experiment employs
NVIDIA L40 48GB GPUs. Other detailed settings are provided in the Appendix B.2.
Baselines We compare AGR with four types of approaches:
1) Direct retrieval without QE: BM25 (Robertson and Zaragoza, 2009) is a standard termmatching sparse retriever, and DPR (Karpukhin et al., 2020) is a standard dense retriever based on BERT model.
2) Traditional QE method: BM25+RM3 (Roy et al., 2019) uses a traditional QE method based on extracting tokens from retrieval documents without supervised training or LLMs.
3) Supervised QE models: GAR (Mao et al., 2021) adopts three types of query expansion generators based on trained seq2seq models, EAR further uses trained query rankers to reorganize the QEs, achieving SOTA performance on NQ and TriviaQA datasets as shown in Chuang et al. (2023).
4) LLM-based QE for IR tasks: Query2doc (Q2D) (Wang et al., 2023) uses LLMs to generate answer-oriented passages as QEs, and Q2E (query2keywords) (Jagerman et al., 2023) generates topic-oriented keyword information as QEs.
To ensure a unified basis for comparison, all QE methods employ BM25 for retrieval, and all LLMbased QE baseline methods utilize Mistral-7B as the backbone model. Moreover, for the specific experimental settings during the generation and PRF retrieval processes of the baseline Q2D+PRF and Q2E+PRF, they utilize the same hyperparameters as AGR, involving sampling generation of 10 QEs, retrieving 3 PRF documents for each QE, and a second generation for 10 new QEs based on the retrieved documents. The distinction arises with that AGR implements a third-step refinement for the newly generated QE candidates, whereas Q2D+PRF and Q2E+PRF randomly choose one QE candidate from the candidates.
Metrics Akin to prior research (Brown et al., 2020; Mao et al., 2021), we adopt the RetrieverReader task paradigm and introduce two principal metrics. Hit@k for retrieval accuracy, is utilized for assessing the effectiveness of the retriever, and it is defined as the proportion of questions/queries for which at least one answer span is contained within the top-k retrieved passages. EM@k for exact match score, is employed to evaluate the results of the Reader, serving as an end-to-end performance indicator. The score is the proportion
of instances where the predicted answer span exactly matches one of the true answers after string normalization.

heading: 4.2 Results
text: Retrieval evaluations As presented in Table 1, our principal findings from the retrieval evaluations can be summarized as follows:

heading: 1) Our zero-shot AGR is on par with the
text: strong supervised baseline EAR-RD∗. This equivalence is demonstrated in Table 1. The AGR method, closely matching the performance of SOTA supervised method EAR-RD∗ within a 1% margin, showcases its effectiveness. Particularly noteworthy is AGR’s improved performance over the supervised approach DPR∗ on TriviaQA, highlighting its capability to understand search intent without relying on supervised training.

heading: 2) AGR showcases superior generalization
text: abilities over all baseline methods. As substantiated in Table 1, on the former two datasets our experiment conducts a cross-domain evaluation of GAR and EAR methods, and for the subsequent two datasets we incorporate cross-domain results of DPR, GAR, and EAR from (Chuang et al., 2023) (transferred from NQ/TriviaQA). These results are considered representative of zero-shot outof-domain settings for supervised methods. Evidently, AGR demonstrates consistently superior
performance over all baseline methods in zeroshot out-of-domain settings across various datasets, manifesting its generalization ability beyond domain constraints.

heading: 3) AGR outperforms other LLM-based QE
text: methods in IR tasks due to its QA-specific design. This conclusion is drawn from the comparison to LLM-based QE methods, where AGR consistently leads in performance against counterparts like Query2Doc, Q2E, and their variants. Surpassing the average Hit@5/100 retrieval accuracy of these methods by significant margins, AGR confirms its design specifically tailored for QA tasks, as opposed to general IR objectives.
End-to-end QA evaluations To further compare AGR method with all baseline approaches across the complete end-to-end OpenQA task, we uniformly employ the Fusion-in-Decoder (FiD) model (Izacard and Grave, 2021) pre-trained on NQ/TriviaQA datasets as the reader component. As illustrated in Table 2, we can obtain observations from the results as follows:
1) AGR secures a dominant position in endto-end EM scores across various datasets. This view is supported by the end-to-end evaluation results conducted on the NQ/TriviaQA datasets. The results indicate that our AGR method almost consistently outperforms all other baseline approaches in terms of end-to-end effectiveness. This includes
surpassing the SOTA in-domain supervised method EAR-RD∗, which is notable given that its retrieval accuracy marginally exceeds AGR. This comparison underscores AGR’s comprehensive proficiency, especially in demonstrating a performance advantage in end-to-end OpenQA task.

heading: 2) With fewer input documents, AGR’s end-
text: to-end EM score advantage grows more distinct. This conclusion is derived through conducting multiple comparative tests by varying the input parameters of the FiD reader. In these experiments, we used top-1/10/100 retrieved passages – standard quantity parameters for FiD – as inputs to compute the EM scores for each QE method accordingly. Significantly, with the reduction in the number of input passages from 100 to 1, the AGR method’s superiority becomes progressively evident. This suggests that the passages retrieved through AGR not only demonstrate exceptional retrieval accuracy but also rank higher in delivering gold passages rich with answer content, thereby facilitating FiD in extracting more accurate responses.

heading: 4.3 Analysis
text: Ablation study To better comprehend the utility of AGR, we conduct various experiments on the NQ dataset analyzing the impact and effectiveness of each component within this architecture as follow:
1) Necessity of individual components: In this experiment, we establish three variants to investigate the necessity of each component: a) w/o Analyze: Our proposed framework without the analysis step, replaced by the generate-refine pipeline; b) w/o CRs: Our proposed framework without in-
tegrating contextual references (CRs) in second round sampling generation; c) w/o Refine: Our proposed framework without the refinement step, directly samples a QE from candidates.
From Table 3, we can draw the following conclusions: a) The performance of AGR on the NQ datasets is better than these variants lacking components of this method, affirming the effectiveness of our proposed Analyze, Generate, and Refine framework. A plausible explanation involves decomposing the problem into sub-questions and employing multi-step progressive prompting. This can amplify LLMs’ multi-task reasoning capabilities, thereby facilitating the generation of improved answer-oriented QEs specific to OpenQA. b) When comparing the performance among different variants, we observe that the variant w/o CRs performs the worst, highlighting the critical role of contextual references in enhancing the quality of subsequent sampling generation. By further incorporating the Refine step, it can reduce irrelevant or erroneous information from the initial round of sampling generation, thus enhancing the overall performance of the AGR framework.
2) Quantitative assessment of generation and refinement: At this assessment stage, we statisti-
cally analyze the QEs generated across three distinct stages, to quantitatively assess the impact of integrating contextual references in the sampling generation process between Qe and Qe, as well as the effect of the refinement process between Qe and qeo. A QE is considered high-quality if the retrieval results include the gold passage within the top-5 documents; otherwise, it is categorized as low-quality. The proportion of high-quality QEs per query instance then acts as our fundamental unit of statistical measurement.
Specific statistical results are depicted in Figure 4. We observe an increasing trend where most intermediate data shifted towards higher proportion rates of high-quality QEs after undergoing the generation and refinement processes. This trend suggests that the processes are effective at filtering out some irrelevant or erroneous information produced during the initial sampling generation, thereby improving the quality of the QE. Conversely, less intermediate data moved towards lower proportion rates, indicating that an overabundance of irrelevant or erroneous information can negatively impact subsequent generation attempts.
To further quantify the process, we track the change in the proportion of high-quality QEs across three stages for each query, estimating the probability of transformation: With a threshold set at 0.6, data above this threshold had a 95.1% probability of resulting in high-quality QEs after processing, whereas data below had only a 31.03% probability. These findings highlight the limitations of the effectiveness in the Generate and Refine phase, indicating the need for quality control at each stage to obtain the final high-quality QE.
Hyper-parameter sensitivity We proceed to explore the sensitivity of our method’s performance to hyper-parameters, primarily focusing on two pivotal sets in the Generate phase: a) the number of generated query expansion, denoted as "n", and b) the number of top-ranking documents retrieved from contextual references, labeled as "k". Due to constraint of the LLMs’ token input capacity limitation, the range for both is capped at their maximum values, with "n" up to 15 and "k" up to 5.
The experiments presented in Figure 3 demonstrate that within the constraints of the model’s token input capacity, increasing "n" enhances re-
trieval accuracy. This suggests that employing a more powerful model with a larger token input capacity could yield even better results. On the other hand, marginal gains diminish as k increases, suggesting a point of diminishing returns in terms of retrieval accuracy enhancement. These findings clearly illustrate the AGR method’s performance sensitivity to these two hyper-parameters.
Case illustration As demonstrated by the results in Tables 1 & 2, although AGR marginally lags behind SOTA in-domain supervised method EARRD∗ in terms of retrieval accuracy, it exhibits better end-to-end EM scores, nearly universally surpassing EAR-RD∗ across various setting. This phenomenon underscores the adaptability of AGR’s design for end-to-end OpenQA. As illustrated by an example in Table 4, it can be observed that the top-1 documents retrieved by both methods contain the question’s answer, marked as the gold passage, while the gold passage retrieved by AGR exhibits a stronger relevance to the answer and subject matter compared to EAR-RD∗, allowing the reader to have a better chance to extract the correct answer.
Time overhead Regarding time overhead, since both employ sample generation, Q2D+PRF and Q2E+PRF involve 2 generative inference steps throughout the process, while AGR involves 5, with the retrieval overhead being identical for both. Consequently, AGR exhibits a time overhead in generative inference which is 2 to 3 times higher than that of Q2D+PRF. In the future we plan to emloy more feasible multi-GPU distributed model loading
strategies for the deployment of larger-parameters LLMs under limited computational resources, and then we can merge some LLMs generation steps (combining Equations 1 and 2), without compromising effectiveness, to reduce the number of inferences and lower latency. Additionally, we will also utilize more powerful parallel inference strategies to increase the speed of LLM inference within the pipeline, thereby reducing latency.

heading: 5 Conclusion
text: In this paper, we introduce a multi-step generative framework AGR with analyze, expand and refine phases, which deeply mines the inherent knowledge of LLMs for answer-oriented query expansion. AGR eschews the need for additional in-domain data for supervised training and demonstrates significantly higher retrieval accuracy compared to state-of-the-art QE methods under out-of-domain zero-shot scenes. Moreover, when integrated with the Fusion-in-Decoder as the reader component, AGR achieves a nearly comprehensive lead in endto-end performance, showcasing its effectiveness on OpenQA task. In future work, we plan to explore a more comprehensive framework, further integrating the requirements of the reader component to achieve a unified architecture and enhance the end-to-end performance for OpenQA task.
Limitations
Firstly, as shown in our experimental analysis, the effectiveness of our proposed AGR, an LLMs-based QE generation framework, is intrinsically linked to the quality of the underlying backbone model. It is needful to find out a suitable LLM to fully demonstrate the AGR’s capacity, and carry out further investigations with more types of LLMs, like GPT-4. Secondly, although AGR avoids additional supervised training data and conserves computational resources, its reliance on multiple inference generations with LLMs introduces certain latency. Therefore, further exploration of strategies to accelerate LLM inference and reduce overall latency are left for future work. Finally, the generations from LLMs are associated with the input prompts. Although we have tested multiple configurations to achieve the current performance, the possibility of better configurations exists.

heading: Acknowledgements
text: This work is supported by the National Natural Science Foundation of China (62272439), and the Fundamental Research Funds for the Central Universities.

heading: 2013, Grand Hyatt Seattle, Seattle, Washington, USA,

heading: A meeting of SIGDAT, a Special Interest Group of the
text: ACL, pages 1533–1544. ACL.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Claudio Carpineto and Giovanni Romano. 2012. A survey of automatic query expansion in information retrieval. ACM Comput. Surv., 44(1):1:1–1:50.
Danqi Chen and Wen-tau Yih. 2020. Open-domain question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 34–37, Online. Association for Computational Linguistics.
Yung-Sung Chuang, Wei Fang, Shang-Wen Li, Wen-tau Yih, and James Glass. 2023. Expand, rerank, and retrieve: Query reranking for open-domain question answering. In Findings of the Association for Computational Linguistics: ACL 2023, pages 12131–12147, Toronto, Canada. Association for Computational Linguistics.
W. Bruce Croft, Donald Metzler, and Trevor Strohman. 2009. Search Engines - Information Retrieval in Practice. Pearson Education.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023. A survey for in-context learning. CoRR, abs/2301.00234.
Efthimis N Efthimiadis. 1996. Query expansion. Annual review of information science and technology (ARIST), 31:121–87.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3929–3938. PMLR.
Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume,
EACL 2021, Online, April 19 - 23, 2021, pages 874– 880. Association for Computational Linguistics.
Rolf Jagerman, Honglei Zhuang, Zhen Qin, Xuanhui Wang, and Michael Bendersky. 2023. Query expansion by prompting large language models. CoRR, abs/2305.03653.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. CoRR, abs/2310.06825.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics.
Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. 2023. Challenges and applications of large language models. CoRR, abs/2307.10169.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6769–6781. Association for Computational Linguistics.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: a benchmark for question answering research. Trans. Assoc. Comput. Linguistics, 7:452– 466.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023, pages 611– 626. ACM.
Victor Lavrenko and W. Bruce Croft. 2017. Relevancebased language models. SIGIR Forum, 51(2):260– 267.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training
for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics.
Minghan Li, Honglei Zhuang, Kai Hui, Zhen Qin, Jimmy Lin, Rolf Jagerman, Xuanhui Wang, and Michael Bendersky. 2023. Generate, filter, and fuse: Query expansion via multi-step keyword generation for zero-shot neural rankers. CoRR, abs/2311.09175.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100–114, Dublin, Ireland and Online. Association for Computational Linguistics.
Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2021. Sparse, dense, and attentional representations for text retrieval. Trans. Assoc. Comput. Linguistics, 9:329–345.
Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2021. Generation-augmented retrieval for opendomain question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4089–4100, Online. Association for Computational Linguistics.
Shahrzad Naseri, Jeff Dalton, Andrew Yates, and James Allan. 2021. CEQE: contextualized embeddings for query expansion. In Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28 - April 1, 2021, Proceedings, Part I, volume 12656 of Lecture Notes in Computer Science, pages 467–482. Springer.
OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.
Yujia Qin, Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlun Zhu, Yankai Lin, Xu Han, Ning Ding, Huadong Wang, Ruobing Xie, Fanchao Qi, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2023. WebCPM: Interactive web search for Chinese long-form question answering. In Proceedings of the 61st Annual
Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8968–8988, Toronto, Canada. Association for Computational Linguistics.
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. Rocketqa: An optimized training approach to dense passage retrieval for opendomain question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 5835–5847. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
Stephen E. Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr., 3(4):333–389.
Joseph John Rocchio Jr. 1971. Relevance feedback in information retrieval. The SMART retrieval system: experiments in automatic document processing.
Dwaipayan Roy, Sumit Bhatia, and Mandar Mitra. 2019. Selecting discriminative terms for relevance model. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1253–1256.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian CantonFerrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and finetuned chat models. CoRR, abs/2307.09288.
Liang Wang, Nan Yang, and Furu Wei. 2023. Query2doc: Query expansion with large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9414–9423, Singapore. Association for Computational Linguistics.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. A survey of large language models. CoRR, abs/2303.18223.
Zhi Zheng, Kai Hui, Ben He, Xianpei Han, Le Sun, and Andrew Yates. 2020. BERT-QE: contextualized query expansion for document re-ranking. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 4718–4728. Association for Computational Linguistics.
Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. 2021. Retrieving and reading: A comprehensive survey on open-domain question answering. CoRR, abs/2101.00774.

heading: A Dataset Information
text: Natural Questions (NQ) (Kwiatkowski et al., 2019) introduced as a question answering dataset, is comprised of real, anonymized, aggregated queries submitted to the Google search engine. The dataset includes 79,168 training examples, 8,757 development examples, and 3,610 test examples.
TriviaQA (Trivia) (Joshi et al., 2017) is a substantial and realistic text-based question answering dataset, which encompasses 950,000 questionanswer pairs derived from 662,000 documents sourced from Wikipedia and the web. The dataset is composed of 60,413 training examples, 8,377 development examples, and 11,313 test examples, providing a diverse range of scenarios that test the depth and adaptability of QA systems.
WebQuestions (WebQ) (Berant et al., 2013) designed for question answering tasks, utilizes Freebase as its underlying knowledge base and consists of 6,642 question-answer pairs. This dataset was developed by sourcing questions through the Google Suggest API, followed by obtaining corresponding answers via Amazon Mechanical Turk. It is structured with an original split of 3,778 training examples and 2,032 testing examples. All answers are defined as Freebase entities.
CuratedTREC (TREC) (Baudis and Sedivý, 2015) is a reference question dataset for benchmarking Question Answering systems created from the TREC-8 (1999) to TREC-13 (2004). It comprises a concise yet focused collection of 694 annotated data entries, making it an ideal resource for evaluating the precision and effectiveness of QA systems under test conditions.

heading: B Experimental Details
text: In this appendix section, we elaborate on the more details of the model parameter settings employed during the inference process of the AGR method, along with the precise prompts used in all LLMbased QE methods throughout our experiments.
B.1 Backbone Model
We conducted a preliminary experiment to identify the most appropriate model foundation, evaluating the performance of prominent LLMs including Llama-3B, Llama2-7B, Llama2-13B, and Mistral-7B. The Base method was designed for direct answer-oriented text generation, whereas the AGR method was simplified, excluding contextual
references integration, to expedite evaluation. Notably, prompt adaptability varied across models to ensure compatibility, maintaining a consistent overall framework.
Experimental results, depicted in Figure 5 show that the effectiveness of the AGR method is intrinsically linked to the size and capabilities of underlying model. Under the Llama-3B setup, AGR underperformed due to the model’s limited grasp of complex prompts. While it consistently achieves improvements on larger models, with enhancements scaling alongside model capabilities. This highlights that models with greater robustness and advanced semantic understanding better complement the AGR approach. Specifically, the Mistral-7B model outshines Llama2-7B and Llama2-13B in reasoning tasks, in line with the findings of Jiang et al. (2023) research.
B.2 Additional Parameter Details For the detailed configuration of model parameters within the AGR framework, we utilize the SamplingParams class from vLLM (Kwon et al., 2023) to facilitate precise and optimized settings across
different stages. Key parameters include "temperature" set at 0.2 during the Analysis and Refinement phases and adjusted to 0.8 for the Sampling Generation phase. "max_tokens" is configured to 150 for Analysis, 100 per item in Sampling Generation, and 300 for Refinement. "repetition_penalty" is consistently set at 1.1 across all phases, and "top_p" is maintained at 1.0.
The configuration of these model parameters is established through testing with a very limited set of sample data from NQ validation dataset. However, given that the initial parameter choices were based on empirical estimates, there remains the potential for undiscovered, more effective parameter combinations.
B.3 Prompts In this subsection, we provide a thorough description of the prompts utilized in all LLM-based QE methods employed in our experiment. Table 5 details the prompts used in Q2D, Q2E, and their variants, while Table 6 displays the prompts applied in each sub-module of AGR. The objective of this subsection is to clarify the precise nature of the prompts, facilitating a deeper understanding of our study. It is important to note that these prompts are configured for the Mistral-7B model, and slight adaptability adjustments may be necessary when applying them to other models.
B.4 Contextual References In the experiment of AGR, we explore the impact of deduplicating the retrieved contextual references before proceeding with subsequent steps. The final outcomes reveal that the Top-5 recall rate stood at 68.37%, marginally different from the nondeduplicated result of 68.47%. However, the endto-end Top-1 EM score was 36.52, significantly lower than the non-deduplicated result of 37.73. A possible explanation is that, without deduplication, the gold passages, being repeatedly retrieved across different QE candidates. This makes gold passages more significant informational weight, thereby benefiting the overall end-to-end answer generation.

heading: Future Work
text: In future work, we plan to explore a more comprehensive framework, further integrating the requirements of the reader component to achieve a unified architecture and enhance the end-to-end performance for OpenQA task. Limitations
Firstly, as shown in our experimental analysis, the effectiveness of our proposed AGR, an LLMs-based QE generation framework, is intrinsically linked to the quality of the underlying backbone model. It is needful to find out a suitable LLM to fully demonstrate the AGR’s capacity, and carry out further investigations with more types of LLMs, like GPT-4. Secondly, although AGR avoids additional supervised training data and conserves computational resources, its reliance on multiple inference generations with LLMs introduces certain latency. Therefore, further exploration of strategies to accelerate LLM inference and reduce overall latency are left for future work. Finally, the generations from LLMs are associated with the input prompts. Although we have tested multiple configurations to achieve the current performance, the possibility of better configurations exists