Synergistic Interplay between Search and Large Language Models for Information Retrieval (2024)

Abstract: Information retrieval (IR) plays a crucial role in locating relevant resources from vast amounts of data, and its applications have evolved from traditional knowledge bases to modern retrieval models (RMs). The emergence of large language models (LLMs) has further revolutionized the IR field by enabling users to interact with search systems in natural languages. In this paper, we explore the advantages and disadvantages of LLMs and RMs, highlighting their respective strengths in understanding user-issued queries and retrieving up-to-date information. To leverage the benefits of both paradigms while circumventing their limitations, we propose InteR, a novel framework that facilitates information refinement through synergy between RMs and LLMs. InteR allows RMs to expand knowledge in queries using LLM-generated knowledge collections and enables LLMs to enhance prompt formulation using retrieved documents. This iterative refinement process augments the inputs of RMs and LLMs, leading to more accurate retrieval. Experiments on large-scale retrieval benchmarks involving web search and low-resource retrieval tasks show that InteR achieves overall superior zero-shot retrieval performance compared to state-of-the-art methods, even those using relevance judgment. Source code is available at https://github. com/Cyril-JZ/InteR.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9571–9583 August 11-16, 2024 ©2024 Association for Computational Linguistics
Information retrieval (IR) plays a crucial role in locating relevant resources from vast amounts of data, and its applications have evolved from traditional knowledge bases to modern retrieval models (RMs). The emergence of large language models (LLMs) has further revolutionized the IR field by enabling users to interact with search systems in natural languages. In this paper, we explore the advantages and disadvantages of LLMs and RMs, highlighting their respective strengths in understanding user-issued queries and retrieving up-to-date information. To leverage the benefits of both paradigms while circumventing their limitations, we propose InteR, a novel framework that facilitates information refinement through synergy between RMs and LLMs. InteR allows RMs to expand knowledge in queries using LLM-generated knowledge collections and enables LLMs to enhance prompt formulation using retrieved documents. This iterative refinement process augments the inputs of RMs and LLMs, leading to more accurate retrieval. Experiments on large-scale retrieval benchmarks involving web search and low-resource retrieval tasks show that InteR achieves overall superior zero-shot retrieval performance compared to state-of-the-art methods, even those using relevance judgment. Source code is available at https://github. com/Cyril-JZ/InteR.

heading: 1 Introduction
text: Information retrieval (IR) is an indispensable technique for locating relevant resources in a vast sea of data given ad-hoc queries (Mogotsi, 2010). It is a core component in knowledge-intensive tasks such as question answering (Karpukhin et al., 2020; Cao et al., 2024), entity linking (Gillick et al., 2019)
* Equal Contribution. † Correspondence to: Chongyang Tao
and fact verification (Thorne et al., 2018). Over the years, the techniques of IR have evolved significantly: from the traditional knowledge base (KB) (Lan et al., 2021; Gaur et al., 2022) to modern search engines (SEs) based on neural representation learning (Karpukhin et al., 2020; Yates et al., 2021), IR has become increasingly important in our digital world. More recently, the emergence of cutting-edge large language models (LLMs; e.g., ChatGPT (OpenAI, 2022), GPT-4 (OpenAI, 2023), Bard (Google, 2023), LLaMA (Touvron et al., 2023a,b)) has further revolutionized the NLP community and given intriguing insights into IR applications as users can now interact with search systems in natural languages.
Over the decades, search engines like Google or Bing have become a staple for people looking to retrieve information on a variety of topics, allowing users to quickly sift through millions of documents to find the information they need by providing keywords or a query. Spurred by advancements in scale, LLMs have now exhibited the ability to undertake a variety of NLP tasks in a zero-shot scenario (Qin et al., 2023) by following instructions (Ouyang et al., 2022; Sanh et al., 2022; Min et al., 2022; Wei et al., 2022; Xu et al., 2023). Therefore, they could serve as an alternative option for people to obtain information directly by posing a question or query in natural languages (OpenAI, 2022), instead of relying on specific keywords. For example, suppose a student is looking to write a research paper on the history of jazz music. They could type in keywords such as “history of jazz” or “jazz pioneers” to retrieve relevant articles and sources. However, with LLMs, this student could pose a question like “Who were the key pioneers of jazz music, and how did they influence the genre?” The LLMs could then generate a summary of the relevant information and sources, potentially sav-
9571
ing time and effort in sifting through search results. As with most things in life, there are two sides to every coin. Both IR technologies come with their own unique set of advantages and disadvantages. LLMs excel in understanding the context and meaning behind user-issued textual queries (Mao et al., 2023; Feng et al., 2023; Niu et al., 2024), allowing for more precise retrieval of information, while RMs expect well-designed precise keywords to deliver relevant results. Moreover, LLMs have the capacity to directly generate specific answers to questions (Ouyang et al., 2022), rather than merely present a list of relevant documents, setting them apart from RMs. However, it is important to note that RMs still have significant advantages over LLMs. For instance, RMs can index a vast number of up-to-date documents (Nakano et al., 2021), whereas LLMs can only generate information that falls within the time-scope of the data they were trained on, potentially leading to hallucinated results (Shuster et al., 2021; Ji et al., 2023; Zhang et al., 2023b,c). Additionally, RMs can conduct quick and efficient searches through a vast amount of information on the internet, making them an ideal choice for finding a wide range of data. Ultimately, both paradigms have their own unique set of irreplaceable advantages, making them useful in their respective areas of application.
To enhance IR by leveraging the benefits of RMs and LLMs while circumventing their limitations, we consider bridging Retrieval Augmented Generation (RAG) and LLM-augmented Retrieval. Fortunately, we observe that textual information refinement can be performed between two counterparts and boost each other. On the one hand, RMs can gather potential documents with valuable information, serving as demonstrations for LLMs. On the other hand, LLMs generate concise summaries using well-crafted prompts, expanding the initial query and improving search accuracy. To this end, we introduce InteR, a novel framework that facilitates information refinement through synergy between RMs and LLMs. Precisely, the RM part of InteR receives the knowledge collection from the LLM part to refine and expand the information in the query. While the LLM part involves the retrieved documents from the RM part as demonstrations to enrich the information in prompt formulation. This two-step refinement procedure can be seamlessly repeated to augment the inputs of RM and LLM. Implicitly, we assume that the outputs of both components supplement each other, leading
to more accurate retrieval. We evaluate InteR on public large-scale retrieval benchmarks involving web search and lowresource retrieval tasks following prior work (Gao et al., 2023). The experimental results show that InteR can conduct zero-shot retrieval with overall better performance than state-of-the-art methods, even those using relevance judgment1, and achieves new state-of-the-art zero-shot retrieval performance. Overall, our main contributions can be summarized as follows:
• We introduce InteR, a novel IR framework bridging two cutting-edge IR products, search systems and large language models, while enjoying their strengths and circumventing their limitations.
• We propose iterative information refinement via synergy between retrieval models and large language models, resulting in improved retrieval quality.
• Evaluation results on zero-shot retrieval demonstrate that InteR can overall conduct more accurate retrieval than state-of-the-art approaches and even outperform baselines that leverage relevance judgment for supervised learning.

heading: 2 Related Work
text: Dense Retrieval Document retrieval has been an important component for several knowledgeintensive tasks (Voorhees et al., 1999; Karpukhin et al., 2020). Traditional techniques such as TFIDF and BM25 depend on term matching and create sparse vectors (Robertson, 2009; Yang et al., 2017; Chen et al., 2017) to ensure efficient retrieval. After the emergence of pre-trained language models (Devlin et al., 2019; Liu et al., 2019), dense retrieval which encodes both queries and documents into low-dimension vectors and then calculates their relevance scores (Lee et al., 2019; Karpukhin et al., 2020; Cai et al., 2022), has recently undergone substantial research. Relevant studies include improving training approach (Karpukhin et al., 2020; Xiong et al., 2021; Qu et al., 2021), distillation (Lin et al., 2021; Hofstätter et al., 2021; Zhang et al., 2023a) and task-specific pre-training (Izacard et al., 2022; Gao and Callan, 2021; Lu et al., 2021;
1In IR tasks, the relevance judgment illustrates the label of relevance between each pair of query and document, which is mainly used for supervised learning of an IR model.
Gao and Callan, 2022; Xiao et al., 2022; Shen et al., 2022) of dense retrieval models which significantly outperform sparse approaches.
Zero-shot Dense Retrieval Many prior works consider training dense retrieval models on highresource passage retrieval datasets like Natural Questions (NQ) (Kwiatkowski et al., 2019) (133k training examples) or MS-MARCO (Bajaj et al., 2016) (533k training examples) and then evaluating on queries from new tasks. These systems (Wang et al., 2022; Yu et al., 2022) are utilized in a transfer learning configuration (Thakur et al., 2021). However, on the one hand, it is time-consuming and expensive to collect such a vast training corpus. On the other hand, even MS-MARCO has limitations on commercial use and cannot be used in a wide range of real-world applications. To this end, recent work (Gao et al., 2023) proposes building zero-shot dense retrieval systems that require no relevance supervision (i.e., relevance label between a pair of query and document), which is considered “unsupervised” as the only supervision resides in the LLM where learning to follow instructions is conducted (Sachan et al., 2022). In this work, we follow this zero-shot unsupervised setting and conduct information refinement through synergy between RMs and LLMs without any relevance supervision to handle the above issues.
Enhance Retrieval Through LMs Recent works have investigated using auto-regressive language models to generate intermediate targets for better retrieval (Cao et al., 2021; Bevilacqua et al., 2022) while identifier strings still need to be created. Other works consider “retrieving” the knowledge stored in the parameters of pre-trained language models by directly generating text (Petroni et al., 2019; Roberts et al., 2020). Some researchers (Mao et al., 2021; Anantha et al., 2021; Wang et al., 2023) utilize LM to expand the query and incorporate these pseudo-queries for enhanced retrieval while others choose to expand the document (Nogueira et al., 2019). Besides, LMs can also be exploited to provide references for retrieval targets. For instance, GENREAD (Yu et al., 2023) directly generates contextual documents for given questions.
Enhance LMs Through Retrieval On the contrary, retrieval-enhanced LMs have also received significant attention. Some approaches enhance the accuracy of predicting the distribution of the next word during training (Borgeaud et al., 2022) or
inference (Khandelwal et al., 2020) through retrieving the k-most similar training contexts. Alternative methods utilize retrieved documents to provide supplementary context in generation tasks (Joshi et al., 2020; Guu et al., 2020; Lewis et al., 2020). WebGPT (Nakano et al., 2021) further adopts imitation learning and uses human feedback in a textbased web-browsing environment to enhance the LMs. LLM-Augmentor (Peng et al., 2023) improves LLMs with external knowledge and automated feedback. REPLUG (Shi et al., 2023) prepends retrieved documents to the input for the frozen LM and treats LM as a black box. Demonstrate–Search–Predict (DSP) (Khattab et al., 2022) obtains performance gains by relying on passing natural language texts in sophisticated pipelines between a LM and a RM, which is most closely related to our approach. However, they rely on composing two parts with in-context learning and target on multi-hop QA. While we aim at conducting information refinement via multiple interactions between RMs and LLMs for large-scale retrieval.

heading: 3 Preliminary
text: Document Retrieval: the RM Part Zero-shot document retrieval is a crucial component of search systems. Given the user query q and the document set D = {d1, ..., dn} where n is the number of document candidates, the goal of a retrieval model (RM) is to retrieve documents that are relevant to satisfy the user’s real search intent of the current query q. To accomplish such document retrieval, prior works can be categorized into two groups: sparse retrieval and dense retrieval. Both lines of research elaborate on devising the similarity function ϕ(q, d) for each query-document pair.
The sparse retrieval, e.g., TF-IDF and BM25, depends on lexicon overlap between query q and document d. This line of RMs (Zhou et al., 2022; Thakur et al., 2021) ranks documents D based on their relevance to a given query q by integrating term frequency and inverse document frequency. Another works (Qu et al., 2021; Ni et al., 2022; Karpukhin et al., 2020) focus on dense retrieval that uses two encoding modules to map an input query q and a document d into a pair of vectors ⟨vq,vd⟩, whose inner product is leveraged as a similarity function ϕ:
ϕ(q, d) = ⟨EQ(q), ED(d)⟩ = ⟨vq,vd⟩ (1) Then the top-k documents, denoted as D̄ that have the highest similarity scores when compared with
the query q, are retrieved by RMs regardless of whether the retrieval is sparse or dense. Noting that as for dense retrieval, following existing methods (Gao et al., 2023), we pre-compute each document’s vector vd for efficient retrieval and build the FAISS index (Johnson et al., 2019) over these vectors, and use Contriever (Izacard et al., 2022) as the backbone of query encoder EQ and document encoder ED.
Generative Retrieval: the LLM Part Generative search is a new paradigm of IR that employs neural generative models as search indices (Tay et al., 2022; Bevilacqua et al., 2022; Lee et al., 2022). Recent studies propose that LLMs further trained to follow instructions could zero-shot generalize to diverse unseen instructions (Ouyang et al., 2022; Sanh et al., 2022; Min et al., 2022; Wei et al., 2022). Therefore, we prepare textual prompts p that include instructions for the desired behavior to q and obtain a refined query q′. Then the LLMs G such as ChatGPT (OpenAI, 2022) take in q′ and generate related knowledge passage s. This process can be illustrated as follows:
s = G(q′) = G(q ⊕ p) (2)
where ⊕ is the prompt formulation operation for q and p. For each q′, if we sample h examples via LLM G, we will obtain a knowledge collection S = {s1, s2, ..., sh}.

heading: 4 InteR
text: On top of the preliminaries, we introduce InteR, a novel IR framework that iteratively performs information refinement through synergy between RMs and LLMs. The overview is shown in Figure 1. During each iteration, the RM part and LLM part refine their information in the query through interplay with knowledge collection (via LLMs) or retrieved documents (via RMs) from previous iteration. Specifically, in RM part, InteR refines the information stored in query q with knowledge collection S generated by LLM for better document retrieval. While in LLM part, InteR refines the information in original query q with retrieved document D̄ from RM for better invoking LLM to generate most relevant knowledge. This twostep procedure can be repeated multiple times in an iterative refinement style.

heading: 4.1 RM Step: Refining Information in RM via LLM
text: When people use search systems, the natural way is to first type in a search query q whose genre can be a question, a keyword, or a combination of both. The RMs in search systems then process the search query q and retrieve several documents D̄ based on their relevance ϕ(q, d) to the search query q. Ideally, D̄ contains the necessary information related to the user-issued query q. However, it may include irrelevant information to query as the candidate documents for retrieval are chunked and fixed (Yu et al., 2023). Moreover, it may also miss
some required knowledge since the query is often fairly condensed and short (e.g., “best sushi in San Francisco”).
To this end, we additionally involve the generated knowledge collection S from LLM in previous iteration and enrich the information included in q with S. Specifically, we consider expanding the query q by concatenating each si ∈ S multiple times2 to q and obtaining the similarity of document d with:
ϕ(q, d;S) = ϕ([q; s1; q; s2; ..., q; sh], d)
= ⟨EQ([q; s1; q; s2; ..., q; sh]), ED(d)⟩ (3)
where [·; ·] is a concatenating operation for query expansion. Now the query is knowledge-intensive equipping with S from LLM part that may be supplement to q. We hope the knowledge collection S can provide directly relevant information to the input query q and help the RMs focus on the domain or topic in user query q.

heading: 4.2 LLM Step: Refining Information in LLM via RM
text: As aforementioned, we can invoke LLMs to conditionally generate knowledge collection S by preparing a prompt p that adapts the LLM to a specific function (Eq. 2). Despite the remarkable text generation capability, they are also prone to hallucination and still struggle to represent the complete long tail of knowledge contained within their training corpus (Shi et al., 2023). To mitigate the aforementioned issues, we argue that D̄, the documents retrieved by RMs, may provide rich information about the original query q and can potentially help the LLMs make a better prediction.
Specifically, we include the knowledge in D̄ into p by designing a new prompt as:
Given a question {query} and its possible answering passages {passages} Please write a correct answering passage:
where “{query}” and “{passages}” are the placeholders for q and D̄ respectively from last RM step:
s = G(q′) = G(q ⊕ p⊕ D̄) (4)
Now the query q′ is refined and contains more plentiful information about q through retrieved docu-
2In our preliminaries, we observed that concatenating each si ∈ S multiple times to q can lead to improved performance, as the query is the most crucial component in IR.
ments D̄ as demonstrations. Here we simply concatenate D̄ for placeholder “{passages}”, which contains k retrieved documents from RM part for input of LLM G.

heading: 4.3 Iterative Interplay Between RM and LLM
text: In this section, we explain how iterative refinement can be used to improve both RM and LLM parts. This iterative procedure can be interpreted as exploiting the current query q and previous-generated knowledge collection S to retrieve another document set D̄ with RM part for the subsequent stage of LLM step. Then, the LLM part leverages the retrieved documents D̄ from previous stage of RM and synthesizes the knowledge collection S for next RM step. A critical point is that we take LLM as the starting point and use only q and let D̄ be empty as the initial RM input. Therefore, the prompt of first LLM step is formulated as:
Please write a passage to answer the question. Question: {query} Passage:
We propose using an iterative IR pipeline, with each iteration consisting of four steps listed below:
1. Invoke LLM to conditionally generate knowledge collection S with prompt q′ on Eq. 4. The retrieved document set D̄ is derived from previous RM step and set as empty in the beginning.
2. Construct the updated input for RM with knowledge collection S and query q to compute the similarity of each document d.
3. Invoke RM to retrieve the top-k most “relevant” documents as D̄ on Eq. 3.
4. Formulate a new prompt q′ by combining the retrieved document set D̄ with query q.
The iterative nature of this multi-step process enables the refinement of information through the synergy between the RMs and the LLMs, which can be executed repeatedly M times to further enhance the quality of results.

heading: 5 Experiments

heading: 5.1 Datasets and Metrics
text: Following Gao et al. (2023), we adopt widely-used web search query sets TREC Deep Learning 2019
(DL’19) (Craswell et al., 2020) and Deep Learning 2020 (DL’20) (Craswell et al., 2021) which are based on the MS-MARCO (Bajaj et al., 2016). Besides, we also use six diverse low-resource retrieval datasets from the BEIR benchmark (Thakur et al., 2021) strictly consistent with Gao et al. (2023) including SciFact (fact-checking), ArguAna (argument retrieval), TREC-COVID (bio-medical IR), FiQA (financial question-answering), DBPedia (entity retrieval), and TREC-NEWS (news retrieval). It is worth pointing out that we do not employ any training query-document pairs, as we conduct retrieval in a zero-shot setting and directly evaluate our proposed method on these test sets. Consistent with prior works, we report MAP, nDCG@10, and Recall@1000 (R@1k) for TREC DL’19 and DL’20 data, and nDCG@10 is employed for all datasets in the BEIR benchmark.

heading: 5.2 Baselines
text: Methods without relevance judgment We consider several zero-shot retrieval models as our main baselines, because we do not involve any query-document relevance scores (denoted as w/o relevance judgment) in our setting. Particularly, we choose heuristic-based lexical retriever BM25 (Robertson and Zaragoza, 2009) (and with pseudo relevance feedback3, denoted as BM25 + PRF), and Contriever (Izacard et al., 2022) that
3As reported in the official Anserini report: https://github.com/castorini/anserini/blob/ 279fc3eecaed4d07c0a9c42017447b6ae87b820c/docs/ regressions/
is trained using unsupervised contrastive learning. We also compare our model with the state-of-theart LLM-based retrieval model HyDE (Gao et al., 2023) which shares the exact same embedding spaces with Contriever but builds query vectors with LLMs.
Methods with relevance judgment Moreover, we also incorporate several systems that utilize fine-tuning on extensive query-document relevance data, such as MS-MARCO, as references (denoted as w/ relevance judgment). This group encompasses some commonly used fully-supervised retrieval methods, including DPR (Karpukhin et al., 2020), ANCE (Xiong et al., 2021), and the finetuned Contriever (Izacard et al., 2022) (denoted as ContrieverFT).

heading: 5.3 Implementation Details
text: As for the LLM part, we evaluate our proposed method on two options: closed-source models and open-source models. In the case of closed-source models, we employ the gpt-3.5-turbo, as it is popular and accessible to the general public. As for the open-source models, our choice fell upon the Vicuna models (Chiang et al., 2023) derived from instruction tuning with LLaMa-1/2 (Touvron et al., 2023a,b). Specifically, we assessed the most promising 13B version of Vicuna from LLaMa2, namely, Vicuna-13B-v1.5. Additionally, we evaluated the current best-performing 33B version of Vicuna derived from LLaMa-1, which is Vicuna-33B-v1.3. As for the RM part, we con-
sider BM25 for retrieval since it is much faster. For each q′, we sample h = 10 knowledge examples via LLM. After hyper-parameter search on validation sets, we set k as 15 for gpt-3.5-turbo, and 5 for Vicuna-13B-v1.5 and Vicuna-33B-v1.3. We also set M as 2 by default. We use a temperature of 1 for LLM part in generation and a frequency penalty of zero. We also truncate each RM-retrieved passage/document to 256 tokens and set the maximum number of tokens for each LLMgenerated knowledge example to 256 for efficiency.

heading: 5.4 Main Results
text: Web Search In Table 1, we show zero-shot retrieval results on TREC DL’19 and TREC DL’20 with baselines. We can find that InteR with selected LLMs surpass state-of-the-art zero-shot baseline HyDE with significant improvement on most metrics. Specifically, InteR with gpt-3.5-turbo has an > 8% absolute MAP gain and > 5% absolute nDCG@10 gain on both web search benchmarks. Moreover, InteR is also superior to models with relevance judgment on most metrics, which verifies the generalization ability of InteR on large-scale retrieval. Note that our approach does not involve any training process and merely leverages off-theshelf RMs and LLMs, which is simpler in practice but shown to be more effective.
Low-Resource Retrieval In Table 2, we also present the zero-shot retrieval results on six diverse low-resource retrieval tasks from BEIR benchmarks. Firstly, we find that InteR is especially competent on TREC-COVID and TREC-NEWS and even significantly outperforms baselines with relevance judgment. Secondly, InteR also brings considerable improvements to baselines on SciFact
and DBPedia, which shows our performance advantages on fact-checking and entity retrieval. Finally, it can be observed that the performance of FiQA and ArguAna falls short when compared to the baseline models. This could potentially be attributed to the LLM’s limited financial knowledge of FiQA and the RM’s marginal qualification to effectively handle relatively longer queries for ArguAna (Thakur et al., 2021).

heading: 5.5 Further Discussions
text: The impact of the size of knowledge collection (h) We conducted additional research to examine the impact of the size of knowledge collection (i.e., h) on the performance of InteR. Figure 2 illustrates the changes in MAP and nDCG@10 curves of InteR with gpt-3.5-turbo on TREC DL’19 and DL’20, with respect to varying numbers of knowledge examples. Our observations reveal a consistent pattern in both benchmarks: as the number of generated knowledge increases, the performance metrics demonstrate a gradual improvement until reaching 10 knowledge examples. Subsequently, the performance metrics stabilize, indicating that additional knowledge examples do not significantly
enhance the results. This phenomenon could be attributed to the presence of redundant knowledge within the surplus examples generated by LLMs.
The impact of the number of information refinement iterations (M ) We also investigated the effect of different numbers of information refinement iterations (M ) on the performance of InteR. The results of InteR with gpt-3.5-turbo presented in Table 3 indicate a notable enhancement in retrieval capacity as M increases from 0 to 2, which verifies the effectiveness of multiple iterative information refinement between RMs and LLMs. However, if we further increase M , the performance may not improve, possibly due to a decrease in the diversity of retrieved documents from RMs. Here if we set M to 0, InteR will degenerate into BM25.
Dense retrieval v.s. sparse retrieval Furthermore, we delve into the impact of the retrieval strategy for constructing D̄ on Eq. 4 on the performance of InteR. Table 4 shows the experimental results of InteR with gpt-3.5-turbo, where we initiate the RM with an unsupervised sparse RM (i.e., BM25) or an unsupervised dense RM (i.e., Contriever). Additionally, we introduce a hybrid retrieval paradigm that combines half of the documents from the sparse RM with the other half from the dense RM. These components are then merged to generate the output of the hybrid RM, namely InteR (Hybrid). We can observe that the dense model is the best strategy for InteR and even outperforms the hybrid counterpart, which is consistent with the
observations from prior works (Karpukhin et al., 2020). Therefore, we use a dense RM to construct D̄ as default. It is noteworthy that we only use a dense RM for constructing D̄ during information refinement and leave the final RM as a sparse BM25 for efficiency.
Case: query_id: 1112341, query: “what is the daily life of thai people”
LLM outputs of M = 1: The daily life of Thai people is heavily influenced by their culture and traditions. The majority of the population practices Buddhism, and daily prayer and meditation are common. Family is also extremely important, and many Thai people live with extended family members. In terms of food, Thai cuisine is known for its spicy and savory flavors, and meals often revolve around rice and noodles. Street food is also a big part of daily life, with vendors selling snacks and meals on every corner. Work is an important aspect of daily life as well, and many Thai people work long hours in industries such as agriculture, tourism, and manufacturing. Despite the demands of work, there is a strong emphasis on taking breaks throughout the day to socialize and interact with others. Overall, the daily life of Thai people is a harmonious blend of religious practice, family values, flavorful cuisine, and hard work.
LLM outputs of M = 2: Thailand’s daily life is influenced by its diverse geography and cultural influences from India, China, Cambodia, and the rest of Southeast Asia. The country’s national religion, Theravada Buddhism, is central to modern Thai identity. Social occasions in Thailand often revolve around food, which is ordered and eaten in a communal style. Thai food presentation is among the most exquisite in the world, developed primarily in the palace to please the King of Siam. Thai people love to have fun together and often ask each other if they had a good time. The population of Thailand is approximately 67.5 million people, including ethnic groups such as Chinese, Malay, Lao, Burma, Cambodia, and Indians. Unemployment rate in Thailand is relatively low, standing at 0.84% according to Thailand’s National Economic and Social Development Board (NESDB). Thai time is divided into five sections, with different names for each section of the day. The country’s primary religion is Theravada Buddhism, and important holidays include Thai New Year, or Songkran. Overall, the daily life of Thai people is diverse, vibrant and deeply rooted in their rich cultural heritage.
Case Study on LLM Outputs To better illustrate how RMs enrich the outputs of LLMs, we selected an example from TREC DL’19 and compared those outputs of gpt-3.5-turbo after the first and second iterations. Due to the limited space, we only show the first LLM-generated knowledge (i.e., s1). In the initial iteration (M = 1), the LLM demonstrated its capacity to generate paragraphs by leveraging its commonsense encoded within its parameters. After taking the search results of RM (documents about ‘Thailand’) in the prompt, LLM outputs in the second iteration M = 2 became more informative. Specifically, the text highlighted
in yellow elaborated on Thailand’s population and unemployment rate, which was absent in the first iteration, and facilitated the next RM step.

heading: 6 Conclusion
text: In this work, we present InteR, a novel framework that harnesses the strengths of both large language models (LLMs) and retrieval models (RMs) to enhance information retrieval. By facilitating information refinement through synergy between LLMs and RMs, InteR achieves overall superior zero-shot retrieval performance compared to state-of-the-art methods, and even those using relevance judgment, on large-scale retrieval benchmarks involving web search and low-resource retrieval tasks. With its ability to leverage the benefits of both paradigms, InteR may present a potential direction for advancing information retrieval systems.
Limitations
While InteR demonstrates improved zero-shot retrieval performance, it should be noted that its effectiveness heavily relies on the quality of the used large language models (LLMs). If these underlying components contain biases, inaccuracies, or limitations in their training data, it could impact the reliability and generalizability of the retrieval results. In that case, one may need to design a more sophisticated method of information refinement, especially the prompt formulation part. We leave this exploration for future work.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for their constructive comments and suggestions. This work was supported by the National Key R&D Program of China (No.2022YFC3301905). Jiazhan Feng and Dongyan Zhao are also with the Wangxuan Institute of Computer Technology, Peking University.

heading: Future Work
text: We leave this exploration for future work