INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning (2024)

Abstract: Large language models (LLMs) have demonstrated impressive capabilities in various natural language processing tasks. Despite this, their application to information retrieval (IR) tasks is still challenging due to the infrequent occurrence of many IR-specific concepts in natural language. While prompt-based methods can provide task descriptions to LLMs, they often fall short in facilitating a comprehensive understanding and execution of IR tasks, thereby limiting LLMs’ applicability. To address this gap, in this work, we explore the potential of instruction tuning to enhance LLMs’ proficiency in IR tasks. We introduce a novel instruction tuning dataset, INTERS, encompassing 20 tasks across three fundamental IR categories: query understanding, document understanding, and query-document relationship understanding. The data are derived from 43 distinct datasets with manually written templates. Our empirical results reveal that INTERS significantly boosts the performance of various publicly available LLMs, such as LLaMA, Mistral, and Falcon, in IR tasks. Furthermore, we conduct extensive experiments to analyze the effects of instruction design, template diversity, few-shot demonstrations, and the volume of instructions on performance. We make our dataset and the fine-tuned models publicly accessible at https: //github.com/DaoD/INTERS.

heading: 1 Introduction
text: Large language models (LLMs) have shown remarkable capabilities across various natural language processing (NLP) tasks. While these models have learned vast knowledge from large text corpora, their (pre-)training objective is not aligned with human’s objective: the latter requires models to “follow human instructions and perform tasks”
*This work was done when Chenghao Zhang and Yifei Chen were doing internships at Renmin University of China.
†Corresponding author.
rather than “predict the next token”. To address this mismatch, instruction tuning is proposed, serving as an effective technique to align LLMs with human tasks and preferences (Ouyang et al., 2022; Wei et al., 2022; Chung et al., 2022; Mishra et al., 2022; Wang et al., 2022, 2023b). After instruction tuning, LLMs can better understand users’ intent and show impressive generalization to new tasks.
In the area of information retrieval (IR), the introduction of LLMs has also led to notable developments (Wang et al., 2023a; Tang et al., 2023; Sun et al., 2023; Ma et al., 2023). Due to the high cost of fine-tuning, many existing studies leverage prompting methods to apply LLMs in IR tasks. However, some of them have reported that LLMs cannot consistently outperform fine-tuned smaller models in this manner (Sun et al., 2023; Gao et al., 2023). For example, RankGPT (Sun et al., 2023) based on gpt-3.5-turbo underperforms monoBERT with 340M parameters on passage ranking tasks. This discrepancy may stem from the complexity of IRspecific concepts like queries, relevance, and search intent, which are infrequently encountered in pretraining corpora and are inherently challenging to comprehend.
To fill the gap, in this work, we build a novel INstruction Tuning datasEt foR Search (INTERS).1
1This paper will use the terms “search-related tasks” and information retrieval tasks” interchangeably.
2782
This dataset is designed to specifically enhance the search capabilities of LLMs. Since the search process involves various tasks, we choose to focus on three pivotal aspects: query understanding, document understanding, and the understanding of the relationship between queries and documents. We collect 43 datasets covering 20 distinct tasks, ensuring the dataset’s comprehensive coverage and richness. In order to improve diversity and broaden applicability, we manually craft 12 unique templates for each dataset and consider both zero-shot and few-shot examples in data generation. To further improve the models’ generalizability, we also manually write a detailed task description for each task, which serves as a bridge to connect each dataset under the same task. Finally, we plan to release all data, templates, model checkpoints, experimental results, and source codes. We wish the openness of our dataset could support more future research and development in this field.
We conduct experiments by fine-tuning several open-sourced LLMs using the INTERS dataset. Experimental results show that INTERS consistently enhances the performance of LLMs of different sizes across a spectrum of search tasks. Notably, this improvement is observed not only in tasks that are directly learned in the training data (in-domain) but also in tasks that are unseen in the training set (out-of-domain). Our further experiments highlight several key insights: (1) customized templates and task descriptions effectively improve model performance; (2) the diversity of templates can enhance model generalizability; (3) instruction tuning specifically tailored for search tasks addresses the existing gap of NLP instructions for such tasks; (4) combining instruction tuning and few-shot prompting can further improve performance; and (5) the substantial data volume can benefit the efficacy of instruction tuning.

heading: 2 Related Work
text: Large Language Models for Information Retrieval LLMs possess a remarkable capacity for language understanding, enabling them to be highly valuable in comprehending user queries and documents. Therefore, many researchers have explored applying LLMs to IR tasks (Zhu et al., 2023). Existing studies can be roughly categorized into two groups. The first group of studies treats LLMs as search agents to accomplish search tasks (Nakano et al., 2021; Qin et al., 2023a; Liu et al., 2023). A
typical method is WebGPT (Nakano et al., 2021), which employs imitation learning to teach an LLM (i.e., GPT-3) to use search engines and answer questions like a human. The other group of studies mainly focuses on applying LLMs to specific IR tasks, such as query reformulation (Wang et al., 2023a; Srinivasan et al., 2022; Tang et al., 2023; Mao et al., 2023) and document ranking (Sun et al., 2023; Zhang et al., 2023b; Ma et al., 2023; Zhuang et al., 2023). Most of these studies rely on prompting LLMs in a zero-shot or few-shot manner. However, due to the inherent complexity of the IR task and the relative scarcity of IR-related concepts in natural language texts, LLMs often cannot achieve superior performance to fine-tuned smaller models in IR tasks (Sun et al., 2023; Gao et al., 2023).
Different from existing studies, our research focuses on using instruction tuning to improve the overall performance of LLMs on various search tasks. This involves enhancing the models’ abilities to interpret and respond to search-related instructions more effectively, thereby improving their utility in complex IR scenarios.
Instruction Tuning for LLMs Instruction tuning (IT) aims at fine-tuning pre-trained LLMs on a collection of formatted instances in the form of natural language (Wei et al., 2022; Mishra et al., 2022; Wang et al., 2022, 2023b). After IT, LLMs can better follow instructions and perform human tasks. This approach bears a close resemblance to supervised fine-tuning (Ouyang et al., 2022) and multitask prompt training (Sanh et al., 2022). Instruction tuning’s efficacy lies in its ability to not only enhance LLMs’ performance on tasks they have been directly trained on but also to equip them with the ability to generalize to new, unseen tasks (Sanh et al., 2022; Wei et al., 2022).
In this work, we leverage IT to specifically enhance LLMs’ performance on search-related tasks. Our dataset is designed with a deep understanding of the task characteristics. Experiments will show that IT is also an effective way to improve LLMs’ overall performance on search tasks.

heading: 3 Instruction Tuning for Search
text: Instruction tuning has proven to be effective for LLMs in responding to instructions. This method essentially involves training LLMs through supervised learning to execute particular tasks based on provided instructions. A notable benefit of this approach is that, after fine-tuning, LLMs can compre-
hend and execute instructions not only for similar tasks but also for tasks they have not learned before. However, it is important to note that search tasks, which are the focus of our study, differ significantly from typical NLP tasks in terms of their objectives and structures. Search tasks primarily revolve around two key elements: queries and documents. Therefore, as shown in Figure 2, we consider collecting tasks and datasets in three categories: query understanding, document understanding, and query-document relationship understanding. We consider that tasks within these categories are instrumental in refining LLMs’ abilities to interpret queries, comprehend documents, and understand their relationships. Below, we introduce the tasks, datasets, and data construction process.

heading: 3.1 Tasks & Datasets
text: Developing a comprehensive instruction-tuning dataset covering a wide range of tasks is very resource-intensive. To address this, we follow the previous studies (Wei et al., 2022; Chung et al., 2022) and choose to convert existing datasets from the IR research community into an instructional format. We consider tasks under the categories of query understanding, document understanding, and query-document understanding. All tasks and datasets we used are shown in Figure 2. Their detailed descriptions, evaluation metrics, licenses, and examples are provided in Appendix A and G.
Query Understanding In IR, a query is a userinitiated request for information, typically composed of keywords, phrases, or natural language questions. It aims at retrieving relevant information from a retrieval system (e.g., a search engine). The effectiveness of a query is measured by its ability to accurately reflect the user’s intent and retrieve the most relevant documents. During the retrieval process, query understanding is a critical component in determining the efficiency and user satisfaction of the IR systems. Therefore, we collect a group of tasks (eight in total) that require models to understand the semantics of queries and capture the underlying user search intent.
Document Understanding In IR, a document refers to any piece of information that can be retrieved in response to a query, such as web pages in search engines. Document understanding is the process by which an IR system interprets and comprehends the content of these documents. Enhanced document understanding leads to better search results and an overall more efficient and user-friendly retrieval process. In INTERS, we collect datasets for four tasks that require a deep understanding of documents.

heading: Query-document Relationship Understanding
text: Query-document relationship understanding is the process of determining how well the content of a document matches or satisfies the intent behind a
user’s query. This involves interpreting the query’s semantics, context, and purpose, and then assessing the relevance of documents based on how closely they correspond to these aspects. It is the core task of information retrieval. We collect eight tasks specifically designed to enhance models’ capability of determining various query-document relationships, e.g., the question answering task involves understanding the relationship between questions and supporting evidences. It is important to recognize the variety of architectures available for modeling the query-document relationship. In this research, we focus on the reranking architecture, which is the most straightforward way to apply LLMs. The candidate documents for reranking are retrieved by BM25 (Robertson and Zaragoza, 2009).
3.2 INTERS Construction After determining the tasks and datasets we plan to use, we start to construct INTERS. The construction process is illustrated in Figure 3, which can be divided into four steps.
(1) Preprocessing. We download all datasets from publicly available resources, filter out unnecessary attributes and invalid data samples, and then convert them into the JSONL format for further processing.
(2) Template collection. We manually craft 12 distinct templates for each dataset to ensure the diversity and richness of the generated data. These templates use natural language instructions to describe the specific task associated with each dataset (two example templates are shown in the second part of Figure 3). To further improve the diversity of the templates, following the design of FLAN (Wei et al., 2022), we integrate up to two “inverse” templates per dataset. For example, for the query expansion task, we include templates that prompt for simplifying a query. In particular, for query-document relationship understanding tasks, there are three typical methods (Zhu et al., 2023; Qin et al., 2023b), i.e., pointwise, pairwise, and listwise. We consider all of them when writing templates (i.e., four for each type) to support a wider range of application scenarios (related discussion is presented in Section 4.4.5). Additionally, to enhance the LLMs’ task comprehension, we provide detailed descriptions for each task. These task descriptions serve a dual purpose: offering a granular understanding of the task’s objectives and establishing a linkage among datasets under the same task. The efficacy of this design will be
demonstrated through our experiments presented in Section 4.4.1.
(3) Example generation. For each sample in the preprocessed data, we use the corresponding task description and a randomly selected template to generate n-shot examples (where n ∈ [0, 5] in our experiments). The third part of Figure 3 shows a zero-shot example generated from the CANARD dataset. For few-shot examples (where n ≥ 1), we insert the n examples between the task description and the input, where the examples are separated by special tokens (i.e., “\n\n”). All few-shot examples are randomly selected from the training set. Moreover, to ensure that the few-shot examples are within the learnable scope of LLMs, we apply a length filter to exclude examples that exceed a predefined length threshold (2,048 tokens in our experiments).
(4) Example mixture. To compile INTERS, we randomly select examples from our entire collection until we accumulate a total of 200k examples.2
To balance the different sizes of datasets, we apply an examples-proportional mixing strategy (Raffel et al., 2020) with a mixing rate maximum of 5k. Under this scheme, any dataset contributing more than 5k examples does not receive extra weighting for the additional samples, thus preventing the dominant influence from larger datasets.

heading: 4 Experiments
text: We fine-tune several open-sourced LLMs on our INTERS, and evaluate their performance in different settings. Our experiments will investigate the following research questions: (1) Can the model obtain the capability to solve search tasks through instruction tuning on INTERS? (§4.2) (2) Is this capability generalizable? (§4.3) (3) Is our instruction design effective? (§4.4.1) (4) Are there any advantages compared to existing instruction sets? (§4.4.2) (5) Is the model still effective with fewshot demonstrations? (§4.4.3) (6) What is the impact of data volume? (§4.4.4) (7) What are the effects of different ranking strategies? (§4.4.5)

heading: 4.1 Backbone Models
text: We consider four LLMs in different sizes, ranging from 1B parameters to 7B parameters: Falcon-RW1B (Penedo et al., 2023), Minima-2-3B (Zhang et al., 2023a), Mistral-7B (Jiang et al., 2023), and
2This number is determined to strike a balance between efficacy and training costs.
LLaMA-2-7B (Touvron et al., 2023). These models are publicly available and widely used in many studies. Their detailed introduction and implementation details are provided in Appendix D.

heading: 4.2 In-domain Evaluation
text: We first perform an in-domain evaluation to validate the effectiveness of instruction tuning with INTERS on search tasks. In this experiment, we split all data into training, validation, and test sets (details are presented in Appendix B). The models are fine-tuned on the training set and evaluated on the test set. As all tasks and datasets are exposed during training, we call it an in-domain evaluation.
The experimental results are shown in Figure 4. Generally, after fine-tuning on INTERS, all models of varying sizes can achieve significantly better performance, demonstrating the effectiveness and broad applicability of instruction tuning in enhancing LLMs’ search performance. Besides, we have the following observations.
(1) On most datasets, larger models tend to perform better than smaller ones. For instance, LLaMA (7B) and Mistral (7B) show superior performance compared to Minima (3B) and Falcon (1B). Intriguingly, in query-document relationship understanding tasks, larger models without finetuning can even outperform the smaller models after fine-tuning (e.g., LLaMA-Chat > INTERSFalcon). This confirms the inherent advantages of larger-scale parameters in model performance. (2) We notice that INTERS-Falcon exhibits inferior performance compared to untuned Falcon in tasks related to understanding the query-document relationship. We attribute this to the complex nature of these tasks. (3) Notably, in document understanding tasks, INTERS-Minima (3B) outperforms INTERS-Mistral (7B), suggesting that fine-tuning
smaller models could serve as a cost-effective approach for particular tasks. (4) Before fine-tuning, LLaMA-Chat, which has already been optimized for dialogue scenarios, exhibits superior performance compared to LLaMA-Base. This advantage is attributed to LLaMA-Chat’s better capability of understanding instructions and performing tasks. However, after instruction tuning with INTERS, the performance gap diminishes. This shows the broad generality of our instruction tuning for various types of LLMs.

heading: 4.3 Out-of-domain Evaluation
text: Instruction fine-tuned LLMs have demonstrated a remarkable zero-shot performance on unseen tasks (Wei et al., 2022; Chung et al., 2022). We also
investigate the generalizability of the models after fine-tuned on INTERS. Specifically, we consider the following three scenarios. • Category-level generalizability: In this scenario, we exclude an entire category of tasks (e.g., query understanding) from INTERS. Then, we fine-tune the models on the remaining data and test them on all datasets. This experiment can help us understand how distinct categories of tasks relate to each other and contribute to overall model performance. • Task-level generalizability: In this scenario, we remove specific tasks (e.g., query intent classification) from INTERS. Similarly, we fine-tune the models on the remaining data and evaluate them on all datasets. The goal is to assess whether fine-tuned models can generalize to unseen tasks effectively. • Dataset-level generalizability: In this scenario, due to the large number of datasets in INTERS, we exclude several datasets at once (including TRECRobust, QReCC, MIMICS-Duo, Climate-FEVER, XSum, Quora, and NQ) from INTERS. Then, we fine-tune the models on the remaining data and test them on all datasets. This experiment aims to evaluate the fine-tuned models’ ability to generalize to unseen datasets within the scope of learned tasks.
We analyze the experimental result as follows:
(1) In the category-level ablation study (Table 1), the models fine-tuned with the full INTERS outperform those trained on ablated versions, verifying the efficacy of comprehensive fine-tuning in improving search task performance. We can also see that models trained on a subset of tasks still surpass the performance of the untrained models. For example, the performance of “w/o Q” is higher than the untuned model “(No FT)” on query understanding tasks. This result indicates that the different task categories are effectively complementary.
(2) Table 2 shows that the models exhibit tasklevel generalization. For instance, models finetuned without the query intent classification (QIC) task still outperform the untrained ones in this task. This implies that knowledge learned from other search tasks helps understand query intent. Furthermore, the query reformulation (QR) task’s performance also drops when the query intent classification task is removed, further supporting the interdependence of these tasks. Overall, task-level generalization indicates that LLMs fine-tuned on INTERS can be better applied to other search tasks.
(3) Some results of the third scenario is illustrated in Table 3 (full results in Appendix F). Compared to the previous two scenarios, this scenario is much easier for the fine-tuned model as all tasks have been learned during training. Generally, the models exhibit good generalizability among datasets, as evidenced by the superior performance of “w/o Ds” compared to the untuned model (“No FT”) across all datasets. We also notice that removing XSum from training leads to improved performance on MultiNews, highlighting the complex relationship between different datasets and suggesting a need for further exploration into the optimal dataset combinations for instruction tuning.

heading: 4.4 Further Analysis
text: We also conduct a series of experiments to investigate the impact of different settings in INTERS. All the experiments are conducted based on fine-tuning the LLaMA-2-Chat-7b model.

heading: 4.4.1 Impact of Task Description & Templates
text: INTERS includes a detailed description and 12 distinct templates for each task to enhance task comprehension and increase data diversity. We examine their effectiveness by the following experiments.
The result shown in Figure 5 demonstrates that the use of task descriptions significantly improves model performance across most datasets. This strongly supports our hypothesis that detailed task descriptions aid in task understanding. Besides, the task description appears to enhance the instruction tuning process, leading to substantial improvements in some cases (e.g., a 51.8% performance improvement on the MIMICS query clarification dataset as shown in Appendix F). We speculate that these task descriptions not only clarify individual tasks but also facilitate more effective cross-dataset knowledge transfer.
In the construction of INTERS, a key component is the development of 12 distinct templates for each
dataset, aiming at guiding the models in task comprehension. It is also interesting to study the influence of these templates on model performance. At first, we compare the performance when training with or without these templates. For the no template setup, we retain the keywords to indicate the different parts of the input. For the example shown in Figure 3, we keep only “Context: ... Query: ...” as input. Besides, we follow FLAN and use the INTERS instructions for zero-shot testing (because if we use no template, the model cannot know what task to perform). The results, shown in Figure 5, reveal that omitting templates leads to suboptimal performance, highlighting the instructional templates’ critical role in task learning.
Next, we study the impact of different templates. By default, we use random templates for evaluation, while in this experiment, we use each template to build test samples for query-understanding tasks and compare their performance differences. Besides, to simulate the real application scenario, we manually write a new unseen template for testing. The results are shown in Table 4. We can see that while the model can achieve significantly better performance than the untuned model on any template, template selection is still vital for some tasks, such as query suggestion (maximum 55.87 vs. minimum 42.57). This reflects the importance of deliberate template design. Remarkably, models tested on unseen templates can still show superior performance. This demonstrates again that our instruction-tuned models have good robustness and generalizability.

heading: 4.4.2 Comparison with FLAN
text: FLAN (Wei et al., 2022; Chung et al., 2022) is a commonly used dataset for fine-tuning LLMs on NLP tasks. We compare its effectiveness on searchrelated tasks with that of our INTERS. Given the significantly larger size of FLAN, we randomly sample 200k data examples from it for a fair comparison.3 Besides, to ensure fairness, as FLAN does not include the search-related tasks tested in this exper-
3https://huggingface.co/datasets/Open-Orca/ FLAN
iment, we also remove these tasks from INTERS for comparison (denoted as INTERS-T). By this means, both models trained on FLAN and INTERS are evaluated on tasks not seen during training. The results are shown in Table 5.
We find that both FLAN and INTERS can enhance LLMs’ performance on the three tasks, demonstrating again the effectiveness of instruction tuning in unlocking LLM potential for search tasks. Notably, INTERS yields a more substantial improvement in search tasks, particularly in query-document relationship understanding. This is consistent with our expectations, as INTERS is specifically tailored for search tasks. Although the tested tasks are unseen in training, other searchrelated tasks can provide relevant knowledge for these tasks. Finally, we can see training on FLAN achieves better performance on fact verification. The potential reason is that this task is very close to other NLP tasks included in FLAN, enabling effective knowledge transfer. Unfortunately, we do not observe further improvement by combining FLAN and INTERS, thus the results are omitted.

heading: 4.4.3 Zero-shot vs. Few-shot Demonstrations
text: LLMs have a strong ability of few-shot learning (also known as in-context learning), which enables them to quickly adapt to a wide range of tasks. Given that INTERS comprises a mix of zero-shot and few-shot, it is critical to examine the few-shot performance of the LLMs fine-tuned on INTERS. We choose datasets for few-shot (n = 5) testing that fit within the models’ input length limit (2,048 tokens in our case). The results are shown on the left side of Figure 6. Generally, few-shot demonstrations bring a consistent improvement in performance across all datasets, compared to zeroshot scenarios. Few-shot demonstrations are particularly beneficial in tasks with complex output spaces, such as reading comprehension (BoolQ), potentially because these examples help the model better understand the task and output format.

heading: 4.4.4 Impact of Data Volumes
text: The quantity of training data plays a pivotal role in the success of instruction tuning. To explore this, we conduct experiments using 25%, 50%, and 75% of the data sampled from INTERS for training. The results on the right side of Figure 6 clearly demonstrate that increasing the volume of instructional data generally enhances model performance. However, the sensitivity to data volume varies across
tasks. For instance, while the query understanding task shows consistent performance across data volumes, increasing data volume cannot effectively improve the performance of query-document relationship understanding. This highlights the need for further research to optimize the mix and volume of instructional data for diverse tasks.

heading: 4.4.5 Impact of Ranking Strategy
text: In our data construction (Section 3.2), we consider three typical methods for query-document understanding tasks, namely pointwise, pairwise, and listwise. Consequently, we test fine-tuned models across these different ranking strategies. Due to the limited space, we report the findings directly: the pointwise methods outperform the pairwise, which in turn exceeds the listwise in effectiveness (so we report pointwise performance in our previous experiments). Moreover, models with 7B or fewer parameters cannot handle the listwise evaluation. This may be due to the fact that the listwise method requires comparing multiple documents simultaneously and employs a sliding window method, presenting a complexity beyond the capability of such models. The comprehensive results are presented in Appendix C.

heading: 5 Conclusion
text: In this paper, we investigated the application of instruction tuning to augment the capabilities of LLMs in performing search tasks. Our instruction tuning dataset INTERS demonstrated its effectiveness in consistently enhancing the performance of various open-sourced LLMs across both in-domain and out-of-domain settings. Our extensive experiments delved into several critical aspects, including the structure and design of instructions, the effects of few-shot learning, and the significance of data volumes in instruction tuning. It is our aspiration that this paper will serve as a catalyst for further
research in the realm of LLMs, particularly in their application to IR tasks, and will encourage continued exploration into the optimization of instructionbased methods for enhancing the performance of these models.

heading: Limitation
text: In this study, we introduce a novel dataset specifically designed for instruction tuning on searchrelated tasks, along with models fine-tuned using this dataset. We acknowledge several limitations in our current work that offer avenues for future research.
First, while our dataset encompasses 20 tasks across 43 datasets, there are still many tasks and datasets that have not been included. Our experimental results suggest that incorporating more data sources can improve the richness and diversity of the dataset, potentially improving overall model performance. Second, due to our limited resources, we cannot conduct experiments with larger LLMs, such as those with 13B, 30B, or even 70B parameters. It is interesting to investigate the influence of instruction tuning on these models and compare their search performance with close-sourced LLMs such as GPT-4 if possible. Third, in the querydocument relationship understanding part, we only consider the reranking architecture. It is valuable to explore the application of LLMs to other architectures, such as retrieval.

heading: Acknowledgments
text: This work was supported by Beijing Natural Science Foundation No. L233008, National Natural Science Foundation of China No. 62272467, the fund for building world-class universities (disciplines) of Renmin University of China, and Public Computing Cloud, Renmin University of China. The work was partially done at the Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE.

heading: Data Mining, KDD 2023, Long Beach, CA, USA,
text: August 6-10, 2023, pages 4549–4560. ACM.
Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2023. Fine-tuning llama for multi-stage text retrieval. CoRR, abs/2310.08319.
Iain Mackie, Paul Owoicho, Carlos Gemmell, Sophie Fischer, Sean MacAvaney, and Jeffrey Dalton. 2022. CODEC: complex document and entity collection. In SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 - 15, 2022, pages 3067–3077. ACM.
Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. Www’18 open challenge: Financial opinion mining and question answering. In Companion of the The Web Conference 2018 on The Web Conference 2018, WWW 2018, Lyon , France, April 23-27, 2018, pages 1941–1942. ACM.
Kelong Mao, Zhicheng Dou, Fengran Mo, Jiewen Hou, Haonan Chen, and Hongjin Qian. 2023. Large language models know your contextual search intent: A prompting framework for conversational search. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 1211–1225. Association for Computational Linguistics.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3470–3487. Association for Computational Linguistics.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021. Webgpt: Browserassisted question-answering with human feedback. CoRR, abs/2112.09332.
Ramesh Nallapati, Bowen Zhou, Cícero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-tosequence rnns and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, August 11-12, 2016, pages 280–290. ACL.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 1797–1807. Association for Computational Linguistics.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, December 9, 2016, volume 1773 of CEUR Workshop Proceedings. CEUR-WS.org.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In NeurIPS.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The refinedweb dataset for falcon LLM: outperforming curated corpora with web data, and web data only. CoRR, abs/2306.01116.
Gustavo Penha, Alexandru Balan, and Claudia Hauff. 2019. Introducing mantis: a novel multi-domain information seeking dialogues dataset. CoRR, abs/1912.04639.
Yujia Qin, Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlun Zhu, Yankai Lin, Xu Han, Ning Ding, Huadong Wang, Ruobing Xie, Fanchao Qi, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2023a. Webcpm: Interactive web search for chinese long-form question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 8968–8988. Association for Computational Linguistics.
Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. 2023b. Large language models are effective text rankers with pairwise ranking prompting. CoRR, abs/2306.17563.
Jun Quan, Deyi Xiong, Bonnie Webber, and Changjian Hu. 2019. GECOR: an end-to-end generative ellipsis and co-reference resolution model for task-oriented dialogue. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 4546– 4556. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2383–2392. The Association for Computational Linguistics.
Sudha Rao and Hal Daumé III. 2018. Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 2737–2746. Association for Computational Linguistics.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pages 3505–3506. ACM.
Stephen E. Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr., 3(4):333–389.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022. Multitask prompted training enables zero-shot task generalization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Ivan Sekulic, Mohammad Aliannejadi, and Fabio Crestani. 2021. Towards facet-driven generation of clarifying questions for conversational search. In ICTIR ’21: The 2021 ACM SIGIR International Conference on the Theory of Information Retrieval, Virtual Event, Canada, July 11, 2021, pages 167–175. ACM.
Krishna Srinivasan, Karthik Raman, Anupam Samanta, Lingrui Liao, Luca Bertelli, and Michael Bendersky. 2022. QUILL: query intent with large language models using retrieval augmentation and multi-stage distillation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: EMNLP 2022 - Industry Track, Abu Dhabi, UAE, December 7 - 11, 2022, pages 492–501. Association for Computational Linguistics.
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and
Zhaochun Ren. 2023. Is chatgpt good at search? investigating large language models as re-ranking agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 14918–14937. Association for Computational Linguistics.
Yanran Tang, Ruihong Qiu, and Xue Li. 2023. Promptbased effective input reformulation for legal case retrieval. In Databases Theory and Applications - 34th Australasian Database Conference, ADC 2023, Melbourne, VIC, Australia, November 1-3, 2023, Proceedings, volume 14386 of Lecture Notes in Computer Science, pages 87–100. Springer.
Leila Tavakoli, Johanne R. Trippas, Hamed Zamani, Falk Scholer, and Mark Sanderson. 2022. Mimicsduo: Offline & online evaluation of search clarification. In SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 - 15, 2022, pages 3198–3208. ACM.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogenous benchmark for zero-shot evaluation of information retrieval models. CoRR, abs/2104.08663.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and verification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 809–819. Association for Computational Linguistics.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian CantonFerrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and finetuned chat models. CoRR, abs/2307.09288.
Ellen M. Voorhees. 2004. Overview of the TREC 2004 robust retrieval track. NIST Special Publication. National Institute of Standards and Technology (NIST).
Ellen M. Voorhees. 2005. Overview of the TREC 2005 robust retrieval track. In Proceedings of the Fourteenth Text REtrieval Conference, TREC 2005, Gaithersburg, Maryland, USA, November 15-18, 2005, volume 500-266 of NIST Special Publication. National Institute of Standards and Technology (NIST).
Ellen M. Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R. Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2020. TREC-COVID: constructing a pandemic information retrieval test collection. SIGIR Forum, 54(1):1:1– 1:12.
Henning Wachsmuth, Shahbaz Syed, and Benno Stein. 2018. Retrieval of the best counterargument without prior topic knowledge. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 241–251. Association for Computational Linguistics.
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 7534–7550. Association for Computational Linguistics.
Liang Wang, Nan Yang, and Furu Wei. 2023a. Query2doc: Query expansion with large language models. pages 9414–9423.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023b. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 13484–13508. Association for Computational Linguistics.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022. Super-naturalinstructions: Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on
Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 5085–5109. Association for Computational Linguistics.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 2369–2380. Association for Computational Linguistics.
Hamed Zamani, Gord Lueck, Everest Chen, Rodolfo Quispe, Flint Luu, and Nick Craswell. 2020. MIMICS: A large-scale data collection for search clarification. In CIKM ’20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020, pages 3189–3196. ACM.
Chen Zhang, Dawei Song, Zheyu Ye, and Yan Gao. 2023a. Towards the law of capacity gap in distilling language models. CoRR, abs/2311.07052.
Longhui Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Meishan Zhang, and Min Zhang. 2023b. Rankinggpt: Empowering large language models in text ranking with progressive enhancement. CoRR, abs/2311.16720.
Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji-Rong Wen. 2023. Large language models for information retrieval: A survey. CoRR, abs/2308.07107.
Shengyao Zhuang, Bing Liu, Bevan Koopman, and Guido Zuccon. 2023. Open-source large language models are strong zero-shot query likelihood models for document ranking. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 8807–8817. Association for Computational Linguistics.

heading: A Details about Tasks & Datasets
text: We introduce the tasks and datasets as follows. Some examples of our generated data are shown in Appendix G.

heading: A.1 Query Understanding
text: In IR, a query is a user-initiated request for information, typically composed of keywords, phrases, or natural language questions. It aims at retrieving
relevant information from a retrieval system (e.g., a search engine). The effectiveness of a query is measured by its ability to accurately reflect the user’s intent and retrieve the most relevant documents. During the retrieval process, query understanding is a critical component in determining the efficiency and user satisfaction of the IR systems. Therefore, we collect a group of tasks addressing aspects of query understanding to enhance LLMs’ capability of understanding the semantics of queries and capturing the underlying user search intent. Specifically, we consider the following eight tasks. • Query description: The query description task involves describing the documents potentially relevant to a user-provided query. Queries typically comprise keywords reflecting the user’s information needs. The objective of the task is to articulate the characteristics and content of documents that would be considered pertinent to these keywords, aiding in the understanding and retrieval of relevant information. We use the following four datasets: GOV2,4 TREC-Robust (Voorhees, 2004, 2005), TREC-COVID (Voorhees et al., 2020), and FIRE 08, 10-12.5 Taking the dataset GOV2 as an example, the query and its description are directly provided. • Query expansion: The query expansion task involves elaborating an original, brief query into a longer, more detailed version while preserving the original search intent. This process enhances the search engine’s understanding of the user’s needs, leading to more accurate and relevant document retrieval. We use the following seven datasets: GOV2, TREC-Robust, TREC-COVID, FIRE, Query2Doc (Wang et al., 2023a), TRECCAsT (Dalton et al., 2020), and TREC-Web 09- 14.6 Taking the dataset GOV2 as an example, the query and its expansion are directly provided. • Query reformulation: The query reformulation task enhances user-input queries to be more explicit and comprehensible for search engines. It addresses omissions typical of user queries, which often exclude common sense or contextually implied information. The refined query, therefore, includes all the necessary details to guide the search engine towards retrieving the most relevant documents. We use the following datasets: CODEC (Mackie et al., 2022), QReCC (Anantha et al., 2021), CANARD (Elgohary et al., 2019), TREC-CAsT, and
4https://ir-datasets.com/gov2.html#gov2 5https://www.isical.ac.in/~fire/data.html 6https://trec.nist.gov/data/webmain.html
GECOR (Quan et al., 2019). Taking the dataset CODEC as an example, the queries and their reformulations are provided in two files, and we can connect them by query IDs. • Query intent classification: User queries can have various search intents, such as informational (seeking knowledge about a topic), transactional (aiming to purchase a product), or navigational (looking to find a specific website). The intents can also be more specific in certain scenarios. Accurately discerning the type of intent behind a query is crucial for search engines to tailor and refine their results effectively. We use the following three datasets: ORCAS-I (Alexander et al., 2022), MANtIS (Penha et al., 2019), and TREC-Web 09-14. Taking the dataset ORCAS-I as an example, each query is associated with an attribute “query type”, which indicates the query’s intent. • Query clarification: The query clarification task addresses unclear or ambiguous user queries by asking for further details or providing clarification options. This process helps refine the query, resulting in clearer and more precise search terms for improved search engine results. We use the following datasets: MIMICS (Zamani et al., 2020), MIMICSDuo (Tavakoli et al., 2022), ClariQ-FKw (Sekulic et al., 2021), and RaoCQ (Rao and III, 2018). Taking the dataset MIMICS as an example, each query is labeled with a list of clarification options. • Query matching: The query matching task involves determining whether two queries or texts, despite differing in expression, convey the same meaning. This is crucial in search tasks where identifying synonymous queries can enhance the relevance and accuracy of results. We use the dataset: MSRP.7 It provides a label for each pair of sentences, indicating whether they convey identical content. • Query subtopic generation: The query subtopic generation task addresses the ambiguity of web searches by identifying and presenting various aspects of the initial query. This approach aids search engines in understanding the query’s breadth, leading to more diverse and relevant search results. We use the dataset: TREC-Web 09-14. It contains subtopic annotations for queries. • Query suggestion: In search sessions, users often input a series of queries to fulfill a specific information need. The query suggestion task aims
7https://www.microsoft.com/en-us/download/ details.aspx?id=52398
to analyze these queries and associated search behaviors to understand the user’s intent and predict the next likely query, thereby enhancing the search experience. We use the AOL dataset.8 This dataset contains a large number of search sessions. Within each session, a query at a specific position is randomly chosen to represent the “next query”. Subsequently, the preceding queries in the session, optionally inclusive of the clicked documents, are utilized as the search context.

heading: A.2 Document Understanding
text: In IR, a document refers to any piece of information that can be retrieved in response to a query, such as web pages in search engines. Document understanding is the process by which an IR system interprets and comprehends the content and context of these documents. The importance of document understanding lies in its direct impact on the effectiveness and accuracy of information retrieval. Enhanced document understanding leads to better search results, more effective organization of information, and an overall more efficient and user-friendly retrieval process. Therefore, we collect the following four tasks to enhance LLMs’ capability of document understanding. • Fact verification: The fact verification task involves assessing whether a claim is supported or refuted by the given evidence. It requires a clear analysis of the relationship between the claim and the evidence, with a careful check to determine if there is sufficient information for a conclusive judgment. Such detailed understanding aids search engines in achieving a deeper comprehension of the documents, enhancing their ability to deliver accurate and relevant results. We use the three datasets: FEVER (Thorne et al., 2018), Climate-FEVER (Diggelmann et al., 2020), and SciFact (Wadden et al., 2020). Taking the dataset FEVER as an example, it provides claims, their labels, and the corresponding evidences. • Summarization: The text summarization task seeks to create a concise summary of one or more lengthy documents, encapsulating all vital information while omitting extraneous details. The summary must accurately reflect the content of the original documents without introducing any new information. Achieving this necessitates a profound understanding of the documents, which can signifi-
8The AOL dataset has been officially withdrawn. However, as it is the most commonly used dataset for query suggestion, we still include it in INTERS.
cantly enhance the performance of search engines by providing distilled, relevant content. We use four datasets: CNN/DM (Nallapati et al., 2016), WikiSum (Liu et al., 2018), Multi-News (Fabbri et al., 2019), and XSum (Narayan et al., 2018). Taking the dataset CNN/DM as an example, it provides articles and their summaries. • Reading comprehension: The reading comprehension task requires generating an answer to a question using information from a given context. It necessitates a deep understanding of the text’s context and semantics, enabling search engines to more accurately rank the relevance of retrieved documents based on this nuanced comprehension. We use the following six datasets: SQuAD (Rajpurkar et al., 2016), HotpotQA (Yang et al., 2018), MS MARCO (Nguyen et al., 2016), TriviaQA (Joshi et al., 2017), BoolQ (Clark et al., 2019), and WebGLM-QA (Liu et al., 2023). Taking the dataset SQuAD as an example, it provides questions, their answers, and the corresponding context. • Conversational question-answering: Conversational question-answering involves responding to a series of interrelated questions based on a given context. As these questions might build upon shared information, some details may be implicitly understood rather than explicitly stated. By comprehensively understanding and analyzing this dialogue structure, search engines can enhance their interpretation of user queries and their connections to relevant documents, thereby improving result accuracy and relevance. We use these two datasets: CoQA (Choi et al., 2018) and QuAC (Choi et al., 2018). Taking the dataset CoQA as an example, each data sample contains a story, a series of questions about the story, and the corresponding answers.

heading: A.3 Query-document Relationship Understanding
text: Query-document relationship understanding in information retrieval is the process of determining how well the content of a document matches or satisfies the intent behind a user’s query. This involves interpreting the query’s semantics, context, and purpose, and then assessing the relevance of documents based on how closely they correspond to these aspects. It is the core task of information retrieval. The relationship between queries and documents varies in different scenarios. For example, in question answering, the model needs to understand the relationship between the question and
its potential relevant materials. In fact checking, the model is required to examine the relationship between the claim and its supporting evidence.
We use the MS MARCO passage ranking dataset and the datasets in the BEIR (Thakur et al., 2021) benchmark across multiple domains (such as biomedical, finance, and social media), which includes Touché-2020 (Bondarenko et al., 2020), ArguAna (Wachsmuth et al., 2018), TREC-COVID, NFCorpus (Boteva et al., 2016), SciDocs (Cohan et al., 2020), Quora,9 CQADupStack (Hoogeveen et al., 2015), DBPedia (Auer et al., 2007), FEVER, Climate-FEVER, SciFact (Wadden et al., 2020), NQ (Kwiatkowski et al., 2019), FiQA (Maia et al., 2018), and HotpotQA. Note that some datasets do not have queries in the training set, so we use the generated queries provided by BEIR.10
It is important to recognize the variety of architectures available for modeling the query-document relationship. In this research, we focus on the reranking architecture, which is the most straightforward way to apply LLMs. More details about applying LLMs to document reranking are provided in Appendix C. The primary objective of document reranking is to rerank a list of candidate documents according to their relevance to the user’s query. The most relevant documents, those that best cover the user’s information needs, are ranked at the top of the list. In our experiments, we use the documents retrieved by BM25 (Robertson and Zaragoza, 2009) as the candidates.
The statistics of all datasets and their evaluation metrics are reported in Table 8.

heading: A.4 Licenses
text: We plan to release our data under the license of CC BY-SA 4.0.11 The authors of 10 out of the 43 datasets in INTERS (FIRE, TREC-Web, MANtIS, ClariQ-FKw, RaoCQ, AOL, Climate-FEVER, WikiSum, TriviaQA, and WebGLM-QA) do not report the dataset license in the paper or a repository. The rest is over view as follows: • Apache License 2.0 license: GOV2, TREC-
Robust, CODEC, CNN/DM • MIT license: TREC-CAsT, GECOR, ORCAS-
I, MIMICS, MIMICS-Duo, XSum • CC BY 4.0: Query2Doc, MSRP, SQuAD 9https://quoradata.quora.com/ First-Quora-Dataset-Release-Question-Pairs 10https://huggingface.co/BeIR 11https://creativecommons.org/licenses/by-sa/4. 0/
• CC BY-SA 4.0: CANARD, BEIR, HotpotQA, QuAC • CC BY-SA 3.0: QReCC, FEVER, BoolQ • CC BY-NC 2.0: SciFact • Provided under the “Dataset License Agree-
ment”: TREC-COVID, Multi-News, MS MARCO Note that CoQA contains several datasets under
different licenses. They are listed on the HuggingFace page.12
B In-domain Evaluation Details
In this evaluation, we split the full dataset into training, validation, and test sets. The split process is designed based on the size and structure of the original datasets. Specifically, if the original datasets do not contain a test set, then: For original datasets with over 10,400 samples, we randomly select 10,000 samples for constructing training data, 200 samples for validation, and 200 samples for testing. In instances where the datasets comprise between 2,000 and 10,400 samples, we randomly select 200 samples each for validation and testing, with the remainder constructing the training samples. For smaller datasets contain fewer than 2,000 samples, we use the ratio of 8:1:1 to obtain the training, validation, and test sets. When the original dataset includes a test set, we use the test set to construct test samples, extracting only the validation set from the samples in the training dataset. The extraction rule is similar to the previous case.

heading: C LLMs for Reranking
text: To apply LLMs for the document reranking task, there are three typical methods: pointwise, pairwise, and listwise (shown in Figure 7). They use different prompts to perform reranking. A brief overview of these methods is presented below, with further details accessible in the literature (Zhu et al., 2023; Qin et al., 2023b).
(1) Pointwise methods measure the relevance between a query and a single document. As illustrated in Figure 7 (a), a common method is prompting the LLMs to judge whether a query and a document are relevant. The relevance score is computed based on the generation probability of “Yes” and “No” tokens: r = pyes/(pyes + pno).
(2) Pairwise methods require LLMs to determine which of two documents is more relevant to the given query, as shown in Figure 7 (b). To get
12https://huggingface.co/datasets/stanfordnlp/ coqa
a ranking list of all candidate documents, aggregation methods, such as PRP-Allpair (Qin et al., 2023b), are applied.
(3) Listwise methods directly prompt LLMs to generate a reranked list of documents, as shown in Figure 7 (c). However, due to the limited input length of LLMs, it is often impractical to include all candidate documents in a single prompt. To address this, a sliding window strategy is commonly applied (Sun et al., 2023).
To support various application scenarios, we consider all three methods when collecting the templates for the query-document relationship understanding tasks in INTERS. Concretely, we collect four distinct templates for each of these methods. The performance comparison of different methods has been presented in Table 6. Note that we only report the results on a select number of datasets due to the substantial computational costs of pairwise methods. From the results, we can see:
First, no matter which reranking method is used, INTERS can consistently improve LLMs’ performance on query-document relationship understanding tasks, reflecting its broad applicability. Second, comparing the three methods, the pointwise methods generally outperform the pairwise methods,
which in turn exceed the listwise methods in effectiveness. Moreover, models with 7B or fewer parameters cannot handle the listwise evaluation. This may be due to the fact that the listwise method requires comparing multiple documents simultaneously and employs a sliding window method, presenting a complexity beyond the capability of such models. Third, in terms of inference cost, the pairwise method is the most resource-intensive. This is attributed to its requirement for pairwise document comparisons and an additional algorithm for deriving the final result. The cost of listwise methods relies on the sliding window algorithm, but its performance relies on the quality of the initial ranking list (Sun et al., 2023). Based on these observations, we consider that the pointwise method is the most suitable one for query-document relationship understanding tasks on LLMs with 7B or fewer parameters, which provides a good balance between efficacy and computational costs.

heading: D Backbone Models & Implementation Details
text: We employ four LLMs in different sizes, ranging from 1B parameters to 7B parameters: • Falcon-RW-1B (Penedo et al., 2023) is a language model developed by the Technology Innovation Institute, trained on 600B tokens of English data. The model is designed for researching large language models and the impact of adequately fil-
tered and deduplicated web data on their properties, such as fairness, safety, limitations, and capabilities.13 • Minima-2-3B (Zhang et al., 2023a) is a novel language model designed to achieve a new computeperformance frontier on common benchmarks by distilling knowledge from a large teacher language model (LLaMA-2-7B). The model uses a data mixture of 126 billion tokens from various sources for distillation.14 • Mistral-7B (Jiang et al., 2023) is a language model engineered for superior performance and efficiency. It leverages mechanisms such as groupedquery attention (Ainslie et al., 2023) and sliding window attention (Beltagy et al., 2020; Child et al., 2019) to outperform other language models in various benchmarks.15 • LLaMA-2-7B (Touvron et al., 2023) is language model trained on around 2T tokens. It has shown exceptional performance across multiple benchmark tests and has been widely used for LLM research. In our experiments, we find that the LLaMA-2-Chat model performs slightly better than the LLaMA-2-Base after fine-tuning (the result is reported in Section 4.2). Therefore, we use LLaMA-2-Chat in our main experiments and further investigation.16
For all backbone models, we used their publicly available checkpoints on Huggingface. The finetuning process was implemented using PyTorch and Colossal-AI frameworks (Li et al., 2023). To optimize memory usage and accelerate training, we applied Deepspeed ZeRO stage 2 (Rasley et al., 2020) and BFloat16 mixed precision techniques. Additionally, Flash attention (Dao et al., 2022) was used to further improve training efficiency. The training was conducted with a batch size of 32, a learning rate of 1e-5, and a maximum length setting of 2,048 tokens. Though some backbone models support longer inputs, we limit the input length to reduce training costs. All models were trained on 8 Tesla A100-40G GPUs. It is important to note that the hyperparameters were set based on empirical observations, as the primary aim was to validate the effectiveness of INTERS. Comprehensive hyperparameter tuning was beyond the scope of this study
13https://huggingface.co/tiiuae/falcon-rw-1b 14https://huggingface.co/GeneZC/MiniMA-2-3B 15https://huggingface.co/mistralai/
Mistral-7B-v0.1 16https://huggingface.co/meta-llama/ Llama-2-7b, https://huggingface.co/meta-llama/ Llama-2-7b-chat-hf
due to resource limitations.

heading: E Comparison with Larger Models
text: We also attempt to evaluate the zero-shot performance of GPT-4 and LLaMA-2-70B. However, due to the limited computational resources, we only evaluate them on query understanding and document understanding tasks. Besides, as some tasks require generation logits (such as BoolQ) for computing evaluation metrics, we do not include them in this evaluation. The results are shown in Table 7. From the results, we can observe that while larger LMs generally perform better on NLPrelevant tasks (such as summarization), they still struggle with IR tasks (such as query clarification). This highlights again the importance of our INTERS dataset for IR tasks.

heading: F Additional Results
text: We present the full evaluation results in Table 9 – Table 14.

heading: G Data Examples
text: We present an example per task in Table 15 – Table 26.

heading: Target

heading: Input

heading: Query Description

heading: Target

heading: Input

heading: Query Subtopic Generation

heading: Target

heading: Input

heading: Query Clarification

heading: Target

heading: Input

heading: Query Reformulation

heading: Target

heading: Input

heading: Query Expansion

heading: Query Suggestion

heading: Input
text: Query Intent Classification

heading: Input
text: User queries can have various search intents, such as informational (seeking knowledge about a topic), transactional (aiming to purchase a product), or navigational (looking to find a specific website). Accurately discerning the type of intent behind a query is crucial for search engines to tailor and refine their results effectively.

heading: Input

heading: Query Matching
text: Fact Verification

heading: Input
text: The fact verification task is to assess whether a claim is supported or refuted by the given evidence. It requires a clear analysis of the relationship between the claim and the evidence, with careful examination to determine if there is enough information for making a judgment. It aids search engines in achieving a deeper comprehension of the documents.
“U2 is a Scottish rock band.” Based on “(1) Title: U2 Content: U2 are an Irish rock band from Dublin formed in 1976 . (2) Title: U2 Content: Within four years , they signed with Island Records and released their debut album , Boy 1980 . (3) Title: U2 Content: Subsequent work such as their first UK number-one album , War 1983 , and the singles “ Sunday Bloody Sunday ” and “ Pride In the Name of Love ” helped establish U2 ’s reputation as a politically and socially conscious group . (4) Title: U2 Content: The group ’s fifth album , The Joshua Tree 1987 , made them international superstars and was their greatest critical and commercial success . (5) Title: U2 Content: Topping music charts around the world , it produced their only number-one singles in the US , “ With or Without You ” and “ I Still Haven’t Found What I ’m Looking For ” . (6) Title: U2 Content: Beginning with their acclaimed seventh album , Achtung Baby 1991 , and the multimedia intensive Zoo TV Tour , the band integrated influences from alternative rock , electronic dance music , and industrial music into their sound , and embraced a more ironic , flippant image . (7) Title: U2 Content: This experimentation continued through their ninth album , Pop 1997 , and the PopMart Tour , which were mixed successes . (8) Title: U2 Content: U2 regained critical and commercial favour with the records All That You Ca n’t Leave Behind 2000 and How to Dismantle an Atomic Bomb 2004 , which established a more conventional , mainstream sound for the group . (9) Title: U2 Content: The group ’s thirteenth album , Songs of Innocence 2014 , was released at no cost through the iTunes Store , but received criticism for its automatic placement in users’ music libraries.”, which label support or refute should be assigned?
Target refute

heading: Conversational QA

heading: Input
text: The area of Africa now known as Malawi was settled by migrating Bantu groups around the 10th century. Centuries later in 1891 the area was colonised by the British. In 1953 Malawi, then known as Nyasaland, a protectorate of the United Kingdom, became a protectorate within the semi-independent Federation of Rhodesia and Nyasaland. The Federation was dissolved in 1963. In 1964 the protectorate over Nyasaland was ended and Nyasaland became an independent country under Queen Elizabeth II with the new name Malawi. Two years later it became a republic. Upon gaining independence it became a one-party state under the presidency of Hastings Banda, who remained president until 1994, when he lost an election. Arthur Peter Mutharika is the current president. Malawi has a democratic, multi-party government. The country has a Malawian Defence Force that includes an army, a navy and an air wing. Malawi’s foreign policy is pro-Western and includes positive diplomatic relations with most countries and participation in several international organisations, including the United Nations, the Commonwealth of Nations, the Southern African Development Community (SADC), the Common Market for Eastern and Southern Africa (COMESA), and the African Union (AU). Questions: (1) Who settled in Malawi? (2) when? (3) What does the defence force include? (4) What does COMESA stand for? (5) Is it a large country? (6) When was it colonized? (7) By who? (8) What was it called before Malawi? (9) when was the federation dissolved? (10) What is the capital city? (11) What is the biggest lake? (12) how much of Malawi’s space does it take up? (13) What is AU?

heading: Target
text: (1) Bantu groups (2) 10th century (3) army, a navy and an air wing (4) Common Market for Eastern and Southern Africa (5) No (6) 1891 (7) British (8) Nyasaland (9) 1963 (10) Lilongwe (11) Lake Malawi (12) a third of Malawi’s area (13) African Union
Table 24: A data example of the conversational QA task. Data source: CoQA.

heading: Target

heading: Input

heading: Reranking

heading: Target

heading: Input

heading: Reading Comprehension

heading: Future Work
text: We acknowledge several limitations in our current work that offer avenues for future research. First, while our dataset encompasses 20 tasks across 43 datasets, there are still many tasks and datasets that have not been included. Our experimental results suggest that incorporating more data sources can improve the richness and diversity of the dataset, potentially improving overall model performance. Second, due to our limited resources, we cannot conduct experiments with larger LLMs, such as those with 13B, 30B, or even 70B parameters. It is interesting to investigate the influence of instruction tuning on these models and compare their search performance with close-sourced LLMs such as GPT-4 if possible. Third, in the querydocument relationship understanding part, we only consider the reranking architecture. It is valuable to explore the application of LLMs to other architectures, such as retrieval