Venue: NAACL 2025

# IFIR: A Comprehensive Benchmark for Evaluating

Instruction-Following in Expert-Domain Information Retrieval

 Tingyu Song\({}^{\clubsuit}\)  Guo Gan\({}^{\diamondsuit}\)   Mingsheng Shang\({}^{\diamondsuit}\)   Yilun Zhao\({}^{\spadesuit}\)

###### Abstract

We introduce IfIR, the first comprehensive benchmark designed to evaluate instruction-following information retrieval in expert domains. IFIR includes 2,426 high-quality examples and covers eight subsets across four specialized domains: finance, law, healthcare, and science literature. Each subset addresses one or more domain-specific retrieval tasks, replicating real-world scenarios where customized instructions are critical. IFIR enables a detailed analysis of instruction-following retrieval capabilities by incorporating instructions at different levels of complexity. We also propose a novel LLM-based evaluation method to provide a more precise and reliable assessment of model performance in following instructions. Through extensive experiments on 15 frontier information retrievers, including those based on LLMs, our results reveal that current models face significant challenges in effectively following complex, domain-specific instructions. We further provide in-depth analyses to highlight these limitations, offering insights to guide future advancements in retriever development.

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 10186–10204

[table]capposition=topsep=*, arcindeposition=left, colbacktitle=black!75!white, colframe=gray!30!black, boxrule=0pt, leftline=0pt, rightrule=0pt, toprule=0pt, title=0pt, bottomrule=0pt, title=0pt, frame=single, linewidth=left, colframe=gray!

_end_ manner. Consequently, users must break down their complex information-seeking needs into several simpler search queries and manually filter the retrieved cases Liu et al. (2021), which is time-consuming and inefficient.

On the other hand, existing instruction-following IR benchmarks typically employ simplified instructions, such as single sentence Su et al. (2023) or a set of keywords Zhao et al. (2024). This simplification in evaluation leads to an incomplete assessment of the model's real-world performance. Although concurrent works like FollowIR Weller et al. (2024, 2025), InstructTR Oh et al. (2024) and BRIGHT Su et al. (2024) incorporate more complex instructions, they do not establish explicit complexity levels to evaluate retrievers' _fine-grained_ abilities in following instructions. Moreover, these studies primarily focus on general domains, leaving the evaluation of instruction-following retrieval in _expert domains_ largely underexplored.

In this work, we introduce **IrfR**, a comprehensive benchmark designed to evaluate the Instruction- Following capabilities of Information Retrievers, particularly in the context of specialized domains. IFIR includes eight subsets covering four specialized domains: finance, scientific literature, law, and healthcare. To provide a more granular evaluation, we create three levels of instruction complexity for each domain, representing a range of real-world information retrieval scenarios in expert domains. IFIR includes 2,426 instruction-following queries, each averaging 6.14 ground-truth passages. To ensure the dataset's high quality, we conduct a comprehensive human expert validation during its construction. Moreover, recognizing the limitations of traditional evaluation methods in measuring instruction-following IR performance, we introduce a novel LLM-based metric, InstFol, designed to more accurately assess how well retrievers follow instructions.

Through extensive experiments on 15 frontier retrievers, including those based on LLMs, we derive three key findings: (1) BM25 performs relatively well because the instructions in expert domains contain more glossary terms. (2) Instruction-tuned retrievers like InstructOR Su et al. (2023) do not perform significantly better than their base models, GTRs Ni et al. (2022). This demonstrates that current instruction-tuned retrievers may not be suitable for complex instructions. (3) Most evaluated models experience performance declines as the complexity of the instructions increases. (4) LLM-based retrievers demonstrate more robust performance on both nDCG and InstFol, highlighting their potential in managing more complex retrieval tasks in specialized domains.

We conclude our main contributions as follows:

* We introduce IfIR, a comprehensive IR benchmark to evaluate the instruction-following ability of information retrievers across specialized domains, meeting their specific demands. The experiments provide insights into _end-to-end_ retrieval in specialized domains.
* We propose InstFol, the first LLM-based evaluation method to measure the instruction-following ability of information retrievers.
* We conduct extensive experiments encompassing a wide range of retrievers, deriving key findings about their instruction-following abilities. Our experimental results reveal the potential of LLM-based retrievers in instruction-following retrieval.

## 2 Related Work

### Expert-domain IR Benchmarks

Information retrieval plays a crucial role in expert domains by enabling efficient access to domain-specific knowledge, facilitating evidence-based decision-making, and accelerating research. In recent years, there has been a growing emphasis on developing IR benchmarks in specialized domains, such as law Xiao et al. (2019); Li et al. (2023), finance Jangid et al. (2018); Chen et al. (2021); Zhao et al. (2024), scientific literature Wu et al. (2024); Ajith et al. (2024), and healthcare Roberts et al. (2020); Tamine and Goeuriot (2021); Xiong et al. (2024). However, existing benchmarks primarily employ oversimplified queries, lacking the depth and specificity required in real-world specialized domains. For instance, in healthcare, practitioners often formulate complex, context-rich queries that integrate patient-specific information (_e.g.,_ medical history and treatment plans) to retrieve relevant clinical passages. Such queries go beyond simple keyword search and require models to handle complex, user-customized needs. This gap underscores the need for a new benchmark that thoroughly evaluates the ability of retrievers to handle instruction-following IR tasks in specialized domains 



### Instruction-Following IR

Several studies have proposed novel training techniques to improve retrievers' instruction-following capabilities [23, 14, 5]. However, due to the lack of instruction-specific IR benchmarks during model development, these approaches have typically been evaluated using traditional benchmarks like BEIR [6] and MTEB [7], which contain queries without complex instructions. To address this gap, new instruction-following IR benchmarks have been proposed. Specifically, INTRCIR [8] reformulates queries in existing retrieval datasets by incorporating user-aligned instruction. However, it limits each query to a single relevant passage, which does not reflect the complexity of real-world scenarios where multiple relevant passages may exist. While FollowIR [9] introduces more complex instructions and passage setups, it focuses on the reranking task. Moreover, these studies primarily focus on instruction-following retrieval in general domains, leaving the evaluation of instruction-following IR in _expert domains_ largely underexplored.

## 3 IrIR Benchmark

We introduce IFIR, a comprehensive benchmark designed to assess and enhance retrievers' instruction-following capabilities in expert domains. The construction process of IFIR, illustrated in Figure 2, employs a semi-automated, human-in-the-loop pipeline that ensures both scalability and high quality. Specifically, we expand the queries in existing specialized-domain IR benchmark (detailed in SS3.1) by incorporating detailed instructions that closely mirror real-world scenarios in relevant expert domains (SS3.2). Each example is thoroughly validated by domain experts (details of annotator selection and assigned tasks are provided in Table 4 in Appendix), ensuring that the instructions are contextually relevant and represent real-world challenges (SS3.2). Additionally, the corresponding relevant passages are carefully verified for completeness and accuracy (SS3.3). In the following subsections, we provide a detailed description of each step in the construction pipeline.

### Retrieval Dataset Collection

To facilitate comprehensive evaluations of instruction-following capabilities across diverse expert domains, IFIR spans four specialized domains: finance, scientific literature, law, and healthcare. These domains are chosen for their significant demand for precise information retrieval and the complexity of their often nuanced queries. For each domain, we extend one or two well-established traditional IR benchmarks (see Table 1), adapting them to instruction-following IR tasks. This ensures that IFIR captures the real-world challenges for each expert domain.

### Instruction Annotation

We now describe the process of augmenting the original IR benchmark queries with instructions that mirror real-world demands and challenges spe

Figure 2: Dataset Construction Pipeline: We derive a specific task according to the dataset, which then guides the generation of instructions based on the original query and task conditions. An LLM is used to assess whether the corpora are relevant to these instructions. As illustrated in the figure, different colors in the “Task” section correspond to the conditions outlined in the “Instruction” section.



cific to each domain. Details of instruction generation are shown in A.1. These instructions introduce complex, often implicit conditions that significantly increase the difficulty of retrieval tasks, as they require the model to not only identify relevant passages but also interpret and follow specific instructions tied to nuanced, task-specific information retrieval. To ensure high-quality instruction annotation, we employ a _human-in-the-loop_ pipeline. We first use LLMs (_i.e.,_ GPT-4o 2024-05-13 version) to generate instructions at varying levels of complexity for each query. These instructions are then reviewed and refined by domain experts to ensure they are precise and aligned with the real-world demands of the respective domains. We detail the annotation process for each domain as follows:

FinanceIn the finance domain, we focus on the instruction-following IR task centered around personal finance inquiries, simulating scenarios where users seek guidance in making informed financial decisions. We design three complexity levels of instructions in queries: The first level involves simple instructions, _e.g.,_ "Please help me to find a financial suggestion for the query \({}^{\text{}}\)Squery\({}^{\text{}}\)." The second level includes additional personal information such as age, occupation, and financial status. The third level builds upon the second level by incorporating specific financial goals. For example, "As a 40-year-old accountant with a steady income and moderate savings, I am seeking advice on the best business structure for taxes when combining full-time work with running a small side business. I am looking for insights on how to optimize tax efficiency while balancing the demands of my full-time job and side business."

Scientific LiteratureIn the science literature field, we focus on the science passage searching queries, simulating a person working in a relevant area(_e.g.,_ teacher, student, etc.) trying to find passages related to scientific claims or problems. We recognize that query instructions can vary in research topics (_e.g.,_ society, history, biomedical, etc.) and research objectives (_e.g.,_ influence, reasoning process). We use the Scifact-open dataset to generate three different levels of instructions. The first level of instruction might state, "Please help me find relevant evidence to support the scientific claim." The second level uses previously annotated "SUPPORT" and "CONTRADICT" tags to generate instructions like "Please help me to find supporting evidence for this scientific claim." The third level includes more customized requirements like research topics and objectives.

LawIn the legal field, we focus primarily on the legal case retrieval task, simulating someone(_i.e.,_ a lawyer) attempting to find relevant references from previous legal cases. We have two types of instructions. One type is to retrieve prior cases that support the reasoning process for the current case, which originates from the FIRE2017 dataset. The other, derived from AILA2019, is designed to retrieve similar cases according to the demands of legal professionals. For the first type, we construct instructions based on the context in the passage that requires citation. For the second type, we construct three different levels of instructions. The first level, similar to previous domains, is "Please help me to find cases similar to the current legal case." The second level adds conditions including whether the case is beneficial to the defendant or plaintiff. The third level constructs instructions searching for cases relevant to some details of the current case while still satisfying the previous two levels.

HealthcareIn the healthcare domain, we focus on retrieving healthcare-relevant passages (_e.g.,_ clinical trials, diagnoses), simulating a doctor finding passages suitable for patients. The seed

\begin{table}
\begin{tabular}{l l c r r p{168.0pt}} \hline \hline
**Domain** & **Adopted Datasets** & **\# Qry** & **Corpus Size** & **\# RP** & **Designed Tasks Reflecting Real-world Challenges** \\ \hline Finance & FiQA [2] & 1,718 & 57,638 & 3.54 & Retrieve financial suggestions based on user-specific needs to support informed decision-making. \\ Scientific Literature & SciFiatect-open [2] & 152 & 500,000 & 4.84 & Retrieve relevant scientific literature tailored to \\ Literature & NFcorpus [1] & 86 & 3,633 & 2.81 & specific scientific research needs. \\ Law & AILA [1] & 85 & 2,914 & 2.01 & Retrieve legal cases that satisfy customized demanders. \\  & FIRE [1] & 168 & 1,745 & 3.36 & Retrieve legal cases to support the judicial decision. \\ Healthcare & TREC-PM [1] & 172 & 241,006 & 15.61 & Retrieve relevant clinical trials based on patient’s \\  & TREC-CDS [1] & 43 & 633,955 & 10.84 & demographics (_e.g.,_ age, gender, medical history) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Overview of the adopted datasets and designed instruction-following IR tasks in IFIR, along with basic statistics for each domain. “**# RP” represents the average relevant passage number per instruction-following query.

 

datasets are derived from TREC-CDS and TREC-PM. Given the two different datasets and corresponding tasks in the biomedical field, TREC-CDS provides a summary accompanied by a detailed description, which we directly use as the instruction. Inspired by the TREC-CDS track, we expand the basic information provided in the TREC-PM track and construct three levels of instructions. The first level contains conditions of the patient's disease and gene variation. The second level adds conditions about the patient's demographics, including age and gender. The third level allows the LLM to create information about the patient's treatment history and family medical history.

Table 1 presents the data statistics of IFIR. It includes a total of 2,426 instruction-following queries across four expert domains. Examples of queries for each domain are provided in Appendix A.2. Each query is carefully reviewed by one of domain experts (Table 4 in Appendix).

### Relevant Passage Annotation

We next discuss the process of annotating relevant passages. For each instruction-following query in IFIR, we select relevant passages from its original dataset. The key insight is that if a passage is annotated as relevant to a query, it may also be relevant to an instruction constructed based on that query. Therefore, we need to verify whether these query-relevant passages satisfy the conditions outlined in the corresponding instructions. Specifically, we first use LLM (_i.e.,_ GPT-4o) to assess the relevance of each original relevant passage to the instruction. The LLM is tasked with generating justification explanations alongside its relevance assessments (we present prompt in Appendix B). Human annotators then review the relevance of each passage and the justifications provided by the LLM. If a passage is found to be misaligned with the instruction, the annotators exclude it.

### Dataset Analysis

Table 1 presents the basic statistics of IifIR. We also conduct a final human evaluation of data quality on 50 examples from each domain subset. For each domain, an external expert (not involved in the data annotation) assesses the quality of each example, providing ratings across several criteria on a scale of 1 to 5. The results, shown in Table 2, indicate consistently high scores (> 4) for instructions following query and relevant passages. These evaluation results indicate the high quality of the benchmark proposed by us, further demonstrating its reliability for assessing instruction-following capabilities in relevant tasks.

## 4 Experiment Setup

This section outlines the experimental setup of our study. We first introduce the two automated metrics used for evaluating IFIR, and then discuss the evaluated retrieval systems. Implementation details can be found in Appendix B.

### Evaluation Metrics

nDCGWe use the widely-adopted IR metric, nDCG Jarvelin and Kekalainen (2002), to evaluate _retrieval performance_. Specifically, given a query with instruction \(Q\), golden passages \(G\), the retrieved passages \(P\) are compared against the \(G\) using nDCG to quantify the accuracy and relevance of the retrieval. While nDCG offers a broad assessment of a model's retrieval capabilities, it does not capture the fine-grained aspects of a model's ability to follow instructions.

InstFolTo address the aforementioned limitation, we introduce a new LLM-based metric, InstFol, specifically designed to evaluate _instruction-following capabilities_ on IfIR. The core idea is to assess the improvement a retriever demonstrates when instructions are incorporated into the query, compared to when they are not. We adopt the evaluation prompt from G-Eval Liu et al. (2023), as shown in Figure 8 in the Appendix, and use GPT-4o-mini as the base evaluator to assess the alignment between each retrieved passage and the given instruction. Considering the potential biases and

\begin{table}
\begin{tabular}{l l} \hline \hline
**Evaluation Criteria** & **Score (1-5)** \\ \hline _Instruction-Following Query_ & \\ Naturalness & 4.24 \\ Fluency & 4.81 \\ Expertise & 4.87 \\ \hline _Relevant Passage (RP)_ & \\ Recurrent Passage Agreement & 4.43 \\ _Excluded RP_ & \\ Exclusion Agreement & 4.32 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Human Validation Results. Naturalness of instructions evaluates how well the instructions align with real-world demands. The Relevant Passage Agreement Score refers to human annotators’ agreement with the LLM on identifying a golden passage, while the Exclusion Agreement Score reflects human annotators’ agreement on excluding a passage.

 

inaccuracies introduced by LLMs, we use the Probability Normalization technique (as detailed in Appendix B ) to reduce overestimation, which has been proven effective in current works Liu et al. (2023); Liusie et al. (2024). For each passage in the Top-\(K\) retrieval results, the LLM is instructed to produce the relevance score. We then average the scores as the final matching score \(S\). The InstFol@\(K\) for the Top-\(K\) retrieved passages is then calculated as:

\[{\tt InstFol@K}=(S_{\text{inst}}-S_{\text{q}})\cdot\alpha \tag{1}\]

where \(S_{\text{q}}\) measures how well the passages retrieved by the original query (without instruction) align with the given instruction; \(S_{\text{inst}}\) measures how well the passages retrieved by the query (with instruction) meet the same instruction. The factor \(\alpha\) is a normalization function that ensures the InstFol score ranges between -1 and 1. In practice, we set \(\alpha\) as \(\frac{1}{3-S_{\text{q}}}\). The implementation details of the InstFol metric are provided in Appendix B.

Our in-depth analysis in Section 6 demonstrates that InstFol exhibits a high correlation with human expert evaluations, highlighting its reliability.

### Evaluated Retrievers

We evaluate a wide range of retrievers, with model sizes ranging from 110M to 7B parameters. These models are categorized into two main types:

Non-instruction-tuned modelsWe include the following commonly-used non-instruction-tuned models for the experiments: (1) **BM25**Robertson et al. (2009), which is a lexical retriever; (2) **ColBERT**Khattab and Zaharia (2020), which encodes queries and documents separately and introduces a mechanism of delayed interaction to be more effective; (3) **Contriever**Izacard et al. (2022), which is a BERT-based model trained by contrastive learning; and (4) **GTR**Ni et al. (2022), which uses the encoder from the T5 model and are pre-trained on MS MARCO Bajaj et al. (2018).

Instruction-tuned retrieversFor the instruction-tuned models, we select: (1) **InstructOR**Su et al. (2023b), which are finetuned on the GTR family using MEDI datasets, and can be utilized for various tasks including retrieval; (2) **E5-mistral-7b-instruct**Wang et al. (2023b), which is a retriever based on a Mistral model and trained on synthetic data. (3) **GritLM-7B**Muennighoff et al. (2024), which is also a Mistral model, trained on the synthetic data from E5-mistral-7b-instruct and MEDI2, capable of performing both generation and retrieval tasks; (4) **Promptriever**Weller et al. (2024b), is an instruction-trained bi-encoder retriever specialized for instruction-following tasks. The model we choose is Promptriever-7B. (5) **NV-Embed**Lee et al. (2024) is a retriever trained by a two-stage contrastive instruction-tuning method. (6) **Propiterary Retrievers**, including OpenAI's Text-Embedding-v3-Large and Text-Embedding-v3-Small.

## 5 Experimental Results

This section first presents the key experimental results, followed by an in-depth analysis of model

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{FQA} & SciFast-open & NFCorpus & AILA & FIRE & TREC-PM & TREC-CDS & **Average** \\ \cline{2-13}  & nDCG & InsITFol & nDCG & InsITFol & nDCG & InsITFol & nDCG & InsITFol & nDCG & InsITFol & nDCG & InsITFol \\ \hline \multicolumn{13}{c}{_Non-instruction-tuned Models_} \\ GTR-XL & 0.44 & -4.85 & 0.54 & -2.93 & 0.60 & 7.30 & 0.05 & -0.13 & 0.54 & 0.40 & 0.31 & -0.85 & 0.15 & 7.69 & 0.37 & 0.95 \\ BM25 & 0.25 & 1.01 & 0.49 & -0.40 & 0.43 & 1.46 & 0.10 & 0.02 & 0.55 & 0.03 & 0.47 & 3.43 & 0.07 & -2.14 & 0.34 & 0.50 \\ GTR-Large & 0.39 & -6.47 & 0.50 & 0.05 & 0.51 & -3.37 & 0.07 & -0.36 & 0.49 & -5.21 & 0.28 & -1.46 & 0.09 & 11.38 & 0.33 & -0.78 \\ GTR-Base & 0.33 & -4.62 & 0.47 & -0.62 & 0.47 & -0.08 & 0.05 & **0.54** & 0.52 & 0.81 & 0.27 & -0.65 & 0.13 & 7.03 & 0.32 & 0.35 \\ Comtiver & 0.13 & 0.52 & 0.29 & -8.22 & 0.36 & 0.17 & 0.08 & 0.09 & 0.51 & 2.82 & 0.09 & 1.24 & 0.04 & 2.46 & 0.21 & -0.13 \\ ColBERT & 0.07 & 0.17 & 0.14 & 0.34 & 0.16 & 0.06 & 0.07 & -0.01 & 0.39 & 1.44 & 0.02 & 1.03 & 0.00 & -0.94 & 0.12 & 0.30 \\ \hline \multicolumn{13}{c}{_Instruction-tuned Models_} \\ NV-Embed-v2 & **0.68** & 2.76 & **0.65** & -1.10 & **0.71** & 13.70 & 0.07 & -0.35 & 0.54 & 0.60 & 0.54 & 0.72 & 0.40 & -5.19 & **0.51** & 1.59 \\ GritLM-7B & 0.63 & 3.09 & 0.63 & -0.06 & 0.70 & 15.10 & 0.10 & -0.32 & 0.51 & 4.01 & **0.57** & -0.09 & **0.42** & -0.32 & **0.51** & 3.06 \\ ES-mistral-7B & 0.54 & 4.26 & 0.63 & 0.05 & 0.69 & 14.14 & 0.10 & 0.08 & **0.57** & **0.31** & 0.56 & 0.92 & 0.28 & -4.42 & 0.48 & 3.05 \\ Intrractor-XL & 0.48 & 1.03 & 0.49 & -2.36 & 0.53 & 0.35 & 0.07 & -0.30 & 0.53 & 1.96 & 0.17 & -2.06 & 0.19 & 0.18 & 0.35 & -0.17 \\ Intractor-large & 0.49 & 3.65 & 0.46 & 0.20 & 0.56 & 3.68 & 0.07 & 0.29 & 0.51 & 2.19 & 0.15 & -3.86 & 0.17 & 6.70 & 0.34 & 1.84 \\ Promptriever-7B & 0.22 & **8.95** & 0.34 & **3.69** & 0.60 & **18.17** & 0.09 & -0.31 & 0.52 & 5.18 & 0.35 & **1.26** & 0.09 & 0.77 & 0.32 & **8.00** \\ Intructor-Base & 0.39 & 3.31 & 0.45 & 0.42 & 0.48 & 2.06 & 0.06 & 0.18 & 0.51 & -2.70 & 0.17 & 1.34 & 0.09 & **13.93** & 0.31 & 2.65 \\ \hline \multicolumn{13}{c}{_Propietary Models_} \\ OpenAI-v3-large & 0.54 & 1.57 & 0.59 & -0.48 & 0.58 & 0.31 & **0.11** & -0.03 & **0.57** & -0.03 & 0.52 & 0.18 & 0.30 & -5.72 & 0.46 & -0.60 \\ OpenAI-v3-small & 0.46 & 2.31 & 0.58 & -0.94 & 0.56 & 0.83 & 0.08 & -0.29 & 0.53 & 3.26 & 0.41 & -1.21 & 0.24 & -0.81 & 0.41 & 0.45 \\ \hline \end{tabular}
\end{table}
Table 3: Performance of retrievers on IFIR measured by nDCG@20 and InstFol@20 (%). Fine-grained results(_i.e.,_ hybrid) can be found at Appendix C. Model performance is ranked based on average results with the nDCG metric.

 

performance, an error case study, and a reliability evaluation of the proposed IFIR metric.

### Main Results

Table 3 presents the main results, from which we derive the following key findings:

Non-instruction-tuned modelsBM25 demonstrates relatively good performance, suggesting possible lexical bias in the datasets. Moreover, GTR models outperform BERT-based models. Unlike ColBERT and Contriever, which are trained solely on the MSMARCO dataset, GTR models also utilize the collected community question-answer pairs and Natural Question Kwiatkowski et al. (2019) datasets. These datasets, which are more closely aligned with human interactions, may contribute to the superior performance of GTR models.

Instruction-tuned models(1) The InstructOR models show minimal improvement over their backbone (_i.e.,_ GTR), and in some cases, even perform worse. This may indicate that the InstructOR models could be overfitting on specific datasets or are better suited to shorter instructions. (2) GritLM-7B, which is of the same size as E5-mistral-7b-instruct, demonstrates stronger performance on healthcare domains where E5-mistral-7b-instruct encounters difficulties. This performance gap may stem from GritLM-7B's inclusion of training data in extra domains, which likely boosts its capacity to handle healthcare content more effectively. (3) Notably, Promptriever-7b outperforms other open-source retrievers in the InstFol metric, while NV-Embed-v2 and GritLM-7B demonstrate outstanding performance in the nDCG metric. Promptriever-7b excels in InstFol due to its targeted training on instance-level instructions, enabling it to adjust relevance based on user input. (4) The proprietary retriever, OpenAI-v3-Large, achieves relatively strong performance on nDCG. However, both OpenAI's retrievers do not demonstrate superior performance on InstFol compared to other retrievers. Unfortunately, the technical details of them, including their training processes, are confidential, which limits our ability to fully understand the factors contributing to their performance.

OverallThe current training methodologies that integrate instructions are not yet perfect solutions for handling long instructions across various domains. From the relatively good performance of BM25 on both metrics, we can deduce that lexical search may serve as an auxiliary tool for complex instructions in specific domains. Meanwhile, although some LLM-based retrievers do not perform well in traditional metrics like nDCG, they exhibit a superior and stable instruction-following ability compared to other retrievers. Additionally, the instruction-trained method introduced by Promptriever is highly intuitive and effective, holding promise for future integration into IR systems.

### Analysis

Scaling up model size leads to better retrieval performance.From Table 3, we can conclude that the scaling law applies to retrievers as well. Specifically, as model sizes increase from 110M to 1B, both the GTR and InstructOR models demonstrate improved nDCG metrics. Additionally, LLM-based retrievers(_e.g.,_ E5-mistral-7b-instruct, GritLM-7B) exhibit relatively strong performance on average. However, when considering instruction-following ability, the scaling law does not apply when the model size is below the 1B threshold. Given the strong performance of GritLM-7B and Promptriever in instruction-following ability, it can be inferred that the current retrieval system can be further enhanced by LLMs finetuned for retrieval tasks.

Existing instruction training methods are still limited.Currently, some retrievers, such as InstructOR and GritLM-7B, are trained with instructions like "Retrieve document from Wikipedia" or "Classify the question's topic" to fit the varying demands of different domains. We investigate how significantly such training methods can enhance performance across various domains. Accordingly, as described in these works, we incorporate these instructions as prompts in both the query and embedding processes. We format the input as "[Prompt] [Query] [Instruction]". The prompt is actually the instruction in these works which gives hints to target tasks and domains, _e.g.,_ "Represent the science question for retrieval. " which is different from our instruction. We use instruction in these works as a prompt to check whether this is an enhancement compared to embedding with no prompt. The results are shown in Figure 3, with detailed outcomes available in the Appendix C.2. We observe that adding instructions does not significantly impact the final performance. Additionally, for LLM-based retrievers (_i.e.,_ GritLM), performance even declines.

 

Therefore, for various domains, merely adding minimal instructions is insufficient. Domain-specific datasets and more complex instructions are required for different domains.

Increase in instruction complexity results in performance decline.As discussed in SS3.2, to test the instruction-following abilities of different retrieval systems with finer granularity, we construct instructions with varying complexity levels. As shown in Figure 4, there is a noticeable performance degradation with level2 and level3 instructions compared to level1. Interestingly, in some subsets, level3 performs better than level2. This improvement is attributed to the fact that level3 instructions are longer and contain more explicit conditions, providing more hints about possible candidates. Overall, we observe that retrievers fine-tuned from LLMs exhibit robust and superior performance compared to other models. This excellent performance may result from LLMs' general capabilities and long-context abilities. Detailed results can be found in the Appendix C.3.

### Error Analysis

We select those instructions with both low nDCG scores and InstFol scores and create a taxonomy of these instructions, categorizing them as (1) Long Instructions. In the legal domain, some instructions exceed 1,024 tokens due to the inclusion of lengthy legal case. Current retrievers are typically trained with a maximum token length of 512, which cannot perfectly handle these lengthy instructions. (2) Dense with Specialized Knowledge. For instructions that require specialized knowledge, especially in the science literature and healthcare domains, common training data do not cover all the expert knowledge needed in specialized domains. (3) Highly Customized Instructions, as illustrated in B. In finance and healthcare domains, users or doctors have several prioritized goals and needs that traditional retrievers may not recognize.

## 6 Reliability Analysis of InstFol

To evaluate the reliability of the proposed LLM-based evaluation metric, InstFol, we conduct a thorough analysis. Our focus is on InstFol's core component: the LLM's ability to score the relevance between an instruction-following query and a passage. We validate these LLM-based relevance scores by comparing them with human evaluation scores. Specifically, we randomly sample 400 query-passage pairs and assign each to one domain expert for evaluation. Both the human evaluators and the LLM (_i.e.,_ GPT-4o) are instructed to rate relevance on a scale of 1 to 5. After collecting the scores from both sources, we evaluate their alignment at the instance level. Notably, we observe a strong Pearson correlation coefficient of 0.704, indicating that InstFol can effectively assess a retriever's instruction-following capabilities.

## 7 Conclusion

We introduce IFIR, a novel IR benchmark designed to assess the instruction-following capabilities of retrievers. IFIR focuses on domain-specific instructions, reflecting the diverse needs across various fields. Our experiments reveal that current instruction-tuned models struggle with long, complex instructions. Moreover, as the complexity increases, a noticeable performance decline occurs across all tested retrieval systems. However, LLVM-based retrievers demonstrate more robust performance and relatively better results compared to other models. This suggests potential solutions for end-to-end specialized-domain retrieval scenarios.

Figure 4: Average nDCG@20 performance on different levels of instructions in different domains.

Figure 3: The nDCG improvement when the model is provided with detailed instructions for retrieval.

 



### Limitations

In this section, we list three limitations of this study, each of which opens avenues for future improvements. First, our dataset has a limited number of queries accompanied by instructions, and we do not provide a training dataset for future works to train their models. Second, we do not compare the performance of domain-specific retrievers like BioBERT [10] with general-purpose retrievers. Domain-specific retrievers, equipped with specialized training data, may achieve superior results in niche fields. Future work should explore the integration and evaluation of such specialized models, particularly in domains like healthcare or law, where domain knowledge is crucial. Finally, there are some relevance judgment gaps [24] in the selected seed datasets. Since we select relevant passages based on previously annotated passages, this approach may lead to some relevant passages being ignored. Future work could investigate ways to improve relevance judgment, potentially through the use of more crowd-sourced human annotation.

## References

* Abbasiantaeb et al. (2024) Zahra Abbasiantaeb, Chuan Meng, Leif Azzopardi, and Mohammad Aliannejadi. 2024. Can we use large language models to fill relevance judgment holes? _arXiv preprint arXiv:2405.05600_.
* AI@Meta (2024) AI@Meta. 2024. The llama 3 herd of models.
* Ajith et al. (2024) Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, and Tianyu Gao. 2024. Listsearch: A retrieval benchmark for scientific literature search. _arXiv preprint arXiv:2407.18940_.
* Asai et al. (2023) Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. 2023. Task-aware retrieval with instructions. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 3650-3675.
* Bajaj et al. (2018) Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018. Ms marco: A human generated machine reading comprehension dataset.
* Bhattacharya et al. (2019) Paheli Bhattacharya, Kripabadhu Ghosh, Saptarshi Ghosh, Arindam Pal, Parth Mehta, Arnab Bhattacharya, and Prasenjit Majumder. 2019. Fire 2019ila track: Artificial intelligence for legal assistance. In _Proceedings of the 11th annual meeting of the forum for information retrieval evaluation_, pages 4-6.
* Boteva et al. (2016) Vera Boteva, Demian Gholipour, Artem Sokolov, and Stefan Riezler. 2016. A full-text learning to rank dataset for medical information retrieval.
* Chan et al. (2024) Chi-Min Chan, Chunpu Xu, Ruibin Yuan, Hongyin Luo, Wei Xue, Yike Guo, and Jie Fu. 2024. Rq-rag: Learning to refine queries for retrieval augmented generation. _arXiv preprint arXiv:2404.00610_.
* Chen et al. (2021) Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R Routledge, et al. 2021. Finqa: A dataset of numerical reasoning over financial data. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 3697-3711.
* Cohan et al. (2020) Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld. 2020. Specter: Document-level representation learning using citation-informed transformers. In _ACL_.
* Goebel et al. (2024) Randy Goebel, Yoshinobu Kano, Mi-Young Kim, Juliano Rabelo, Ken Satoh, and Masaharu Yoshioka. 2024. Overview and discussion of the competition on legal information extraction/entailment (colicee) 2023. _The Review of Socionetwork Strategies_, 18(1):27-47.
* Groeneveld et al. (2023) Dirk Groeneveld, Iz Beltagy, Evan Walsh, AKshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jia, Hamish Ivison, Ian Magnussson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Armen Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Valentine Pyatkin, Abhihasi Ravichander, Dustin Schwenk, Saurabh Shah, William Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah Smith, and Hannaneh Hajishirzi. 2024. OLMo: Accelerating the science of language models. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 15789-15809, Bangkok, Thailand. Association for Computational Linguistics.
* Ionescu et al. (2024) Bogdan Ionescu, Henning Muller, Ana-Maria Drugulinescu, Johannes Ruckert, Asma Ben Abacha, Alba Garcia Seco de Herrera, Louise Bloch, Raphael Brungel, Ahmad Idrissi-Yaghir, Henning Schafer, et al. 2024. Overview of the imageclef 2024: multimedia retrieval in medical applications. In _International Conference of the Cross-Language Evaluation Forum for European Languages_, pages 140-164. Springer.
* Izacard et al. (2022) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. _Transactions on Machine Learning Research_.
* Jangid et al. (2018) Hitkl Jangid, Shivangi Singhal, Rajiv Ratn Shah, and Roger Zimmermann. 2018. Aspect-based financial

sentiment analysis using deep learning. In _Companion Proceedings of the The Web Conference 2018_, pages 1961-1966.
* Jarvelin and Kekalainen (2002) Kalervo Jarvelin and Jaana Keskalaainen. 2002. Cumulated gain-based evaluation of ir techniques. _ACM Trans. Inf. Syst._, 20(4):422-446.
* Jiang et al. (2023) Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaptlot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L'elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothee Lacroix, and William El Sayed. 2023. Misrtal 7b. _ArXiv_, abs/2310.06825.
* Johnson et al. (2016) Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. Mimic-iii, a freely accessible critical care database. _Scientific data_, 3(1):1-9.
* Ke et al. (2024) YuHe Ke, Liyuan Jin, Kabilan Elangovan, Hairil Rizal Abdullah, Nan Liu, Alex Tiong Heng Sia, Chai Rick Soh, Joshua Yi Min Tung, Jasmine Chiat Ling Ong, and Daniel Shu Wei Ting. 2024. Development and testing of retrieval augmented generation in large language models-a case study report. _arXiv preprint arXiv:2402.01733_.
* Khattab and Zaharia (2020) Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval_, pages 39-48.
* Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: a benchmark for question answering research. _Transactions of the Association of Computational Linguistics_.
* Lee et al. (2024) Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. Nv-embed: Improved techniques for training llms as generalist embedding models. _arXiv preprint arXiv:2405.17428_.
* Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyun Kim, Chan Ho So, and Jaewoo Kang. 2020. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. _Bioinformatics_, 36(4):1234-1240.
* Li et al. (2023) Haitao Li, Yunqiu Shao, Yueyue Wu, Qingyao Ai, Yixiao Ma, and Yiqun Liu. 2023. Lecardv2: A large-scale chinese legal case retrieval dataset. _arXiv preprint arXiv:2310.17609_.
* Liu et al. (2021) Bulou Liu, Yueyue Wu, Yiquiu Liu, Fan Zhang, Yunqiu Shao, Chenliang Li, Min Zhang, and Shaoping Ma. 2021. Conversational vs traditional: Comparing search behavior and outcome in legal case retrieval. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_, pages 1622-1626.
* Liu et al. (2024) Hao Liu, Ali Soroush, Jordan G Nestor, Elizabeth Park, Betina Idnay, Yih Fang, Jane Pan, Stan Liao, Marguerite Bernard, Yifan Peng, et al. 2024. Retrieval augmented scientific claim verification. _JAMIA open_, 7(1):ooae021.
* Liu et al. (2023) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chengunya Zhu. 2023. G-eval: NLg evaluation using gpt-4 with better human alignment. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 2511-2522.
* Lisnie et al. (2024) Adian Lisnie, Potsawee Manakul, and Mark Gales. 2024. LLm comparative assessment: Zero-shot nlg evaluation through pairwise comparisons using large language models. In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 139-151.
* Ma et al. (2021) Yixiao Ma, Yunqiu Shao, Yueyue Wu, Yiqun Liu, Ruizhe Zhang, Min Zhang, and Shaoping Ma. 2021. Lexcard: a legal case retrieval dataset for chinese law system. In _Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval_, pages 2342-2348.
* Mandal et al. (2017) Arpan Mandal, Kripabandhu Ghosh, Arnab Bhattacharya, Arindam Pal, and Saptarshi Ghosh. 2017. Overview of the fire 2017 irled track: Information retrieval from legal documents. In _FIRE (Working Notes)_, pages 63-68.
* Muenhighoff et al. (2024) Niklas Muenhighoff, Hongjun Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amamperet Singh, and Douwe Kiela. 2024. Generative representational instruction tuning. _arXiv preprint arXiv:2402.09906_.
* Muennighoff et al. (2023) Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. Metbe: Massive text embedding benchmark. In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_, pages 2014-2037.
* Msysore et al. (2022) Sheshera Msysore, Arman Cohan, and Tom Hope. 2022. Multi-vector models with textual guidance for fine-grained scientific document similarity. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4453-4470, Seattle, United States. Association for Computational Linguistics.
* Ni et al. (2022) Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, et al. 2022. Large dual encoders are generalizable retrievers. In _Proceedings 

of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 9844-9855.
* Oh et al. (2024) Hanseok Oh, Hyunji Lee, Seonghyeon Ye, Haebin Shin, Hansol Jang, Changwook Jun, and Minjoon Seo. 2024. Instructir: A benchmark for instruction following of information retrieval models. _arXiv preprint arXiv:2402.14334_.
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In _Advances in Neural Information Processing Systems_.
* Roberts et al. (2020) Kirk Roberts, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, Kyle Lo, Ian Soboroff, Ellen Voorhees, Lucy Lu Wang, and William R Hersh. 2020. Trec-covid: rationale and structure of an information retrieval shared task for covid-19. _Journal of the American Medical Informatics Association_, 27(9):1431-1436.
* Roberts et al. (2018) Kirk Roberts, Dina Demner-Fushman, Ellen M. Voorbees, William R. Hersh, Steven Bedrick, and Alexander J. Lazar. 2018. Overview of the TREC 2018 precision medicine track. In _Proceedings of the Twenty-Seventu Text RErieval Conference, TREC 2018, Gaithersburg, Maryland, USA, November 14-16, 2018_, volume 500-331 of _NIST Special Publication_. National Institute of Standards and Technology (NIST).
* Roberts et al. (2017) Kirk Roberts, Dina Demner-Fushman, Ellen M Voorhees, William R Hersh, Steven Bedrick, Alexander J Lazar, and Shubham Pant. 2017. Overview of the trec 2017 precision medicine track. In _The... text REtrieval conference: TREC. Text REtrieval Conference_, volume 26. NIH Public Access.
* Roberts et al. (2015) Kirk Roberts, Matthew S Simpson, Ellen M Voorhees, and William R Hersh. 2015. Overview of the trec 2015 clinical decision support track. In _TREC_.
* Robertson et al. (2009) Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. _Foundations and Trends@ in Information Retrieval_, 3(4):333-389.
* Saxena et al. (2022) Shreya Saxena, Raj Sangani, Siva Prasad, Shubham Kumar, Mihir Athale, Rohan Awhad, and Vishal Vaddina. 2022. Large-scale knowledge synthesis and complex information retrieval from biomedical documents. In _2022 IEEE International Conference on Big Data (Big Data)_, pages 2364-2369. IEEE.
* Su et al. (2023a) Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A Smith, Luke Zettlemoyer, and Tao Yu. 2023a. One embedder, any task: Instruction-finetuned text embeddings. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 1102-1121.
* Su et al. (2023b) Hongjun Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A Smith, Luke Zettlemoyer, and Tao Yu. 2023b. One embedder, any task: Instruction-finetuned text embeddings. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 1102-1121.
* Su et al. (2023c) Hongjun Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Haisu Liu, Quan Shi, Zachary S Siegel, Michael Tang, Ruoxi Sun, Jinsuing Yoon, Sercan O'Arik, Danqi Chen, and Tao Yu. 2023c. Bright: A realistic and challenging benchmark for reasoning-intensive retrieval.
* Tamine and Goeuriot (2021) Lynda Tamine and Lorraine Goeuriot. 2021. Semantic information retrieval on medical texts: Research challenges, survey, and open issues. _ACM Computing Surveys (CSUR)_, 54(7):1-38.
* Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Ruckle, Abhishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
* Wadden et al. (2022) David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Iz Beltagy, Lucy Lu Wang, and Hannaneh Hajishirzi. 2022. Scifact-open: Towards open-domain scientific claim verification. In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 4719-4734.
* Wang et al. (2023a) Jianyou Wang, Kaicheng Wang, Xiaoyue Wang, Prudhwiraj Naidu, Leon Bergen, and Ramamohan Paturi. 2023a. Doris-mae: scientific document retrieval using multi-level aspect-based queries. In _Proceedings of the 37th International Conference on Neural Information Processing Systems_, pages 38404-38419.
* Wang et al. (2023b) Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023b. Improving text embeddings with large language models. _arXiv preprint arXiv:2401.00368_.
* Wang et al. (2023c) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023c. Self-instruct: Aligning language models with self-generated instructions. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 13484-13508.
* Weller et al. (2024a) Orion Weller, Benjamin Chang, Sean MacAvaney, Kyle Lo, Armen Cohan, Benjamin Van Durme, Dawn Lawrie, and Luca Soldaini. 2024a. Followir: Evaluating and teaching information retrieval models to follow instructions. _arXiv preprint arXiv:2403.15246_.
* Weller et al. (2025a) Orion Weller, Benjamin Chang, Eugene Yang, Mahsa Yamohammadi, Sam Barham, Sean MacAvaney, Araman Cohan, Luca Soldaini, Benjamin Van Durme, and Dawn Lawrie. 2025a. mfollowir: a multilingual benchmark for instruction following in retrieval 

Orion Weller, Kathryn Ricci, Eugene Yang, Andrew Yates, Dawn Lawrie, and Benjamin Van Durme. 2025b. Rank1: Test-time compute for reranking in information retrieval.
* Weller et al. (2024) Orion Weller, Benjamin Van Durme, Dawn Lawrie, Ashwin Paranjape, Yuhao Zhang, and Jack Hessel. 2024b. Prompritever: Instruction-trained retrievers can be prompted like language models. _arXiv preprint arXiv:2409.11136_.
* Wu et al. (2024) Siwei Wu, Yizhi Li, Kang Zhu, Ge Zhang, Yiming Liang, Kaijing Ma, Chenghao Xiao, Haoran Zhang, Bohao Yang, Wenhu Chen, et al. 2024. Scimmir: Benchmarking scientific multi-modal information retrieval. _arXiv preprint arXiv:2401.13478_.
* Xiao et al. (2019) Chaojun Xiao, Haoxiang Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Tianyang Zhang, Xianpei Han, Zhen Hu, Heng Wang, and Jianfeng Xu. 2019. Cail2019-scm: A dataset of similar case matching in legal domain. _ArXiv_, abs/1911.08962.
* Xiong et al. (2024) Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. 2024. Benchmarking retrieval-augmented generation for medicine. _arXiv preprint arXiv:2402.13178_.
* Yang et al. (2019) An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guunting Dong, Haoran Wei, Huan Lin, Jiulong Tang, Jialin Wang, Jianxing Tu, Jianwei Zhang, Jianxin Yao, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzhen He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruiz Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinnan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuahu Zhou, Xingzhang Ren, Xinyun Zhang, Xipin Wei, Xuancheng Ren, Xueijing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. 2024. Qwen2 technical report.
* Zhao et al. (2024a) Xinran Zhao, Tong Chen, Siiao Chen, Hongming Zhang, and Tongshuang Wu. 2024a. Beyond relevance: Evaluate and improve retrievers on perspective awareness. _arXiv preprint arXiv:2405.02714_.
* Zhao et al. (2024b) Yilun Zhao, Hongjun Liu, Yitao Long, Rui Zhang, Chen Zhao, and Arman Cohan. 2024b. Financemath: Knowledge-intensive math reasoning in finance domains. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 12841-12858, Bangkok, Thailand. Association for Computational Linguistics 



## Appendix A IrIR Benchmark

### Instruction Generation

We first ask LLM (_i.e.,_ GPT-4o) to generate an instruction according to the reality demands, with the prompts on dataset NFCoprus shown in Figure 5. And then we check whether previous annotated passages for the query satisfy the generated instruction, with the prompts shown in Figure 6.

For the datasets with multi-levels, including FiQA, SciFact-open, AILA, and TREC-PM, we do not construct a fully complex instruction all at once. Since we have different levels of reasoning, we ask the LLM to detail the instruction level by level, akin to a bottom-up approach. For example, for a given instruction at level 2, such as "Please help me to find the plaintiff's beneficial legal case." we request the LLM to generate a more complex instruction for the next level that also includes the "plaintiff's beneficial" condition. The prompt for instruction generation on dataset AILA is shown in Figure 7.

### Details of Instructions in Each Domain

Instruction examples are shown in Table 5. For the legal datasets, which belong to the legal domain, the query part consists of only a summary or is omitted due to the length context of legal cases.

Different complexity instruction examples are shown in Table 6. To emphasize, we describe the content of each level again. As the level increases, so do the conditions. Target passages in Level 3 must satisfy the conditions of Levels 1 and 2, and Level 2 candidates must satisfy Level 1 conditions. (1) **FiQA**: The first level simply asks for financial suggestions. The second level includes information about personal financial status. The third level incorporates personal financial purposes.

(2) **Scifact-open**: The first level involves asking for science passages relevant to a given science claim. The second level seeks evidence that either contradicts or supports this claim. The third level is tailored for students or researchers who need to find evidence based on customized demands.

(3) **AILA**: The first level involves searching for similar cases. The second level requires that the relevant case be beneficial for the plaintiff or defendant. The third level adds more explicit conditions such as the details of the current cases, potential goals, and more similar scenarios.

(4) **TREC-PM**: The first level includes information about the patient's disease. The second level adds the patient's demographics, including age and gender. The third level incorporates additional information about the patient's treatment history and family history.

## Appendix B Implementation Details

EmbeddingTo accommodate the long context of certain passages, we employ a sliding window of 512 tokens with an overlap of 128 tokens, using mean pooling to generate embeddings. For LLM-based retrievers, however, we do not apply mean pooling due to their extended context window. Given hardware constraints, we run LLM-based retrievers in FP16 mode to reduce GPU memory usage. When querying, we concatenate the query and instruction with a space character. Because of the large number of passages and the correspondingly large embeddings, we use LangChain and Elasticsearch in our experiments. In addition, we also provide the BEIR code.

Implementation Details of InstFolGiven \(C_{\text{q}}\), which is the retrieved passages set for a query, and \(C_{\text{inst}}\), which is the retrieved corpus set for the query combined with instructions, we evaluate each passage in both sets using LLM (_i.e.,_ gpt-4o-mini) with some evaluation criteria. This approach is inspired by G-Eval Liu et al. (2023) and TREC's principles of data collection. We then obtain two sets of weighted scores, \(S_{q}\) and \(S_{\text{inst}}\). \(S_{\text{q}}\) measures how well the passages retrieved by the original query (without instruction) align with the given instruction; \(S_{\text{inst}}\) measures how well the passages retrieved by the query (with instruction) meet the same instruction. Each score in \(S_{q}\) and \(S_{\text{inst}}\) is calculated as follows, where \(s_{k}\) is an element of \(S_{q}\) or \(S_{\text{inst}}\), and \(p_{k}\) represents the logarithmic probability of each score as determined by the API:

\[\text{weighted\_score}=\frac{\sum^{10}_{k=1}(s_{k}\times e^{p_{k}})}{\sum^{10 }_{k=1}e^{p_{k}}}\]

Here, \(e^{p_{k}}\) converts the log probabilities back to standard probabilities for calculation purposes. This formula accounts for the inherent probabilistic nature of LLMs, where predictions for each token are based on a statistical probability distribution influenced by configurations such as temperature and top_p.

We use the average of \(S_{\text{q}}\) and \(S_{\text{inst}}\) to calculate the instruction-following ability through a metric we propose, called InstFol. The insight is to consider the maximum improvement a retriever can



**[system prompt]**

You are an expert in science.

**[user input]**

Given the scientific claim: {claim}, Imagine you are a student or researcher seeking information on a specific topic. Based on the conditions listed below, construct a detailed retrieval instruction tailored to the claim. You do not need to incorporate all of the conditions, but ensure your instruction is relevant.

* Research fields

* Research topics

* Research objectives

* Customized needs (For example, experimental subjects, experimental methods, etc.)

The instruction should target a single type of information and be both coherent and logical. It should also be detailed and specific, presented in the first person, and narrated naturally in one paragraph.

Please return your answer as follows:

Instruction: ...

Figure 5: Prompt for generating instruction on NFCorpus dataset.

Figure 6: Prompt for evaluating corpus on NFCorpus dataset.



achieve. Consider a case with two students, A and B. Student A has a rank of 300 and a previous rank of 500, while student B has a rank of 10 and a previous rank of 40. Traditionally, we would calculate improvement through absolute differences. However, student B has less room to improve his rank. Based on this insight, we propose the InstFol metric to evaluate the retriever's instruction-following ability. The factor \(\alpha\) is a normalization function that ensures the InstFol score ranges between -1 and 1. In practice, we set \(\alpha\) as \(\frac{1}{3-S_{\text{q}}}\).

InstFol@K = \((S_{\text{inst}}-S_{\text{q}})\cdot\alpha\) (2)

When calling the API to evaluate the InstFol, we use top_p = 0.7 and top_logprobs = 5. We set the temperature to 0.0 to reduce the overestimation by the LLM. The prompt for evaluation is shown in Figure8.

Error AnalysisThe example of error analysis is illustrated in Table7.

## Appendix C Details of Experimental Results

### Detailed Results of IfIR

In addition to the results obtained using InstFol, we also present the outcomes of traditional IR metrics, as shown in Table8. Furthermore, beyond the _end-to-end_ retrieval method, we implement a _hybrid_ retrieval approach utilizing BM25. An overall improvement in performance can be observed for GTR-xl, Instructor-xl, and Promptriever compared to the _end-to-end_ method. However, as shown in Table8, performance varies across tasks. This highlights the importance of designing a carefully tailored hybrid retrieval pipeline to meet the demands of real-world scenarios.

### Detailed Results of Retrievers with Instructions as Prompts

As shown in Table9, we present the instructions as prompts, as described in these papers. However, we use these instructions as prompts to differentiate from our instructions. The input to the retrievers should be formatted as "[Prompt] [Query] [Instruction]." Additionally, there may be slight differences in the input format due to different models.

### Detailed results of different retrievers on different levels.

The detailed result for each domain is shown in Table10. The result of InstFol is shown in Table11. From Table11, we can find that current information retrieivers are not good at long instructions and instructions with highly dense expert knowledge.

\begin{table}
\begin{tabular}{c l l c} \hline \hline ID & Fluent in English & Major & Annotation tasks \\ \hline
1 & > 10 yrs & Legal & Legal subset annotation \\
2 & > 7 yrs & Legal & Legal subset annotation \\
3 & > 10 yrs & Biology & Healthcare subset annotation; Science Literature subset annotation; Annotation Validation \\
4 & > 7 yrs & Pharmacy & Healthcare subset annotation; Science Literature subset annotation \\
5 & > 10 yrs & Biomedical engineering & Healthcare subset annotation; Science Literature subset annotation; Annotation Validation \\
6 & > 7 yrs & Mathematics & Financial subset annotation; Annotation Validation. \\
7 & > 6 yrs & Finance & Financial subset annotation; \\
8 & Native Speaker & Finance & Financial Subset Annotation \\
9 & > 10 yrs & Legal & Legal subset annotation \\ \hline \hline  & & & \\ \end{tabular}
\end{table}
Table 4: Human annotator’s tasks 

* [14] M. C. Bader, A. D. Kessler, and R. J. S. Levitt, "A review of the role of a new class of social networks in social media: The role of social interactions in social media," _Social Networks_, vol. 30, no. 2, pp. 57-68, 2009.

[MISSING_PAGE_POST]

 



\begin{table}
\begin{tabular}{c l p{142.3pt} p{142.3pt}} \hline Dataset & level1 & level2 & level3 \\ \hline FiQA & Please help me to find the financial suggestions for my query. & I am a 40-year-old account with a steady income and moderate savings. & As a 40-year-old account with a steady income and moderate savings. \\  & & study income and moderate savings. & As a 40-year-old account with a small seeking advice on the best business structure for taxes when combining full-time work, with running a small side business. I am particularly interested in understanding the tax implications, legal considerations, and potential benefits of different business structures. Additionally, I am looking for insights on how to optimize tax efficiency while balancing the demands of pay full-time job and side business \\ SciFact-open & Please find the science passage which related to the claim & Please help me to find the contradict evidence. & As an expert in the field of science, I need to find a peer-reviewed research article or a review paper that presents contracting evidence regarding the relationship between folate deficiency and homocysteine levels in the blood. The passage should offer evidence that proposes the claim stating that a deficiency of folate results in decreased blood levels of homocysteine. \\ AILA & Please help me find the relevant legal cases. & As a defendant player, I want the case where the defendant is beneficial. \\  & & As a defendant player, I want the case where the defendant is beneficial. \\ TREC-PM & \(\Gamma\)m looking for clinical trials suitable for a 38-year-old male patient diagnosed with LDosarcoma with CDK4 Amplification. Please focus on trials specifically targeting Liposarcomas on riskals specified byraeling Liposarcomas. In it crucial that the trials consider the presence of CDR4 Amplification. & This report was not reported as follows: An agent has been diagnosed with Liposarcomas with CDK4 Amplification. It is crucial that the trials consider the presence of CDR4 Amplification in the patient’s condition. Additionally, the patient’s age and gender should be taken into account when selecting suitable clinical trial options. \\  & & As a defendant player, I want the case where the defendant is beneficial. \\ TREC-PM & \(\Gamma\)m looking for clinical trials suitable for a 38-year-old male patient diagnosed with LDosarcoma with CDK4 Amplification. Please focused with CDK4 Amplification. Please focus on trials specifically targeting Liposarcomas on risks specifically targeting Liposarcomas on riskals specified byraeling Liapestron or related soft tissue sarcomas. It is crucial that the trials consider the presence of CDR4 Amplification in the patient’s one and gender should be taken into account when selecting suitable clinical trial options. \\  & & As a defendant player, I want the case where the defendant is beneficial. \\ TREC-PM & \(\Gamma\)m looking for clinical trials suitable for a 38-year-old male patient diagnosed with LDosarcoma with CDK4 Amplification. Please focus on trials specifically targeting Liposarcomas on riskals specified byraeling Liapestron or related soft tissue sarcoma. It is crucial that the trials consider the presence of CDR4 Amplification in the patient’s one and gender should be taken into account when selecting similar clinical trial options. \\  & & As a defendant player, I want the case where the defendant is beneficial. \\  & & As a defendant player, I want the case where the defendant is beneficial. \\ TREC-PM & \(\Gamma\)m looking for clinical trials suitable for a 38-year-old male patient diagnosed with LDosarcoma with CDK4 Amplification. Please focus on trials specifically targeting Liposarcomas on riskals specified byraeling Liapestron or related soft tissue sarcomas. It is crucial that the trials consider the presence of CDR4 Amplification in the patient’s one and gender should be taken into account when selecting similar clinical trial options. \\  & & As a defendant player, I want the case where the defendant is beneficial. \\  & & As a defendant player, I want the case where the defendant is beneficial. \\ TREC-PM & \(\Gamma\)m looking for clinical trials suitable for a 38-year-old male patient diagnosed with LDosarcoma with CDK4 Amplification. Please focus on trials specifically targeting Liposarcomas on riskals specified byraeling Liapestron or related soft tissue sarcomas. It is crucial that the trials consider the presence of CDR4 Amplification in the patient’s one and gender should be taken into account when selecting similar clinical trials outside the patient’s age and gender. As well as any potential influences from his treatment background, smoking history, and obesity. \\  & & As a defendant player, I want the case where the defendant is beneficial. \\  & & As a defendant player, I want the case where the defendant is beneficial. \\ TREC-PM & \(\Gamma\)m looking for clinical trials suitable for a 38-year-old male patient diagnosed with LDosarcoma with CDK4 Amplification. Please focus on trials specifically targeting Liposarcomas on riskals specified byraeling Liapestron or related soft tissue sarcomas. It is crucial that the trials consider the presence of CDR4 Amplification in the patient’s one and gender should be taken into account when selecting similar clinical trial options. \\  & & As a defendant player, I want the case where the defendant is beneficial. \\ TREC-PM & \(\Gamma\)m looking for clinical trials suitable for a 38-year-old male patient diagnosed with LDosarcoma with CDK4 Amplification. Please focus on trials specifically targeting Liposarcomas on riskals specified byraeling Liapestron or related soft tissue sarcomas. It is crucial that the trials consider the presence of CDR4 Amplification in the patient’s one and gender should be taken into account when selecting similar clinical trials outside the patient’s age and gender. As well as any potential influences from his treatment background, smoking history, and obesity. \\  & & As a defendant player, I want the case where the defendant is beneficial. \\  & & As a defendant player, I want the case where the defendant is beneficial. \\  & & As a defendant player, I want the case where the defendant is beneficial. \\ TREC-PM & \(\Gamma\)m looking for clinical trials suitable for a 38-year-old male patient diagnosed with LDosarcoma with CDK4 Amplification. Please focus on trials specifically targeting Liposarcomas on riskals specified byraeling Liapestron or related soft tissue sarcomas. It is crucial that the trials consider the presence of CDR4 Amplification in the patient’s one and gender should be taken into account when selecting similar clinical trials outside the patient’s age and gender. As well as any potential influences from his treatment background, smoking history, and obesity. \\  & & As a defendant player, I want the case where the defendant is beneficial. \\  & & As a defendant player, I want the case where the defendant is beneficial. \\  & & As a defendant player, I want the case where the defendant is beneficial. \\  & & As a defendant player, I want the case where the defendant is beneficial. \\ TREC-PM & \(\Gamma\)m looking for clinical trials suitable for a 38-year-old male pati diagnosed with LDosarcoma with CDK4 Amplification. Please focus on trials specifically targeting Liposarcomas on riskals specified byraeling Liapestron or related soft tissue sarcomas. It is crucial that the trials consider the presence of CDR4 Amplification in the patient’s one and gender should be taken into account when selecting similar clinical trials outside the patient’s age and gender. As well as any potential influences from his treatment background, smoking history, and obesity. \\  & & As a defendant player, I want the case where the defendant is beneficial. \\  & & As a defendant player, I want the case where the defendant is beneficial.





**[system prompt]**

You are an expert in legal domain.

**[user input]**

Given an instruction: instruction, and a prior case: corpus, please evaluate the prior case according to the instruction and Evaluation Criteria and return a JSON object with the score and reason.

There are 3 relevant levels to evaluate the case regarding the instruction:

1. The prior case is similar to the one in the instruction.

2. The prior case satisfies the instruction at the 'plaintiff' or 'defendant' beneficial level.

3. The prior case totally matches the instruction, including the detailed requirements in the instruction.

Evaluation Criteria:

1. If the prior case only meets the instruction at the first level, the score is 1.

2. If the prior case meets the instruction at the first and second levels, the score is 2.

3. If the prior case meets the instruction at all three levels, the score is 3.

4. If the prior case does not meet any of the levels, the score is 0.

Please give a score between 0 and 3.

**

IMPORTANT: Please make sure to only return in JSON format, with the "score" and "reason" key. No additional words or explanations are needed.

Please think step by step about the reason and give the score according to the Evaluation Criteria.

Example JSON:

"score": 1,

"reason": "The corpus only matches the instruction in terms of research field and research topics."

**

JSON:

Figure 8: Prompt for instruction generation on AILA dataset.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c c} \hline \hline  & FiQA & SciFact-open & NFCorpus & AILA & FIRE & TREC-PM & TREC-CDS & Average \\ \cline{2-19}  & nDCG & MRR & nDCG & MRR & nDCG & MRR & nDCG & MRR & nDCG & MRR & nDCG & MRR & nDCG & MRR \\ \hline \multicolumn{19}{c}{_End-to-end retrieval_} \\ \hline GTR-XL & 0.44 & 0.40 & 0.54 & 0.49 & 0.60 & 0.57 & 0.05 & 0.04 & 0.54 & 0.50 & 0.31 & 0.27 & 0.15 & 0.12 & 0.37 & 0.34 \\ BM25 & 0.25 & 0.21 & 0.49 & 0.45 & 0.43 & 0.40 & 0.10 & 0.08 & 0.55 & 0.51 & 0.47 & 0.43 & 0.07 & 0.05 & 0.34 & 0.30 \\ GTR-Large & 0.39 & 0.34 & 0.50 & 0.46 & 0.51 & 0.46 & 0.07 & 0.07 & 0.49 & 0.41 & 0.28 & 0.23 & 0.09 & 0.06 & 0.33 & 0.29 \\ GTR-Base & 0.33 & 0.28 & 0.47 & 0.43 & 0.47 & 0.42 & 0.05 & 0.04 & 0.52 & 0.47 & 0.27 & 0.24 & 0.13 & 0.09 & 0.32 & 0.28 \\ Contiver & 0.13 & 0.10 & 0.29 & 0.24 & 0.36 & 0.29 & 0.08 & 0.06 & 0.51 & 0.48 & 0.09 & 0.06 & 0.04 & 0.04 & 0.21 & 0.18 \\ ColBERT & 0.07 & 0.05 & 0.14 & 0.12 & 0.16 & 0.13 & 0.07 & 0.05 & 0.39 & 0.35 & 0.02 & 0.01 & 0.00 & 0.00 & 0.12 & 0.10 \\ \(\cdot\)NV-Embed-v2 & **0.68** & **0.66** & **0.65** & **0.62** & **0.71** & **0.70** & 0.07 & 0.04 & 0.54 & 0.51 & 0.54 & 0.53 & 0.40 & 0.36 & **0.51** & **0.49** \\ GrLM-7B & 0.63 & 0.61 & 0.63 & 0.60 & 0.70 & 0.69 & 0.10 & 0.06 & 0.51 & 0.46 & **0.57** & **0.54** & **0.42** & **0.38** & **0.51** & 0.48 \\ ES-mistral-7b-inst & 0.54 & 0.51 & 0.63 & 0.59 & 0.69 & 0.67 & 0.10 & 0.06 & 0.57 & **0.54** & 0.56 & 0.52 & 0.28 & 0.23 & 0.48 & 0.45 \\ Instructor-XL & 0.48 & 0.44 & 0.49 & 0.44 & 0.53 & 0.47 & 0.07 & 0.05 & 0.53 & 0.49 & 0.17 & 0.12 & 0.19 & 0.15 & 0.35 & 0.31 \\ Instructor-large & 0.49 & 0.45 & 0.46 & 0.42 & 0.56 & 0.51 & 0.07 & 0.06 & 0.51 & 0.45 & 0.15 & 0.11 & 0.17 & 0.12 & 0.34 & 0.30 \\ Prompriteiver-7B & 0.22 & 0.17 & 0.34 & 0.28 & 0.60 & 0.59 & 0.09 & 0.07 & 0.52 & 0.48 & 0.35 & 0.29 & 0.09 & 0.06 & 0.32 & 0.28 \\ Instructor-base & 0.39 & 0.34 & 0.45 & 0.42 & 0.48 & 0.42 & 0.06 & 0.05 & 0.51 & 0.46 & 0.17 & 0.13 & 0.09 & 0.07 & 0.31 & 0.27 \\ OpenAI-v3-large & 0.54 & 0.51 & 0.59 & 0.54 & 0.58 & 0.55 & **0.11** & **0.08** & **0.57** & 0.54 & 0.52 & 0.46 & 0.30 & 0.26 & 0.46 & 0.42 \\ OpenAI-v3-small & 0.46 & 0.41 & 0.58 & 0.52 & 0.56 & 0.52 & 0.08 & 0.06 & 0.53 & 0.48 & 0.41 & 0.38 & 0.24 & 0.21 & 0.41 & 0.37 \\ \hline \multicolumn{19}{c}{_Hybrid retrieval_} \\ \hline GtrlM-7B & **0.59** & **0.54** & **0.62** & **0.58** & **0.57** & **0.51** & **0.10** & 0.07 & 0.55 & 0.50 & **0.59** & **0.56** & **0.39** & **0.32** & **0.49** & **0.44** \\ GTR-XL & 0.43 & 0.38 & 0.57 & 0.51 & 0.52 & 0.47 & 0.06 & 0.06 & **0.56** & **0.53** & 0.33 & 0.29 & 0.14 & 0.13 & 0.37 & 0.34 \\ Instructor-XL & 0.46 & 0.41 & 0.54 & 0.51 & 0.52 & 0.46 & 0.09 & **0.07** & 0.56 & 0.52 & 0.23 & 0.20 & 0.20 & 0.16 & 0.37 & 0.33 \\ Prompriteiver-7B & 0.25 & 0.21 & 0.42 & 0.38 & 0.54 & 0.49 & 0.09 & 0.07 & 0.56 & 0.53 & 0.42 & 0.37 & 0.11 & 0.09 & 0.34 & 0.31 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Performance of retrievers on IFIR measured by nDCG@20 and MRR@20.





\begin{table}
\begin{tabular}{l c c c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{FiQA} & \multicolumn{3}{c}{AILA} & \multicolumn{3}{c}{TREC-PM} & \multicolumn{3}{c}{Scifact-open} \\ \cline{2-10}  & Level1 & Level2 & Level3 & Level1 & Level2 & Level3 & Level1 & Level2 & Level3 & Level1 & Level2 & Level3 \\ \hline BM25 & 0.282 & 0.221 & 0.239 & 0.158 & 0.060 & 0.030 & 0.505 & 0.437 & 0.482 & 0.568 & 0.438 & 0.480 \\ Contriever & 0.146 & 0.121 & 0.111 & 0.144 & 0.018 & 0.012 & 0.112 & 0.084 & 0.077 & 0.306 & 0.252 & 0.325 \\ ColBERT & 0.078 & 0.043 & 0.107 & 0.111 & 0.052 & 0.012 & 0.012 & 0.023 & 0.037 & 0.132 & 0.121 & 0.165 \\ GTR-Base & 0.422 & 0.215 & 0.337 & 0.096 & 0.017 & 0.000 & 0.269 & 0.280 & 0.268 & 0.511 & 0.448 & 0.456 \\ GTR-Large & 0.479 & 0.279 & 0.391 & 0.117 & 0.023 & 0.048 & 0.293 & 0.312 & 0.219 & 0.538 & 0.467 & 0.513 \\ GTR-XL & 0.530 & 0.366 & 0.413 & 0.073 & 0.032 & 0.023 & 0.350 & 0.324 & 0.243 & 0.595 & 0.512 & 0.515 \\ \hline Instructor-Base & 0.424 & 0.361 & 0.387 & 0.110 & 0.024 & 0.000 & 0.119 & 0.208 & 0.197 & 0.481 & 0.429 & 0.449 \\ Instractor-Large & 0.531 & 0.454 & 0.472 & 0.110 & 0.024 & 0.048 & 0.144 & 0.157 & 0.147 & 0.480 & 0.404 & 0.513 \\ Instructor-XL & 0.558 & 0.435 & 0.445 & 0.122 & 0.012 & 0.042 & 0.190 & 0.181 & 0.135 & 0.536 & 0.461 & 0.476 \\ E5-mistral-7b-inst & 0.628 & 0.490 & 0.488 & 0.162 & 0.041 & 0.060 & 0.582 & 0.567 & 0.537 & 0.673 & 0.620 & 0.601 \\ GritLM-7B & 0.705 & 0.594 & 0.581 & 0.185 & 0.014 & 0.017 & 0.618 & 0.578 & 0.527 & 0.699 & 0.620 & 0.583 \\ Prompriver-7B & 0.164 & 0.144 & 0.377 & 0.140 & 0.051 & 0.048 & 0.231 & 0.406 & 0.417 & 0.287 & 0.276 & 0.449 \\ NV-Embedv-2 & 0.759 & 0.657 & 0.594 & 0.116 & 0.026 & 0.029 & 0.563 & 0.547 & 0.493 & 0.714 & 0.634 & 0.625 \\ \hline OpenAI-v3-small & 0.529 & 0.410 & 0.429 & 0.148 & 0.032 & 0.013 & 0.428 & 0.436 & 0.371 & 0.631 & 0.545 & 0.563 \\ OpenAI-v3-large & 0.616 & 0.488 & 0.511 & 0.158 & 0.056 & 0.062 & 0.553 & 0.523 & 0.483 & 0.621 & 0.578 & 0.587 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Detailed nDCG@20 results of different retrievers on different levels.

\begin{table}
\begin{tabular}{l l c c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{FiQA} & \multicolumn{3}{c}{AILA} & \multicolumn{3}{c}{TREC-PM} & \multicolumn{3}{c}{Scifact-open} \\ \cline{2-10}  & Level1 & Level2 & Level3 & Level1 & Level2 & Level3 & Level1 & Level2 & Level3 & Level1 & Level2 & Level3 \\ \hline BM25 & -3.76 & -2.48 & 9.53 & -0.08 & -0.15 & 0.29 & 0.83 & 2.28 & 7.17 & -0.70 & -5.36 & 4.85 \\ Contiever & -2.04 & -0.69 & 4.28 & 0.07 & -0.11 & 0.30 & 0.07 & 1.91 & 1.75 & -3.20 & -11.25 & -10.21 \\ ColBERT & -4.43 & -7.28 & 12.21 & 0.24 & 0.85 & -1.13 & 0.14 & 1.11 & 1.84 & -0.23 & -1.06 & 2.31 \\ GTR-Base & -3.76 & -12.39 & 2.30 & -0.06 & 0.51 & 1.18 & -4.56 & -0.12 & 2.73 & -1.09 & -0.39 & -0.39 \\ GTR-Large & -6.40 & -13.20 & 0.18 & 0.12 & -0.31 & -0.88 & -3.37 & -0.28 & -0.73 & -0.18 & 0.64 & -0.30 \\ GTR-XL & -4.45 & -10.59 & 0.49 & 0.15 & -0.20 & -0.34 & -3.21 & -0.73 & 1.40 & -0.14 & 0.46 & -9.12 \\ \hline Instructor-Base & -1.43 & 1.52 & 9.83 & 0.16 & -0.23 & 0.61 & -4.31 & 1.64 & 6.70 & -0.94 & 0.66 & 1.54 \\ Instructor-Large & -0.34 & 2.49 & 8.79 & -0.22 & 0.04 & 1.04 & -2.38 & -2.77 & -6.42 & -1.15 & -2.07 & 3.83 \\ Instructor-XL & -0.66 & -1.93 & 5.67 & 0.03 & -0.07 & -0.86 & -2.00 & -0.82 & -3.37 & 0.53 & 2.34 & -9.95 \\ ES-mistral-7b-inst & 0.14 & -0.13 & 12.78 & 1.11 & -0.08 & -0.79 & -0.89 & 1.10 & 2.55 & -0.22 & 0.19 & 0.18 \\ GridM-7B & 0.25 & 1.68 & 7.32 & -0.37 & 0.06 & -0.66 & -0.75 & 0.14 & 0.33 & -0.36 & 0.92 & -0.73 \\ Prompriveer-7B & -2.41 & 2.18 & 27.09 & -1.22 & 0.37 & -0.07 & 2.09 & 12.49 & 25.21 & -2.02 & -5.17 & 18.27 \\ NV-Embedv-v2 & 0.14 & 1.13 & 7.02 & -0.83 & -0.20 & -0.02 & 0.37 & 0.44 & 1.35 & -0.54 & 0.51 & -3.27 \\ \hline OpenAI-v3-small & -0.60 & -0.62 & 8.16 & 0.12 & 0.10 & -1.09 & -1.93 & -0.80 & -0.88 & 0.26 & 1.72 & -4.79 \\ OpenAI-v3-large & -0.95 & -0.97 & 6.63 & 0.54 & -0.12 & -0.50 & -1.72 & -1.37 & 3.64 & -0.19 & 0.75 & -1.99 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Detailed nDCG@20 results of adding instructions as prompt. The first line is without instruction as a prompt, the second is with instruction as a prompt.