Beyond Content Relevance: Evaluating Instruction Following in Retrieval Models
Authors: Jianqun Zhou
Yuanlei Zheng
Wei Chen
Qianqian Zheng
Shang Zeyuan
Wei Zhang
Rui Meng
Xiaoyu Shen
Venue: ICLR 2025 Poster (2025)
Keywords: LLM
Instruction-Following
Retrieval Model
Benchmark
Primary area: datasets and benchmarks
TL;DR: We propose InFoSearch, a novel benchmark, along with an evaluation protocol to assess the depth of instruction-following capabilities in information retrieval (IR) models.

Abstract: Instruction-following capabilities in large language models (LLMs) have progressed significantly, enabling more complex user interactions through detailed prompts. However, retrieval systems have not matched these advances, most of them still relies on traditional lexical and semantic matching techniques that fail to fully capture user intent. Recent efforts have introduced instruction-aware retrieval models, but these primarily focus on intrinsic content relevance, which neglects the importance of customized preferences for broader document-level attributes. This study evaluates the instruction-following capabilities of various retrieval models beyond content relevance, including LLM-based dense retrieval and reranking models. We develop InfoSearch, a novel retrieval evaluation benchmark spanning six document-level attributes: Audience, Keyword, Format, Language, Length, and Source, and introduce novel metrics -- Strict Instruction Compliance Ratio (SICR) and Weighted Instruction Sensitivity Evaluation (WISE) to accurately assess the models' responsiveness to  instructions. Our findings indicate that although fine-tuning models on instruction-aware retrieval datasets and increasing model size enhance performance, most models still fall short of instruction compliance. We release our dataset and code on https://github.com/EIT-NLP/InfoSearch.

Published as conference paper at ICLR 2025
BEYOND CONTENT RELEVANCE: EVALUATING IN- STRUCTION FOLLOWING IN RETRIEVAL MODELS
Jianqun Zhou'; ; Yuanlei Zheng"! Wei Chen
Qiangian Zheng', Hui Su², Wei Zhang', Rui Meng", Xiaoyu Shen
Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative, Institute of Digital Twin, Eastern Institute of Technology, Ningbo "Meituan Inc. *Salesforce Research "School of Software Engineering, Huazhong University of Science and Technology EEngineering Research Center of Chiplet Design and Manufacturing of Zhejiang Province ru ruimengesalesfoot com, syshen@eiten edu. on
ABSTRACT
cntructiio-following capabilities LLMs have progressed significantly, enabling more complex user interactions through detailed prompts. However, retrieval systems have not matched these advances, most of them still relies on traditional lexical and semantic matching techniques that fail to fully capture user intent. Recent efforts have introduced instruction-awarg retrieval models, but these pri- marily focus intrinsic content relevance, which neglects the importance of customized preferences for broader document-level attributes. This study evaluates the
capabilities of various retrieval models beyond content relevance, including LLM- based dense retrieval and reranking models. We develop InfoSearch, novel retrieval evaluation benchmark spanning six document-level attributes: Audience, Keyword, Format, Language, Length, and Source, and intro- duce novel metrics Strict Instruction Compliance Ratio (SICR) and Weighted Instruction Sensitivity Evaluation (WISE) to accurately assess the models' respon- siveness instructions. Our findings indicate that although fine-tuning models on instruction-awes -wwrre retrieval datasets and increasing model size enhance performance, most models still fall short instruction compliance|
INTRODUCTION
The advent of instruction-followinn in large language models (LLMs) has greatly expanded their generative capabilities (Brown 2020 Lou al., 2023), allowing users express more complex intentions through detailed instructions Black et al. 2022 Touvron et al. 2023). However, retrieval systems have not kept pace with these advancements, continuing to rely traditional lexical or semantic matching techniques (Xiong et al. 2020 Wang et al. 2022 Xiao et al. 2023). As result, while users have grown accustomed to interacting with generative models using intricate instructions (Team et al. 2023 Bai et al. 2023), Achiam et al. 2023), their retrieval behavior remains limited to keyword- based queries followed by manual filtering of results find the desired information. Several studies have started to explore instruction-awant retrievers that can interact with users as seamlessly as generative models, but these primarily focus on task-level instructions (Asai et al. 2023) Wang et al., 2023] |Png et al., 2024), guiding retrievers with one instruction for each task. While this task-level instruction essential for adapting single retrieval model to multiple predefined scenarios, it falls short of meeting users' customized demands beyond standard tasks (Weller et al. 2024b).
Recent works have shifted from task- to instance-level instructions, providing tailored instructions for each instance to better align with customized needs (Weller et al. 22024a] Oh et al. 2024). These approaches specify instructions that which content to include or exclude, thus clarifying user intent. While they greatly enrich the diversity of instructions, their primary focus remains content relevance. When searching for certain documents, users typically care about two aspects: the informational content and its presentation (Taylor 1962 Mizzaro 1998), including document-level
Authors contributed equally. & Corresponding authors.
We release our dataset and code on
com/k: "-NLP/ WNIP/InfoSearch

---

Published as conference paper at ICLR 2025
attributes such as length, language, and format. A sole focus content relevance neglects the importance of customized preferences for broader document-leeee attributes.
We believe that truly instruction-awwrr retrieval system must go beyond content relevance to accommodate a variety of user-defined document attributes. To further research in this direction, propose InfoSearch, novel benchmark designed to evaluate IR models based their ability to follow customized instructions across six structured dimensions: Audience, Keyword, Format, Language, Length, and Source. These dimensions encompass key document-level features that address user needs beyond content relevance. Additionally, we include both instructed and reverse- instructed modes to assess the model's ability to comprehend instructions in both affirmative and negative formats. Each instruction is carefully crafted and manually validated to ensure naturalness and representativeness of complex real-world scenarios.
Beyond the comprehensiveness of datasets, well-defined evaluation metrics are essential to thor- oughly assess the instruction-folloiing capabilities of retrieval models. While traditional IR metrics like nDCG and MRR are primarily effective for assessing content relevance in ad-hoc retrieval tasks (Weller et al. 2024a), we propose new metrics specifically designed to measure the depth of instruction-follove
capabilities in retrieval models. By structuring the evaluation across these separate dimensions and modes, we offer a detailed analysis of how well models follow instructions on each condition, providing clearer insights into their strengths and limitations. Overall, fine-tuning on instruction-awart retrieval datasets and increasing model parameters improve instruction-followinn capabilities, with re-ranking models outperforming retrieval models in this aspect. However, both approaches still show considerable room for improvement in meeting the standards set by our benchmark.
Our contributions can be summarized as follows:
We propose InfoSearch, an evaluation framework that covers six key dimensions: Audience, Keyword, Format, Language, Length, and Source, to assess retrieval models' ability to follow complex instructions beyond content matching.
We introduce two novel metrics Striet Instruction Compliance Ratio (SICR) and Weighted Instruction Sensitivity Evaluation (WISE) that provide more nuanced and accurate assessment of retrieval models' instruction adherence compared to traditional IR metrics.
We evaluate 15 retrieval models, encompassing I sparse retrieval model, 8 bi-encoder-based dense models, and 6 LLM-based reranking models. This thorough evaluation enables com- prehensive comparison across diverse methodologies, delivering valuable insights into their instruction-followiin effectiveness.
2 INFOSEARCH: CONSTRUCTION AND EVALUATION
We construct a benchmark, Instruction-Following Following Search (InfoSearch), to evaluating the search models' ability to follow instructions. InfoSearch is composed of query-doc pairs across six dimen- sions and two novel metrics to measure models' responsiveness to instructions. In this section, $2. details the dimension settings and retrieval modes under the InfoSerach framework, $2.2 explains the dataset construction process, and 2.33 describes the design logic behind our proposed metrics.
2.1 DATASET FRAMEWORK
In real-world scenarios, users exhibit a wide range of complex and diverse search intentions. The use of tailored instructions can link the specific search query content the requirements and preferences of the user. Instructions typically contain detailed information or document-leved characteristics that align with user needs, aiming to enhance the precision and relevance of search results. As shown in Figure [ we conducted an extensive analysis of real users and their underlying intentions to identify six factors (dimensions) influencing search behaviors: user background (Audience), specific search terms topics (Keyword), preferred format for information presentation (Format), required response length (Length). language requirement (Language), and information origin (Source). To enhance the diversity of instructions across these six dimensions, established multiple conditional branches within each dimension, allowing instructions to dynamically adapt and expand based on different conditions. Moreover, within the same dimension and conditions, we create varied

---

Published as conference paper at ICLR 2025
Dimension
Condition
Retrieval Mode
Audience
[Layman], [Expert]
Original Mode:
"How can reduce stress? Retrive relevant passages that unswer the query.
Kyword
[Keyword]
Format
[Post), [Code], [Manual]
Instructed Mode:
"Tell me effective ways to reduce stross. Please provide the answer English."
Language
[Chinese], [English]
Length
[Sentence], [Paragraph], [Article]
Reversely Instructed Mode:
"Tell effective ways reducee stress. Please not provide answers in English."
Source
[Blog]. [Forum], [News]
Figure 1: InfoSearch consists of six dimensions, each representing a document-leve feature with values drawn from predefined conditions. Queries are paired with one dimension and evaluated in three retrieval modes based on the given instructions.
instructions using diverse wording and expressions. This approach not only enriches the dataset but also strengthens the robustness and reliability of the evaluation framework by simulating broad range of potential user inputs.
Drawing inspiration from (Zhang al., 2024), we incorporate semantic negation into the dataset by reversing the meaning of instructions across each dimension. This approach allows each query to be associated with multiple instructions, covering various conditions and offering both positive and negative semantic contexts. This ensures that the model is exposed to three distinct retrieval modes:
Original Mode: This mode serves as a baseline that evaluates the model's basic retrieval ability to find pertinent information without any specific constraints.
Instructed Mode: this mode, the model required to find documents that are content relevant and satisfy the condition specified in the instruction.
In this mode, the model is required to find documents that are content relevant and do not satisfy the condition specified in the instruction, which tests the model's ability to understand negation.
Reversely Instructed Mode:
By integrating six dimensions and three distinct retrieval modes, we have developed the com- prehensive evaluation dataset InfoSearch. This dataset serves as a robust tool for systematically systematically evaluating model's ability to interpret and respond accurately to diverse instructions during retrieval.
2.2 CONSTRUCTION PROCESS
The primary objective of developing InfoSearch is to bridge queries with diverse instructions, ensuring precise alignment between instructions and their corresponding target documents. We achieve this by collecting Question-Answer (Q-A) pairs for each dimension and expanding the target document pool through web-retrieved content. Figure Souuee]oon outlines the 7-step construction process. Data sources, methodological details (c.g., GPT-4 prompts), implementation challenges and dataset statistics are provided Appendix A
Step 1: Condition Determination:
Queries are diversified via multiple conditions, enabling single query to yield distinct relevant documents depending on contextual requirements.
Step 2:
Data Collection: ensure query naturalness and minimize generation costs, we con- sciously integrate conditions when filtering Q-A pairs from existing datasets web pages.
Step 3: Instruction Generation: Requires producing precise, concise, and natural instructions that reflect natural conversational patterns, aligning with users' tendency express intents.
Step 4: Document Rewriting: When queries or documents inadequately address instruction re- quirements, GPT-4 refines existing content to produce contextually appropriate documents.
Step 5: Instruction Reversal: To verify whether ranking improvements stem from instruction comprehension, instruction semantics are systematically reversed, testing model robustness against persistent high-ranking results.

---

Published as conference paper at ICLR 2025
Step 2:
Data Collection
Step 3: Instruction G Geseration
Step Instraction Reversal
Step & Hard Negative Gener attonaii
Care Query: A Advanoes artificiall inlelliennce
Blog
Blog
Iou 1. but ar the resenl
Recarsad Ins 1: How are q Reversed Ins N: W hat are the resent artificial intelligence
Hard Neg I: In recent years, the ar sonrdd hos been. hurzine weith
Thic blen
intelliosone
Slep 4:
Acticle polommenns
Doc N: Science News- iio-inepim camens and
Ins N: How are recent drvelopments artificial intallligene? Please stare article novering
Document Rewriting
Hard Neg N: (CNN NEW S)- Advanoes
intelligence have sparked consers amony privayy advocatess A
Al belp drives detecl
advansemenss? Pleuse cut
padestrinns and obstacles
sham article covering these all atlvrnomm!tss
laster...
these aivancumentss?
survey conducted by the
Step 7: Manual Review
IustructionFoolloiinng Search Search
InfoSearch
Figure Overview of the dataset construction process for InfoSearch.
Step 6: Hard Negative Generations: Adversarial examples are added to resist the model's tendency to depend on superficial document features rather than query-document relationships.
Step 7: Manual Review: Anomalous outputs were excluded, prioritizing documents that consis- tently underperformed or had low relevance scores, followed by expert verification.
By applying the data construction process described above, the InfoSearch benchmark comprises 600 core queries, 1,598 instructed queries, 1,598 reversely instructed queries, and 6,392 documents.
2.3 EVALUATION METRICS
In real-world search systems, user experience hinges on the relevance of top-K results, which directly reflects model efficacy. Thus, instruction-followin models must be evaluated based both original query rankings and their responsiveness to instructions. While metrics like Robustness @k (Oh
al. 2024) and p-MRR (Weller et al.) 2024a) assess instruction compliance, they exhibit five critical limitations: I Robustness vulnerability to single anomalies. 2 Neglects instruction-respons variations. 3 Ignores top-K ranking importance. ④ Insensitive high-rank changes. 2 Inadequate
handling of edge cases. A detailed analysis of these limitations is provided in Appendix B
We define two metrics to quantify the model's responsiveness to instructions: Striet Instruction Com- pliance Ratio (SICR) and Weighted Instruction Sensitivity Evaluation (WISE) metric. Assuming that in the original mode, the core query q has n positive documents, denoted as P p= (Pi, Pg...., p- When it comes to the instructed mode where the core query is designates single gold document P'out of P. demoting others to negatives. When In the reversely instructed mode, P becomes negative. Let Rori, Rine and Rree denote pi 's rankings and Soris Sins and Srev its relevance scores across original, instructed, and reversed modes.
Strict Instruction Compliance Ratio The SICR metric introduces strict criterion for evaluating sensitivity instructions. Ideally, for retrieval result that strictly adheres the instruction, the gold document's 's ranking and relevance score in the instructed mode should be higher than in the original mode, denoted as (Ryns < Rori & Sins > Sori). Simultaneously, in the reversely instruction mode, its ranking and relevance score should be lower than those in the original mode, denoted as (Rori < Rrev Sort > Sree). A query that strictly satisfies these criteria assigned score of 1. Implementing rigorous scoring criteria ensures that all changes of relevant documents taken into account, thereby effectively addressing the issue low-score sensitivity (defect ()) and and incomplete evaluation (defect ()) The formula for this criterion is follows:
[1, (Rins Rori) and (Sins Sori) and (Rori Ryev) and (Sors Srev); 0, otherwise,
I(()
The SICR score calculated the ratio of the number of queries meeting the instruction followwnn criteria to the total number of queries in the test set, represented by the following formula:
SICR Σ)=1((1)
(2)
Q
Where |Q] represents the total number of queries in the test set. This formula calculate the percentage of retrievals that strictly adhere the specified instructions relative to the total results.

---

Published as conference paper at ICLR 2025
Weighted Instruction Sensitivity Evaluation The SICR metric reflect the proportion of model results that comply with instructions but lacks a detailed quantification the degree of compliance. On this basis, the WISE metric relaxes the evaluation criteria by focusing only the ranking changes, regards the results that meet (Rans S Rori < Rrer) as following the instructions, and provides more levels of rewards penalties for the results. It can be calculated using the following formula:
F()) { (Rois
Borin Bins),
if Rins Rori < Brevs
(3)
Spenalty Rori: (Roris Rine),
otherwise.
When defining the reward component, the model is expected to comprehend and execute the in- structions, effectively optimizing the rankings of the top K results accordingly. This implies that significant ranking changes within the top K results should be given greater weight to address defects 3 and ④ these changes more likely to be noticed and utilized by users. It is essential to consider both the absolute ranking Rins and the relative ranking (Rori Rsns). To achieve this, introduced the /Rin.
term, generously rewarding smaller Rans values. Simultaneously. through
the (1 Ron Parr-R.am Runs term, grant higher rewards results that demonstrate substantial ranking improvements. Additionally, a uniform value of 0.01 assigned to results beyond the Top K. More extreme cases are considered (defects )): if core query contains N positive documents in the original mode and meets the conditions Rovi S N and Rins = 1, it will be granted reward of 1. This is based on the premise that, for an ideal retriever, the N positive documents would likely rank at the top in the original mode. Accordingly, results that rank higher and exhibit more significant changes should be assigned greater weight, The reward formula defined as:
if Rors S N and Ryns = =1, if Rori S K, otherwise,
freward
(1
(4)
Rins
0.01,
where K = 20 signifies that our primary focus is on the top 20 retrieval results. Some of the reward values are visualized in the F
For the penalty component, we reference the design of p-MRR, emphasizing the magnitude of the ranking drop and apply stricter demerit points for gold documents that experience a larger decline in ranking. However, unlike p-MRR, our Rins. Roris and Rrev yield six possible permutations. To account for the remaining five cases aside from (Rins S Rori < Rreew, we formulated the following penalty formula:
-1,
if Rrev < Rori < Rines
f Spenalty
Rusi- Rinz, Rins Beay Rari Rori
if Rort S Riner
(5)
if Rree S Rori:
In summary, for test set with queries, the overall evaluation formula can be expressed as:
(())(())
WISE J
(6)
3 EXPERIMENTS
This section first introduces the experimental settings 3.1
followed by description of the overall
retrieval results of different types of search models in 3.2 Lastly, it discusses models' performance across individual dimensions in $3.3
3.1 EXPERIMENTAL SETUP
The goal of the benchmark is to determine how effectively retrieval models adjust their retrieval behavior in response to instructions. To thoroughly assess how state-of-the-ant retrieval models follow instructions, we evaluate 15 models across four categories of models:
We selected 15 models representing the four model architectures:
Rort may he equal to
Rori < Rins covers two cases: (Rori S Rins Rrev) and (Rori S Rrev Rins). Similarly, Pree Rort covers two (Prev S. Rort Rine) ) and (Rrev Rins Rrev).

---

Published as conference paper at ICLR 2025
Table I: Performance comparison of different retrieval models averaged six dimensions. The last three columns display the ranking of the gold document in the original query (Rori), and the relative rank change after applying the instructed and reversed instructions.
1D
Robustmess 10
Gold Document Rank
Model
Oni
Ins
Rev
Ori
Ins
Rev
P-MRR
WISH f
SICR
Pari
Rise
Ryw
BM25
47.5
39.1
38.5
47.5
17.7
20.9
7.0
-12.0
0.0
18.4
18.0
18.3
Dense Retrieval
Bge-Large-v155 1.5
53.2
34.9
34.9
53.2
15.8
21.0
21.3
-29.5
1.0
20.4
25.0
24.9
ES- E5-Large-v2
60.4
52.0
49.9
60.4
26.6
30.2
5.6
-23.3
0.8
14.7
12.8
13.9
Instructor-XL
62.6
39.3
62.6
17.5
23.4
-29.8
30.5
30.5
36.3
Mistral-ins-v0.2
19.4
25.5
29.2
19.4
8.5
12.7
-32.4
-9.22
0.0
236.0
153.0
153.1
GTE GTE-QWeen2
43.6
43.1
48.5
43.6
18.7
26.5
-21.7
-39.0
0.1
104.3
75.3
71.3
E5-Mistral-ins
78.3
64.3
66.0)
78.3
41.8
46.4
-16.3
6.6
5.4
5.6
GritLM
70.8
66.3
70.8
-11.1
14.4
5.8
8.9
70.7
62.2
60.1
70.7
40.7
43.2
4.8
-18.1
2.1
7.4
5.7
5.6
NY-Emmedd-v2
69.5
54,5
52.2
69.5
33.3
36.0
17.7
-13.5
2.8
8.1
8.7
9.3
Point- wise Reranking
Mistral-ins-v0.2
62.0
58.4
59.0
62.0
38.0
44.7
4.1
6.5
4.7
8.8
Llama-3.1
74.8
74.8
14.4
5.4
3.7
8.2
FollowIR
72.4
66.3
65.5
72.4
46.2
50.0
4.1
13.4
12.5
5.5
3.8
7.6
List-wise Reranking (Fine-tuned)
Zephyr-betu
70.8
55.9
58.0
70.8
32.0
36.4
1.7
-3.2
8.7
6,4
6.1
7.0
Rank Vicuna-vi
65.4
55.2
55.2
65.4
31.2
35.7
-6.5
5.6
7.3
6.3
7.0
RankZophyr-vl
75.0
64.7
75.0
41.8
47.5
14.5
10.5
4.5
4.4
5.4
List-wise Reranking (Instructional Zero-shot)
Mistral-ins-v0.2 GPT-4o
74.5
64.4
61.6
74.5
40.5
42.2
8.1
22.0
5.7
4.8
7.2
83.8
74.2
74.2
83.8
53.0
58.0
15.0
33.5
32.1
2.6
1.7
4.3
Sparse retrieval: model, BM25 [Robertson et al. 2009).
Dense retrieval: models, including BGE-large-v
(Xiao al. 2023), E5-large-v2
(Wang et al.] 2022), Instructor-XI, (Su et al. 2023). E5-Mistral (Wang et al.
2023). Gritl M
(Muennighofi al. 2024), NV-Embed-v2 (Lee et al. 2024a), GTE:-Qwe22 et 2023).
and SFR-Emmeedding-i [Meng et al. 2024).
Fine-tuned ranking models: models, including FollowIR (Weller et al. 2024a), Rank cuna (Pradeep al., (023a), RankZephyr (Pradeep al. 2023b), where FollowIR is point-wise model and the other two are list-wise.
Instruction-tuned generation models used for reranking: models, including Mistral- 7B- IB-Instruut-v0.22 (Jiang al. 2023). Zephyr (Tunstall et al. (023). and GPT-4o (Achiam) et al. 2023).
For dense retrieval models, we compute the dot product between query and document vectors to determine retrieval rankings. For reranking models, the top 100 results from F5-mistral (Wang et al.) 2023) are re-ranked based on the models' interpretation of the instruction. For general large language models, we use two settings: In the point-wise setting, both the query and document are inputs, with the output probabilities of True or False used as similarity scores. In the list-wise setting, following (Pradeep al., 2023b). list of documents is provided as a prompt (see Appendix C and the model returns the ranked document IDs in list.
Based on Mistral-7B 7--Instructt-O.2 (Tiagg et al. 2023 we conduct specialized experiment to evaluate its zero-shot performance in three retrieval settings: dense retrieval, point-wise reranking, and list-wise reranking. strate instruction- followiinnn abilities in retrieval tasks without fine-tuning. This experiment explores Mistral's potential as a zero-shot retrieval model, assessing whether can naturally generate strong embeddings act effective reranker to identify caaructiio-relevan documents from the list of candidates. For the mode of dense retrieval, we use mean pooling to obtain the sentence level representation.
highly capable instruction-tur
model, Mistral expected to demon-
We include GPT-4o as a strong baseline due to its demonstrated instruction custruction-following capabilities and to set high performance benchmark for all models.

---

Published as conference paper at ICLR 2025
3.2 RESULT OVERVIEW
Table provides a detailed comparison of different retrieval models across six dimensions using nDCG@ @ 10, Robustness@e @ 10, p-MRR, WISE, and SICR. The table also includes the average rankings of the golden documents in the instruction mode across the three retrieval models. Notably, almost all models achieved relatively high nDCG, indicating that relying solely on nDCG is insufficient to capture the impact of instructions on ranking changes. Although Robustness can be used for model comparison, it is unable to assess the extent of performance changes before and after instructions because the relevant documents corresponding to the three retrieval modes differ. p-MRR can partially reflect the model's responsiveness to different instructions; however, due to the limitations of this metric, the results are not expressed with sufficient accuracy. For instance, according to p-MRR evaluations, the instruction instruction-following performance of vargeaag---55 and Instructor models is significantly better than that of GPT-4o. Meanwhile, the WISE and SICR scores closely align with the ranking changes of the Gold Document and can clearly distinguish the mnstruction-followinn capabilities of the models as well as the performance differences between them. The results reveal cnttuuttio-following performance across different model categories, which can be summarized as follows: list-wise reranking models >> point-wise reranking models > dense retrieval models sparse retrieval models. Larger model architectures typically outperform smaller models in both WISE and SICR.
distinct patterns
The WISE and SICR scores of the sparse retrieval model BM25 indicate that models relying solely on lexical matching, without sensitivity to instruction-based retrieval or context-aware instructions, struggle to interpret and act on complex instructions. BM25's 's inability to adapt underscores the limitations of traditional sparse retrieval for instruction-following tasks.
In contrast, dense retrieval models show greater sensitivity to instructions, though their performance varies. For instance, BGE-Large-v1.. 1.5 and Instructor-XL demonstrate significant performance degra- dation under instructions, as reflected in their negative WISE scores. However, models like GritLM,, 25-Mistral-ins and NV-Embed-v v2 demonstrate greater adaptability. Notably, GritLM achieves the highest WISE and SICR score among the dense models, indicating that, benefiting from joint training on both encoding and generative tasks, GritL is better equipped to handle complex instructions. In contrast, models primarily trained on task-specific instructions, such as BGF--Lrreevl and Instructor-XL, encounter difficulties when addressing a broader range of instructions.
Point-wise reranking models generally outperform dense retrieval models. Among them, Llama- 3.1 achieves the highest WISE and SICR scores. Although it has not been specifically fine-tuned for retrieval tasks. Llama-3.1 benefits from its extensive understanding of language, granting it certain degree of instruction instruction-folloiingg capabilities. FollowIR also demonstrates competitiveness; by fine-tuning with content-aware instructions, FollowIR achieves comparable scores to Llama-3.1 with fewer model parameters.
Among list-wise reranking models, GPT-40 performs the best, achieving the highest scores in WISE and SICR across all models, demonstrating its exceptional capability handling and adhering to complex instructions. Additionally, RankZephyr shows decent performance but remains closer to pointwise re-ranking models in terms of instruction following, possibly due to limitations in its training data. Although Mistral-ins: in0 has the second-highed SICR score after GPT-4o, its WISE score is not as remarkable, indicating that while the model can comprehend instructions, it struggles to effectively elevate the rankings of the corresponding documents.
3.3 PERFORMANCE ANALYSIS ACROSS DIMENSIONS
The radar plots in Figure 3 offer a visual summary of how different models perform in these dimensions, highlighting their strengths and weaknesses in instruction following. Across all models, both retrieval and reranking models show significant room for improvement. Particularly, certain dimensions such as format and audience consistently present challenges. Performance these remains suboptimal, indicating that models struggle with instructions requiring specific formatting audience adaptation. The difficulty likely arises from insufficient exposure structured data formats such as (StackOverflow Post], [Code Snippet]. or [Offical Manual], and a lack of nuanced understanding of diverse audience contexts during training.

---

Published as conference paper at ICLR 2025
WWAmmbed42 1D-Mistral
SFR-Embedding-2
GPT-4a RankZeshyr
Liam-3311 FollowiR
(ist-wise) Metral (po intwwiee
Um25
E5-Large-22
Language
Length
Language
Length
Format
Source
Format
Source
Keyword
Audience
Keyword
Audience
(a) Retrieval
(b) Reranking
Figure 3: Radar plots comparing the WISE scores of various models across different dimensions, highlighting the strengths and weaknesses of each model in handling different types of instructions. Among retrieval models, GritLM demonstrates the strongest instruction-follwiii capability, while GPT-4 consistently performs the best across dimensions the reranking category.
Retrieval models show notable variability in performance across dimensions. GritLM stands out, leading in overall dnstruciionfolloviin ability. Retrieval models generally perform well the language and keyword dimensions, but they struggle significantly on the format and audience dimensions. This indicates that retrieval models handle text-based instructions effectively but struggle with structural and contextual cues.
Compared retrieval models, reranking models generally perform better across all dimensions. This improvement is particularly evident in the keyword dimension. largely because reranking models, during inference, directly verify keyword presence within the context. GPT-4 stands out in the language, source, and keyword dimensions, consistently outperforming other models. However, even nop-eeforminng models like GPT-4o face challenges with audience-related instructions. Despite the overall performance gap in the Source and Audience dimensions, RankZephyr performs comparably to GPT-4o the length, audience, and keyword dimensions, demonstrating the effectiveness of fine-tuning for reranking tasks.
4 ANALYSIS
p-MRR WISE. While both metrics aim to measure models' instruction-followinn abilities by considering rank changes, p-MRR does not consistently reflect real performance. Many models in this study received p-MRR scores that were inconsistent with the ranking trends of the gold documents; for instance, GPT-4o scored 15.0, which was lower than both Instructor-X XI. and NV-Embed-v2. Eventually, most models scored even lower than BM25. This discrepancy arises because p-MRR evaluates only relative ranking changes (Rina - Rori), disregarding absolute ranking shifts (Rori). In contrast, the proposed WISE metric strictly enforces instruction following standards by accounting for both absolute and relative ranking changes. GPT-4o achieved the highest WISE score, as it was able further elevate the rankings of top golde documents when instructions were added and lower the rankings under semantically inverse instructions (R.ri 3.51, Rins Rori. Rori < Rrew). makes WISE more reliable metric for evaluating instruction- folletttiiooooowwningg capabilities.
Dense retrieval model vs Reranking model. Reranking models(representte by red and gray row Table significantly outperform most dense retrieval models(representee by green row in Table in instruction-followingg tasks due to their ability to evaluate documents in relation one another, optimizing the final ranking based on contextual relevance and nuanced understanding. By considering the entire list of retrieved documents, rerankers effectively adjust the order based on the specific needs of the query, capturing subtle distinctions that dense retrieval models may overlook. This leads more accurate rankings that align with user intent, especially in

---

Published as conference paper at ICLR 2025
Table 2: Performance comparison of different retrieval models across six dimensions using the WISE and SICR metrics. The dimensions are: DI (Audience). D2 (Keyword), D3 (Format), D4 (Language), D5 (Length), and D6 (Source). Higher scores indicate stronger ceastructtin-follwwin capabilities.
WiSe
SICR
Model
D1
D2
D3
D4
D5
D6
Avg.
D1
D2
D3
D4
D5
D6
Avg.
BM25
-3.0
-2.1
-2.8
-7.2
-7.5
1.97
-12.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Dense Retrieval
Bge-Large-v1.5
-16.8
-38.2
-42.1 1
-20.7
-28.6
-30.7
-9..5
0.5
0.0
0.3
2.0
1.0
2.3
1.0
E5-Large-v2
-38.3
-15.5
-25.3
-21.6
23.2
23.3
Instructor-XL
-27.7
-34.7
-30.5
-20.5
-35.5
-29.8
-2..8
5.7
2.1
0.3
4.0
0.3
4.0
2.7
Mistral-ins-v0.22
-35.8
-67.8
-29.7
-31.9
-66.6
-63.3
-4.22
0.0
0.0
0.0
0.0
0.0
0.0
0.0
GTE-Qwen2
-34.0
-36.5
-44.3
-18.0
-56.4
-44.6
-39.0
0.0
0.0
0.1
E5-Mistrul-ing
-7.3
-4.5
-19.9
0.1
-13.4
-13.0
-16.3
2.9
0.0
0.0
10.5
0.0
3.3
2.8
GritLM
-3.4
6.8
-36.0
-6.7
-25.8
-1.5
-11.1
11.4
11.8
1.3
11.7
6.9
FER-Embeddiing2-1
-7.8
-45.9
-13.0
-15.4
-13.5
-13.2
-18.1
2.9
2.1
NV-Embed--2
-9.8
-27.7
-18.1
-7.3
-9.3
-8.7
-13.5
2.4
0.7
1.0
3.5
0.3
9.0
2.8
Point-wise Reranking
Mistral-ins-v0.2
.8.9
34.5
-9.3
21.4
-12.6
-0.3
4.1
1.4
28.6
2.7
6.5
4.7
4.7
8.1
Llama-3.1
-6.2
38.7
29.0
-5.9
40.2
14.4
6.2
38.3
36.7
19.3
FollowIR
-233
47.7
-2.3
20.9
-2.6
18.8
13.4
3.3
27.2
7.0
19.5
1.7
16.3
12.5
List wise Reranking
Mistral-ins-v0.2
-6.3
46.0
-6.6
7.6
-1.9
10.0
8.1
10.5
39.2
9.7
23.0
8.0
21.7
22.0
Zephyr-beta
2.77
14.1
-13.9
-3.2
8.7
RankVicuna-vy
-2.5
-8.5
-9.8
-11.8
-4.3
-2.2
-6.5
5.2
10.5
3.3
4.5
2.3
8.0
5.6
RankZephyr-vl
7.4
53.9
7.8
10.6
7.8
40.3
14.5
4.3
42.5
1.0
5.5
4.3
5.3
10.5
GPT-4o
complex scenarios where the relationship between documents and the query is crucial for effective instruction-followinng Consequently, while dense retrieval models excel in efficiently retrieving relevant documents, reranking models provide the precision necessary to enhance the overall ranking quality, resulting in superior performance in tasks requiring sophisticated language comprehension.
Point-wise reranking vs. List-wise reranking. Point-wise reranking evaluates each document independently, predicting relevance without considering other documents. In contrast, list-wise reranking considers the entire set, optimizing the overall ranking order by leveraging relative relation- ships between documents. As shown in Table list-wise ranking (gray row) generally outperforms point-wise ranking (red row) for instruction-followii context and the relative importance of documents. This makes list-wise reranking more effective in organizing documents to align with complex instructions, improving relevance and coherence across different queries.
tasks, as it better captures the broader query
Zephyr vs. RankZephyr. RankZephyr outperforms Zephyr in both WISE and SICR because of its more sophisticated training process, better robustness to initial document order, multiple reranking passes that help correct ranking errors. Compared Zephyr, RankZephyr learns from RankGPT, which allows it to adopt more sophisticated ranking strategies. Besides, RankZephyr benefits from multiple passes, allowing to adjust the ranking more effectively compared to a single-pas passeeepssssss strategy like Zephyr's, which might not optimize the ranking as thoroughly. These factors combine to ensure that RankZephyr minimizes penalties for ranking important documents too low, leading significantly better WISE and SICR scores.
Mistral (retrieval) vs. (point-wise) ys (list-wise). Mistral-ins-v0.2 shows poor retrieval performance, highlighting its limitations in handling complex, instruction-driven ranking scenarios. Without specific training for retrieval, it fails to rank documents effectively on nuanced instructions. On the other hand, Mistral-ins-vol ins-v0.2 in point-wise ranking demonstrates improved performance in both WISE and SICR, scores individual documents independently, allowing to better adhere to instructions, though it lacks the depth to consider relationships between documents. However, Mistral-ins-ve in list-wise ranking truly excels, as it optimizes the entire list and takes document interactions into account, enabling to handle more sophisticated instruction-follwwee significantly better WISE and SICR scores, making list-wise Mistral-ins--O.22 the most effective approach for instruction-follen following tasks where ranking coherence is critical.
tasks. This results in

---

Published as conference paper at ICLR 2025
5 RELATED WORK
Dense Retrieval The development of dense retrieval models has significantly enhanced the seman- tic understanding and efficiency of retrieval systems. Existing dense models can be categorized into two types based on their architecture: Bidirectional Embedding Models and Decoder-only Embedding Models. Bidirectional Embedding Models are typically base on BERT (Devlin 2018 or T5 (Raffel
et al. 2020) encoders, performing general embedding tasks. Early models that base on BERT or T5 for efficient text embeddings include Sentence BERT (Reimers] 2019), SimCSE (Gao et al. 2021) and Sentence T5 (Ni et al., 2021). To better accommodate the requiremente of text embeddings, researchers have pre-trained these encoders using contrastive learning (Izacard et al. 2021 Wang
et al. 2022). Furthermore, these models are fine-tuned using various supervised datasets to enhance
their performance in retrieval tasks or other downstream applications (Lee et al. 2024b] Li & Li 2023). Compared to bidirectional embedding models, decoder-only embedding models initially perform relatively poorly in general embedding tasks, primarily due to their limited capacity to com- prehensively capture and utilize contextual information (Brown) 2020). However, many researchers have sought to optimize these models' performance by introducing contrastive learning methods to address their deficiencies in embedding tasks (Neelakantan et al. 2022). Currently, researchers have explored not only the use of synthetic data (Wang et al. 20023) but also a hybrid strategy combining real and synthetic data (Meng et al. 2024, [BehnamGhader et al.) 2024), achieving significant success in text embedding tasks. Collectively, these advancements in contrastive pre-training, model scaling, and leveraging weak supervision and synthetic data significantly propel the field of retrieval.
CnatructionFFollviin for Retrieval The notion of relevance often varies among users (Mizzaro 1998). Consequently, queries alone may not fully address all users' information needs (Ruthven &
Lalmas 2003), while instructions can expand these intentions beyond the scope of the queries. Recent information retrieval research has recognized this and tried train retrieval models by combining instructions with queries to enhance their instruction-folooiin
capabilities. In general, existing catruction-following models can be divided into two categories based instruction design methods: task-aware and vontent-awars instruction retrieval models. TART first proposed a general retrieval system with task-level instruction, setting specific instructions for different retrieval tasks to query corresponding results (Asai et al. 2023). Subsequently, Instructor expanded the scope of instructions so that text embeddings can not only retrieve but also classify and diagnose duplicate problems (Su et al. 2023). However, these task-aware instructions are too general and lack the specificity of user instructions real scenarios. On this basis, other researchers have developed content-level instructions. InstructIR set instructions to adapt to query-text pairs based on user background (such as work, hobbies) (Oh et al. 2024). ExcluIR set exclusionary instructions based on the content differences between query results, accounting for users' exclusionary needs in queries (Zhang et al.) 2024). FollowIR IR set instructions to distinguish query results by combining exclusion and incl lusion the ability of retrievers to recognize and respond to different perspectives in queries (Zhao et al., 2024).. However, real user intentions involve both internal (content, audience, language) and external (format, length) answer attributes. MAIR proposed scale instruction retrieval benchmark covering 126 different information retrieval tasks, but lacks an explicit evaluation of the model's 's instruction following ability (Sun et al., 2024).
(Weller et al. 2024a). PIR focuses
6 CONCLUSION
Despite leveraging LMS as the backbone for training retrieval models, most existing IR models cannot truly understand the instructions in query. Further, traditional score indicators(e.g,, nDCG) cannot reflect whether the model has the ability to follow instructions and most existing dataset with instructions are designed with a only single dimension, so we propose InfoSearch and two novel meIS,, SICR). The choice of dimensions in our dataset takes into account the instructions that users may give in actual scenarios. Additionally, our metrics consider the combined performance of the model in three Original mode, Instructed mode, Reversely instructed mode), with increasing difficulty across modes as each introduces more complex challenges. We hope this work helps the future instruction-folloiin retrieval task.

---

Published as conference paper at ICLR 2025
ACKNOWLEDGEMENT
We thank EIT and IDT High Performance Computing Center for providing computational resources for this project. This work is supported by 2035 Key Research and Development Program of Ningho City under Grant 20242122
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. 2023.
Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. Task-aware retrieval with instructions. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 3650-3675, 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge,
Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 2023.
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Ma- jumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv: 1ete::1111...,... 2016.
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. LIm2vec: Large language models are secretly powerful text encoders. arXiv preprint arXiv:2404. 05961, 2024.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, al. Gpt-neox- An open-source autoregressive language model. arXiv preprint arXiv:2204.06744 2022.
Tom B Brown. Language models are few-shot learners. arXiv preprini arXiv:2005. 14165, 2020.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 2018.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.0882) 2021.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:21 2021.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv-embed: Improved techniques for training lims as generalist embedding models. arXiv preprint arXiv:2405. /7428, 2024a.
Sean Lee, Aamir Shakir. Darius Koenig, and Julius Lipp. Open source strikes bread new fluffy embeddings model, 2024b. URL https: /www wtwp://wxxxeeeee www.mmlxedrreac.ai//..... mixbai-embed--argee
Xianming Li and Jing Li. Angle-optimized text embeddings. arXiv preprint arXiv:2309.12871, 2023.
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.08. 03281, 2023.
Renze Lou, Kai Zhang, Jian Xie, Yuxuan Sun, Janice Ahn, Hanzi Xu, Yu Su, and Wenpeng Yin. Muffin: Curating multi-facee faceted instructions for improving instruction following. In The Twelfth International Conference Learning Representations, 2023.

---

Published as conference paper at ICLR 2025
Rui Meng. Ye Liu, Shafiq Rayhan Joty, Caiming Xiong. Yingbo Zhou, and Semih Yavuz. Sfr- embedding-2: Advanced text embedding with stati-stage training, 2024. URL https://
Stefano Mizzaro. How many relevances information retrieval? Interucting with computers, 10(3):
303-320, 1998.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316, 2022.
Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning, arXiv preprint arXiv:2402.09006, 2024.
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, et al. Text and code embeddings by contrastive pre-training. arXiv preprint (rxXiv:221..00000 2022.
Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith B Hall, Daniel Cer, and Yinfei Yang. Sentence-15: Scalable sentence encoders from pre-traines text-to-text models. arXiv preprint arXiv:2/08.08871 2021.
Hanseok Oh, Hyunji Lee, Seonghyeon Ye, Haebin Shin, Hansol Jang, Changwook Jun, and Minjoon Seo. Instructir: A benchmark for instruction following of information retrieval models. arXiv preprint orXiv:2402.143:3 2024.
Letian Peng, Yuwei Zhang, Zilong Wang, Jayanth Srinivasa, Gaowen Liu, Zihan Wang, and Jingbo Shang. Answer is all you need: Instruction-followee text embedding via answering the question. arXiv preprini arXiv:2402.09644 2024.
Ronak Pradeep. Sahel Sharifymoghaddam, and Jimmy Lin. Rankvicuna: Zero-shot listwise document reranking with open-source large language models. arXiv preprint arXiv:2309. 15088, 2023a.
Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. Rankzephyr: Effective and robust zero-shot listwise reranking a breeze! arXiv preprint arXiv:2312.02724, 2023b.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
N Reimers. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 10084, 2019.
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends@ in Information Retrieval, 3(4):333 -389,, 2009.
Ian Ruthven and Mounia Lalmas. A survey on the use of relevance feedback for information access systems. The Knowledge Engineering Review, 18(2):95-145 2003.
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A Smith, Luke Zettlemoyer, and Tao Yu. One embedder, any task: iestruction-finctumne text embeddings. In Findings of the Association for Computational Linguistics: ACL 2023, pp.
1102-1121, 2023.
Weiwei Sun, Zhengliang Shi, Jiulong Wu, Lingyong Yan, Xinyu Ma, Yiding Liu, Min Cao, Dawei Yin, and Zhaochun Ren. Mair: A massive benchmark for evaluating instructed retrieval. arXiv preprint arXiv:2410.1011 2024.
Robert S Taylor. The process of asking questions. American documentation, 13(4):391-39, 1962.
Gemini Team, Rohan Anil. Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk. Andrew M Dai, Anja Hauth, et al. Gemini: a family highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.

---

Published as conference paper at ICLR 2025
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310.16944 2023.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervin contrastive pre-training. arXiv preprint arXiv:2212.03353, 2022.
Liang Wang, Nan Yang. Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00366. 2023.
Orion Weller, Benjamin Chang, Sean MacAvaney, Kyle Lo, Arman Cohan, Benjamin Van Durme, Dawn Lawrie, and Luca Soldaini. Followir: Evaluating and teaching information retrieval models to follow instructions. arXiv preprint arXiv:2403. /5246, 2024a.
Orion Weller, Benjamin Van Durme, Dawn Lawrie, Ashwin Paranjape, Yuhao Zhang, and Jack Hessel. Promptriever: cnntuctioo-traineed retrievers can be prompted like language models. arXiv preprint arXiv:2409.11 11166, 2024b.
Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian- Yun Nie. C-pack: Packaged resources to advance general chinese embedding. arXiv preprint arXiv:230..07597 2023.
Wenhan Xiong, Xiang Lorraine Li, Srini Iyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Wen-tau Yih, Sebastian Riedel, Douwe Kiela, et al. Answering complex open-domain questions with multi-hop dense retrieval. arXiv preprint arXiv:2009. 12756, 2020.
Wenhao Zhang, Mengqi Zhang, Shiguang Wu, Jiahuan Pei, Zhaochun Ren, Maarten de Rijke, Zhumin Chen, and Pengjie Ren. Excluir: Exclusionary neural information retrieval. arXiv preprins arXiv:2404. 17288, 2024.
Xinran Zhao, Tong Chen, Sihao Chen, Hongming Zhang, and Tongshuang Wu. Beyond relevance: Evaluate and improve retrievers on perspective awareness. arXiv preprint 40Xiv22405.02714, 2024.

---

Published as conference paper at ICLR 2025
A MORE DETAILS OF INFOSEARCH
A.1 CONDITION DETERMINATION
The query for each dimension can be diversified and expanded through various conditions, allowing the same query to correspond to different relevant documents under different conditions. Except for the Keyword dimension, the other five dimensions have fixed conditions. The condition of the Keyword dimension is the keyword in the document that is relevant to the query. Therefore, the condition of the Keyword dimension needs be determined after filtering out the Query-Documan (Q-D) pairs. To achieve this, both the query and its corresponding document were input into GPT-4, generating a unique condition for each Q-D pair. However, GPT-4 occasionally selected irrelevant words, such as "and" or "what", that did not align with the user scenario when generating keyword conditions. To address this, we meticulously crafted prompt templates (Table 3 for condition extraction, ensuring that the conditions were both unique and representative of each document, accurately reflecting the document's s core theme.
Table 3: template that generates the specific conditions required for the keyword dimension
Prompt for Condition Generation
### TASK ###
Your task is to generate a condition that refines a given query in relation to a provided document. The condition should be:
Relevant to the document's core topic must align with the central theme or key content of the document.
2. A meaningful constraint on the query It should introduce specific aspect, subtopic, or perspective that naturally extends the original query while still being directly supported by the document.
3. Not generic arbitrary restriction The condition must be logically derived from the document's s content and should not be trivial or overly broad constraint.
### INPUT ###
You will receive a query and a document input:
Query: {query}
Document: {document}
### FORMATTING ###
Condition: the condition your generated >
A.2 DATA COLLECTION
To ensure the queries realistic and reduce the human cost, we consciously integrate conditions when filtering Q-A pairs from existing datasets or web pages. For instance, in the Format dimension, due to the lack of available multi-format Q&A datasets, we selectively extract Q-D pairs from StackOverflow posts. For posts containing code and detailed official documentation responses, we use their titles as queries and the complete responses as documents under the [StackOverflow] condition. The pure code snippets within the answers and references to official documentation are separately extracted and used as documents under the [Code Snippet] and [Manual] conditions, respectively. Table shows the source of datasets used to collect query-documen pairs for each dimension.
A.3 INSTRUCTION GENERATION
The generation of accurate, concise, and natural instructions is crucial. When searching, users tend express their intentions using simple, naural language, the generated instructions must remain

---

Published as conference paper at ICLR 2025
Table 4: Structure and source of the dataset
Dimension
Source Data
Condition Value
Audience
BioASQ: seifact Muennighoft al.
2022
[Layman], [Expert]
Keyword
MSMARCO Bajaj et al. 2016
[keyword]
Format
Stackoverflow. various office doc
[Stackoverflow Post]. [Code Snippet], [Official Manual]
Language
publichealth-qu
[Chinese], [English]
Length
medical.gg MMuennighof' et al. 2022 google search
[Sentence], [Paragraph], [Article]
Source
CNN-englisb-news, google scarch
[Blog]. [Forum Postj. [News Article]
brief and clear, closely mirroring conversational style. To achieve this, we employed words such as "smooth", "natural", and "realistic" in the prompts (see Table [ to guide GPT-4 in crafting instructions that emphasize not only semantic accuracy but also the emulation of authentic user expressions. Furthermore, two-sentence structure for the instructions, first rephrasing the query and then appending specific constraints. This structure effectively separates the core query from the conditions, enhancing the diversity of generated instructions. For example, "What are the most effective exercises for losing weight? Please find discussions from forum posts only." This two-sentence structure ensures logical clarity and semantic coherence.
Table 5: A prompt template for Generating Instruction
Prompt for Instruction Generation
### TASK ###
You tasked with generating natural query with an instruction based the query and the condition provided by the user. You will be provided with query and condition and you need to:
1. Rephrase the core query as the first sentence, making it sound like natural human query without changing its meaning.
2. Create second sentence that specifies the search restriction.
Ensure each sentence is smooth, concise, reasonable, natural, and realistic, mimicking a real human tone.
### INPUT ###
Core Query: {core query)
Condition: {condition}
### FORMATTING ###
Core query: <<the core query give you > Condition: <hhe condition I give you
Query with Instruction: <the query with instruction you generated
A.4 DOCUMENT REWRITING
When the query and relevant documents fail meet the instruction requirements, we use GPT-4 to rewrite the existing documents to generate relevant documents. The documents that need to be modified are mainly concentrated in the source, length and audience dimensions, so we set specific prompts for these dimensions respectively (see Table 66. In this process, we experimented documents from the query, but these often exhibited redundant expressions and inconsistent formatting. Therefore, we adjusted the existing documents, ensuring they meet the instruction requirements while preserving naturalness and authenticity in the language.
with directly generating condition-satisyyl

---

Published as conference paper at ICLR 2025
Table 6: Prompt templates for document rewriting
Dimension
Prompt Template
Source
### TASK ####
For a core query, need documents from a blog, forum post, or news article. I will provide you with query, the corresponding document from news article. Your task is to rewrite the document as blog and forum post content.
###
CAUTION ###
For the blog you generated, you cannot use the core query as blog title directly, You need to rephrase it, but do change the semantics of this query. Besides, you need to give various information in the line under the title, such as the author, when was published, the word "Blog", and the section it belongs All the above information must be random.
For forum post, must be a form of discussion among multiple users. The usernames need to be random rather than use "usel", "use2" etc.
###
INPUT ###
Core Query: {core query)
Document: fdocument)
### FORMATTING ####
Your output should be in the following format:
Blog: <the blog you generated >
Forum: <the forum post you generated >
Audience
### TASK ####
will provide you with a core query and its corresponding document. The target audience for this document is experts. Your task is to Rewrite this document to make it easily understandable for laymen.
### CAUTION ###
1. Keep the semantics of the document intact.
2. Do not any technical jargon in the rewritten document for layman.
### INPUT ###
Query: {query)
Document for expert: {expert}
### FORMATTING ###
Query: <the query give you >
Layman: <the rewritten document for layman you generated
Length
### TASK ###
will provide you with a core query and its corresponding paragraph. Your task is rewrite this paragraph into single sentence and an article.
### CAUTION ###
1. Ensure that both the sentence and article retain the original meaning of the paragraph.
### INPUT ###
Core Query: {core query}
Paragraph: {paragraph)
### FORMATTING ####
Your output should be in the following format:
Sentence: <the rewritten single sentence that answers the query >
Article: <the multi halti-par rewritten document that answers the query>

---

Published as conference paper at ICLR 2025
A.5 INSTRUCTION REVERSAL
In real retrieval, we observed that results already ranked highly tend to remain at the top even after instructions applied. This makes it difficult determine whether the improvement in ranking is due to the model's understanding of the instructions or simply a result of detailed keyword and semantic matching. To address this, we validate the model's comprehension of the instructions by reversing the semantic meaning of the instructions. For example, "Please answer in Chinese" is reversed "Please do not answer in Chinese."
Table 7: A prompt Template for Instruction Reversing
Prompt for Instruction Reversing
### TASK ###
Your expertise lies in interpreting and transforming direct instructions into their opposite or negative forms while maintaining clarity and coherence in the transformed instructions. Your task is to reverse the instruction I give you.
### CAUTION ###
While reversing instruction, ensure that the new instruction conveys the opposite meaning accurately. Please keep in mind that the transformation should remain clear and easy to understand, avoiding any ambiguity.
### INPUT ###
Instruction: {instruction}
### FORMATTING ###
Reverse Instruction: <the instruction your reverse >
A.6 HARD NEGATIVE GENERATIONS
While positive documents for the same query under varying conditions may act as negative examples for one another (instruction negatives), we still need to prevent the model from relying solely on prominent features for simple retrieval, thereby neglecting the subtle relationships between the query and the documents. To address this, we use GPT-4 to generate documents that appear to be related to the query topic the surface but cannot actually answer the query, serving as hard negative documents (query negatives).

---

Published as conference paper at ICLR 2025
Table 8: prompt templates for generating hard negative
Prompt for Hard Negative Generation
### TASK ###
You are tasked with generating hard negative document based on given query. A hard negative document should appear superficially relevant to the query but contain critical inaccuracies, misleading details, subtle contradictions. Follow these steps:
Understand the core intent of the query and identify key entities, relationships, or requirements.
2. Generate a document that incorporates some keywords from the query but does not provide direct or indirect answer the query. The document should maintain plausible structure and stay on related topic while ensuring that no information within it can be used to infer construct correct response the query.
3. Ensure the document is coherent, natural, and realistic, mimicking genuine but incorrect response.
### INPUT ###
Core Query: {core query]
### FORMATTING ###
Core query: <the core query give you > Hard negative document:
<generated document
A.7 MANUAL REVIEW
We filtered out anomalous documents from the outputs 12 retrieval models, selecting those that failed to rank within the top 50 in six or more models or had relevance score helow 0.5 for the query, followed by manual screening. This process aimed to eliminate mislabeled Q-D pairs selected from other datasets or documents inaccurately retrieved through manual search. For these mismatched documents, we proceed with manual replacement. After multiple rounds screening to ensure the quality of the InfoSearch dataset, the statistical results are summarized in Table 9
Table 9: InfoSearch dataset statistics. IQ, /7| and | represent the word lengths of the original query, instructed query and reversely instructed query respectively.
Dimension
Q
Avg [Q]
I
Avg [|
# R
Avg
R
# D
Audience
100
9.02
210
20.46
210
15.91
840
Keyword
100
6.30
288
17.90
288
18.92
1152
Format
100
9.16
300
16.65
300
19.31
1200
Language
100
8.75
200
14.09
200
15.74
800
Length
100
8.52
300
15.94
300
16.26
1200
Source
100
7.38
300
18.19
300
15.58
1200
Total
600
1598
1598
6392

---

Published as conference paper at ICLR 2025
To make the data more intuitive, Table t0 Table 13 the InfoSearch dataset.
provide specific examples from each dimension
Table 10: An example Audience dimension
Core Query
How to Prevent Heart Disease
Instructed 1
Explore effective strategies for preventing heart disease. Please explain in terms that are easy for the general public to understand.
Instructed 2
Investigate the latest preventive measures against heart disease. Make detailed discussion suitable for professional audience.
Reversed 1
How to Prevent Heart Disease. I'm looking for response that is more technical than layman.
Reversed 2
How Prevent Heart Disease. Please keep your answer simple and clear.
Document
To prevent heart disease, consider the following strategies
Adopt Vegan Diet: Vegan diets, particularly those rich in soy and other plant- based proteins, can reduce the risk of cardiovascular disease. These proteins are high in non-essential amino acids, which promote glucagon activity. Glucagon helps regulate lipid levels and cholesterol synthesis, leading to healthier heart conditions.
Increase Glucagon Activity:
Document
Vegan proteins may reduce risk of cancer, obesity, and cardiovascular disease by promoting increased glucagon activity. glucagon promotes (and insulin inhibits) AAMP-deppender mechanisms that down-regulate lipogenic enzymes and cholesterol synthesis, while up-regulating hepatic LDL receptors and production of the IGF-I antagonist IGFBP-1, The insulin-sensitzzing properties of many vegan diets-high in fiber, low in saturated fat

---

Published as conference paper at ICLR 2025
Table 11: An example Keyword dimension
Core Query
What helps for acne?
Instructed
What treatments are effective for acne? Ensure your answer includes information specifically about "progesterone".
Instructed 2
Can you tell me what helps reduce acne symptoms? Focus on the effects of "mint" in your response.
Instructed 3
What natural remedies are beneficial for managing acne? Please include details about "Chamomile".
Reversed
What helps for acne? Can you provide a response that does involve the term "progesterone"?
Reversed
What helps for acne? Can you give me reply that does not entail the use of the term "mint"?
Reversed 3
What helps for acne? Can you provide a response avoiding the term "Chamomile"?
Document
Progesterone helps with acne that occurs in the late 30's and early 40's. Also, if the acne varies with the period, climination of xenoestrogens (environmental estrogens) and phytoestrogens and taking progesterone cream helps with this type of acne well.
Document 2
Acne home remedy: Mint. Mint can help remove pore-clogging oil. To help clear acne before it begins, mix tablespoons of finely chopped fresh mint with two tablespoons each of plain yogurt and oatmeal (use blender pulverize the oatmeal to powder). Leave the concoction on your face for 10 minutes, then rinse off with water.
Document
Acne home remedy: Chamomile, Chamomile helps decrease inflammation from acne. In blender or coffee grinder, combine the contents of chamomile tea bag with enough water to form a paste, and apply that to acne. Alternately, steep two chamomile tea bags with cup boiled water for 15 minutes.

---

Published as conference paper at ICLR 2025
Table 12: An example in Format dimension
Core Query
How can I access environment variables in Python?
Instructed I
How can I access environment variables in Python? Limit the search to Stackover- flow posts.
Instructed 2
How can I access environment variables in Python? I need code snippets to solve the problem.
Instructed 3
How can access environment variables in Python? Only consider official manuals.
Reversed
How can I access environment variables in Python? Provide me with an answer that is not Stackoverflow post..
Reversed
How can access environment variables in Python? Could you deliver response that isn't It in the form of code snippet?
Reversed
How can access environment variables Python? I'm seeking a reply that isn't official manual.
Document
Environment variables accessed through ['os.environ']
"pyyhonn import os
print(os.environ| HHME'J)
To see list of all environment variables:
"python
print(os.environ)
If key is not present, attempting to access it will raise 'KeyError'. To avoid this:
"python
# Returns 'None" if the key doesn't 't exist
print(os.environ.geg KEY-THAT MIGHT EXISTT)
Document
" "python import os
print(os.environ| HHOMEJ])
Document
os. ** **evvrro***** **
A [mapping] object where keys and values strings that represent the process environment. For example, 'environ[" "oviron("HOOE'J is the pathname of your home directory (on some platforms), and is equivalent to 'getenv("HOMEE") in C This mapping is captured the first time the ['os"] module is imported, typically during Python startup as part of processing 'site.py'. Changes the environment made after this time are not reflected in ['os.environ") ] except for changes made by modifying "'os.environn ] directly.
On Windows, the keys are converted to uppercase. This also applies when getting, setting, deleting an item. For example, 'environ[ "environ["monty"] "python" maps the key "MNTTY" to the value "python"".

---

Published as conference paper at ICLR 2025
Table 13: An example in Language dimension
Core Query
What is diabetes?
Instructed I
Tell me what diabetes Please use Chinese.
Instructed 2
Tell me the answer to what is Siabetes.Please use English.
Reversed
What is diabetes? Please respond in a language other than Chinese.
Reversed 2
What is diabetes? I'd rather have a response in language other than English.
Document
# (R diabetes mellitus, 9 DM diaabeees - 1E
* サークッサド 3
Hai
- 2
| $22 1 T
-
K -/ $ $ #
7 E IN 11
1 + 1 MR BRRMIE
Document 2
Diabetes is chronic disease that occurs either when the pancreas does not produce enough insulin or when the body cannot effectively use the insulin it produces.
Table 14: An example in Length dimension
Core Query
How many calories are martini?
Instructed
How many calories are martini? Please give me sentence answer. What's the calorie count of martini? I'd like paragraph explaining it. Can you tell me the calories in a martini? Please provide detailed article.
Instructed 2
Instructed 3
Reversed 1
How many calories are in a martini. Please provide a detailed response, not just single sentence.
Reversed 2
How many calories are in martini. Please avoid giving me paragraph as your response.?
Reversed 3
How many calories are in martini. don't 't structure your answer as an article.
Document Document
2.25 oz (67 mL) Martini (extra dry): 140 calories.
The amount of Calories in a martini cocktail can vary based on how you make it. A martini cocktail technically only has two ingredients, vodka and vermouth, so Calorie count depends on your proportions. GREY GOOSE@ Vodka contains 66 Calories per 30 ml serving* Try mixing up our Classic Dry Vodka Martini Cocktail recipe.
Document
Vodka Martini Calories
Depending on the size of your cocktail, and the extras you mix in, one serving of vodka martini is approximately 202 calories. Vodka martini calories can be much higher if the drink has more than the two basic liquors. To figure out the calories in vodka teaspoon of French vermouth has approxi- mately 7.8 calories...

---

Published as conference paper at ICLR 2025
Table 15:
An example in Source dimension
Core Query
Effective exercises for weight loss
Instructed I
What's the best way to do exercises for weight loss effectively? Please provide blog post on this topic. How can I perform exercises effectively for weight loss? I'd like forum post on this subject. Tell me how to do effective exercises for weight loss. Give me something from News Articles.
Instructed 2
Instructed 3
Reversed |
Effective exercises for weight lleseelease provide a response that is not from a blog-
Reversed 2
Effective exercises for weight loss.I'm looking for an answer that's not based on a forum thread.
Reversed 3
Effective exercises for weight loss,Please avoid using a news article as your source..
Document
What Are the Best Exercises for Weight Loss? May 6, 2024 Blog
Losing weight can be challenging journey, but incorporating exercise into your routine can make a significant difference. Not only does exercise help you burn calories, but it also boosts your metabolism, improves your mood and increases your overall health and well-being.
But with many different types of exercises there, it can be overwhelming figure out which ones are the best for weight loss.
How Exercise for Weight Loss Walking exercise for weight loss Walking low-impact exercise that is perfect for beginners... superMario _ilt:
Document 2
myself enjoy going on long walks (anywhere from 30 minutes hours). It's easy on the joints, can listen to music or stick to my thoughts, and you get fresh air away from being cooped up in a gym. It definitely as helped me trim up some over time.
Individual Ad 2701:
do 1-2 hours of lifting day hate cardio well After lifting do how much should walk after I lift like would 20-30 minutes work I'm gaining muscle and can see that my arms and chest bigger but my belly getting bigger also did try eating less calories but idk. Proudscobi:
If you are going to choose one for weight loss, go for weight lifting. It will improve your body composition. Even if you don't lose weight you will look better. NBC HEALTH
Document 3
workouts may be better for weight loss, study finds. People who got their exercise in between a.m. and 9 a.m. had lower BMIs than those who opted to exercise later in the day. Is morning the best time of day to exercise? Research published Tuesday in the journal Obesity finds that early morning activity between a.m. and 9 a.m. could help with weight loss. "My cautious suggestion from this study is that if we choose to exercise the early morning, before eat, can

---

Published as conference paper at ICLR 2025
Moreover, Figure 4 illustrates the distribution of data across six dimensions in the FollowIR, Instruc- UR, and InfoSearch datasets. The chart highlights the varying proportions of query- eeery-document pairs based on dimensions like Audience, Keyword, Language. Length, Source, and Format.
A udience
Keyword
Language
Length
Source
Format
Other
FollowIR
InstructIR
InFoSearch
Figure 4: Comparison of the InfoSearch dataset with FollowIR and InstructIR in terms of data distribution across six dimensions.
B EVALUATION METRICS ANALYSIS
To measure instruction-folloviie performance for retrieval models is challenge. Two metrics were specifically proposed previous studies for this purpose: Robustness@K (Oh et al. 2024) and p-MRR (Weller et al., 2024a). We argue that neither of them effectively reflects true instruction- following performance.
Robustness@K, @k is designed to assess model's performance the same query under different instructions. Specifically, it groups instances the same query, calculates the minimum nDCG @k score within each group, and averages the group scores generate the overall Robustness @k k score. Let Q = {91, 42, }n} be set of queries. For each query there are m, distinct instruction variants {1, L2 ..., Lim, calculate the minimum nDCG@k score across all its instruction:
min-nDCG((g) = min Sij
jE(1.2.......)
where Sij represents the nDCG@k score for query qi under instruction 4új. Compute the overall Robustness@ score as average of these minimum scores across all queries:
Robustness@l:
min-nDCG((g)
(
However, the Robustness @k metric oversimplifies the evaluation of model's ability follow instructions. ( Even a model demonstrates strong performance across the majority of queries, single anomalously low score reduce the overall robustness score. 2 Furthermore, focusing solely on the lowest score disregards variations in the model's responses to different instructions, thus failing to capture the overall performance trend.
As for p-MRR, it is based on the MRR metric and quantifies the model's ability to follow instructions by comparing the rankings of relevant documents in the original mode and the instruction mode. The following formula is applied to calculate the score for each relevant document within query:
M MR
if Rog > Ruew otherwise,
M RRR
MRRR
MRR MRR.g
'For instance, the nDCG@t @ scores for group A are {0.8. 0.5. 0.3, 0.2), while those for group B are (0.9, 0.9, 0.9, 0.2). Although group B exhibits significantly better overall performance, Rohustness @ k assigns the same both groups

---

Published as conference paper at ICLR 2025
where MRR is mean reciprocal rank, Rog is the rank of the document in the original retrieval mode, and Rnew is the new rank the instruction mode. However, 3 p-MRR fails to distinguish the importance of ranking, neglecting highlight the critical role that top K results play in retrieval. 4 Moreover, the linear discount mechanism of p-MRR insufficiently sensitive to changes in higher rankings, making it ineffective in capturing subtle movements the top. ( Lastly, p-MRR demonstrates limitations when addressing special cases and extreme performances.
C PROMPT FOR LIST- LIST-WISE RERANKING MODELS
Table 16: Prompt for List-wise Reranking Models. The input consists of a list of documents or passages, and the model is prompted to return ranked list of document IDs based their relevance to the query.
TASK
Prompt Template
Rank

> >>
You are RankGPT, intelligent assistant that ranks passages based their relevance to query.
< <<user|>
will provide you with [num) passages, each indicated by a number identifier [ Rank the passages based their relevance the query: {query).
[1] passage 1
[2] passage 2}
[num] (passage [num}}
Search Query: {query).
Rank the {num} passages above based their relevance the search query. The passages should be listed in descending order using identifiers. The most relevant passages should be listed first. The output format should be >[] e.g. [1] > Only respond with the ranking results, do not say any word or explain.
<lassistant] >
Model Generation: [9] [4] > 120] >... >[1]]
FFor example, the performance of model 1 is Rog 10 and Knew = 5, yielding p-MRR of -0.5, while model 2's performance is Rog 100 and Rnew 50, resulting in a p-MRR of -0.5. Although both models receive the score, is evident that model has greater impact the retrieval results.

---

Published as conference paper at ICLR 2025
SAMPLING OF WISE SCORE REWARD COMPONENT D
02
61
91
51
11
000
000
080-0
0000
5000
EWR
L2V0
13V0
00V0
6000
1900
0800
0000
000
0000
0000
001
035
1900
005
0050
0600
1050
890
1290
9091
9890
1950
4399
22990
0000
0V1
855-0
200-1
0000
SP0
2P0
000O
1670
980
010
235
1000
5
000
010-0
50000
019-
S10
LIPO
500
022
0500
SRR
0000
600
190
P90
LOED
585
X00
003
532'
1PE0
LPED
195
6350
XM0 1
0000
000 1
010
2/5'0
1250
52E0
92E0
0950
6550
0000
000 11
SKC'
10E
10ED
SIED
115
0000
000 11
S/2/0
03C0
8620
0000
105
OEE
000
3920
0900
1420
1850
1826
000
000
552
650
5520
1120
5120
0020
5326
0000
03
SVCO
USCO
9520
OSED
952
9120
0000
00
9950
050O
550
550
6900
0000
11
0013
S020
USEO
9P29
150
0000
11
001
6250
1300
OFEO
STED
000
C22
0000
00000
20
0
222
SCC-0
0000
11
000
612-
PCC
81
000'
000
11
00110 O
0000
1000
R 12
Figure 5: Heatmap of the rewards component
THE COMPLETE RESULTS OF EVALUATING WITH INFOSEARCH DATASET
Table Table
Table 21 and Table 2 show all the results of the 15 retrieval models in InfoSearch dataset. Ori indicates models evaluate in Original mode. Ins indicates models evaluate Instructed mode. Rev indicates models evaluate in Reversely instructed mode. Act. indicates the actual performance of the model and ideal indicates the ideal performance. Per. indicates
18 Table 19 Table 20

---

Published as conference paper at ICLR 2025
how far the actual performance is from the ideal performance a proportion of the ideal performance. lower percentage indicates that the actual performance closer to the ideal performance, while higher percentage indicates a greater deviation from the ideal performance. The calculation formula Per. ideal actual
ideal
Table 17: Audience Results
Audience-(Layman, Expert)
nDCG@ 10
MRR@ @
WISE
Model
SICR
Or
Ch
Re
Or
Ch
Re
Act.
Idcal
Per.j
BM25
46.1
38.7
36.4
21.0
11.9
13.3
-3.0
65.9
104.6
0.0
Dense Retrieval
Bge-Large-v1.5
48.6
38.1
37.6
22.9
12.9
11.9
-16.8
67.5
124.9
0.5
E5- E5-Large-v2
53.9
45.3
42.6
32.4
16.7
16.2
-15.6
71.7
121.7
1.4
Instructor-XI.
48.3
30.1
31.2
29.5
8.6
10.0
-27.7
64.6
142.9
5.7
Mistral-ins-v0.2.
31.1
35.6
37.5
20.0
17.1
17.1
-35.8
40.6
188.3
0.0
E5-Mistral-ins
78.9
63.3
64.3
72.4
34.8
35.2
-7.3
86.1
108.5
2.9
GritLM
56.2
56.7
57.1
41.9
31.4
30.0
-3.4
70.2
104.9
11.4
GTE-Qwent
56.4
57.0
57.3
46.7
35.2
35.2
-34.0
65.3
152.0
0.0
SFR-Fmmeedding-211
63.2
51.6
52.0
41.9
24.8
22.9
-7.8
79.2
109.9
2.9
NV-Embed-v2
65.3
47.6
47.5
44.8
17.6
17.1
-9.8
80.5
112.2
2.4
Point-wise Reranking
Mistral-ins-v0.2
75.8
60.9
63.6
62.9
28.1
35.2
-8.9
85.0
110.4
1.4
Llama-3.1
79.9
65.1
67.4
68.6
36.7
41.4
-6.2
88.4
107.0
6.2
FollowIR
76.9
64.9
63.6
69.5
35.7
35.2
-2.3
85.7
102.6
3.3
List-wisc Reranking
Mistral-ins-v0.2
68.7
58.9
58.6
51.4
29.0
28.6
-6.3
81.0
107.8
10.5
Zephyr-beta
77.0
62.1
62.6
71.4
35.2
37.6
-2.7
84.8
103.2
1.0
Rank Vicuna-vl
62.2
52.5
51.2
50.5
27.1
25.7
-2.5
75.1
103.3
5.2
RankZephyr-vl
71.0
58.9
59.2
56.2
31.0
30.0
7.4
82.6
91.1
4.3
GPT-4o
87.7
72.5
72.6
88.6
48.6
48.6
7.4
95.9
92.2
15.2

---

Published as conference paper at ICLR 2025
Table 18: Keyword Results
Keywords-(Inclut (keywords))
nDCG@ 10
MRR
WISE
Model
SICR
Or
Ch
Re
Or
Ch
Re
Act.
Ideal
Per.
BM25
70.4
70.0
54.1
64.5
45.3
24.7
-42.1
77.7
154.2
0.0
Dense Retrieval
Bge-Large-v1.5
46.2
39.7
29.4
25.8
11.8
11.5
-38.2
66.2
157.8
0.0
E5-Large-v2
60.1
70.6
45.0
43.2
46.7
16.0
-38.3
75.6
150.7
0.7
Instructor-XL
68.5
48.7
38.4
56.8
19.9
14.6
-34.7
79.4
143.7
2.1
Mistral-ins-v0.2
31.7
28.5
37.0
30.7
7.3
19.9
-67.8
36.0
288.4
0.0
E5-Mistral-ins
72.3
79.5
71.8
60.6
57.1
33.1
-4..5
80.7
155.2
0.0
GritLM
85.9
79.4
67.2
89.2
58.2
46.0
6.8
86.6
92.2
11.8
GTF-Qwen2
58.9
43.5
49.3
58.2
18.1
32.8
-36.5
64.9
156.2
0.0
SFR-Embeddding-2 2-R
47.3
64.5
47.7
30.7
38.3
24.4
-5..9
65.4
170.3
1.0
NV-Embed-v2
61.5
61.4
40.2
49.8
34.8
17.1
-27.7
74.7
137.1
0.7
Point-wise Reranking
Mistral-ins-v0.2
39.9
63.6
38.0
16.7
40.1
16.0
34.5
62.4
44.7
28.6
Llama-3.1
61.7
76.9
48.4
48.4
54.0
28.2
38.7
74.2
47.8
38.3
FollowIR
51.2
78.1
45.7
34.1
59.9
25.8
47.7
68.7
30.6
27.2
List-wise Reranking
Mistral-ins-v0..
67.4
79.3
43.8
59.2
64.1
27.2
46.0
76.8
40.1
59.2
Zephyr-beta
68.9
65.2
47.8
59.9
47.0
32.8
14.1
77.0
81.7
27.5
Vicunacun
66.8
75.6
51.9
65.2
57.8
32.1
-8.5
75.7
111.2
10.5
RankZephyr-vi
72.6
77.3
52.0
79.1
60.6
34.5
53,9
92.3
41.6
42.5
GPT-4o
71.8
78.8
61.9
66.2
70.7
51.6
63.0
86.0
26.7
60.6

---

Published as conference paper at ICLR 2025
Table 19: Format Results
Format- (StckooeSta Post, Code Snippet, Official Manual
nDCG@10
MRR@ @1
WISE
Model
SICR
Act.
Ideal
Per.
BM25
22.6
15.3
19.1
16.0
4.7
8.3
-2.8
30.7
109.0
0.0
Dense Retrieval
Bge-Large-vl. 1.5
58.0
25.9
31.4
46.0
5.3
11.3
-2.1
65.8
163.9
0.3
E5-Large-v2
59.3
44.9
52.0
44.0
20.3
36.7
-15.5
68.9
122.5
0.0
Instructor-XL.
64.2
35.7
40.6
54.0
11.7
20.3
-30.5
72.5
142.1
0.3
Mistral-ins-v0.2
2.4
3.2
4.3
0.0
1.0
1.3
-29.7
5.6
630.7
0.0
E5-Mistral-ins
72.2
46.7
58.9
72.0
17.7
37.0
19.9
75.9
126.3
0.0
GrilLM
45.5
48.4
53.8
31.0
21.3
38.7
-36.0
54.6
165.9
1.3
GTE-Qwen2
14.3
14.6
19.0
13.0
5.3
12.3
-44.3
18.2
343.6
0.0
SFR-Embeedding-24
75.4
53.0
63.1
76.0
23.7
46.3
-13.0
78.5
116.5
2.0
NV-Embed-v2
67.5
41.5
53.1
59.0
15.3
30.7
-18.1
73.4
124.7
1.0
Point-wise Reranking
Mistral-ins-v0.2
62.2
50.8
61.7
43.0
21.3
35.7
-9.3
74.2
112.6
2.7
Llama-3.1
68.0
51.2
59.5
58.0
18.3
32.7
-9.5
77.3
112.3
10.0
FollowIR
72.2
54.9
68.5
62.0
23.3
50.0
-2.3
79.7
102.9
7.0
List-wise Reranking
Mistral-ins-v0.2
69.1
50.8
58.3
69.0
24.3
42.3
-6.6
74.6
108.8
9.7
Zephyr-beta
48.8
35.2
42.3
51.0
15.3
36.7
-13.9
58.3
123.8
8.0
Rank Vicuna-v1
44.4
33.3
41.3
32.0
12.7
25.3
-9.8
56.7
117.2
3.3
RankZephyr-vl
73.0
52.6
63.5
61.0
20.3
41.7
7.8
81.2
90.4
1.0
GPT-4o
78.7
59.4
67.7
84.0
32.0
52.3
21.9
94.3
76.8
11.3

---

Published as conference paper at ICLR 2025
Table 20: Language Results
Language- (Chinese-(Chinese, English)
nDCG @ 10
MRR @
WISE
Model
SICR
Or
Ch
Re
Or
Ch
Re
Act.
Ideal
Per.
BM25
36.1
30.3
28.8
28.0
14.0
13.0
-7.2
43.7
116.5
0.0
Dense Retrieval
Bge-Large-v1.5
42.7
36.1
32.0
38.0
19.0
13.5
-20.7
51.5
140.1
2.0
E5-Large-v2
52.4
50.2
44.6
58.0
35.5
33.5
-25.3
56.7
144.7
0.5
Instructor-XL
47.6
37.7
35.0
54.0
25.0
19.5
-20.5
50.0
141.1
4.0
Mistral-ins-v0.2
20.7
30.0
29.5
19.0
21.0
19.0
-31.9
26.5
220.6
0.0
E5-Mistral-ins
81.5
73.4
62.4
80.0
48.5
40.0
0.1
87.4
99.9
10.5
GritLM
82.6
81.0
75.9
78.0
57.0
48.0
-6.7
87.7
107.7
4.5
GTF-Qwen2
38.5
36.8
37.9
43.0
27.0
28.5
-18.0
41.1
143.7
0.0
SFR-Embeddiin-22 2-R
81.5
81.7
64.3
77.0
61.0
31.5
-15.4
88.1
117.4
1.0
NV-Embed-v2
68.3
67.4
59.6
72.0
39.0
33.5
-7.3
76.8
109.5
3.5
Point-wise Reranking
Mistral-ins-v0.2
58.9
63.3
60.8
32.0
38.5
32.0
21.4
76.6
72.0
6.5
Llama-3.1
67.9
71.9
64.2
61.0
44.5
36.5
29.0
79.4
63.5
22.0
FollowIR
68.4
70.6
64.7
54.0
48.0
37.0
20.9
81.2
74.3
19.5
List-wise Reranking
Mistral-ins-v0.2
73.7
70.7
63.5
69.0
49.5
38.5
7.6
81.9
90.7
23.0
Zephyr-beta
70.9
58.7
58.1
68.0
38.5
37.5
-6.9
79.3
108.7
10.5
Vicunaaun
63.4
55.5
53.2
54.0
29.5
29.5
-1 -11.8
76.3
115.5
4.5
RankZephyr-vi
79.5
66.6
66.9
69.5
38.5
37.5
10.6
88.0
87.9
5.5
GPT-4o
83.2
86.2
82.2
83.0
75.5
65.5
53.1
91.3
41.8
55.5

---

Published as conference paper at ICLR 2025
Table 21: Length Results
Length-(Sentence, Paragraph, Article)
nDCG@ 10
MRR@ @1
WISE
Model
SICR
Or
Ch
Re
Or
Ch
Re
Act.
Ideal
Per.
BM25
63.3
43.4
54.1
64.0
19.3
40.7
-7.5
71.5
110.4
0.0
Dense Retrieval
Bge-Large-v1.5
62.7
35.1
42.0
46.0
9.0
15.7
-28.6
74.3
138.4
4.7
E5-Large-v2
73.9
52.6
63.0
66.0
26.3
48.0
-21.6
80.3
126.9
2.7
Instructor-XL
75.7
38.1
47.3
74.0
12.3
25.3
-35.5
81.3
143.7
1.7
Mistral-ins-v0.2
11.8
22.6
27.6
13.0
11.7
25.0
-66.6
13.7
586.2
0.0
E5-Mistral-ins
86.2
64.2
76.2
92.0
33.0
60.3
-13.4
86.0
115.6
0.0
GritLM
74.0
65.4
76.6
76.0
36.0
61.7
-25.8
79.0
132.6
0.3
GTF-Qwen2
34.4
47.3
55.0
32.0
22.3
42.7
-56.4
40.7
238.6
0.3
SFR-Embedding-22 2-RR
75.7
59.1
70.7
70.0
27.7
53.3
-13.5
81.7
116.5
1.0
NV-Embed-v2
81.9
55.3
65.6
83.0
26.7
48.0
-9.3
84.6
111.0
0.3
Point-wise Reranking
Mistral-ins-v0.2
60.3
52.3
60.2
42.0
24.0
38.0
-12.6
74.4
116.9
4.7
Llama-3.1
84.4
64.1
73.0
86.0
36.7
56.0
-5.9
87.6
106.8
2.7
FollowIR
80.3
61.9
71.5
80.0
35.0
55.7
-2.6
84.6
103.0
1.7
List-wise Reranking
Mistral-ins-vO.3
88.7
66.0
76.8
92.0
38.7
64.7
-1.9
89.1
102.2
8.0
Zephyr-beta
85.8
61.5
74.6
93.0
31.7
63.0
-5.7
86.5
106.6
2.0
Vicunavul
84.8
61.8
74.1
93.0
32.7
60.7
-4.3
85.4
105.0
2.3
RankZephyr-vi
86.1
63.5
75.7
90.0
34.0
59.7
7.8
88.2
91.1
4.3
GPT-4o
89.1
68.8
79.0
95.0
42.0
67.3
10.2
94.4
89.2
10.3

---

Published as conference paper at ICLR 2025
Table 22: Source Results
Source-(Blog, Forum Post, News Article)
nDCG@10
MRR@
WISE
Model
SICR
Act.
Ideal
Per.
BM25
45.8
37.0
38.3
25.0
15.7
15.7
-9.4
63.4
114.9
0.0
Dense Retrieval
Bge-Large-vl.. 1.5
59.1
34.6
36.8
49.0
12.3
16.7
-30.7
71.4
143.0
2.3
E5-Large-v2
61.1
48.6
52.1
49.0
22.7
29.0
-23.2
72.6
132.0
1.3
Instructor-XL.
70.5
40.0
43.7
66.0
13.0
15.3
-29.8
77.7
138.3
4.0
Mistral-ins-v0.2
19.9
32.9
39.2
18.0
12.0
24.0
-63.3
23.9
364.7
0.0
E5-Mistral-ins
76.2
58.9
62.4
70.0
30.7
36.0
-13.0
82.5
115.8
3.3
GrilLM
80.3
66.3
67.5
76.0
40.7
46.0
-1.5
84.4
101.8
11.7
GTE-Qwen2
58.2
59.2
72.4
55.0
29.0
53.0
44.66
63.6
170.1
0.0
BFR--Embedding-24
78.9
63.1
62.6
74.0
37.0
35.3
-13.2
84.0
115.7
4.7
NV-Embed-v2
71.3
53.8
47.5
67.0
29.7
23.3
-8.7
78.5
111.1
9.0
Point-wise Reranking
Mistral-ins-v0.2
71.5
59.7
69.7
56.0
24.3
48.0
-0.3
81.0
100.4
4.7
Llama-3.1
84.9
71.4
79.9
91.0
46.0
75.7
40.2
84.8
52.6
36.7
FollowIR
81.9
67.4
79.3
74.0
35.7
65.0
18.8
86.1
78.2
16.3
List-wise Reranking
Mistral-ins-v0.2
77.7
60.8
68.6
78.0
34.3
57.0
10.0
81.0
87.6
21.7
Zephyr-beta
70.4
52.9
62.8
65.0
23.7
44.0
-3.9
78.2
105.0
3.0
Rank Vicuna-v1
68.7
52.7
59.5
74.0
30.3
50.0
-2.2
72.6
103.0
8.0
RankZephyr-vl
82.8
61.8
71.0
85.0
33.7
56.0
-0.3
83.5
100.4
5.3
GPT-4o
89.4
79.5
82.0
93.0
65.7
77.3
45.2
95.5
52.7
39.7


Reviews:
[{"review_id": "SpAg6pkr29", "reviewer": "Reviewer_L8BM", "summary": "This paper proposes a new dataset and evaluation metrics to evaluate instruction-based information retrieval. Their dataset, InfoSearch proposes 6 different subsets of evaluation: Audience, Keyword, Format, Language, Length, and Source. \n\nThey build the dataset by collecting query and document pairs from existing IR datasets as well as Google. They then have GPT-4 extract keywords from the documents, generate an instruction via some specific style, and then re-write the document as needed. They also prompt GPT-4 to generate instruction reversals, which are the opposite. They then manually review them. \n\nThey propose two new metrics to evaluate instruction following (WISe and SICR) and show results on a wide range of models. They show that larger models do better, rerankers do better, list-wise models do better, and instruction-trained models do better. They show that in some dimensions, such as audience, are still difficult for retrievers.", "questions": "### Re: Weakness 1, dataset construction\n\na. How were the number of instructions decided/created?\n\nb. How was the number of document in the corpus decided? \n\nc. It seems like all documents are merged together from all subsets to create the corpus, how are we sure there are not false negatives in the other sets? A qualitative analysis or GPT-based analysis would be very valuable here.\n\nd. How were these documents gathered, just any that were from the original pairs (but then where did the remainder come from?)\n\n### Re: Weakness 2, metrics\n\nA core motivation from the paper of WISe is:\n> Intuitively, if the ranking of Pi improves in the instructed mode (Rins < Rori) and worsens in the reversely instructed mode\n(Rori < Rrev), this demonstrates the model’s effective compliance with the provided instructions\n\na: Why is this true? I don't necessarily understand why the query is more relevant with the instruction, weren't these gathered from already relevant query and doc pairs? So I see no specific reason why the rank should increase with the instruction as compared to no-instruction.\n\nb: Arbitrary values: why does WISe not go to 0? Why use 20 as the number for the conditional?\n\nc: Doesn't WISe/SICR have the same problem as p-MRR in terms of rank of 1 for the original and instructed query?\n\nd: Working through some examples for WISe, that I found unusual:\n- If the rank is 1 for both, the score is 0 correct? (From Eq 1)\n- From r_{ori} of 20 to r_{ins} of 1, we have a score of 1.0. If you change r_{ins} to 2, now the score is 0.07? Perhaps I did the math wrong, but this seems like a pretty large gap\n- From r_{ori} of 20 to r_{ins} of 2 we have 0.07 but from r_{ori} of 21 to r_{ins} of 1, we have 0.01 (an even lower score, even though arguably the model improved more)\n- Overall, the N=20 distinction seems to introduce some weird disfluencies in the possible range of scores for this metric. Even the overall range seems somewhat odd, you can get scores from -1 to 1, but oddly distributed.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "I think this paper has a lot of strengths:\n- A needed dataset for instruction-based IR, covering a wide range of topics, including some models still struggle on\n- A wide set of models they evaluated, including larger models like GPT-4\n- Interesting analysis from the metrics and various comparisons", "weaknesses": "Although I lean positively, there are a few weakness/area to improve\n\n1. I am still confused slightly on the dataset creation. For example, what decided how many instructions there were per query? In Table 1 I see somewhere between 2-3 per query.  The instructions are also fairly short (20 words, which is almost standard query-length), but this is not a dealbreaker.  See the questions for these.\n2. The two new introduced metrics are somewhat difficult to understand and I am not sure I fully agree with them. See the questions, from where I worked through some examples and see some ways of assigning scores that seem odd to me.\n3. Although I wouldn't have expected the authors to have done this since the deadline was so soon after the model's releases, there are some models that claim to do better on this, including ICL-based models (BGE-ICL, RARe) and zero-shot models (Promptriever). It would be interesting to see their performance.\n\nOverall, I do like the paper and would be open to increasing my score if some of these weaknesses/questions are addressed (mostly metrics based).\n\nUpdate after response: I have increased my soundness score and most of my concerns have been addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aMWe0W8cac", "reviewer": "Reviewer_r4CY", "summary": "The paper, “Beyond Content Relevance: Evaluating Instruction Following in Retrieval Models,” introduces **InfoSearch**, a benchmark that assesses how well retrieval models follow user instructions beyond simple content relevance. Traditional retrieval models have limitations in understanding and executing detailed user instructions, often focusing on keyword matching rather than instruction adherence. InfoSearch spans six key dimensions—Audience, Keyword, Format, Language, Length, and Source—representing user-specific document attributes. The study evaluates 16 retrieval models using this benchmark and introduces two novel metrics, **Weighted Instruction Sensitivity evaluation (WISe)** and **Strict Instruction Compliance Ratio (SICR)**, designed to capture the depth of instruction-following capabilities. The results reveal that current retrieval models, even those fine-tuned, often struggle with adhering to specific instructions, providing insights for advancing instruction-aware retrieval in future research.", "questions": "- What is the computational feasibility of using WISe and SICR metrics in large-scale production environments?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The writing is easy to follow and the logic is clear.\n- InfoSearch covers six essential dimensions, moving beyond traditional content relevance. This multidimensional approach provides a holistic evaluation of models, accounting for varied user-specific needs that reflect real-world scenarios.\n- The introduction of WISe and SICR metrics adds significant value by capturing nuanced differences in instruction-following. These metrics are specifically designed to evaluate instruction adherence, overcoming limitations in traditional metrics like nDCG or pMRR.", "weaknesses": "- The benchmark’s focus on instruction-following may overlook other critical aspects of retrieval quality, such as semantic understanding and contextual relevance. Overemphasizing instruction compliance may risk sidelining these equally important factors.\n- The InfoSearch benchmark relies heavily on synthetic data generation, primarily using LLMs like GPT-4 to create instruction-query pairs and rephrased documents. This reliance may introduce artificial biases in the data that do not accurately reflect real-world queries or instruction styles, potentially limiting the benchmark’s validity for evaluating real-world retrieval tasks.\n- While the benchmark tests models on reversely instructed queries (negative instructions), the analysis does not deeply explore how models interpret and handle negation or exclusion conditions. Understanding these nuances would be valuable for practical applications where users often specify what not to include in results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nZwfcoBanj", "reviewer": "Reviewer_RSj2", "summary": "This paper introduce InfoSearch, a novel benchmark aims to evaluate retriever models' instruction following capabilities. The authors find that existing metrics like nDCG, Robustness@k, and pMRR are not sufficient for evaluating instruction following capabilities. To more rigorously evaluate this capability, this paper introduce 2 WISe and SICR. This paper conduct extensive analysis regarding different ranking method.", "questions": "1. One minor issue but it's prevalent throughout the paper: consider using ~\\citep rather than \\cite for in-text citation, e.g., L145\n\n2. L161, where's your table 2.2? \n\n3. L198, $\\textbf{P}^i$ is unclear, what $i$ refers to? \n\n4. L203, 2 periods \n\n5. eq(2), why you select 20 and 0.01, how you decide 2K? Treat it as normalization factor? \n\n6. L310-L311, for reranking model, can you explain more how it works? \n\n7. It might be super helpful if you can include your benchmark example in the main text. \n\n8. Do you consider using general-purpose LLM for doing point-wise reranking? Like llama-3.2, mixtral, qwen2, etc.?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper bridge the gap that no benchmarks are evaluating retriever models' instruction following capabilities \n2. This paper introduces 2 metrics for rigorously evaluating instruction following capabilities\n3. Extensive experimentations \n4. Interesting interpretation and analysis of the main result\n5. reverse mode is interesting!", "weaknesses": "1. Sec 2.3 Evaluation Metric, the motivation for designing WISe and SICR is not clear. In L181 - L183, weakness of Robustness@k is not clearly explained. \n\n2. Sec 2.3 Evaluation Metric, since you are comparing with your metrics with pMRR and Robustness@k, it's good to have a comparison table showing your metric is better in handling cases that pMRR and Robustness@k fail, in order to convince your audience that your metrics are effective. Additionally, it would be helpful if you could compare and explain the difference between WISe and SICR. \n\n^ 2 follow-up, I saw Table 2 comparing all metrics, I think it might be good to have a table listing all strengths/weakness at the first place.\n\n3. In eq (1), $F(q) = f_{\\textit{penalty}}(R_{ori}, R_{ins})$ if $\\lnot R_{ins} < R_{ori} < R_{rev}$; whereas, eq (3) only gives $R_{ori} < R_{ins}$. More clarifications will be helpful.\n\n4. One of my major concern is $\\textbf{you do not explain anything about the dataset construction pipeline}$, which I will treat the core of this paper. For example, you lack details \n\n    4.1 How you select web pages?\n\n    4.2 Prompt for step 2? No reference at all. How can you check the quality?\n\n    4.3 Quality check for all GPT-generated responses. There's no details. If they are in the appendix please reference them in the main text. \n\n    4.4 Only saying $\\textbf{manual review}$ is NOT acceptable. Biographies of annotators? Inter-annotator agreement score?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oWkFPA3R6R", "reviewer": "Reviewer_S77t", "summary": "In this paper, the authors evaluate retrieval models' instruction-following capabilities and develop the InfoSearch benchmark covering several dimensions of documents such as audience, keyword, format, language, length, and source. They also introduce new metrics to assess the models’ responsiveness to instructions.\n\nPros:\n- The proposed dataset is somewhat useful to the research community.\n- The exploration to document attributes for instruction-aware retrieval is interesting.\n- The experimental study is insightful.\n\nCons:\n- It would be great if the authors could introduce more on the motivation and application scenario for instruction following retrieval. Based on my understanding, if the instruction following functionality over document-level features can be implemented by introducing LLMs for understanding the query intent then rewriting the query with customized filters.  There is no need for an advanced retrieval model.\nAgain, some examples might help reviewers understand the problem of instruction following retrieval and confirm whether this is a true research problem.\n- Focusing only on the document-level instructions is not enough for evaluating instruction-followed retrieval systems.  Even for document-level instructions, some additional dimensions such as temporal and location, should also be considered.\n- For the experiments, I would expect to add some customized filters (such as language requirement) on existing retrieval models.\n- It is unfair to compare retrieval models with reranking models.  I think the most challenging part is the retrieval part, which may fail to retrieve relevant results without good instruction understanding.", "questions": "- What is the underlying retrieval model for the reranking models in Table 3? (How the initial results are prepared?)", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Pros:\n- The proposed dataset is somewhat useful to the research community.\n- The exploration to document attributes for instruction-aware retrieval is interesting.\n- The experimental study is insightful.", "weaknesses": "Cons:\n- It would be great if the authors could introduce more on the motivation and application scenario for instruction following retrieval. Based on my understanding, if the instruction following functionality over document-level features can be implemented by introducing LLMs for understanding the query intent then rewriting the query with customized filters.  There is no need for an advanced retrieval model.\nAgain, some examples might help reviewers understand the problem of instruction following retrieval and confirm whether this is a true research problem.\n- Focusing only on the document-level instructions is not enough for evaluating instruction-followed retrieval systems.  Even for document-level instructions, some additional dimensions such as temporal and location, should also be considered.\n- For the experiments, I would expect to add some customized filters (such as language requirement) on existing retrieval models.\n- It is unfair to compare retrieval models with reranking models.  I think the most challenging part is the retrieval part, which may fail to retrieve relevant results without good instruction understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wqfzFEuzgB", "reviewer": "Reviewer_7Ty8", "summary": "This paper introduces a new dataset for evaluating instructable retrievers. The dataset is based on existing IR data and utilizes GPT-4 to generate instructions across six dimensions. Additionally, it rewrites the original documents to align with the new instructions and includes reverse instructions. The paper proposes two new metrics to evaluate models’ sensitivity to instructions: (i) WISe, which rewards search results that comply with instructions and penalizes those that do not; and (ii) SICR, which considers both the relevance score predicted by the model and the ranking information. The paper evaluates different types of retrieval models, finding that the LLM-based listwise reranker performs the best, followed by the pointwise reranker and dense retriever.", "questions": "SICR uses relevance scores, but listwise models do not output relevance scores. How do you define the score for these models?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Compared to previous data, the new data considers more dimensions, providing better coverage for evaluation.\n\n- This paper demonstrates the weaknesses of previous metrics for evaluating instruction following, such as Robustness@k and pMRR, and introduces two well-motivated alternative metrics.\n\n- This paper conducts an extensive evaluation of different retrieval methods, with insightful results and analysis.", "weaknesses": "- The proposed data is incremental compared to existing data. For example, InstructIR includes the dimension of Audience, and this paper essentially reuses their data generation pipeline, adding more dimensions.\n\n- Similar to InstructIR, both the instruction and the positive examples are generated by GPT-4. Although the authors mention they manually reviewed the quality, I am still concerned about the data’s reliability: the positive document is rewritten by GPT-4 to align with the instructions. However, GPT-4 itself serves as a baseline model with only moderate performance in determining whether a document follows an instruction.\n\n- The proposed metric assumes there are n positive documents under the original mode, with only one remaining positive under the instruction mode. However, this assumption does not always hold in real-world tasks, which limits this metric’s applicability to their dataset alone. For instance, we might have multiple positives under the instruction mode or find that some positives are not strictly positive under the default mode.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)