Beyond Content Relevance: Evaluating Instruction Following in Retrieval Models

Authors: Jianqun Zhou, Yuanlei Zheng, Wei Chen, Qianqian Zheng, Shang Zeyuan, Wei Zhang, Rui Meng, Xiaoyu Shen
Primary area: datasets and benchmarks
Keywords: LLM, Instruction-Following, Retrieval Model, Benchmark

Abstract:
Instruction-following capabilities in large language models (LLMs) have progressed significantly, enabling more complex user interactions through detailed prompts. However, retrieval systems have not matched these advances, most of them still relies on traditional lexical and semantic matching techniques that fail to fully capture user intent. Recent efforts have introduced instruction-aware retrieval models, but these primarily focus on intrinsic content relevance, which neglects the importance of customized preferences for broader document-level attributes. This study evaluates the instruction-following capabilities of various retrieval models beyond content relevance, including LLM-based dense retrieval and reranking models. We develop InfoSearch, a novel retrieval evaluation benchmark spanning six document-level attributes: Audience, Keyword, Format, Language, Length, and Source, and introduce novel metrics -- Strict Instruction Compliance Ratio (SICR) and Weighted Instruction Sensitivity Evaluation (WISE) to accurately assess the models' responsiveness to  instructions. Our findings indicate that although fine-tuning models on instruction-aware retrieval datasets and increasing model size enhance performance, most models still fall short of instruction compliance. We release our dataset and code on https://github.com/EIT-NLP/InfoSearch.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 5, 3

Reviews and discussion:
<thread>
creation date:
1730599091016

thread_type:
official_review

<thread_object>
creator: reviewer_7ty8

<content>
summary:
This paper introduces a new dataset for evaluating instructable retrievers. The dataset is based on existing IR data and utilizes GPT-4 to generate instructions across six dimensions. Additionally, it rewrites the original documents to align with the new instructions and includes reverse instructions. The paper proposes two new metrics to evaluate models’ sensitivity to instructions: (i) WISe, which rewards search results that comply with instructions and penalizes those that do not; and (ii) SICR, which considers both the relevance score predicted by the model and the ranking information. The paper evaluates different types of retrieval models, finding that the LLM-based listwise reranker performs the best, followed by the pointwise reranker and dense retriever.

strengths:
- Compared to previous data, the new data considers more dimensions, providing better coverage for evaluation.

- This paper demonstrates the weaknesses of previous metrics for evaluating instruction following, such as Robustness@k and pMRR, and introduces two well-motivated alternative metrics.

- This paper conducts an extensive evaluation of different retrieval methods, with insightful results and analysis.

weaknesses:
- The proposed data is incremental compared to existing data. For example, InstructIR includes the dimension of Audience, and this paper essentially reuses their data generation pipeline, adding more dimensions.

- Similar to InstructIR, both the instruction and the positive examples are generated by GPT-4. Although the authors mention they manually reviewed the quality, I am still concerned about the data’s reliability: the positive document is rewritten by GPT-4 to align with the instructions. However, GPT-4 itself serves as a baseline model with only moderate performance in determining whether a document follows an instruction.

- The proposed metric assumes there are n positive documents under the original mode, with only one remaining positive under the instruction mode. However, this assumption does not always hold in real-world tasks, which limits this metric’s applicability to their dataset alone. For instance, we might have multiple positives under the instruction mode or find that some positives are not strictly positive under the default mode.

questions:
SICR uses relevance scores, but listwise models do not output relevance scores. How do you define the score for these models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730766631773

thread_type:
official_review

<thread_object>
creator: reviewer_s77t

<content>
summary:
In this paper, the authors evaluate retrieval models' instruction-following capabilities and develop the InfoSearch benchmark covering several dimensions of documents such as audience, keyword, format, language, length, and source. They also introduce new metrics to assess the models’ responsiveness to instructions.

Pros:
- The proposed dataset is somewhat useful to the research community.
- The exploration to document attributes for instruction-aware retrieval is interesting.
- The experimental study is insightful.

Cons:
- It would be great if the authors could introduce more on the motivation and application scenario for instruction following retrieval. Based on my understanding, if the instruction following functionality over document-level features can be implemented by introducing LLMs for understanding the query intent then rewriting the query with customized filters.  There is no need for an advanced retrieval model.
Again, some examples might help reviewers understand the problem of instruction following retrieval and confirm whether this is a true research problem.
- Focusing only on the document-level instructions is not enough for evaluating instruction-followed retrieval systems.  Even for document-level instructions, some additional dimensions such as temporal and location, should also be considered.
- For the experiments, I would expect to add some customized filters (such as language requirement) on existing retrieval models.
- It is unfair to compare retrieval models with reranking models.  I think the most challenging part is the retrieval part, which may fail to retrieve relevant results without good instruction understanding.

strengths:
Pros:
- The proposed dataset is somewhat useful to the research community.
- The exploration to document attributes for instruction-aware retrieval is interesting.
- The experimental study is insightful.

weaknesses:
Cons:
- It would be great if the authors could introduce more on the motivation and application scenario for instruction following retrieval. Based on my understanding, if the instruction following functionality over document-level features can be implemented by introducing LLMs for understanding the query intent then rewriting the query with customized filters.  There is no need for an advanced retrieval model.
Again, some examples might help reviewers understand the problem of instruction following retrieval and confirm whether this is a true research problem.
- Focusing only on the document-level instructions is not enough for evaluating instruction-followed retrieval systems.  Even for document-level instructions, some additional dimensions such as temporal and location, should also be considered.
- For the experiments, I would expect to add some customized filters (such as language requirement) on existing retrieval models.
- It is unfair to compare retrieval models with reranking models.  I think the most challenging part is the retrieval part, which may fail to retrieve relevant results without good instruction understanding.

questions:
- What is the underlying retrieval model for the reranking models in Table 3? (How the initial results are prepared?)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730843880411

thread_type:
official_review

<thread_object>
creator: reviewer_rsj2

<content>
summary:
This paper introduce InfoSearch, a novel benchmark aims to evaluate retriever models' instruction following capabilities. The authors find that existing metrics like nDCG, Robustness@k, and pMRR are not sufficient for evaluating instruction following capabilities. To more rigorously evaluate this capability, this paper introduce 2 WISe and SICR. This paper conduct extensive analysis regarding different ranking method.

strengths:
1. This paper bridge the gap that no benchmarks are evaluating retriever models' instruction following capabilities 
2. This paper introduces 2 metrics for rigorously evaluating instruction following capabilities
3. Extensive experimentations 
4. Interesting interpretation and analysis of the main result
5. reverse mode is interesting!

weaknesses:
1. Sec 2.3 Evaluation Metric, the motivation for designing WISe and SICR is not clear. In L181 - L183, weakness of Robustness@k is not clearly explained. 

2. Sec 2.3 Evaluation Metric, since you are comparing with your metrics with pMRR and Robustness@k, it's good to have a comparison table showing your metric is better in handling cases that pMRR and Robustness@k fail, in order to convince your audience that your metrics are effective. Additionally, it would be helpful if you could compare and explain the difference between WISe and SICR. 

^ 2 follow-up, I saw Table 2 comparing all metrics, I think it might be good to have a table listing all strengths/weakness at the first place.

3. In eq (1), $F(q) = f_{\textit{penalty}}(R_{ori}, R_{ins})$ if $\lnot R_{ins} < R_{ori} < R_{rev}$; whereas, eq (3) only gives $R_{ori} < R_{ins}$. More clarifications will be helpful.

4. One of my major concern is $\textbf{you do not explain anything about the dataset construction pipeline}$, which I will treat the core of this paper. For example, you lack details 

    4.1 How you select web pages?

    4.2 Prompt for step 2? No reference at all. How can you check the quality?

    4.3 Quality check for all GPT-generated responses. There's no details. If they are in the appendix please reference them in the main text. 

    4.4 Only saying $\textbf{manual review}$ is NOT acceptable. Biographies of annotators? Inter-annotator agreement score?

questions:
1. One minor issue but it's prevalent throughout the paper: consider using ~\citep rather than \cite for in-text citation, e.g., L145

2. L161, where's your table 2.2? 

3. L198, $\textbf{P}^i$ is unclear, what $i$ refers to? 

4. L203, 2 periods 

5. eq(2), why you select 20 and 0.01, how you decide 2K? Treat it as normalization factor? 

6. L310-L311, for reranking model, can you explain more how it works? 

7. It might be super helpful if you can include your benchmark example in the main text. 

8. Do you consider using general-purpose LLM for doing point-wise reranking? Like llama-3.2, mixtral, qwen2, etc.?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730994323222

thread_type:
official_review

<thread_object>
creator: reviewer_r4cy

<content>
summary:
The paper, “Beyond Content Relevance: Evaluating Instruction Following in Retrieval Models,” introduces **InfoSearch**, a benchmark that assesses how well retrieval models follow user instructions beyond simple content relevance. Traditional retrieval models have limitations in understanding and executing detailed user instructions, often focusing on keyword matching rather than instruction adherence. InfoSearch spans six key dimensions—Audience, Keyword, Format, Language, Length, and Source—representing user-specific document attributes. The study evaluates 16 retrieval models using this benchmark and introduces two novel metrics, **Weighted Instruction Sensitivity evaluation (WISe)** and **Strict Instruction Compliance Ratio (SICR)**, designed to capture the depth of instruction-following capabilities. The results reveal that current retrieval models, even those fine-tuned, often struggle with adhering to specific instructions, providing insights for advancing instruction-aware retrieval in future research.

strengths:
- The writing is easy to follow and the logic is clear.
- InfoSearch covers six essential dimensions, moving beyond traditional content relevance. This multidimensional approach provides a holistic evaluation of models, accounting for varied user-specific needs that reflect real-world scenarios.
- The introduction of WISe and SICR metrics adds significant value by capturing nuanced differences in instruction-following. These metrics are specifically designed to evaluate instruction adherence, overcoming limitations in traditional metrics like nDCG or pMRR.

weaknesses:
- The benchmark’s focus on instruction-following may overlook other critical aspects of retrieval quality, such as semantic understanding and contextual relevance. Overemphasizing instruction compliance may risk sidelining these equally important factors.
- The InfoSearch benchmark relies heavily on synthetic data generation, primarily using LLMs like GPT-4 to create instruction-query pairs and rephrased documents. This reliance may introduce artificial biases in the data that do not accurately reflect real-world queries or instruction styles, potentially limiting the benchmark’s validity for evaluating real-world retrieval tasks.
- While the benchmark tests models on reversely instructed queries (negative instructions), the analysis does not deeply explore how models interpret and handle negation or exclusion conditions. Understanding these nuances would be valuable for practical applications where users often specify what not to include in results.

questions:
- What is the computational feasibility of using WISe and SICR metrics in large-scale production environments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731093646557

thread_type:
official_review

<thread_object>
creator: reviewer_l8bm

<content>
summary:
This paper proposes a new dataset and evaluation metrics to evaluate instruction-based information retrieval. Their dataset, InfoSearch proposes 6 different subsets of evaluation: Audience, Keyword, Format, Language, Length, and Source. 

They build the dataset by collecting query and document pairs from existing IR datasets as well as Google. They then have GPT-4 extract keywords from the documents, generate an instruction via some specific style, and then re-write the document as needed. They also prompt GPT-4 to generate instruction reversals, which are the opposite. They then manually review them. 

They propose two new metrics to evaluate instruction following (WISe and SICR) and show results on a wide range of models. They show that larger models do better, rerankers do better, list-wise models do better, and instruction-trained models do better. They show that in some dimensions, such as audience, are still difficult for retrievers.

strengths:
I think this paper has a lot of strengths:
- A needed dataset for instruction-based IR, covering a wide range of topics, including some models still struggle on
- A wide set of models they evaluated, including larger models like GPT-4
- Interesting analysis from the metrics and various comparisons

weaknesses:
Although I lean positively, there are a few weakness/area to improve

1. I am still confused slightly on the dataset creation. For example, what decided how many instructions there were per query? In Table 1 I see somewhere between 2-3 per query.  The instructions are also fairly short (20 words, which is almost standard query-length), but this is not a dealbreaker.  See the questions for these.
2. The two new introduced metrics are somewhat difficult to understand and I am not sure I fully agree with them. See the questions, from where I worked through some examples and see some ways of assigning scores that seem odd to me.
3. Although I wouldn't have expected the authors to have done this since the deadline was so soon after the model's releases, there are some models that claim to do better on this, including ICL-based models (BGE-ICL, RARe) and zero-shot models (Promptriever). It would be interesting to see their performance.

Overall, I do like the paper and would be open to increasing my score if some of these weaknesses/questions are addressed (mostly metrics based).

Update after response: I have increased my soundness score and most of my concerns have been addressed.

questions:
### Re: Weakness 1, dataset construction

a. How were the number of instructions decided/created?

b. How was the number of document in the corpus decided? 

c. It seems like all documents are merged together from all subsets to create the corpus, how are we sure there are not false negatives in the other sets? A qualitative analysis or GPT-based analysis would be very valuable here.

d. How were these documents gathered, just any that were from the original pairs (but then where did the remainder come from?)

### Re: Weakness 2, metrics

A core motivation from the paper of WISe is:
> Intuitively, if the ranking of Pi improves in the instructed mode (Rins < Rori) and worsens in the reversely instructed mode
(Rori < Rrev), this demonstrates the model’s effective compliance with the provided instructions

a: Why is this true? I don't necessarily understand why the query is more relevant with the instruction, weren't these gathered from already relevant query and doc pairs? So I see no specific reason why the rank should increase with the instruction as compared to no-instruction.

b: Arbitrary values: why does WISe not go to 0? Why use 20 as the number for the conditional?

c: Doesn't WISe/SICR have the same problem as p-MRR in terms of rank of 1 for the original and instructed query?

d: Working through some examples for WISe, that I found unusual:
- If the rank is 1 for both, the score is 0 correct? (From Eq 1)
- From r_{ori} of 20 to r_{ins} of 1, we have a score of 1.0. If you change r_{ins} to 2, now the score is 0.07? Perhaps I did the math wrong, but this seems like a pretty large gap
- From r_{ori} of 20 to r_{ins} of 2 we have 0.07 but from r_{ori} of 21 to r_{ins} of 1, we have 0.01 (an even lower score, even though arguably the model improved more)
- Overall, the N=20 distinction seems to introduce some weird disfluencies in the possible range of scores for this metric. Even the overall range seems somewhat odd, you can get scores from -1 to 1, but oddly distributed.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a new dataset and evaluation metrics for instruction-following in information retrieval focusing on 6 dimensions: Audience, Keyword, Format, Language, Length, and Source. The dataset is built by collecting query and document pairs from existing IR datasets as well as Google, and uses GPT-4 to generate instructions and re-write documents. The paper also proposes two new metrics to evaluate instruction following, namely, WISe and SICR that aim to better evaluate this task. Results show that larger models, rerankers, and list-wise models perform better. 

*Strengths*:  
-The reviewers acknowledged the importance of the dataset in measuring progress in instruction-based IR  
-They found the experiments to be comprehensive  
-The proposed metrics were recognized as a notable contribution   
-The reviewers found the analysis from metrics and comparisons to be interesting and insightful  

*Weaknesses*:
-There were several issues and places were the descriptions were confusing or there were lack of details.  Although the authors have responded to and clarified most of the issues, the fact that there were so many writing and clarity issues in the submitted version remains a concern.   
-One of the reviewers mentioned that the updated manuscript violated the format, although the authors corrected that  
-The motivation needed to be expanded   
-One reviewer finds the dataset construction pipeline to be incremental with respect to existing datasets (see below more discussion about this).   
-Some related work needed to be discussed and differentiated    

This was a challenging case, involving completely opposite evaluations by reviewers and extensive discussions between reviewers and authors. To resolve the disagreements and come to a fair decision, I initiated an internal discussion between reviewers. One reviewer strongly argued that the dataset is incremental and the proposed metrics aren't well justified, deeming the contribution insufficient. In contrast, another reviewer disagreed strongly, emphasizing that by the same logic, many valuable datasets in the community could also be dismissed as incremental, arguing that the dataset provides a practical and meaningful resource for advancing instruction-following IR.  

After reviewing the discussions, rebuttals, and the paper itself, I concluded that the dataset offers value to the IR and RAG communities. Instruction-following IR is an emerging area, and this resource provides a foundation for measuring and driving progress. While I acknowledge some of the opposing reviewer’s points, I believe the dataset and metrics collectively form a significant contribution.

That said, the need for numerous clarifications and revisions during the rebuttal highlights the manuscript’s lack of readiness at submission. This remains a concern and detracts from the overall state of the work at submission time.