Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers

Authors: Shijie Chen, Bernal Jimenez Gutierrez, Yu Su
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: large language model, information retrieval

Abstract:
Information retrieval (IR) systems have played a vital role in modern digital life and have cemented their continued usefulness in this new era of generative AI via retrieval-augmented generation. With strong language processing capabilities and remarkable versatility, large language models (LLMs) have become popular choices for zero-shot re-ranking in IR systems. So far, LLM-based re-ranking methods rely on strong generative capabilities, which restricts their use to either specialized or powerful proprietary models. Given these restrictions, we ask: is autoregressive generation necessary and optimal for LLMs to perform re-ranking? We hypothesize that there are abundant signals relevant to re-ranking within LLMs that might not be used to their full potential via generation. To more  directly leverage such signals, we propose in-context re-ranking (ICR), a novel method  that leverages the change in attention pattern caused by the search query for accurate and efficient re-ranking. We assume that more relevant documents should receive more attention weights when an LLM is processing the query tokens, and leverage such signals for re-ranking. To mitigate the intrinsic biases in LLMs, we propose a calibration method using a content-free query. Due to the absence of generation, ICR only requires two ($O(1)$) forward passes to re-rank $N$ documents, making it substantially more efficient than generative re-ranking methods that require at least $O(N)$ forward passes. Our novel design also enables ICR to be applied to any LLM without specialized training while guaranteeing a well-formed ranking. Extensive experiments with two popular open-weight LLMs on standard single-hop and multi-hop information retrieval benchmarks show that ICR outperforms RankGPT while cutting the latency by more than 60% in practice. Through detailed analyses, we show that ICR's performance is specially strong on tasks that require more complex re-ranking signals, such as handling contextualization and contradiction between the query and passages, as well as information integration across multiple passages. Our findings call for further exploration on novel ways of utilizing open-weight LLMs beyond text generation.

Decision: accept (poster)
Reviewer scores: 3, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730162466847

thread_type:
official_review

<thread_object>
creator: reviewer_fyfw

<content>
summary:
This paper proposes an unsupervised ranking method that leverages the attention scores of query tokens over each document token. Extensive experiments, including nine single-hop and three multi-hop datasets, were conducted to prove the improvements over unsupervised baseline RankGPT. Ablation studies were conducted to prove the effectiveness of calibration and aggregation.

strengths:
1. This paper is well-written and easy to follow.
2. Extensive experiments, including nine single-hop and three multi-hop datasets, were conducted to prove the improvements over unsupervised baseline RankGPT
3. Ablation studies were conducted to prove the effectiveness of calibration and aggregation. 
4. Clear analyses were presented to help explain why calibration helps and what re-ranking signals the proposed method captures.

weaknesses:
1) Missing important unsupervised baseline UPR:

Sachan, Devendra Singh, et al. "Improving passage retrieval with zero-shot question generation." arXiv preprint arXiv:2204.07496 (2022).

UPR ranks documents based on the log-likelihood of generating the ground-truth query given the document. Although it is less efficient than ICR, UPR operates in an unsupervised manner and should be compared with ICR.

2) The impact of document order has not been studied. ICR ranks all documents within the context, but the paper does not specify how documents are concatenated. Are they randomly shuffled before concatenation? To what extent does document order impact ranking performance?

3) For single-hop evaluations, only the weaker BM25 retriever is applied. It remains unclear whether ICR can enhance the performance of stronger retrievers.

questions:
1) UPR is a popular unsupervised reranking method that ranks documents based on the log-likelihood of generating a query from the input document. How does ICR's performance compare to that of UPR?

Sachan, Devendra Singh, et al. "Improving passage retrieval with zero-shot question generation." arXiv preprint arXiv:2204.07496 (2022).

2) Are the documents randomly shuffled before concatenation? To what extent does document order impact ranking performance?

3) Why is the top 20 instead of the top 100 utilized for multi-hop evaluations?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730525379732

thread_type:
official_review

<thread_object>
creator: reviewer_cyay

<content>
summary:
This paper proposed in-context re-ranking (ICR), an efficient re-ranking method based on the attention distributions of LLMs. This method can achieve better performance compared to RankGPT while maintaining lower latency.

strengths:
The motivation is "whether autoregressive generation necessary and optimal for LLMs to perform re-ranking?", which is quite interesting and worth discussing. 
The proposed method is technical sound and is effective on various datasets of re-ranking tasks. 
The paper is well-written and easy to follow.
The experiments and discussions are thorough.

weaknesses:
The paper lacks more high-level intuition and explanations especially about why the proposed method works intuitively. 
What are the advantages and disadvantages of the proposed methods intuitively.
Why the proposed method works well although the model's instruction following ability is poor?

questions:
1. In line 216, "We use N/A as a calibration query". I was wondering why use this as the special token, how about NA or None or blank space or <>? Could these be alternatives?
2. In line 226-227, "the final ranking score sd_i, which measures the change of attention weights that each document receives when the query changes from the content-free calibration query to the actual query". I was wondering whether sd_i can be considered as distance between the actual query and calibration query? It would be better to have more high-level intuition and explanation about this.
3. In line 233-234, "since we place the query tokens at the end of the re-ranking prompt, ICR can share the KV cache of document tokens when computing s_{di, Q} and s_{di, Qcal}". I was wondering why the shared KV cache is in effective when you put query tokens at end? I didn't get the point here.
4. In line 297, "ICR's performance advantage is more prominent on Mistral, which has weaker instruction-following capabilities...". I was wondering what's the advantages of the proposed method when instruction following is involved? Whether the proposed method can better leverage important information in the query? I'm looking forward to more explanation here.
5. In ablation study/discussion, I'd love to see how the overall distribution looks like in terms of calibration scores.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730645665515

thread_type:
official_review

<thread_object>
creator: reviewer_d1gb

<content>
summary:
This paper proposes a novel re-ranking method based on LLMs called ICR. ICR first aggregates the attention weights received by each document token from all query tokens. To mitigate intrinsic biases of LLMs, ICR calibrates the ranking scores by subtracting the attention scores obtained by using a content-free query. ICR is evaluated on both single-hop and multi_-hop re-ranking tasks using open-source LLMs, Mistral and LLaMA 3.1.

strengths:
1. The problem is important, and the overall writting is clear. 
2. The proposed method is easy to follow and demonstrates effectiveness on two public LLMs.

weaknesses:
My primary concern is the insufficient comparison with other methods and a lack of depth in experimental analysis.

1）Although the paper compares ICR with RankGPT, highlighting improvements, it would be strengthened by comparisons with more methods, especially other zero-shot listwise methods. 

[1]  RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models.

[2] A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models.

2） The paper could benefit from additional analysis to deepen the understanding of ICR’s behavior. First, for open-source LLMs, it would be helpful to provide a clear comparison between ICR and fine-tuning (FT) methods, highlighting the performance gap and cases where fine-tuning might be preferable. Second, the paper does not analyze the sensitivity of the proposed algorithm to changes in the input document order. It would be valuable to examine how much the re-ranking results fluctuate when the order of input documents is adjusted.

questions:
1. The analysis in Figure 4 lacks details on how results would change if the document input order were adjusted. Additionally, if the document set contains similar documents, would these similar documents receive higher weights?

2. In Section 3.3, how does the method address biases introduced by documents of varying lengths?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730719616144

thread_type:
official_review

<thread_object>
creator: reviewer_nmvy

<content>
summary:
The authors focus on adopting large language models (LLMs) as zero-shot rerankers. I believe this is crucial because if we also need to train the LLMs as rerankers, there would be no difference from the previous framework. The authors propose leveraging the intrinsic attention scores to aggregate into a document score, along with a standard debias score. ***The authors claim that the proposed method operates in O(1) LLM forward passes (This is a point I am skeptical about and would like the authors to clarify in their discussion).*** I will first assign a threshold score and then adjust it based on the authors' rebuttal.

strengths:
1. The writing of this paper is well-crafted, allowing readers with relevant backgrounds to quickly follow and provide feedback.
2. The proposed method is versatile and applicable, making it suitable for use with open-source LLMs.

weaknesses:
1. The claim of O(1) LLM forward passes raises significant questions. Please refer to my Question 1 for a detailed explanation, as this will determine the overall quality of the paper.
2. A limitation of this method is that ordinary users cannot implement it using advanced commercial large language models, especially when compared to RankGPT. However, I believe this is due to commercial factors rather than technical ones.
3. When only comparing against a single baseline, RankGPT, the classic datasets TREC 19 and 20 are notably missing.

questions:
1. A listwise LLM-based ranker typically requires O(N) forward calls because it often assumes that the number of candidate documents, N, is too large, exceeding the LLM's context limit. This necessitates the use of a sliding window for multiple forward passes. Given the same number of documents, how does the proposed ICR achieve this with only two forward passes? If RankGPT requires O(N), I believe the proposed ICR would also require O(N) with the same LLM.
2. Since RankGPT is the only used baseline, the TREC 19 and 20 datasets tested in the RankGPT paper should also be included in the experiments.
3. ICR utilizes attention scores from all transformer layers. Could you provide results from an ablation study showing the outcome when using only the attention scores from the final layer?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732564419063

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Submission Updated By Authors

comment:
We thank all reviewers for their insightful comments and have revised our submission accordingly. In addition to minor proof-reading, we summarize the major changes below (highlighted in blue in the revision):
1. We added an explanation to our intuition behind ICR’s design in the abstract (reviewer CYaY).
2. We enriched explanations of ICR’s design in Section 3 for better readability (reviewer CYaY and d1gB).
3. For Section 4, we enriched single-hop experiment results (table 1) with results on DL19 and DL20, which enhances the comprehensiveness of our evaluation setup (reviewer NMvY).
4. For Section 5.1, we added evaluations and analysis on the sensitivity of ICR to different input order to Appendix C, which shows ICR is more robust than RankGPT to different input orders (reviewer d1gB and FYfW).
5. We added re-ranking results based on ColBERTv2 to Appendix D, which shows that ICR also works with stronger base retrievers (reviewer FyfW).
6. We added comparison between ICR, RankVicuna, and Setwise re-ranking to Appendix E along with relevant discussions (reviewer d1gB).

We hope our responses and the submission update can resolve the concerns raised by the reviewers. 

**As the discussion period approaches its end, we kindly ask the reviewers to read our response and adjust their evaluations. We remain open to addressing any further concerns.**
</content>
</thread_object>

</thread>
<thread>
creation date:
1733277645552

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the author response period and additional clarification on complexity

comment:
We're glad that the reviewers acknowledged several strengths of this work, specifically the good writing quality (all reviewers), ICR's novelty and interesting motivation (Reviewers d1gB and CYaY), effectiveness (Reviewers d1gB and CYaY) and versatility (Reviewer NMvY). Additionally, Reviewers CYaY and FYfW think our experiments and analysis were thorough.

In the author response period, we have addressed most concerns from reviewers:
- Reviewer NMvY mainly raised concerns on the validity of our claim on complexity improvement and the lack of experiments on TREC 19 and 20. We have updated our draft with additional experiments on the two datasets and provided detailed explanations for complexity to address these concerns. 
- Reviewer d1gB asked for comparisons with more methods and evaluation on ICR's sensitivity to different document order. We responded with a comparison between ICR and two methods named by the reviewer, as well as evaluation of ICR's robustness to varying document orders, which addressed the reviewer's concerns.
- Reviewer CYaY was mainly concerned with the lack of explanations for the intuition behind ICR. We provided detailed explanations in our response and updated the paper draft accordingly.
- Reviewer FYfW raised concerns about one missing baseline method, ICR's robustness to document order, and its applicability to stronger retrievers. We responded with justification of our baseline choice. We also demonstrated ICR's robustness and effectiveness with stronger retrievers through additional experiments.

No new concerns have been raised during the author response period. The only remaining point of concern is whether ICR reduces the complexity of LLM-based zero-shot re-ranking from O(N) forward passes to O(1), as we claimed. This was first raised by Reviewer NMvY and echoed by Reviewers CYaY and FYfW. We have already provided detailed discussion on this matter and would like to add to our previous discussion summary with some final clarifications.

To our understanding, Reviewer NMvY’s concern arises from two aspects:

- Firstly, the reviewer thinks that ICR also requires a sliding window strategy to handle the same amount of documents as RankGPT, thus ICR also needs an O(N) number of forward passes. 

    We have responded to this aspect in our rebuttal. We acknowledge that using ICR assumes the LLM has enough context length to process all documents. It is important to note that ICR’s design only relies on the attention pattern of LLMs, which is orthogonal to the efforts in improving context length and inference efficiency of LLMs. We believe the increase of LLM context length and the advancement of inference infrastructures will continue to weaken the limitation imposed by this assumption. 
- Secondly, the reviewer thinks RankGPT also only needs O(1) forward passes if used without the sliding window mechanism.

    We have provided detailed explanations in our discussions and showed this is not true. RankGPT still requires O(N) forward passes in this case.

We hope our explanations can help the reviewers and other audience to clear potential confusion and resolve this concern.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes an approach to zero-shot reranking for information retrieval. The idea is to measure how much attention is placed on a particular document, relative to some dummy calibration query. Notably, this approach does not require any auto-regressive decoding, unlike some prior works. Consequently, for re-ranking large document sets, the approach offers a sizable latency reduction, while possessing good quality.

Reviewer opinion was divided. On the one hand, the paper studies a topical problem, and has an interesting solution which is intuitive. The empirical results are encouraging, and showcase the flexibility of LLMs for complex tasks.

On the other hand, there were concerns about the paper's claims about reducing $O(N)$ to $O(1)$ forward passes for the re-ranking. Following a lengthy discussion, some salient points emerged:
- there is a distinction between "forward passes" and "API calls", with the latter favoured in some prior work.
- "forward passes" can encompass both a prefill/context encoding step, and sequential auto-regressive decoding steps. These steps have rather different profiles; e.g., the former tends to be compute-bound.
- for both the proposed method and RankGPT, the input is a (tokenised) query and list of (tokenised) documents.
- in RankGPT, one performs a single parallel prefill/context encoding step; and then $N$ sequential auto-regressive decoding steps, to generate the proposed ranking over the documents.
- in the proposed method, one performs two parallel prefill/context encoding steps; and _no_ auto-regressive decoding steps.

The paper's claim is then that the proposed method can be much faster since it has shaved off $O(N)$ auto-regressive decoding steps. This claim was not explained in detail in the original manuscript (which simply referred to "forward passes", without discussing context encoding versus auto-regressive decoding). We believe it crucial for the paper to properly discuss this issue, and ensure there is precision in the claims. This said, we believe it accurate that the proposal does indeed shave off $O(N)$ auto-regressive decoding steps, and that this can lead to significant cost reductions.

There were also some questions regarding the paper's comparison to other baselines (e.g., listwise ranking), stability analysis, and providing more intuition for the method. The author response on these points were reasonable, and were generally seen as satisfactory.

Overall, the paper's finding is expected to be of interest to the retrieval community. It is however imperative that the comments about "forward passes" are properly qualified. We strongly encourage the authors to incorporate detailed points made in the discussion with reviewers.