Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers
Authors: Shijie Chen
Bernal Jimenez Gutierrez
Yu Su
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Model
Information Retrieval
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose an efficient LLM-based re-ranking method that outperforms RankGPT while only requiring two forward passes without specialized training.

Abstract: Information retrieval (IR) systems have played a vital role in modern digital life and have cemented their continued usefulness in this new era of generative AI via retrieval-augmented generation. With strong language processing capabilities and remarkable versatility, large language models (LLMs) have become popular choices for zero-shot re-ranking in IR systems. So far, LLM-based re-ranking methods rely on strong generative capabilities, which restricts their use to either specialized or powerful proprietary models. Given these restrictions, we ask: is autoregressive generation necessary and optimal for LLMs to perform re-ranking? We hypothesize that there are abundant signals relevant to re-ranking within LLMs that might not be used to their full potential via generation. To more  directly leverage such signals, we propose in-context re-ranking (ICR), a novel method  that leverages the change in attention pattern caused by the search query for accurate and efficient re-ranking. We assume that more relevant documents should receive more attention weights when an LLM is processing the query tokens, and leverage such signals for re-ranking. To mitigate the intrinsic biases in LLMs, we propose a calibration method using a content-free query. Due to the absence of generation, ICR only requires two ($O(1)$) forward passes to re-rank $N$ documents, making it substantially more efficient than generative re-ranking methods that require at least $O(N)$ forward passes. Our novel design also enables ICR to be applied to any LLM without specialized training while guaranteeing a well-formed ranking. Extensive experiments with two popular open-weight LLMs on standard single-hop and multi-hop information retrieval benchmarks show that ICR outperforms RankGPT while cutting the latency by more than 60% in practice. Through detailed analyses, we show that ICR's performance is specially strong on tasks that require more complex re-ranking signals, such as handling contextualization and contradiction between the query and passages, as well as information integration across multiple passages. Our findings call for further exploration on novel ways of utilizing open-weight LLMs beyond text generation.

Published as a conference paper at ICLR 2025
ATTENTION IN LARGE LANGUAGE MODELS YIELDS EFFICIENT SERO-SHOT RE-RANKERS
Shijie Chen Bernal Jiménez Gutiérrez
Su
The Ohio State University
(chen. 1 021 6, imenezgut.ierree. 1, su 809) eosu. edu
ABSTRACT
Information retrieval (IR) systems have played vital role in modern digital life and have cemented their continued usefulness in this new era of generative Al via retrieval -aggeeeede generation. With strong language processing capabilittes and remarkable versatility, large language models (LLMs) have become popular choices for zero-shot re-ranking in IR systems. So far, L.LM-based re-ranking methods rely strong generative capabilities, which restricts their use to either specialized or powerful proprietary models. Given these restrictions, we ask: is autoregressive generation necessary and optimal for LLMs to perform re-ranking? We hypothesize that there are abundant signals relevant to re-ranking within LLMs that might not be used to their full potential via generation. To more directly lever- age such signals, we propose in-context re re-ranking (ICR), a novel method that leverages the change in attention pattern caused by the search query for accurate and efficient re-ranking. We assume that more relevant documents should receive more attention weights when an LLM is processing the query tokens, and leverage such signals for re-ranking. To mitigate the intrinsic biases in LLMs, we propose calibration method using content-free query. Due the absence of generation, ICR only requires two (O(1) forward passes to re-rank N documents, making it substantially more efficient than generative re-ranking methods that require least O(N) forward passes. Our novel design also enables ICR to be applied to any L.M without specialized training while guaranteeing a well-formed ranking. Extensive experiments with two popular open-weight LLMs standard single- hop and multi-hop information retrieval benchmarks show that ICR outperforms RankGPT while cutting the latency by more than 60% in practice. Through de- tailed analyses, we show that ICR's performance is specially strong on tasks that require more complex re-ranking signals, such handling contextualization and contradiction between the query and passages, as well as information integration across multiple passages. Our findings call for further exploration on novel ways of utilizing open-weight LLMs beyond text generation.
INTRODUCTION
Information retrieval (IR) plays pivotal role in numerous real-world applications, ranging from search engines to recommendation systems. In the era of generative AI. IR techniques are widely used to enhance generative models with up-to-date information, process known as retrieval- augmented generation (Lewis et al. 2020). A critical component of modern IR pipelines is re- ranking (Nogueira et al. 2019: 2020 which typically leverages more powerful models to improve search outcomes produced by lightweight, large-scale retrieval systems such BM25 (Robertson) & Zaragoza 2009) and dense retrieval systems (Karpukhin et al., 2020). The advent of Large Lan- guage Models (LLMs) has significantly influenced the technology stack of information retrieval (Zhu et al. 2023 Wang et al., 2024), particularly in their application as high highhpeeformance zero-shot
re-rankers. leading substantial improvements in retrieval performance (Sun et al.) 2023).
Existing LLM- based re-ranking methods predominantly rely on the generative capabilities LLMs.
Listwise methods (Sun al. 2023 Tang al. 2023 Pradeep al. 2023b) requires the LLM to
Code and data: httpt: /ttps:://iiie /gi thub. cww/OSU- -tP.SUUU-P-G-Up/////// w n-Cont Rerakkinnn

---

Published as a conference paper at ICLR 2025
RankGPT
In-context Re-ranking
Query Q
Generative Re-ranking, O(N) FP
69.4
Ranked Passagio: (1) [1 [2]
Score
65
60.4
In-conlext Re-ranking. (1) FP
$1.9
LLM
Dos B
Doc [1]
Docs
Doc [2]
Sin Single-hop Re-ranking
(nDCGg10)
Multi-hop Re-ranking (Recall @ 5)
Dataset
(a) LLM-based re-ranking methods. FP: forward pass.
(b) Re-ranking performance of Llama 3.1 8B.
Figure I: (a) Overview of in-context re-ranking (ICR), (b) Re-ranking performance using Llama-3.1
8B on single-hop and multi-hop re-ranking tasks. We report the micro-average across tasks.
generate a ranking of document identifiers in a pre-defined format. Comparison-based methods, such pointwise(Zhunng
al. 2023a b) Reddy al.] 2024) and pairwise methods(Qin al., 2024 Chen et al.] 2024), either ask the LLM to generate a relevance score for each document separately, or to compare document pairs and form a ranking. Other methods utilize the LLM's output logits to rank document identifiers and avoid expensive generation(Zhuang et al.) (2024; Reddy et al.) 2024). So far, L.LM-based re-rankers depend on strong generative capabilities, and thus have only proven to be effective when using strong proprietary models or supervised fine-tuned models as the backbone, constraining their broader adoption with open-weight models.
Considering the limitations of existing generative approaches, we ask: Is autoregressive genera- tion necessary and optimal for LLMs to perform re-ranking? To answer this research question, we hypothesize that abundant signals related to re-ranking have already emerged during the context en- coding stage of LLMs and directly utilizing such signals may improve re-ranking performance over using them indirectly via generative approaches. In this work, propose in-context re-ranking (ICR), a simple yet effective re-ranking method that harnesses the contextual understanding capac- ity of LLMs through their attention weights, as they correlate closely with context relevance within LLMs (Peysakhovich & Lerer 2023). ICR first aggregates the attention weights received by each document token from all query tokens. To mitigate intrinsic biases of LLMs, we calibrate the rank- ing scores by subtracting the attention scores obtained by using a content- free query (i.e. "N/A"). Finally, we take the sum of the calibrated attention weights from all tokens in a document as the final ranking score. In this way, the ranking score captures the change in total attention received by each document caused by the semantics of the actual query. As shown Figure] T ICR only requires two forward passes to re-rank N documents, substantially reducing the O(N) Torward passes required by generative re-ranking (Qin et al., 2024) to O(1). Additionally, without the need of text gener- ation, ICR guarantees well-formed ranking using any LLM. The token-level ranking scores also provide improved interpretability compared to ranking results generated in black-box processes.
We comprehensively evaluate ICR with two weegtt LLMs and various IR benchmarks, in- cluding the well-established TREC (Craswell et al.] (020) and BEIR benchmark (Thakur et al.] 2021], as well as multi-hop retrieval tasks (Gutiérrez et al. 2024). Our experiments show that ICR outperforms RankGPT on most tasks while reducing lateney by more than 60%. When used with Llama-3.1 8B (Dubey et al.] 2024), ICR outperforms RankGPT (Sun al. 2023) by 8.4 points in single-hop re-ranking and 2.2 points in multi-hop re-ranking on average, closely matching the strong proprietary model GPT-4o mini(Openta 2024).
Through detailed analysis, we demonstrate that LLMs' attention weights capture rich signals rel- evant to re-ranking, including contextualization signals between query and passages, reasoning signals for handling contradiction relations; and information integration signals involving bridge entities. During generation, such signals may be overpowered by others, such as position bias and lexical overlap. Without generation, ICR is able to directly leverage these relevance signals, which substantially improves re-ranking performance, especially on tasks that require handling deeper se- mantic relevance between query and documents, such as evidence retrieval for fact verification, as well as tasks that require integrating information from multiple documents, as is the case with multi-hop re-ranking tasks. Our findings indicate that generation is not always the optimal way of unleashing the full potential of LLMs, and highlights the value of exploring creative ways to better leverage open-weight models across diverse applications.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
2.1 ZERO-SHOT RE-RANKING WITH LARGE LANGUAGE MODELS
Existing LLM-based re-ranking methods typically fall into three categories: pointwise, pairwise, and listwise re-ranking. Pointwise re-ranking asks the LLM to grade the relevance of each docu- ment separately. This can be done via relevance generation (Liang et al. 2023) and query gener- ation (Drozdov al. 2023 Sachan et al., 2022). However, the black- box nature of LLMs makes calibrating the pointwise scores an open challenge. Recently, Qin et al. (2024) propose a pairwise
re-ranking method that first uses LLMs for pairwise document comparisons and then integrates the
results into a complete ranking. A more direct approach is listwise re-ranking (Sun et al. 2023 Ma
al. 2023) which uses the LLM generate a ranking list. As pointed out by Qin et al. (024 all these methods require O(N) to ((N²) API calls for re-ranking N documents. Since each API call generates least (1) tokens, their complexity measured in forward passes remains at least O(N). Please see Appendix A for a more detailed complexity analysis. Apart from cost concerns, two additional issues stand out for generative re-ranking: (1) the relevance scores generated by LLMs lack interpretability, which undermines the trustworthiness of the re-ranking process; (2) there is no guarantee that LLMs always generate well-formed outputs, such as relevance scores or ranking lists. This greatly limits the availability of LLM-basec re-rankers and the usage of language models that are weaker at following instructions.
To enable zero-shot re-ranking with open-weight LLMs, researchers also specialize foundation language models with re-ranking data synthesized by strong proprietary LLMs (Sun et al. 2023] Pradeep et al. 2023a b). Nevertheless, they have been shown to struggle with generalizing to corpus sizes unseen during training (Pradeep et al. 2023b). More recently, FIRST (Reddy et al. 2024) achieves zero-shot re-ranking with one forward pass by sorting document identifiers based on the output logits of the first generated token, which is supposed to represent the most relevant document. However, FIRST still requires specialized fine-tuning, and its logit-based design restricts document identifiers to be single-token (i.e. "A" to "Z"), limiting the number of documents it can handle.
In contrast to generative re-ranking methods, ICR is based on the general language processing abil- ities present in any pretrained language model. Without generation, ICR avoids the instability of generation, guaranteeing well-formed and consistent ranking results. As result, ICR is applicable to any LLM without specialized fine-tuning. Additionally, due to the limitation of context length and the degradation of generation quality for long contexts, existing zero-shot methods typically need to be used with a sliding window mechanism (Sun et al. 2023), which requires O(N) iterations, to handle larger retrieval corpora. In comparison, ICR can fully leverage recent long-context LLMs and re-ranks any number of documents with two (O()) forward passes, substantially lowering latency.
2.2 ATTENTION- BASED INTERPRETABILITY LARGE LANGUAGE MODELS
Attention weights has been used for interpreting the importance of input tokens to language model output since the attention mechanism was first introduced by (Bahdanau et al. (2015). While there ex- ist controversial opinions on how reliable attention weights are for interpretability (Serrano & Smith 2019 Wiegreffe & Pinter 2019), previous work has shown the effectiveness of distilling cross- attention between documents and reader model for training better retrievers (Izacard & Grave, 2021]. More recently. PPeyskhoovich & Lerer (203) propose attention sorting to improve gener- ation quality, where input documents are sorted by the amount of attention they receive when the L.M is generating the next token. In-context re-ranking shares a similar assumption that attention weights correlates with the relevance of documents to the query. However, we aggregate the atten- tion weights over all query tokens, rather than only considering the last token as in previous work. We also propose calibration method to help mitigate LLMs' intrinsic biases. Our experiments on information retrieval benchmarks show the necessity of the proposed components.
3 IN-CONTEXT RE-RANKING
In this section, we introduce the in-context re-ranking (ICR) method. Given query Q and set of N documents D = (di...., ) retrieved by a retriever, a re-ranker ranks the documents by their relevance the query. ICR accomplishes the re-ranking process by producing ranking score
3

---

Published as a conference paper at ICLR 2025
Layer L
<lne>
Calibratio
Agg
<ine>
<ins>
Agg
Figure 2: Illustration for In-context Re-ranking (ICR). ICR first aggregates (Agg) attention weights between document and query tokens to form token-level query ranking scores. Then calibration score is calculated similarly with the calibration query, which is subtracted form the query ranking score. The final score is the of calibrated scores for all tokens for each document.
St for each document d, using the attention distribution within an LLM. More specifically, we first prompt the LLM with the documents and the search query to retrieve attention weights (Section 3.1). Then we aggregate the attention weights across all attention heads and layers to compute a token-level ranking with a content-free query (Section 3.3 Figure Tiguure2illin iillustrates the architecture of ICR.
for each document (Section 3.2). Finally, we calibrate the ranking scores
3.1 PROMPT DESIGN
To leverage the instruction following abilities in modern LLMs, we formulate the re-ranking task into either question answering (QA) or information extraction (IE) tasks, which are more commonly seen by L.MSs. We design two kinds of instruction prompts for different type of queries:
QA style: For search queries that are questions, we prompt the LLM to answer the question based on provided documents, leveraging its QA abilities.
IE style: For non-question queries, we prompt the LLM to find relevant information from provided documents, leveraging its IE abilities.
The prompt consists of the instruction, the documents to be re-ranked, and the search query. We reverse the order of input documents as we find improves performance in our preliminary experi- ments, probably due to the position bias in LLMs. Given the autoregressive nature of decoder-only LLMs, we place the query at the end of the prompt. Please see Appendix B for prompt examples.
3.2 ATTENTION AGGREGATION
We hypothesize that LLMs will, on average, attend more strongly tokens in relevant documents than irrelevant ones when processing the query. Accordingly, we design an attention aggregation method compute the attention weight that each token document gets from the query across all attention heads and layers in the LLM. Using an LLM with L layers and H attention heads, the ranking score sd., for the j-th token in document d, is:
L
H
ΣΣΣX
Σ
8d
IQQ
j,k0
where IQ represents the set of token indices for query Q. aj denotes the attention weight from the k-th token (in the query) to the j-th token (in document d,) by the h-th attention head at layer Since each query token's 's attention weights form a distribution and sum up 1, our aggregation strategy avoids the commonly seen length bias issue where longer documents receive higher scores.
3.3 RANKING SCORE CALIBRATION
LLMs show various intrinsic biases text generation (Gallegos et al. 2024). An ideal re-ranker should assign equal scores to all documents when given content-free free query. Inspired by Zhao et al. (202T), we propose ranking score calibration method to mitigate intrinsic biases LLMs' attention weights. Following Zhao et al. (202T),
"N/A" " the calibration query Qeal and calculate
4

---

Published as a conference paper at ICLR 2025
calibration scores ($d...Quaa (eatt } via attention aggregation for each document. Then subtract it from ranking scores from the actual query ) to obtain the calibrated ranking score {sd.,)) ):
Adi 8d.,5 8d..s Reat
As will be discussed in Section 5.1 the calibration scores capture strong attention weights biased towards meaningless tokens in the documents, such as punctuation, which should not affect the relevance of the documents. After that, we filter out such tokens with abnormally negative calibrated scores. Finally, we sum the calibrated scores for all tokens in each document to obtain the final ranking score &d,, which measures the change of attention weights that each document receives when the query changes from the content-free calibration query to the actual query:
Sd, {din}: 5% [dirlde, > Sd. 22
Sd. E s:
SEST,
where denotes standard deviation. Since we place the query tokens the end of the re-ranking prompt, ICR can share the KV cache of document tokens when computing $d. and Sd., Em- pirically, we observe the overhead for calibration be only approximately 30%.
EXPERIMENTS
We evaluate the proposed in-context re-ranking method both single-hop (Section 4.2) and multi- hop (Section 4.3) re-ranking tasks using open-weight L.Mss. Considering the additional training cost introduced by supervised methods, such as RankVicuma (Pradeep et al. (2023a), and inference cost needed by other listwise methods, such as the setwise method (Zhuang et al. 2024), we mainly use RankGPT as a representative zero-shot baseline (Comparisons with those two methods are in Appendix(F] (F). We also examine the scaling trend of their efficiency and performance (Section 4.4).
Then we show the effectiveness of attention aggregation and calibration in ICR via ablation studies (Section[4.5) As specialized methods do not outperform their teacher models (Pradeep et al. 2023b Reddy et al. 2024), RankGPT with strong proprietary models still strikes the best balance between performance and efficiency for zero-shot re-ranking. We report re-ranking results from RankGPT with GPT3.5-turbo and GPT-4o mini for reference.
4.1 EXPERIMENT SETUP
Base LLMs. As in-context re-ranking requires access the attention distribution of all layers and heads in an LLM, we experiment with open-weight LLMs. We choose two popular instruction- tuned models: Mistral 7B (Jing et al. 2023) and Llama-3.1 8B (Dubey al. 2024 We note that, while we only test ICR with open- weight LLMS in this paper, ICR can also be implemented with proprietary models if their vendors choose to.
Datasets & Metrics. We evaluate our method on both single-hop information retrieval bench- marks, which emphasizes estimating the documents' semantic relevance, and multi-hop retrieval tasks, which demands information integration across multiple documents.
In single-hop evaluations, we experiment on TREC (Craswell et al. 2020 and nine public datasets in the BEIR benchmark (Thakur et al.) (0211) including TREC-COVIN (Voorhees et al. 2021
NFCorpus (Boteva et al. (.Pe)aDBPeii (Hasibi et al. 2017). SeiFact (Wadden et al. 2020
SciDocs (Cohan et al.J 2020), FIQA (Maia et al. 2018). FEVER (Thorne et al. 2018)), Cl Clmate- FEVER (Diggelmann et al. 2020), and NQ (Kwiatkowski et al. 2019). Following Sun et al. 2023),
re-rank the top 100 documents returned by BM25(Robertson & Zaragoza 2009) and report nDCG@ 10. We apply sliding window of size 20 and stride 10 for RankGPT. For ICR, we directly re-rank all documents.
In multi-hop evaluations, we sample 1000 queries from three popular multi-hop question answer- ing datasets in Gutiérrez al. (2024). including MuSiQue (answerable) (Trivedi et al. 2022)
Mooe specifically, we Distral-7Biinstmut-to0 and Ilama-3. 1-8B-instruct.
5

---

Published as a conference paper at ICLR 2025
Table 1: nDCG@ 10 on the BEIR benchmark. SR: Success rate, the chance that a re-ranker's output format is correct. We bold the best performance for each task with each base LLM.
TREC-
NF
DBPedia- Eatity
SciFact SeiDoes RIQA
FEVER
Climatee FEVER
NQ
Macro-
SR
19 20
COVID Corpus
Avg
Avg
BMI5
59.5
32.2
31.8
67.9
14.9
23.6
65.1
16.5
30.6
44.7
40.1
RankGPT (Mistral 7B)
51.3
49.2
60.6
31.0
32,9
65.5
14.7
24.1
62.7
16.7
32.6
44.0
40.1
34.3%
ICR (Mistral 7B)
59.2
53.6
63.9
33.2
31.4
72.4
16.2
31.0
79.8
20.6
51.0
57.3
46.6
100%
RankGPT (Llama-3,1 8B)
64.9
63.9
72.6
33.7
41,4
68.4
17.9
31.5
66.7
22.7
51.1
52.0
48.6
99.2%
ICR (Llama-3,1 8B)
55.7 51.9
72.8
34.7
35,3
76.1
17.1
38.1
84.5
23.1
51.2
60.4
49.1
100%
RankGPT (GPT-3.5 turbo)
64.0
62.2
74.1
32.8
42.0
64.1
15.4
30.5
72.6
22.4
49.7
54.0
44.9
98.2%
RankGPT (GPT-40 mini)
79.1
38.0
44.9
77.0
20.
41.7
80.1
60.5
95.5%
2WikiMuliiHopQA (Ho et al. 2020), and HotpotQA YYnn et al. 2018). Following Gutiérrez et al.
2024), we re-rank the top 20 retrieval results returned by ColBERT v2 (Santhanam al.) 2022) and measure recall @2 and recall @5. We also report the ranking success rate for RankGPT, which measures the chance that it outputs a correctly formatted ranking.
We use the QA style prompt for multi-hop tasks, TREC-DLI9, TREC-DL20, and four tasks in BEIR, including TREC-COVID. DBPedia-Entns FIQA, and NQ. For fair comparison, we use the official prompt for RankGPT and report ColBERT v2 results if the output ranking is unusable.
4.2 SINGLE-HOP RE-RANKING
Compared to the proprietary GPT-3.5 Turbo and GPT-4o mini, open-weight LLMs generally more limited in their instruction following abilities, as shown by their substantially lower perfor- mance and success rate when used with RankGPT. In contrast, ICR guarantees producing usable ranking with any base L.M and delivers strong performance with open-weight LLMs.
Results in Table demonstrates the strong performance of ICR on the single-hop re-ranking tasks, outperforming RankGPT by 13.3 (8.4) points in micro-average score and 6.5 (0.5) points in macro average when used with Mistral 7B (Llama-3.1 8B). ICR's performance advantage is more promi- nent on Mistral, which has noticeably weaker instruction-folloiingg capabilities compared to Llama- 3.1. as shown by its significantly lower RankGPT performance and success rate. This indicates that ICR can effectively leverage re-ranking signals from L.MSs even when the model is weaker at text generation. Impressively, ICR (Mistral 7B) outperforms RankGPT (GPPT3.5-turbo) by 3.3 points in micro-average and 1.7 points in macro-average. Moreover, ICR (Llama-3.1 8B) matches RankGPT (GPT-40-mini) in micro-averag, (60.4 vs 60.5 points) and is slightly behind by 2.1 points in macro- average. These results highlight the effectiveness of ICR in unleashing the potential of open-weight LLMs by leveraging their internal signals. ICR substantially improves the results of BM25. We show that ICR can also work with stronger retrievers, such as Co in Appendix
We observe that ICR strongly outperforms RankGPT with both LLMs on three single-hop datasets: FiQA, SciFact and FEVER. Notably, on FEVER, RankGPT with GPT-4o mini (80.1 points) is closely matched by ICR with Mistral 7B (79.8 points), and further outperformed by ICR with Llama-3.1 8B (84.5 points). As will be detailed in Section 5.2 ICR's exceptional performance on these datasets comes from their need to leverage more complex contextualization signals in ad- dition to similarity cues. In FIQA, the meaning of many retrieved passages can only be fully deter- mined when combined with their respective queries, increasing the task's 'ss's complexity. In SciFact and FEVER, however, relevance is determined by whether a passage supports or contradicts a specific fact, thus they also require reasoning capabilities.
Despite its high performance on most datasets, we also notice that RankGPT consistently performs better on DBPedia-Entity. We attribute much of ICR's 's disadvantage on DBPedia-Entinn to the base LLMs' lexical bias, which hurts ICR's performance when distractor documents have high lexical overlap with the query, which is common in DBPedia-Entin (See Appendix C.I for more details). We also surprisingly observe that ICR (Ilama-3.1 8B) underperforms ICR (Mistral 7B) and notice- ably falls behind RankGPT (Llama 3.1 8B) TREC DL.9 and DL20. We suspect Llama-3. might be fine-tuned on relevant re-ranking data for MSMARCO, the data source of TREC, which leads to high re-ranking success rate but may skew its attention distribution.

---

Published as a conference paper at ICLR 2025
4.3 MULTI-HOP RE- DE-RANKING
Table 2:
Re-ranking performance the multi-hop test set. SR: Success rate, the chance that re-ranker's output format correct. We bold the best performance for each task and base LLM.
MuSiQue
2Wiki
HotpotQA
Average R@2
SR
R@2
R@5
R@2
R@5
R@2
R@5
R@5
ColBERT v2
37.9
49.2
59.2
68.2
64.7
79.3
53.9
65.6
RankGPT (Mistral 7B)
36.2
48.7
56.3
67.1
64.8
79.8
52.4
65.2
20.5%
ICR (Mistral 7B)
39.5
52.6
62.7
71.3
71,2
84.3
57.8
69.4
RankGPT (Llama-3.1 8B)
39.3
51.4
60.8
72.0
69.9
84.8
56.6
69.4
98.2%
ICR (Llama-3. 8B)
44.3
55.9
67.0
72.6
77.3
86.2
62.9
71.6
100%
RankGPT (GPT-3.5 Turbo)
38.7
61.6
84,9
57.4
70.6
98.6%
RankGPT (GPT-40 mini)
45.7
57.0
71.6
75.1
88.8
66.0
73.6
93.7%
Table 3: A on multi-hop test set. ICR shows substantial improvements over ColBERT v2 on all datasets. We bold the best performance for each task and base LLM.
MuSiQue
2Wiki
HotporQA
Avg
ColBERT v2
16.1
37.1
59.0
33.5
RankGPT (Llama3.1 8B)
20.8
46.7
71.2
46.2
ICR (Llama3.1 8B)
23.5
44.6
75.7
47.9
RankGPT (Mistral 7B)
15.9
38.4
62.5
38.9
ICR (Mistral 7B)
21.3
43.4
73.5
46.1
In multi-hop hop evaluations, observe similar trends as in single-hop experiments. Table 2 shows that both RankGPT and ICR substantially improve the retrieval results of ColBERT v2 on all three multi- hop test sets, with ICR outperforming RankGPT by 5.4%-6.3% in Recall @22 and 2.2%-442%% in Recall@5 @5 on average. Impressively, ICR (Mistral 7B) matches RankGPT (GPT-3.5 Turbo) (57.8% vs 57.4% in R@2 and 69.4% vs 70.6% in R@5). ICR (Llama-3.1 8B) further comes close to RankGPT (GPT-40 mini) (62.9% vs 66.0% in R@2 and 71.6% vs 73.6% in R@5), making it competitive alternative proprietary models for multi-hop re-ranking.
To better understand the performance improvement on multi-hop re-ranking, we additionally re- port the All-Recall@5 performance following Gutiérrez et al. (2024), which measures the rate of a retrieval model ranking all relevant documents for a query within top-5. Table 3 shows that ICR substantially improves All-Recall @5 all three multi-hop tasks. This suggests that ICR can better understand the relationships between documents and perform multi-hop information integration. We provide a more detailed analysis in Section 5.2
4.4 SCALING TREND OF SPEED AND PERFORMANCE
As previously discussed, re-ranking N documents using RankGPT requires at least O(N) forward passes. In contrast, ICR only requires two (O(1) forward passes. We empirically compare the speed and performance of ICR and RankGPT on the Sci- Fact dataset with 300 queries, under the same set- tings as above. We note that the choice of dataset should not affect our conclusion. We test seal- ing trend when re-ranking K documents with KE (20, 40, 60, 80, 100)- This test performed on a single NVIDIA RTX A6000 Ada GPU. We Llama-3.1 8B as the base LLM and truncate the doc- uments to 100 words per GPU memory limitations.
86.0
RadSOPT Paforracce
75,0
72.5
Latency
67.5
62.5
20
46
100
Figure 3: Scaling trend of performance and average latency on the SciFact dataset.
7

---

Published as a conference paper at ICLR 2025
Table Ablation studies on single-hop and multi-hop re-ranking, We report nDCG@ SciFact and FiQA, as well as recall@5 on MuSiQue and 2WikiMullihopOA.
Single-hop FiQA
Multi-hop 2Wiki
SciFact
Macro-Avg
MuSiQue
Macro-Avg
ICR (Llama-3.1 8B)
76.1
38.1
57.1
55.9
72.6
64.3
Calibration
71.3
28.8
50.0
52.1
64.4
58.2
Aggregation
68.2
24.4
46.3
41.9
52.6
47.2
Both
60.9
20.1
40.5
41.1
50.3
45.7
We implement ICR with the Transformers library
(Woff et al. 2020) by extracting attention weights from the forward () function. We implement RankGPT using vLLM (Kwon et al. 2023), popular LLM inference infrastructure that is much faster than Transformers to ensure a practical comparison
3 Figure 3 shows that ICR is more than two times faster than RankGPT with lower rate of growth when K becomes larger, despite having a less efficient implementation. Meanwhile, we notice ICR's 's performance keeps improving as K grows, while RankGPT saturates K 40.
4.5 ABLATION STUDY
In this section, evaluate the effectiveness of attention aggregation and score calibration in ICR. We conduct ablation studies on SciFact, FiQA, MuSiQue, and 2WikiMultHHopAA, When both com- ponents removed, ICR ranks the documents only based the attention weights of the last query token, i.c., equivalent to attention sorting (Peysakhovich & Lerer 2023).
As shown in Table both attention aggregation and score calibration are crucial for ICR's high performance, The huge gain brought by attention aggregation (13.7 points in single-hop and 17.1 percent in multi-hop only the last token. The calibration process also has big performance impact (9.3 points in single- hop and 6.1 percent in multi-hop on average). We analyze why calibration helps in Section
average) shows the importance of considering all query tokens rather than
ANALYSIS & DISCUSSION
In this section, we first analyze why score calibration is effective (Section 5.1 Then, we investigate how ICR leverages complex signals in LLMs for complex retrieval tasks using Llama-3.1 8B (Section 5.2)) We analyze ICR's behavior based on token-level ranking scores of documents.
5.1 WHY DOES CALIBRATION IMPROVE IN-CONTEXT RE-RANKING?
To understand the effectiveness of the proposed calibration method, we analyze the distribution of calibration score Seut multi-hop datasets. As the calibration query is irrelevant any document, an unbiased re-ranker should assign equal ranking scores for all documents. Therefore, calibration scores that deviate from uniform distribution can reflect the base LLM's intrinsic biases.
20.0
Table 5: An example of token-level calibration score in ICR (Llama-3.1 8B). Deeper color indi- cates higher calibration score.
17.5
Ranking Score
35.0
32.9
10.0
Avg
Indo ak istani wars and conflicts
The Kashmir issue has been the main cause whether direct or indirect, countries with the exception of the Indo Pakistani War Figure 4: Average calibration scores of ICR at 197 where conflict originated due to turmoil in erst while East Pakistan now Bangladesh
major conflicts between two
Document Position
different document input positions.
We note that ICR is compatible with L.M inference infrastructures like vL..M as well if an interface for getting attention weights is exposed (not available for vI..M of now).
8

---

Published as a conference paper at ICLR 2025
Table 6: In-context re-ranking scores for all tokens in FIQA example with query "How read bond yield quotes? What do the time, coupon, price, yield, and time mean?" B, R, denote ranking from BM25, RankGPT, and ICR respectively.
Passage
B R 1 Support
The month and 1 year columns show the percentage change over that period. Coupon 6 10 1 Truc
(coupon rate) is the amount of interest paid on the bond each period. Price is the nor- malised price of the bond ... Yield the interest rate that you would receive by buying that price....
The 10yr bond pays coupons semi annually. The yield % is what you would get annually 2 1 2 False if you hold the bond full term. The coupon payment won't be exactly 1.65% because of how bond pricing and yields work. The yicld commonly guoted does not tell you how much each interest payment is.
We observe strong position bias towards documents placed at the beginning and the end of the input on both Mistral and 1, as shown in Figure E echoing the lost-inthhe-middee issue discussed in recent literature (Liu et al. 2024). With the proposed calibration method, ICR shows improved robustness against different document input orders compared to RankGPT, which partly explains its superior performance (Quantitative results are available in Appendix DD)
Since ICR determines document's relevance by summing ranking scores over all its tokens, token- level scores could directly reveal ICR's s inner workings. In addition to the position bias, the example in Table that Llama3. I'ss attention distribution is biased towards titles, entities and punctua- tion (Similar results with Mistral can be found in Appendix G. ). By capturing these biased signals in the LLM, calibration helps mitigate intrinsic biases and improve re-ranking performance.
5.2 WHAT COMPLEX RE- RE-RANKING SIGNALS DOES ICR LEVERAGE?
Our experiment results show that ICR dramatically outperforms RankGPT on several single-hop datasets (FiQA, FEVER, SciFact) and all multi-hop re-ranking tasks. In this section, we explain how re-ranking on these datasets requires more complex re-ranking signals than more conventional retrieval datasets and provide a comprehensive investigation of ICR's improved ability to use such signals. We use token-level ranking scores in our analysis similar to the previous section.
Query-Passage Contextualization. As opposed to other QA datasets BEIR like TREC-COVID. NFCorpus and NQ. which use stand-alone text such scientific abstracts and Wikipedia articles as their retrieval corpora, FiQA's retrieval corpus consists of human-written responses to the search queries in the dataset. As shown in the example in Table 6 this makes many passages only intelligi- ble when their corresponding question is present, which makes re-ranking these passages a specially complex challenge that requires deeper contextualization between passages and the query.
Table clearly illustrates ICR's improved detection of such query-passage contextualization. In the passage ranked 1st by ICR, which RankGPT ranks at the 10th the strongest signals come from the phrases "coupon (coupon rate) is", "price is"" "yield is" and "time is". indicating that the model is not only detecting lexical overlap but also attending the word "is". which is essential answering the query. Interestingly, the distractor passage, which ICR ranks 2nd and RankGPT ranks 1st shows weaker and more diffused attention patterns, indicating the model's uncertainty concerning this document's relevance to the query. We provide more such examples in Appendix G.2
Contradiction-Baan Based Reasoning. ICR shows exceptionally strong performance on FEVER and SciFact, two fact verification datasets. By breaking down the performance across claim types, we find that ICR does particularly well on claims which are contradicted by the retrieved passages. On FEVER, ICR using L improves performance over RankGPT on Contradiction examples by 26 points while only by 9 points on Support examples (SciFact sees a similar gap of 15 points to points). We note that the evidence required for refuting claims usually have lower lexical overlap with the query and instead rely on deeper reasoning processes which must consider both the search query and the passages, resulting in more challenging task.
In Table 7 we see that ICR can more effectively leverage these reasoning processes than RankGPT and thereby more accurately rank refuting passages. The first example in Table shows FEVER

---

Published as a conference paper at ICLR 2025
Table 7: In-context re-ranking scores for all tokens in a FEVER example with the query "Night of the Living Dead was originally directed by Krzysztof Kieslowski." B, R, denote ranking from BM25, RankGPT, and ICR respectively.
Passage
B R Support
Night of the Living Dead is 1968 American independent horror film directed by 11 28
True
George A. Romero starring Duane Jones and Judith O'Dea It was completed on
114,000 budget and premiered October
1968
The Scar ( (Blizna is a 1976 Polish film written and directed by Krzysztof Kieslowski 1 4 False and starring Franciszek Pieczka Filmed on location in Olechów Poland the film is about man put in charge of the construction...
Table 8: In-context re-ranking scores for all tokens in a MuSiQue example with the query "Who was the spouse of a leading speaker against slavery and publisher of an antislavery newspaper?" The bridge entity needed to reach all supporting documents is "Frederick Douglass."
First Hop
Sccond Hop
The North Star nineteenth century anti- slavery newspaper published from the Talman Building Rochester, New York by abolitionist Frederick Douglass.
Helen Pitts Douglass (1838-1903) Amer- ican suffragist and abolitionist, known for bo- ing second wife of Frederick Douglass. She also created the Frederick Douglass Memorial and Historical Association.
example in which ICR ranks the passage "Night of the Living Dead" the most relevant one by attending to the true director "George A. Romero", since it refutes the claim that it was directed by "Krzysztof Kieslowski". On the other hand, both RankGPT and BM25 rank another passage mentioning a film directed by "Krzysztof Kieslowski" at the top. probably due to its high lexical similarity. We provide more such examples for both FEVER and SciFact in Appendix G.3 G.3
Multi-Hop Information Integration. ICR obtains strong performance on multi-hop retrieval tasks, which requires integrating information across multiple passages. Through token-level analysis, we show that ICR assigns high ranking scores to bridge enfity tokens that are essential in the integration process. We demonstrate this phenomenon in Table by showing how ICR (Llama-3. 8B) scores each token example from MuSiQue (See more examples in Appendix G.4). The bridge entity "Frederick Douglass" provides a stronger than average signal in both the first hop and second hop documents, suggesting that ICR is leveraging the LLM's multi-hop hop reasoning signals.
6 CONCLUSION
We propose in-context re-ranking (ICR), efficient re-ranking method based on the attention dis- tributions of LLMs. Compared to generative methods, it only requires O(1) forward passes for re- ranking N documents, instead of least O(N) forward passes, significantly cutting the latency of re-ranking. Through comprehensive experiments, we demonstrate the strong zero-shot performance of ICR on both standard single-hop information retrieval benchmarks and challenging multi-hop hopp re- ranking tasks. Compared to RankGPT, ICR is particularly strong at handling more complex retrieval tasks. By directly leveraging internal signals within L.LMs, ICR substantially boosts the re-ranking performance of open-weight LLMs to match strong proprictary models.
Our analysis further reveals the abundant re-ranking signals carried by attention weights, and that ICR can effectively leverages such signals especially in challenging tasks. The evidence presented in our detailed analysis concerning ICR's improved ability to detect complex re-ranking signals strongly supports our hypothesis that the abundant re-ranking signals that exist within LLMs might not be optimally leveraged through generative approaches. These findings also highlight the poten- tial of developing better ways leverage open-weight models.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
The authors would like thank Yao Fu and Kai Zhang for their constructive feedback. We also appreciate the thoughtful comments from colleagues of the OSU NLP group. This work is supported in part by NSF OAC and NIH R01LM014199 The views and conclusions contained herein are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. government. The U.S. government authorized to reproduce and distribute reprints for government purposes notwithstanding any copyright notice herein.
REFERENCES
Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, 2015.
Vera Boteva, Demian Gholipour, Artem Sokolov, and Stefan Riezler. A full-text learning rank dataset for medical information retrieval. In Advances in Information Retrieval: 38th European Conference on IR Research, ECIR 2016, Padua, Italy, March 20-23, 2016, Proceedings 38, pp- 716-722. Springer, 2016.
Yiqun Chen, Qi Liu, Yi Zhang, Weiwei Sun, Daiting Shi, Jiaxin Mao, and Dawei Yin. Tourrank: Utilizing large language models for documents ranking with a tournament-inspired strategy, 2024.
URL https: /tttpp:///axiiee /ttrxvivvv... org abs 2406 11678
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. Specter:
Document-level representation learning using citation-informed transformers. arXiv preprint arXivv2004.077/80. 2020.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. Overview of the trec 2019 deep learning track. arXiv preprint arXiv:2003.07820 2020.
Thomas Diggelmann, Jordan Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. Climate-fever: A dataset for verification of real-world climate claims. arXiv preprint arXiv:2012.00614 2020.
Andrew Drozdov, Honglei Zhuang, Zhuyun Dai, Zhen Qin, Razieh Rahimi, Xuanhui Wang. Dana Alon, Mohit lyyer, Andrew McCallum, Donald Metzler, and Kai Hui. PaRaDe: Passage rank- ing using demonstrations with LLMs. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 14242-14252, Singapore, December 2023, Association for Computational Linguistics. doi: 10.18653/w//0022 findings-emnlp.5501 URL https::/aclanttooo wrg. findings emnlp.
95c
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models, 2024. TtLLhttps://attt
wrg. abs 24 407 21783
Isabel O Gallegos. Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernon- court, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. Bias and fairness in large language models: A survey. Computational Linguistics, pp. 1-79, 2024.
Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. Hipporag: Neuro- biologically inspired long-term memory for large language models, 2024.
Faegheh Hasibi, Fedor Nikolaev, Chenyan Xiong, Krisztian Balog, Svein Erik Bratsberg, Alexander Kotov, and Jamie Callan. Dbpedia-entin v2: test collection for entity search. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1265-1268, 2017.
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing muuli--hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint wrXiv:2011.0006/ 2020.
11

---

Published as a conference paper at ICLR 2025
Gautier Izacard and Edouard Grave. Distilling knowledge from reader to retriever for question answering. In International Conference on Learning Representations, 2021. URL https://
openreview hwtpeeewwwwwweeeeeeeeeeeeeeeeeee
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310006825. 2023.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.00 04906, 2020.
Tom Kw Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- King-Weii Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le. and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the doi:
Association for Computational Linguistics, 7:452-466, 2019,
00276. URL
org. wttpo/ooooooor//1111222200 Q09-10666
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. SOSP '23, pp. 611-626, New York, NY, USA, 2023, Association for Computing Machinery. ISBN 9798400702297. doi: 10.1 145/36000006336116665
URL https:
7/doi. org/1000rg10.. 1141 0111111443100 . 3613165
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-aie -urgeeee- genera- tion for knowledge-intensst nlp tasks. Advances in Neural Information Processing Systems, 33: 1459-9474, 2020.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024.
Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. Zero-shot listwise document rerank- ing with a large language model, 2023.
Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. Www'15 18 open challenge: Financial opinion mining and question answering. In Companion Proceedings of the The Web Conference 2018, WWW '18, pp- 1941-1942, Republic and Canton of Geneva, CHE, 2018. International World Wide Web Con- ferences Steering Committee. ISBN 9781450356404. doi: 10.1145/3184558.11222
URL
httpss://oi.osz orw.:/...o.0... /ttpg//0...11111118888111331
Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. Multi-stage document ranking with bert. arXiv preprint arXiv: /9/0. 1910.1444.. 2019.
Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. Document ranking with a pretrained sequence- to-sequence model. arXiv preprint urXiv:2003.06713 2020.
OpenAl. Hello gpt-4o. July 2024.
URL
gpt mini advancing cost efficient ientttiiteelggeeeee
Alexander Peysakhovich and Adam Lerer. Attention sorting combats recency bias in long context
language models. ArXiv, abs/2310.014227, 2023.
org/CorpusID: 263609111
Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. Rankvicuna: Zero-shot listwise docu- ment reranking with open- large language models, 2023a. URL/btts:://axxy org/
abs/ 200...309... 15088
12

---

Published as a conference paper at ICLR 2025
Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. Rankzephyr: Effective and robust zero- shot listwise reranking is a breeze! arXiv preprint arXiv:2312.02724, 2023b.
Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang. and Michael Bendersky. Large language models are effective text rankers with pairwise ranking prompting, 2024.
Revanth Gangi Reddy, JaeHyeok Doo, Yifei Xu, Md Arafat Sultan, Deevya Swain, Avirup Sil, and Heng Ji. First: Faster improved listwise reranking with single token decoding, 2024. URL htttp::/arxiv.orr//aat
15657
Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr, 3(4):333- April 2009. ISSN 1554-0669, doi: 10.1561/1500000019
URL http:: /ww. wttp.o........00010..00000
Devendra Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. Improving passage retrieval with zero-shot question generation. In Pro- ceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pp. 3781-3797, 2022.
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. Col- BERTv2: Effective and efficient retrieval via lightweight late interaction. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, pp. 3715-3734, Seattle, United States, July 2022. Associa-
tion for Computational Linguisties. doi: 10. 12111166///22222222 naacl-main.211 URL. https:
//aclanthor ooy. org. /ttgg//22222222222/ nwa.l maaac1-main.27.
Sofia Serrano and Noah A. Smith. Is attention interpretable? In Anna Korhonen, David Traum, and Lluís Märquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pp. 1931-2951, Florence, Italy, July 2019. Association for Computational Lin- guistics. doi: 10. 10.18653/v1/P19-1 1282. htt.ht/ps///cllaanooo...oor/////
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is ChatGPT good at search? investigating large language models as re-ranking agents. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 14918-14444 8-14937, Singapore, December 2023. Association for Computational Linguistics. doi: 10. Btt.3///2223.emlp-mmee 923. URL, hittps:/aacanthoolee
2023. emnlp ma in.. 923
Raphael Tang, Xinyu Zhang, Xueguang Ma, Jimmy Lin, and Ferhan Ture. Found in the middle: Permutation self- consisensy improves listwise ranking in large language models. arXiv preprint arXiv:2310.07712 2023.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL hhttps://oppee/ee
wtt.::/ooorrmmmoooooeettere jeJ.
James Thome,, Andreas Vlachos, Christos Christodoulopoulls, and Arpit Mittal. Fever: large-scale dataset for fact extraction and verification. arXiv preprint arXiv: 1.X3:18803 2018.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554, 2022.
Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Cemmner-Fushmant William R Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. Trec-covid: constructing pandemic information retrieval test collection. In ACM SIGIR Forum, volume 54, pp. 1-12. ACM New York, NY, USA, 2021.
13

---

Published as a conference paper at ICLR 2025
David Wadden, Shanchuan Lin, Kyle Lo, Luey Lu Wang, Madeleine van Zuylen, Arman Co- han, and Hannanch Hajishirzi. Fact or fiction: Verifying scientific claims. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference Empir- ical Methods in Natural Language Processing (EMNLP), ) pp. 7534-7550, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.1888 http5/////2000000mmmmmmoo/ 609. URL
https://aclantholoogr wwwp://wwooorgg/.oo.oooo emnip ma mainn. 609
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Large search model: Redefining search stack in the era of llms. In ACM SIGIR Forum, volume 57, pp- 1-16. ACM New York, NY, USA, 2024.
Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.). Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 11-20, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/11/D19-1002 URL attps://aclantinn
org/D19-20 1002
Thomas Wolf, Lysandre Dehut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. As- sociation for Computational Linguistics. htth::/:: www. aclweb. org/anthology/
22200. emn 1omnnl-ddee demas.....
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09000 09600, 2018.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning, pp. 12697-12706. PMLR, 2021.
Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, Zhicheng Dou, and Ji-Rong Wen. Large language models for information retrieval: A survey, arXiv preprini arXiv:2308.07111 000 2023.
Honglei Zhuang, Zhen Qin, Kai Hui, Junru Wu, Le Yan, Xuanhui Wang, and Michael Berdersky. Beyond yes and no: Improving zero-shot Ilm rankers via scoring fine-grained relevance labels. arXiv preprint arXiv:2310. 121.:222..1 2023a.
Shengyao Zhuang, Bing Liu, Bevan Koopman, and Guido Zuccon. Open-source large language models are strong zero-shot query likelihood models for document ranking. arXiv preprint 12X2::3310.13243. 2023b.
Shengyao Zhuang. Honglei Zhuang, Bevan Koopman, and Guido Zuccon. A setwise approach for effective and highly efficient zero-shot ranking with large language models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '24, 2024.
14

---

Published as a conference paper at ICLR 2025
APPENDICES
A COMPLEXITY OF LLM-BASED RE-RANKING METHODS
For fair comparison of complexity in of forward passes (FPs), we assume the LLM used for re-ranking has long enough context window to process all input documents at once. We consider two types of forward passes in the LLM inference process: those in the prefill phase and those in the decode phase. The complexity of re-ranking method is the sum of the number of forward passes needed prefill and decode phases.
We note that, the generation targets of generative methods, such document identifiers ranking scores, are typically much shorter than the input context. Therefore, we can approximately treat the cost of FPs in the decode phase as constant.
#Prefill FPs per API call
#Decode FPs per API call
Method
#API calls
Total FPs
pointwise
O(N)
O(1)
O(1) O(1)
O(N)
pairwise
O(N) ~ O(N2)
0(1)
O(N) ~ O(N3) O(N)
listwise
0(1)
0(1)
O(N)
ICR(6 (ours)
((1)
O(1)
N/A
O(1)
Table A.I: Complexity of different re-ranking methods in terms of forward passes. N is the number of documents to re-rank.
We list the number of API calls and FPs in each phase per API call required by different re-ranking methods in Table A.1 As illustrated, pointwise and pairwise methods require at least O(N) LLM API calls, leading
total of ((N) FPs in both prefill and decode phases. Listwise re-ranking methods only invoke the LLM API once, but needs to use O(N) FPs in the decode phase for gen- erating a list of N document identifiers. Similar to listwise re-ranking, ICR also only needs O(1) LLM API calls. However, by fully leveraging the signals produced in the prefill stage, ICR avoids the decode stage, leading to an overall complexity of only O(1) FPs.
B PROMPTS
We present exmaple prompts used ICR and RankGPT in our experiments.
B.
ICR PROMPT
Table B.2 shows example prompt for ICR HotpotQA using the QA style instruction. Table B.3 shows an example prompt for ICR on FEVER using the IE style instruction. The prompt is wrapped by LLM-specific special tokens <prefix> and <suffix> correctly leverage their instruction following abilities, such as [NNST] and 
 for Mistral 7B.
B.2 RANKGPT PROMPT
Table B.4 shows an example prompt for RankGPT on TREC-COVID. The prompt is also wrapped by LLM-specifie special tokens. In addition, add Ranked Passages: after csuffix" to increase the ranking success rate. Still, open-weight LLMs often fails to generate usable ranking.
15

---

Published as a conference paper at ICLR 2025
Table B.2: An example prompt for ICR on HotpotQA using the QA style instruction.
sprefLxoftere are some pazagraphs. Please answer the question based on the celevant information the paragraphs.
[1] Document 1 Chris Sale
christopher Allen Sale (born March 30, 1969), nicknamed The Condor, is American profeasional baseball pitcher for the Bostonn Red Sox of Major Leaque Raseball selected 13th overall
the 2016 Major League Baseball draft by the Chicago White 8ox and made his MLD debut with them in 2010. lle is a six Lime Al 1-Sta,, and led the American Teague in strikeouta 2015. Prior to playing profeseionally, played college baseball for Florida Gulf Coast Univeraity.
(MBB)
[20] Klay Thompson Klay Alexander Thorpson professienal basketball pleyer for the Solden State Kerriers of the National Basketball Association (NBA), The son of former NBA player Mychal Thorpson, played college basketbell for three seasons Kashington State Universicy, where he was a two-tine first-team all-confereace selection in the Fac-10. Thompson was selected in the first round of the 2011 overall pick. In. 2014, he and teanmate Stephen Curry set a then NBA record with 484 combined three pointers Brochers". Thompsen is three-time NBA All-Star and a two-time All-NBA Third leam henozee, In 2015, he helped lead the Marziora to their fiest NBA Championship since 1975. Thompson helped the Narriors year in 2017, winning his second NBA Championship.
Peorcacy 8, 1990)
American
draft Golder State with the 11th
seascn, as the pair weze giver the
"Splash
to the NBA Pinals for third straight
Query: What reiationehip does Fred Gehrke have to the 23rd overall pick in the 2010 Majoz League Baseball Draft?<suifix>
Table B.3: An example prompt ICR on FEVER using the IE style instruction.
sprefLxollere are some paragraphs. Please find Information that are relevent to the query.
[1] Okraine and the United Nations
The Ikraine was one of the founding members of the United Nations when is joined in 1945 the Ukrainien Sovies Sceielis: Republic along with the Byelcrussian Seviet Socialist Republic signed the United Nations Charter when they ware part of Soviet Union After the dissolution nexly-independent ukraine retained its sest
Soviet Unien in 1991 the
[20] Council Feeple's Ninisters
The Counct of People 's Miniscers of Ckraina was the main exacutive inssitution cuties and fonctiona the Chapter v of the Constitution of the Ukrainian National Rapublic It was the January 25 1918 aftes the retura of the Uxzainian
of Ukrainian People Republic
outlined
reorgani sed out the General Secretariat of Ukraine upon the proelamation 4th Universal and Independence delegation from the preliminary pesce talks from Brest- equal-rightfil participant and
As the preliminary
talks in Brest Ukraine acheduled to finalized the treaty on February 9 19"8 Until the end of the month January 1928 pledged mini sters.
recognized
the General Secretariat continued seive as full
Query: Ukrainian Soviet Socialist Republic founding participant of the UK. U8.<suffix>
C LIMITATIONS OF ICR
C.1 ICR STILL SUFFERS FROM LEXICAL BIAS
Given that ICR sometimes underperforms RankGPT in our experiments, we use the same analysis tools for understanding its limitations.
Firstly, we find that ICR signal is highly concentrated: most of the re-ranking score comes from a small number of tokens in few documents. This characteristic leads to most documents receive too little scores to be differentiated. Secondly, we find that this signal is mostly concentrated in phrases that are lexically similar the query.
16

---

Published as a conference paper at ICLR 2025
Table B.4: An example prompt for RankGPT on TREC-COVID.
spreflx>Thia is an intelligent assiatant that can rank paasayes based on their relovancy to the query.
the following are 20 passages, each indicated by number identifier []. can zank them based their relevance quezy: "what is the origin of coviD-19"
[1jorigin Novel Coronavirus (COVID-19); Computational Blology Study using Artificial Inte:ligence
Crigin of the COVID 19 virus has been intensely debated in the scientific community since the first infected cases were detected in December 2019. The disease has caused global pandemic, leading to deaths thousands of paople acress the world and thus f'nding origin of this novel coronavi rus is important in responding and controlling the pandemic....
[20] Papa Biovanni xxi11 Bergamo Hospital at the time of the C0v13-19 cutbreak: letter from the warfront.
In early December 2019, the 2019 novel corenavirus (COVID 19) identified as the agent responsible for the first pneumonis cases of unkzown origin in Wahan, the dapital of the llbbei region in China. The virus has been identified as a novel that has been promotly named SARS-CoV-2 (seveze
enveloped R&A betacoronavirus? acute respirstory syndreme corenavirus 2). The World Bealth organization (WHO), on January 12, 2020 deslared the COVTD-19 public health emergency of internati onal concera. On March 11, the NHO made the assesament that COVID-19 can be characterized as pandemic.
The search quezy is: "what is the origin of COVID 19", will rank the 20 passages above based their relevance to the search query. The passages will be lissed in descencing order using identifiers, the moat relevant passages should ba listed first the output format should be List all 20 ranked passages and do not explaia your ranking until after the 1ist is dene, fnne.cauff Passages:
[] etc, e.g., [1] [2] etc. Be
Table C.I: In-context re-ranking scores for tokens in several DBPedia-Entity examples. B, R, and denote ranking from BM25, RankGPT, and ICR respectively.
Query
Passage
B
R
Support
Give me all launch pads
71 Launch guil C A launchi padl is an above -ground platform from which rocket powered missile space vehicle vertically launched A space port launch complex is facility which includes, and provides required support for one or more launch puds
29
31
False
operated by
NASA.
192 ( Cape Can av eral Air Force Station Launch Complex 19 C Launch Complex 19 (LC- is desctivated launch on Cape av eral Air
8
2
16
True
Station. Florida used by NASA to launch all of the Gemini manned space fl ights It was also used by unmamned Titm I and Titan II missiles JL C- 19 was in use from 195 during which time
196 manned. The first use of LC - 19 was on August 14, 195 9. This was a Titan I and the mission was declared failure rocket exploded while on the pad.
launches. 10 which
This bias leads to ICR's 's lower performance in standard information retrieval tasks where document relevance depends on much more than pure lexical similarity. Additionally, ICR's performance on DBPedia-Entity and 2WikiMultihopQ) construction that leads to many distractor documents containing entities with high lexical overlap with the query, and therefore getting higher scores from ICR.
suffers from this limitation due to its entity-centric query
While this lexical bias appears to overpower other signals in ICR, it remains unclear whether more comprehensive document understanding signal will be unveiled if we adjust the aggregation strategy. We hope to explore this interesting direction future work.
C.2 OTHER LIMITATIONS
This work is limited to using decoder-only language models, due to their popularity and strong performance. We leave extending ICR to other language model architectures to future work.
17

---

Published as a conference paper at ICLR 2025
The proposed method relies the availability of attention weights from the base LLM, thus lim- iting its applicability to proprictary LLMs with only API access. While in-context re-ranking shows promising performance etary LLMs still has the best performance. It would be interesting to test the performance of ICR on stronger models with sfate-of-the -frt--rr- performance.
recently developed open-weight LLMs, RankGPT with propri-
Our mechanistic analysis of ICR reveals its vulnerability to syntactic similarities. This poses a potential risk in applying ICR practical usage. For example, content providers could inject certain phrases to trick ICR for better ranking. Future work could further enhance ICR's robustness.
D ROBUSTNESS TO INPUT DOCUMENT ORDER
Table D.2: Re-ranking performance given different document input orders. We report nDCG@ 10 on Trec-Covid and Recall @5 on MuSiQue. For random order, report average performance and standard deviation across different random seeds.
Order
Covid
MuSiQue
Retriever
59.5
37.9
RankGPT (Llama3.1 8B)
Retriever Retriever
72.6
52.6
ICR (Llama3.1 8B)
66.8
56.8
RankGPT (Llama3.1 8B)
Reversed Retriever
72.9
50.5
ICR (Llama3.1 8B)
Reversed Retriever
72.8
55,9
RankGPT (Llama3.1 8B)
Random
65.6 (2.58)
49.8
(0.72)
ICR (Llama3.1 8B)
Random
69.8 (1.66)
55.6 (0.45)
To study the robustness of different methods against position bias, we present the re-ranking perfor- mance of RankGPT and ICR on single-hop and multi-hop datasets with different input orders. We evaluate re-ranking performance when input documents are in the order returned by the retriever, in reversed order of the retriever, or in random order. the random order setting, we use random seeds and report average performance well as standard deviation.
As shown in Table D.2, both RankGPT and ICR work the best when the input is ranked by the re- triever. We observe that ICR works better when using the reverse order single-hop datasets, possi- bly due to the recency bias of LLMs. When the input is randomly shuffled, ICR shows considerably higher performance and lower standard deviation than RankGPT, indicating stronger robustness.
E ICR + COLBERTV2 RESULTS ON BEIR
Our main experiments showed the effectiveness of ICR when re-ranking results of BM25. Here, we demonstrate that ICR also improve results produced by stronger dense retrieval methods, such as ColBERT v2. We report results with Llama-3.1 8B on the BEIR benchmark in Table E.3
Table E.3: 1.ama-3.1 8B's re-ranking performance (nDCG@ @ 10) the BEIR benchmark with dif- ferent retrievers. SR: Success rate, the chance that a re-ranker's output format is correct. We bold the best performance for each task with each retriever.
TREC.
NF Corpus
DBPedia- Entity
SeiFact SeiDoes FIQA
FEVER
Climate- FEVER
NQ
Micro- Avg
Macro- Avg
SR
BM25
59.5
32.2
31.8
67.9
14.9
23.6
65.1
16.5
30.6
44.6
38.0
RankGPT
72.6
33.7
41.4
68.4
17.9
31.5
66.7
22.7
51.1
51.9
45.1
99.2%
ICR
72.8
34.7
35,3
76.1
17.1
38.1
84.5
23.1
51.2
60.5
48.1
100%
ColBERT v2
65.7
33.2
44.3
67.5
15.1
35.6
78.3
18.3
54.5
57.6
45.8
RankGPT
77.4
34.1
47.6
66.1
17.4
67.7
21.9
60.1
54.7
47.5
99.2%
ICR
74.6
35.8
40.5
74.4
17.4
40.9
86.6
22.7
59.7
63.7
50.3
100%
When used with ColBERTv2, ICR still brings a substamtial performance improvement of 6.1 points in micro-average and 4.5 points in macro- averreee We also notice that ICR shows the largest advan- tage over RankGPT more reasoning intensive tasks, such as SciFact, FiQA, and FEVER. while underperforms DBPedia-Entit, This is consistent with our observations based on BM25.
18

---

Published as a conference paper at ICLR 2025
F COMPARISON WITH SUPERVISED AND SETWISE APPROACHES
To understand how ICR compares with other more costly re-ranking methods, such as the supervised fine-tuned Rank Vicuna (Pradeep al., 2023a) and the more complex setwise re-ranking method Zhuang et al. 2024), we compare ICR's performance with them in Table
Table F:4: Re-ranking performance (nDCG@@ 10) four BEIR datasets. *: Results copied from
Zhuang et al. 2024
NFCorpus
DBPedia-Enin Entity
SciFact
Trec-Covid
BM25
32.2
31.8
67.9
59.5
RankGPT (Llama 3.1 8B)
33.7
41.4
68.4
72.6
ICR (Llama 3.1 8B)
34.7
35.3
76.1
72.8
Vicuna
34.7
44.4
71.3
80.7
Setwise.heapsort (T5-XL)*
35.2
42.8
67.7
75.7
Setwise.bubblesor (TS-XL)*
35.3
43.8
69.1
75.65
RankGPT (3.5-turbo)
32.8
42.0
64.1
74.1
RankGPT (4o-mini)
38.0
44.9
77.0
79.1
We notice that Vicuna and setwise methods show better performance on Trec-Covid, NFCorpus and DBPedia, as expected. Both methods incur additional cost in training (Rank Vicuna) or inference time (Setwise methods). Similar to our findings in the main text, ICR shows impressive performance on SciFact, outperforming supervised methods without specialized training.
G MORE QUALITATIVE EXAMPLES
G.1 INTRINSIC BIASES OF MISTRAL 7B
We present the intrinsic biases captured by the calibration process with Mistral 7B in the same way in Section 5.1 Similar Llama-3.1 8B, Mistral 7B also exhibit strong position bias both sides of the context window, shifted to the front. Mistral also demonstrate a strong bias towards document identifiers, title, entities, and punctuation.
Table G.5: An example of token-level calibration score in ICR (Mistral 7B). Deeper color indicates higher score.
20.0
17.5
12 J Long Sp ru ce Gener ating Station (0x0A) It was Man it oba H ydro fourth generating station to be built on the Nelson River which flows from Lake inn ipe g Hudson Bay The station built Long Sp ru ce Rap ids The site is approximately east of Gill am Man oba and down ydro K ett le Gener ating Station
Avg
$ 11
Position
of Man it oba H
Figure G.1: Calibration Score of Mistral 7B.
G.2 QUERY-PASSAGE CONTEXTUALIZATTOO
As discussed Section 5.2 due to FIQA's construction, many of its passages are not easily inter- preted without context from their corresponding query, making the re-ranking process more chal- lenging. In T.6l we illustrate FIQA's more complex nature with three additional examples that also include ICR and RankGPT performance well as ICR's per-token re-ranking scores. In the first query, we see that ICR detects that Llama-3.1 strongly attends to the question's 's answer: "Syrian refugee situation in Europe". even though the phrase has direct semantic relation to the query. In contrast, RankGPT ranks this document at position 22 and elevates a distractor to the top position though is not related to the query since mentions the Berkshire Hathaway shareholders
19

---

Published as a conference paper at ICLR 2025
meeting directly. Similarly, in the second and third queries, ICR detects that Llama-3.1 attends to tokens and phrases which are directly related to answering the query: stating that nothing would happen to the user's bank account in the first query and that providing services to Germany from Poland (another EU country) without paying taxes would be allowed. On the other hand, RankGPT unable re-rank these passages correctly and instead opts for passages which have significant semantic overlap with the query but are actually answering completely different queries.
Table G.6: In-context re-ranking scores for all tokens in several FIQA examples. B, R, and denote ranking from BM25, RankGPT, and ICR respectively.
Query
Passage
B
R
Support
What to ask
For whatever it's worth, when went the meeting couple years ago, the ques-
61
22
Truc
Warren
Bon and answer segment is mosily students asking how to pack stock or what book
Buffet
they should read. I'm someone else will ask but would he interesting hear
Berkshire
their take on the Syrtan refugee situation Europe and how it may impact the EU in
Hathaway shareholder
general....
meeting?
"* "Autonomous
would us they spread trucks"" Buifeet sharehold:
ars Berkshire Hathaway (BRK.A) annual mesting in May. If self-driving tracks become predominant on the reads, could steal business from Berkshire owned rail- road Burlington Northern, Bullett hinted. &gt: Bulfett acknowledged that autenemouu cars are "comnig"" "", and could also "hart" Borkshire's 's insurance business Gcico.
If the U.S.
You must mean the current debt geiling debacle. The meaning of it is: US govemmenn is coustantly borrowing money (by issuing treasury bonds) and constantly repaying of the bonds that
16
20
True
defaults on its debt, what will happen bank money?
to maturity..... Going back to bank aecount. most
probably nothing would happen to the money you store therg, Even if the bank hud invested 100% of the money US treasury bonds (which doesn't really happen).
suvereign ileii: What happens when Greece defaults. Here are few things: Every bank in Greece will instantly go insolvent. The Greek govemment will nationalise every bank in Greece. The Greek government wall forbid withdrawals from Greek banks.
Andrew Lilico
likely
for
Greece Refanlss
Does freedom provide services allow
country for than half year you'se te generally considered to be its resident for tax purposes. Germany is no you providé the services from Poland, and never set foot in Germany they won't I word.
You't free provide services,
you stay
16
True
exception
true almost any country
me contracting in
Germany
without
"Yon" 'Il need to read carefully the German laws on tax residency. in many Fumpean (and other) laws loss residency due to absence conditioned noquiring residency elsewbere. But in general. it is possiblc to use treaties and statuses so that you end up not being resident anywhere,
68
13
False
paying taxes there (but in my
doesn't mean that the income no immediately before
longer taxed. Keep in mind, that the treaty has ""who is visiting Contracting State a resident of the other Contracting State"" language in relevant cases, you may still apply it in the US even if longer resident Germany."
country)?
20

---

Published as a conference paper at ICLR 2025
G.3 CONTRADICTION BASED REASONING
In this section, we provide more examples of ICR's improved performance in contradiction-base reasoning examples within fact verification datasets such FEVER and SeiFact, as explained in Section 5.2 Some FEVER examples are provided in Table G.7 The first example shows that ICR leverages Llama-3.1's 's attention patterns on Jiang Wen's many talents as an actor, screenwriter and director to refute the the claim that he is exclusively producer. On the other hand, RankGPT in- correctly re-ranks this document to position 13, opting instead for documents where "Jiang Wen" is mentioned in conjunction with producers because of semantic overlap. In the second example, RankGPT is strongly biased towards matching the year "1995" and thus re-ranks a completely un- related passage to the first position. In contrast, ICR is able to detect more subtle attention patterns placed by L on the year "1975", which is the true year in which Balibo started.
In Table G.8 we include some contradiction-bute based examples from the SciFact dataset. Although they are much more Nomain-speciffr and challenging, we find that ICR also demonstrates the en- hanced ability to leverage deeper reasoning signals which see in the FEVER dataset. In the first query. we are looking for statements related to the enzyme aPKCz and how it affects tumor growth, like the last sentence in the first passage. This sentence claims that "PKC deficiency", not PKC, causes cancer cell enhancement through the use of glutamine, effectively refuting the claim. From the attention patterns shown in the table, it seems that ICR is able to detect that the Llama-3. model attends to this contradictory sentence and the passage as a whole more strongly in its attempt to verify this claim, bringing from position 8 to position while RankGPT brings down to position 9. In contrast, RankGPT elevates the passage in the second row to the first spot, most likely due to more lexical cues since the passage discusses how the KRAS protein affects glutamine metabolism and yields tumor enhancements.
The second example in this table shows a claim concerning how interferon-induced genes reduce granule cell survival when exposed to West Nile virus. Even though both the gold document and the distractor are quite similar to the claim in this case, RankGPT is unable to detect that the first genes" increase granule cell survival, exactly the
passage actually finds that opposite of our claim. From the attention patterns, we see that LIama-3.1 is strongly attending to this passage whole as well concentrating attention the last sentence which is the most relevant to the claim. On the other hand, we see that the attention patterns of Llama 3.1 in the distractor passage are more concentrated on lexical similarities such as the "West Nile virus".
Table G.7: In-context re-ranking scores for all tokens in several FEVER contradiction exam-
ples. B, R, and I denote ranking from BM25, RankGPT, and ICR respectively.
Query
Passage
Support
Janng Wen exclusively producer.
Jinng Wen born January 1963 ) Chinese film actor screenwriter and Birector As director, he sometimes grouped with the Sixth Generation that emerged in the 1990s
2
13
True
Emperot Motion Pictures known as EMP) is Bim produces and distributor part the Emperor Group. . Following the box ollice
32
1
2
False
Elfect Medallion EMP has produced The Sun Also Rises and Forever Enthralled, two works by renowned Chinese auteurs Jiang Wen and Chen Kaige) ...
Balibo (film) starts in the year 1995.
Balibo (ilm) 2009 Australian war film that follows the story the Balibo Five : group of joumalists who captured and killed while reporting activities just prior to the Indonesian invasion of East Timor of 1975
19
True
Bjarnar Teigen (born 29 June 1971) is Norwegian actor; theater director and play- wright.| Teigen studied acting in the Oslo National Academy of the Arts and started working actor Molde Teatret Vart after graduation 1995
26
62
False
21

---

Published as a conference paper at ICLR 2025
Table G.8: In-context re-ranking scores for all tokens in several SciFact contradiction- exam- ples. B, R, and I denote ranking from BM25, RankGPT, and ICR respectively.
Query
Passage
Support
aPKC2 causes
Control of Nutrient StressLadueed Metabolic Reprogramminn by PKCÇ in Tumori- genesis Turnor; cells have high energetic and unabolic needs and known to adapt their metaholism to be able to survive and keep proliterating under conditions of mu- trient stress. We show that PKCC deficiency promotes the plasticity necessary for cancer oolls to reprogram their metabolism to utilize glutamine through the scrine biosynthetic pathway the ahsence glucose. [..]]
8
True
tumour
enhuncement
by affecting
glutamine
metaholism.
Glutamine supports pancreatic caneer growth through Kras-regulated metabolic patlway, Cancer cells have metabolic dependencies that distinguish them from their normal counterparts. Among these depentencies is an incressed use of the amino
Folsc
glotamine fuel anabolic processes. Indeed. the spectrum of glutamine- deni tumours and the mechanisms whereby glutamine suppuets cancer metabolism
remain areas of active investigation, [-.] Moreover, knockdown any component
enzyme in this scries of reactions also results in pronounced suppression of PDAC
growth in vitro and in vivo. Furthermere, we establish that the reprogramming of glu-
jurnine inelabolism mediated vncogenic KRAS, the signature genetic alteration PDAC, through the transcriptional upregulation and repression of key metabolie enzymes in this palhway. I...
Rapid
Differential innate iomone response programs in neuronal subtypes determine sus- ceptibility to infection in the brain positive stranded RNA viruses: Allhough ceptibiliity of neurons in the bruin microbial infection is a mujor determinunt of clin- ical outcome, little known about the molecular factors governing this vulnerahility. Here we show that two types of neurons from distinct brain regions showed differen- tial permissivaty replication of several positive stranded RNA viruses. Granulé neurons of the cerehellum and cortical neuroms from the cerebral contex have unique innate immune programs that confer differential susceptibiiityy to Viral infection ex and viva. By transducing cortical neurons with genes that more highly in granule cell neurons, we identified three interferon stimulated genes (ISGs; 1627, Trgl and Rsad2 (also known Viperin)) that mediated the antiviral ct- focts against different ncurotropic viruses. Morcover, we found that the opigenetic state and microRNA (miRNA))meddiale regulation ISCs correlates enhanced antiviral response in granule cell neurons, [..]]
3
Troe
up-regulation and higher
basal
expression of interferon-
induced
genes reduce survival of granule cell neurons that are infocted by West Nile vinas.
expressed
Beta interferon controls West Nile infection and pathogenesis in mice. Stud- ies with mice lacking the common plasmu membrane receptor for type interferon (IFN-a,3R()X0X have revealed that IEN signaling restricts tropism. dissemination. and lothality after infection with West Nile virus (WNV) or several other pathogenic viruses. Consistent with ral titers in ex vivo cultures of macrophages, deadritic cells, fibroblasts, and cerebellar granulé cell neurons. but not cortical neurons, from IFN-Bt-X01() mice were greater ihan in wild-type cells...
1
False
for I-N-8 control WNV replication. vi-
G.4 MULTI-HOP REASONING
As discussed in Section 5.2 our token specific analysis indicates that ICR's strong multi-hop re- trieval performance comes from the knowledge consolidation abilities of LLMs. In Table which shows several examples of correct in-context re-ranking by Llama-3.1 8B across three multi-hop benchmarks, it is quite evident that the bridge entity provides a strong signal correctly re-rank the second hop document. It is interesting to note that in both the third and fourth examples, the final answer (both years of death) also provides additional strong signals for our method. This suggests that, in some cases, LLMS are going even farther in terms of knowledge consoliaation than merely identifying the correct bridge entity.
22

---

Published as a conference paper at ICLR 2025
Table G.9; In-context re- re-ranking scores for all tokens in several multi-hop hhp examples from all three multi-hop benchmarks which Llama-3 ICR re-ranks correctly.
Usery
Hep
Wis meoct Conrado?
Ougre
State
Reaty Commat aiccoal Dasgat Stad Uxiversity grada: with tack
and slbeaa puseec Oeeag Copical
MasiQue
ixhurtal
coutis Beve Stundry Cloomuas Online Inc. Eine
Ytar wite comquiys
Bocing tweety your
When did milturr turma-- Lary Aloll adacasd?
Unavemitt
in tinuss whokeehhp man Morth Trmes grownd by the patiider
Luny Keal
Axpus
BUTC Phinypows begas when Philip:
MasKja
B the Unbwoddy at the Patigernni The enteenary
Rance Roces, he otraned dagne Buchiler
Boand of Rogann than maale reprorccaton with Lraied
Uhinasite thataponna (UB)
socovll scrvace Usiel States Arst
other who book dutes peotosior Miltior
Seephin
May tlaptih loven relenicin Wor- teck was lean Boatersoon Hanpoirre
This Hilly Billen boms
pesticcionall dincer while
dalling ail the eqaiine skanctype the unde bulki
canc:: Cooars Barzan,
esita, Eigiaaal
1981
fall and dracel by Skppes Dakky with stask: soke
posei Bispiin Wesnee
War Adas Makdizse
Kabaitz Makdiooe Mury Aspedt 12 Murch 1881)
Kabsiias Masildd Murry Benuchange Zaelani maderria dien ovey orier and jowt with waa
Fig rece then 50 hools and thoe an esoos and
sinos on Manoaaoo exdil laaae, pollitss rel glna
bom and trougge up cakonial New Sea seote
rxanbared An his mvscaiie Kallamn Maniiial
New Zeatadd and scakal Engkadd, whbc
schoe he maanee 1919 stoood bashand his
fow aiee fraer wiree eath as Laccorace
1917. de thined lacr
As. the 464s Ouassao
Hlpkax:
Espkaa de Soath glaun by is Assriiaa alcomaate
Scnmi Saace Assy cake strlund ca "I Naice Arsay"
HopeetjA
whin Suizaa wash indadad de asu Seres Natza Anay.
V Recack,
South sihan
call acdldn and consennca nacoon, guttenng qecenal
HElettan in Masch 2003 and rashed candirr dac es
Albust
Aloomane
Mosken
suinasie that postion
Klner
1008
pertirecug
at No. 6 reeta 1. "aaliwer"" them and rogoing the chatt.
ongorite decadion the ure thor trase surl moriove
conmocstall well and was de finazze Horast for
Bir
HaqpotjA
ingleed by the 1975 lis warling when?
stevidee sniles Fut died on CRS from lineary 36,
ahoa a Stuthee faeily non looring Reane
lots Fobwary The skmn alvd of
wox reavsied yours lass popake ken
ipral de 1939 Hill (Mdccrrancc I jracc
zach
rouied Wadden reary shretied civilar
Mutiaun staree wull fether Bulonn
concepts
righeve mataer als focaed mopuuticcen ring wwwllggen ailig
elate fndeal spees
nimas for the caline manie bocn of dh 1911 dusing whids bh
mendiackk reloct
23


Reviews:
[{"review_id": "kUg2r5LXuQ", "reviewer": "Reviewer_NMvY", "summary": "The authors focus on adopting large language models (LLMs) as zero-shot rerankers. I believe this is crucial because if we also need to train the LLMs as rerankers, there would be no difference from the previous framework. The authors propose leveraging the intrinsic attention scores to aggregate into a document score, along with a standard debias score. ***The authors claim that the proposed method operates in O(1) LLM forward passes (This is a point I am skeptical about and would like the authors to clarify in their discussion).*** I will first assign a threshold score and then adjust it based on the authors' rebuttal.", "questions": "1. A listwise LLM-based ranker typically requires O(N) forward calls because it often assumes that the number of candidate documents, N, is too large, exceeding the LLM's context limit. This necessitates the use of a sliding window for multiple forward passes. Given the same number of documents, how does the proposed ICR achieve this with only two forward passes? If RankGPT requires O(N), I believe the proposed ICR would also require O(N) with the same LLM.\n2. Since RankGPT is the only used baseline, the TREC 19 and 20 datasets tested in the RankGPT paper should also be included in the experiments.\n3. ICR utilizes attention scores from all transformer layers. Could you provide results from an ablation study showing the outcome when using only the attention scores from the final layer?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The writing of this paper is well-crafted, allowing readers with relevant backgrounds to quickly follow and provide feedback.\n2. The proposed method is versatile and applicable, making it suitable for use with open-source LLMs.", "weaknesses": "1. The claim of O(1) LLM forward passes raises significant questions. Please refer to my Question 1 for a detailed explanation, as this will determine the overall quality of the paper.\n2. A limitation of this method is that ordinary users cannot implement it using advanced commercial large language models, especially when compared to RankGPT. However, I believe this is due to commercial factors rather than technical ones.\n3. When only comparing against a single baseline, RankGPT, the classic datasets TREC 19 and 20 are notably missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G7M5AVu1oF", "reviewer": "Reviewer_d1gB", "summary": "This paper proposes a novel re-ranking method based on LLMs called ICR. ICR first aggregates the attention weights received by each document token from all query tokens. To mitigate intrinsic biases of LLMs, ICR calibrates the ranking scores by subtracting the attention scores obtained by using a content-free query. ICR is evaluated on both single-hop and multi_-hop re-ranking tasks using open-source LLMs, Mistral and LLaMA 3.1.", "questions": "1. The analysis in Figure 4 lacks details on how results would change if the document input order were adjusted. Additionally, if the document set contains similar documents, would these similar documents receive higher weights?\n\n2. In Section 3.3, how does the method address biases introduced by documents of varying lengths?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The problem is important, and the overall writting is clear. \n2. The proposed method is easy to follow and demonstrates effectiveness on two public LLMs.", "weaknesses": "My primary concern is the insufficient comparison with other methods and a lack of depth in experimental analysis.\n\n1）Although the paper compares ICR with RankGPT, highlighting improvements, it would be strengthened by comparisons with more methods, especially other zero-shot listwise methods. \n\n[1]  RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models.\n\n[2] A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models.\n\n2） The paper could benefit from additional analysis to deepen the understanding of ICR’s behavior. First, for open-source LLMs, it would be helpful to provide a clear comparison between ICR and fine-tuning (FT) methods, highlighting the performance gap and cases where fine-tuning might be preferable. Second, the paper does not analyze the sensitivity of the proposed algorithm to changes in the input document order. It would be valuable to examine how much the re-ranking results fluctuate when the order of input documents is adjusted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2YJjkI6cCE", "reviewer": "Reviewer_CYaY", "summary": "This paper proposed in-context re-ranking (ICR), an efficient re-ranking method based on the attention distributions of LLMs. This method can achieve better performance compared to RankGPT while maintaining lower latency.", "questions": "1. In line 216, \"We use N/A as a calibration query\". I was wondering why use this as the special token, how about NA or None or blank space or <>? Could these be alternatives?\n2. In line 226-227, \"the final ranking score sd_i, which measures the change of attention weights that each document receives when the query changes from the content-free calibration query to the actual query\". I was wondering whether sd_i can be considered as distance between the actual query and calibration query? It would be better to have more high-level intuition and explanation about this.\n3. In line 233-234, \"since we place the query tokens at the end of the re-ranking prompt, ICR can share the KV cache of document tokens when computing s_{di, Q} and s_{di, Qcal}\". I was wondering why the shared KV cache is in effective when you put query tokens at end? I didn't get the point here.\n4. In line 297, \"ICR's performance advantage is more prominent on Mistral, which has weaker instruction-following capabilities...\". I was wondering what's the advantages of the proposed method when instruction following is involved? Whether the proposed method can better leverage important information in the query? I'm looking forward to more explanation here.\n5. In ablation study/discussion, I'd love to see how the overall distribution looks like in terms of calibration scores.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The motivation is \"whether autoregressive generation necessary and optimal for LLMs to perform re-ranking?\", which is quite interesting and worth discussing. \nThe proposed method is technical sound and is effective on various datasets of re-ranking tasks. \nThe paper is well-written and easy to follow.\nThe experiments and discussions are thorough.", "weaknesses": "The paper lacks more high-level intuition and explanations especially about why the proposed method works intuitively. \nWhat are the advantages and disadvantages of the proposed methods intuitively.\nWhy the proposed method works well although the model's instruction following ability is poor?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ipqL6NPHo4", "reviewer": "Reviewer_FYfW", "summary": "This paper proposes an unsupervised ranking method that leverages the attention scores of query tokens over each document token. Extensive experiments, including nine single-hop and three multi-hop datasets, were conducted to prove the improvements over unsupervised baseline RankGPT. Ablation studies were conducted to prove the effectiveness of calibration and aggregation.", "questions": "1) UPR is a popular unsupervised reranking method that ranks documents based on the log-likelihood of generating a query from the input document. How does ICR's performance compare to that of UPR?\n\nSachan, Devendra Singh, et al. \"Improving passage retrieval with zero-shot question generation.\" arXiv preprint arXiv:2204.07496 (2022).\n\n2) Are the documents randomly shuffled before concatenation? To what extent does document order impact ranking performance?\n\n3) Why is the top 20 instead of the top 100 utilized for multi-hop evaluations?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and easy to follow.\n2. Extensive experiments, including nine single-hop and three multi-hop datasets, were conducted to prove the improvements over unsupervised baseline RankGPT\n3. Ablation studies were conducted to prove the effectiveness of calibration and aggregation. \n4. Clear analyses were presented to help explain why calibration helps and what re-ranking signals the proposed method captures.", "weaknesses": "1) Missing important unsupervised baseline UPR:\n\nSachan, Devendra Singh, et al. \"Improving passage retrieval with zero-shot question generation.\" arXiv preprint arXiv:2204.07496 (2022).\n\nUPR ranks documents based on the log-likelihood of generating the ground-truth query given the document. Although it is less efficient than ICR, UPR operates in an unsupervised manner and should be compared with ICR.\n\n2) The impact of document order has not been studied. ICR ranks all documents within the context, but the paper does not specify how documents are concatenated. Are they randomly shuffled before concatenation? To what extent does document order impact ranking performance?\n\n3) For single-hop evaluations, only the weaker BM25 retriever is applied. It remains unclear whether ICR can enhance the performance of stronger retrievers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)