Embedding-Based Context-Aware Reranker
Authors: Ye Yuan
Mohammad Amin Shabani
Siqi Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Passage reranking
Retrieval-Augmented Generation
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose a new reranking method to mitigates the challenges of cross-passage inference.

Abstract: Retrieval-Augmented Generation (RAG) systems rely on retrieving  relevant evidence from a corpus to support downstream generation. 
The common practice of splitting a long document into multiple shorter passages enables finer-grained and targeted information retrieval. 
However, it also introduces challenges when a correct retrieval would require inference across passages, such as resolving coreference, disambiguating entities, and aggregating evidence scattered across multiple sources.
Many state-of-the-art (SOTA) reranking methods, despite utilizing powerful large pretrained language models with potentially high inference costs, still neglect the aforementioned challenges.
Therefore, we propose Embedding-Based Context-Aware Reranker (EBCAR), a lightweight reranking framework operating directly on embeddings of retrieved passages with enhanced cross-passage understandings through the structural information of the passages and a hybrid attention mechanism, which captures both high-level interactions across documents and low-level relationships within each document.
We evaluate EBCAR against SOTA rerankers on the ConTEB benchmark, demonstrating its effectiveness for information retrieval requiring cross-passage inference and its advantages in both accuracy and efficiency.
Our source code is available at https://github.com/BorealisAI/EBCAR.

Published as a conference paper at ICLR 2026
EMBEDDING-BASER CONTEXT-AWARE RERANKER
Ye 2: Mohammad Amin Shabani", Siqi Liu
McGill University, MMila Quebec Al Institute, *RBC Borealis
ABSTRACT
Retrieval-Augee Augmeettd Generation (RAG) systems rely on retrieving relevant ev- idence from corpus to support downstream generation. The common practice of splitting a long document into multiple shorter passages enables finer-grained and targeted information retrieval. However, it also introduces challenges when a correct retrieval would require inference across passages, such as resolving coreference, disambiguating entities, and aggregating evidence scattered across multiple sources. Many state-of-the-ae (SOTA) reranking methods, despite uti- lizing powerful large pretrained language models with potentially high infer- ence costs, still neglect the aforementioned challenges. Therefore, we propose Embedddin-Baand Baxedd Context-A waare Reranker (EBCAR), a lightweight reranking framework operating directly on embeddings of retrieved passages with enhanced cross-passage understandings through the structural information of the passages and a hybrid attention mechanism, which captures both high-level interactions across documents and low-level relationships within each document. We evalu- ate EBCAR against SOTA rerankers on the ConTEB benchmark, demonstrating its effectiveness for information retrieval requiring cross-passage inference and its advantages in both accuracy and efficiency. Our source code is available at http://ghhhubbcom//Boorlll////af
INTRODUCTION
2024 have become Betrieval-Augmentte Generation (RAG) systems (Lewis et al. 2020 Wu et al.
cornertone for enabling language models to incorporate external knowledge complex reasoning tasks such as question answering (Mao et al. 2021 Xu et al. 2024b). fact verification (Adjali) 2024 Yue al. 2024). and dialogue generation (Huang et al., 2023 Wang al. 2024a). A typical RAG pipeline consists of retriever, identifying relevant passages from a large corpus, and a reranker, reordering these candidates to surface the most useful evidence for downstream generation. The reranker plays critical role in filtering out noisy retrieval results and promoting passages that are more faithful, complete, and relevant to the input query (Glass et al. 20022) de Souza P. Moreira et al.) 2024). To support fine-grained retrieval, modern pipelines often rely on passage-level indexing, where a long document is split into shorter fixed-size passages for retrieval. This chunking process refines the granularity of retrieval and improves the readability of retrieved content for downstream 2024). While long-context encoders and embedding models
models (Xu et al. 2024a Jiang et al. have recently emerged (Zhang et al.
2024 Warner al.) 2025 Boizard al.) 2025 Zhu et al.,
2024], passage-level retrieval remains the dominant design choice in many deployed systems due to
its effectiveness and efficiency (Wu et al. 2024
Conti et al. 2025).
Existing reranking methods typically fall into three paradigms (Shailymoghhadaa et al. 2025 (i) Pointwise scoring models evaluate each query-passage pair independently. For example, Nogueira et al. (2019) and Nogueira et al. (2020) score individual query-passage pairs using pretrained lan- guage models (PLMs), by formulating the task as a binary classification problem or as generative task requiring "True" or "False" token as the output. (ii) Pairwise approaches compare pairs of passages to infer relative relevance; for instance, [Pradeep et al. (202T) estimates the probability that one passage is more relevant than another given the query. (iii) Listwise methods, in contrast, op- crate over the entire set of retrieved candidates at once. One common practice is to prompt large language models (LLMs) to directly output reranked list (Sun et al. 2023). Another is to distill su- pervision signals from LLMs into smaller generative models (Pradeep et al. 2023a b Gangi Reddy
Work done while doing an internship RBC Borealis. Corresponding to to-ye.yun ye.yuan3 mail.mcgill.c

---

Published as a conference paper at ICLR 2026
Query
What Waett Criggs birth dona who tax esadled st Marannscha Bopelst Univeesdtyy
Passages
Documents
Centent Wist Passage hi porrs Wos, Wishhr Nj He has
Craig spent earyy years studidd Hararathe Blgpter Univenityy lilk barsh jul 25. 1958.
sudiod in Mircratte Beysst Uniority
Beranking
Embedding Model
Document ID: Passage ID: Content He has stacedd at satas Bustoc Univeriity
Pascage ID:
ID:0
Jul
Concane...
1858
Decument ID: 0. Passage ID:0
Gerl Colleer spent No early years Carnoiton, HO. He soucked Maranatha
Chunking
Decument ID Passage ID 0 Content Ger Coller hp earty yean Carma
Content.
stacked
Passage ID: Content...
Document ID: N. Passaga
Ropest Saptember 28, 197).
Decument Contant
ID.N
Decumentt Paxnage ID: Saytit Univarlyy birsh it Seytemiter 26, 197).
Vector Database
Decumane ID
Passage ID
Retriaving
Figure 1: A Sttrieval-Augmente Generation (RAG) pipeline with passage-level retrieval. Long documents are chunked into passages before being embedded into a dense vector store. At query time, top-k passages are retrieved and reranked to provide evidence for downstream generation.
et al. 2024). Recently, inference-time methods have been proposed that prompt an LLM to compute calibrated -reetton-bbsed scores across the retrieved passages (Chen et al. 22255).
Despite recent progress, two important challenges in reranking remain underexplored. First, across pointwise, pairwise, and listwise paradigms, most rerankers rely on feeding the raw text of retrieved passages and the query into large pretrained language models (PLMs) for scoring. This strategy incurs substantial computational cost and latency during inference, making it less practical for real- world deployments. Second, many existing methods are evaluated on idcalized benchmarks that assume the necessary information for answering query is fully contained within single passage, leaving the capabilities of current reranking strategies underexplored in scenarios that demand cross- passage inference (Conti al., (025)) Thakur et al. 2025 Zhou et al. (025). . As illustrated in Figure consider the query "What is Wiatt Craig's 's birth date, who has studied at Maranatha Baptist University?" ?" Passages related "Wiatt Craig". "birth date", and "Maranatha Baptist University" may all be retrieved individually. However, multiple retrieved passages mention the birth date of a person who studied at Maranatha Baptist University, but without resolving the pronoun "he", the reranker cannot determine which passage truly pertains to "Wiatt Craig", leading to misranking.
To address the dual challenges of inference inefficiency and insufficient modeling of cross-passage context, we propose Embedding- Based Context-Aware Reranker (EBCAR). In contrast to prior rerankers that rely on scoring raw text using large PLMs, EBCAR operates directly on the dense embeddings of queries and retrieved passages. These passage embeddings are readily available, as they are stored a vector database during indexing. The query embedding is obtained at inference time using the same encoder employed during passage indexing, ensuring compatibility in represen- tation space. This design decouples reranking from costly text-to-text inference, substamtially im- proving computational efficiency and enabling low-latency deployment. To facilitate cross-passage inference, EBCAR incorporates structural signals, such as document IDs and the positions of pas- sages within their original documents, via positional encodings. These encodings are processed by a transformer encoder equipped with a hybrid attention mechanism, comprising two complementary multi-head attention modules: shared full attention module and dedicated masked attention mod- ule. Both modules operate on the same input representations but differ in their attention scopes. The shared full attention allows the query and all passage embeddings to attend to one another, capturing global interactions across retrieved passages. In contrast, the dedicated masked attention restricts attention scopes to the passages originating from the same documents, as determined by their doeu- ment IDs. This hybrid attention design enables the model to infer over both inter-document context and fine- ane-grained intra-document dependencies, promoting more accurate evidence aggregation and entity disambiguation. We evaluate EBCAR ConTEB (Conti al., 2025). a challenging bench- mark originally designed to test retrieval models' use of document-wiin context but used in our work to assess rerankers. Our experiments include both in-distribution evaluations and nut-of-distributin zero-shot tests on entirely unseen domains, where EBCAR achieves promising performance.
Our main contributions are summarized as follows:
2

---

Published as a conference paper at ICLR 2026
We highlight the challenges of both efficiency and cross-passage inference in reranking, and introduce a lightweight embedding- framework to addresses them.
We propose a hybrid attention architecture that incorporates structural signals, such as doe- ument identity and passage position, through positional encodings. and combines shared full and dedicated masked attention modules to enhance context-aware reranking.
We conduct extensive evaluations on ConTEB, demonstrating that EBCAR achieves promising performance in both in-distribution and outrof-distribution zero-shot settings, with significantly lower inference latency.
2 RELATED WORK
In this section, we review prior work related to reranking from three perspectives: existing context- aware methods to enhance retrieval systems, the design of reranking methods, and the evaluation protocols used to assess their capabilities.
Contextual Retrieval. Prior work has proposed several methods to enhance retrieval systems with broader contextual understanding beyond isolated passages. For instance, late chunking (Günther et al. 2025), semantic chunking (Qu et al., 2025), and hierarchical retrieval (Arivazhagan et al. 2023 Choe et al. 2025) aim to improve retrieval granularity by better aligning the unit of retrieval with semantic boundaries or document structure. These methods operate at the retrieval or gener- ation stage and have proven effective enhancing RAG pipelines. In contrast, our work focuses on the reranking stage and introduces architectural mechanisms to explicitly model intra and inter- passage dependencies within the reranker. We position EBCAR as complementary to the above techniques: can be integrated seamlessly into RAG pipelines that already incorporate context- aware retrieval or generation strategies, providing an additional layer of cross- poss-passage inference during candidate reranking.
Reranking Methods. Existing reranking approaches can be broadly categorized into three paradigms: pointwise, pairwise, and listwise. (i) Pointwise models treat reranking as a binary classi- fication or regression task and score each query-passage pair independently. MonoBERT (Nogueira) et al. 2019) formulates reranking as binary classification problem using BERT to predict whether a passage relevant a given query. MonoT5 (Nogueira et al. (020)) adopts generative approach, fine-tuning T5 model to generate the tokens "True" or "False" and computing the predicted rel- evance score via softmax over the logits of these two tokens. However, pointwise rerankers ineur high inference costs, as they require separate forward pass for each candidate passage.
(ii) Pairwise methods compare two candidate passages at time to infer their relative relevance to the query. DuoT5 (Pradeep et al.) 2021), for instance, estimates the probability that one passage more relevant than another. The resulting pairwise orders are typically aggregated, using sorting heuristics or tournament-bon based voting, yield final ranking. While pairwise approaches can better capture comparative relevance than pointwise methods, their computational cost grows quadratically with the number of candidates.
(iii) Listwise methods score or reorder the entire set of retrieved passages jointly. While earlier methods treated this holistically using traditional architectures, most recent listwise approaches leverage large language models (LLMs) in different ways, leading to three main subcategories in practice. Prompt-only reranking directly queries powerful LLM, e.g., GPT-3.5 GPT-4, turn a ranked list of passages in a single forward pass. For example, RankGPT (Sun et al., 2023) prompts the model reorder passages based on their relevance to the query. While effective in zero-shot settings, this approach depends on proprictary APIs and incurs high inference latency and computational cost. supe-supe distillation extracts listwise supervision signals from prompt- only LLMs and uses them to fine-tune smaller, spen-source models. Rank Vicuna (Pradeep et al.) 2023a), RankZephyr (Pradeep et al. (202bb), and LitSDistill (Tamber et al. 2023) fine-tune 7B-scale or smaller models using the ranked outputs from GPT-style models. Similarly, FIRST (Gangi Reddy et al. 2024) and FIRST-Mistral (Chen et al. 2024) use the logits of the first generated token as a ranking signal, while PE-rank (Liu et al. (025) compresses passage context into single embedding for efficient listwise reranking. These distilled models preserve much of the performance of larger
L.MMs while offering improved inference efficiency. Inference-tine relevance extraction bypasses
3

---

Published as a conference paper at ICLR 2026
generation entirely by computing relevance scores from internal attention or representation patters of LLMs. ICR (Chen et al. 2025). for instance, derives passage-level relevance using attention- based signals from calibration prompt and two forward passes through the LLM. While more efficient, these methods still rely on large models and full-text processing during inference.
Evaluation Settings, Most existing rerankers are trained on the MS-MARCO passage ranking dataset (Bajaj et al. (016) and evaluated in zero-shot manner benchmarks such as the TREC Deep Learning (DL) tracks (Craswell et al. 2020; 202T) and the BEIR benchmark suite (Thakur al. T02T). While these datasets have driven substantial progress, they predominantly feature self- contained passages that independently provide sufficient information to answer a query. This design simplifies the reranking task and limits the need for inference across multiple passages lever- aging broader document-level context (Conti et al. 2025 Thakur et al. 2025). In addition, struc- tural signals such as document identifiers or passage positions are often absent or unused these benchmarks. To address these limitations, we evaluate our proposed EBCAR on ConTEB (Conti) et al. 2025), a benchmark originally developed to assess retrievers' use of document- context, which we use as challenging testbed to evaluate rerankers' ability to infer across multiple retrieved passages in both in-distribution and -if-o---ittrbuutionn scenarios. In this work, we use the term cross-passage inference to broadly refer to settings where relevant information is distributed across multiple passages, requiring models to aggregate and reconcile evidence across them.
3 METHODOLOGGY
Rerankers aim to refine a list of candidate passages retrieved for query, producing a final rank- ing that better reflects passage relevance. In this work, we propose an embedding- based rerank- ing framework that jointly scores candidate passages while leveraging cross passagas context and document-level structure. Our approach operates entirely in the embedding space, enabling fast and scalable inference. Below, we formally define the embedding- -ased reranking problem and present our proposed method in detail.
3.1
WAREDDING-BASED FRAMEWORK
Given a user-issued query Tg and a set of candidate passages {x1, "2,., } retrieved from vec- tor store, the goal is to reorder them based on their contextual relevance to the query. Each passage I; is associated with pre-computed embedding embedding function E()), i.e., P; = (xx). These passage embeddings remain fixed during infer- ence, and only the query needs to be encoded as q E(x,) using the same embedding function. The reranker aims to produce a permutation (--) over the indices {1, (1.2..... .......,, k} such that the reordered list {I=(1).-- ... . (kk) } places the most relevant passages to the top. This formulation enables efficient and parallelizable inference over dense embeddings.
Rd obtained during indexing via a shared
3.2 EMBEDDING-BANE CONTEXT-AWARS RERANKER
Architecture. To capture both inter-document and intra-document dependencies among candidate passages, we propose Abrrid-attention-beeee reranking architecture, EBCAR, that operates on pre- computed dense embeddings. As shown in Figure 2, EBCAR integrates document identifiers and passage position information into the encoding process, and leverages a hybrid attention mechanism to model both global and document- interactions.
We begin by augmenting the passage embeddings with structural signals. Given k candidate pas- sages {zi...., I with corresponding dense embeddings {PI,...,Phs }. each passage embedding ps E Rd is enriched with two additional embeddings: a document II) embedding doc(i) e Rd indicating the document to which belongs, and passage position encoding pos(i) E Rdd reflecting the position of x; within its original document. These components are combined as p; P; doc(i) + pos(i). The document ID embeddings are only used to distinguish the unique documents that the current retrieved passages belong to, instead of all the documents in the corpus. The i-th document ID embedding corresponds to the i-th unique document from the current k re- trieved passages. As a result, the document ID embedding table has a maximum size of l x d and remains fixed across training and inference. Importantly, the document ID embeddings are relative
4

---

Published as a conference paper at ICLR 2026
and local to each retrieved candidate set, rather than global corpus-level identifiers. At inference time, we re-index document IDs on on-the-fly according to the unique source documents within the retrieved passages. The document ID embeddings table is reused across all queries and kept frozen. For example, suppose the retriever returns k = 6 passages for query q. where 21, D2, I4 come from document A, I3, I5s from document B, and F6 from document C. EBCAR dynamically assigns three document IDs for this candidate set, e.g., doc(1) = doc(2) = doc(4) (4) = e1, doc(3) = doc(5) e2, doc(6) = e3, where ei, e2, and е3 are selected from the same fixed-size embedding table. When processing another query q', the retriever may return different set of six passages z'.... zyg from entirely different documents, e.g.. A' B', C:; these will again be assigned document embeddings e1, e2, e3 from the same table. The primary purpose of this design is to enable the model to learn to recognize which passages originate from the same document through these relative document ID embeddings, thereby facilitating within-document reasoning. As a secondary benefit, this dynamic formulation also allows new documents to be incorporated without retraining: cach new document is simply assigned temporary local document ID during reranking. The passage position encodings follow a standard sinusoidal design. The query embedding E IRd remains unchanged. We con- catenate the query with the enriched passage embeddings along the sequence length axis to form an input sequence, as shown in Figure 2 which is then processed by a stack of M modified Transformer encoder layers.
Each Transformer encoder layer employs a hybrid attention mechanism composed of two modules. The shared full attention module is a standard multi-head attention mechanism that allows the query and all passages to attend to one another freely, capturing global relevance and inter-document re- lationships. In contrast, the dedicated masked attention module restricts each passage's 's attention to only those passages within the same document, plus the query. enabling fine- fine-grained modeling of intra-document coherence. Formally, the attention score of this module is computed as:
(QK) + maskdoc
DedicateelAAunSSooeeee Q, K) softmax
dd
where Q and K are the query and key matrices, respectively, and de is the head dimension. The ad- ditive attention mask maskjoe E ((++1))(+++1) is defined such that the (i.j)-th entry is 0 if passage j belongs to the same document passage i or if j is the query embedding, and -o otherwise. This masking encourages the model to reason locally within documents without interference from unrelated passages. As illustrated in Figure 2 this masking pattern ensures that: (i) the query can attend to all passages, (ii) passage P1 from, e.g., document O can attend to the query and passage P2 from the same document, but not to p3 from, e.g., document N, and (iii) passage pa from document N can attend only to the query and itself. Note that the dedicated attention mechanism always al- lows each passage to attend to the query, ensuring that the model can implicitly recognize the query position as special without any additional signals. To better illustrate how the two attention modules complement each other, consider the example below.
Query: When did Hull City win the 2016 Championship play-off final?
Retrieved passages (p1, P2 are from Document A; Ps, P4 are from Document B):
Pr: Hull City is a professional football club based in Kingston upon Hull, England.
P2: They reached the 2016 Championship play-off final after finishing fourth in the league.
P3: The Championship play-off final determines which team is promoted to the Premier League.
P4: This match of 2016 took place on 28 May at Wembley Stadium.
Within each document, the dedicated masked attention facilitates local reasoning. linking "They" to "Hull City" in Document A, and connecting "This match" to "the Championship play-off final' in Document B. Across documents, the shared full attention enables global interaction, aligning the "Hull City* in Document A with the event context in Document B to identify that passage p4 contains the answer. Together, the two modules complement each other: the dedicated masked attention consolidates intra-document coherence, while the shared full attention integrates evidence across documents for accurate cross-passage reasoning. The outputs of the two modules are summed and passed through feedforward network with residual connections and layer normalization. After M stacked layers, we extract the updated passage embeddings (fI,..., Px from the output sequence. These contextualized embeddings are then used to compute the final relevance scores.
Training Objective. Given a query 9 and a set of candidate passages {21...., 2x ,......,,,,, sage is labeled either positive or negative based on its relevance to the query. Let q E Idd denote
each pas-
5

---

Published as a conference paper at ICLR 2026
Query
Crag's who at Marasatta Bepost University
Embedding Model
Feed Forward
P2 Ps
P1
P1
9
9
Add Norm
P1
M
P:
Shared Attencion
Dedicated Masked Astention
P5
P5
Decument ID: & Passage ID
yerrs Wess, Wisdaor, Ni
Content: Waat Cragg spent Its carty
Full
Dedictted
at
Bigris University
Decument ID:
Content: has studied Mur anstha. Baycoe Univerdiy, dry He birth dine to
P
P1
Pa
Pa
26. 1958.
+
Decument Passage
Passage Encoding
(97).
Embedding
Figure 2: Overview of the EBCAR architecture. Candidate passage embeddings are enriched with document ID and passage position information, then processed jointly with the query embedding through Transformer encoder. The hybrid attention mechanism combines shared full attention module and a dedicated masked attention module. The outputs the two modules are summed with residual connections and layer normalization.
the original query embedding, and {pi..... p& the updated passage embeddings obtained from the final Transformer encoder layer in EBCAR. Our training objective encourages the updated represen- tation of the positive passage to be close to the query embedding, while pushing the negatives away. Therefore, we adopt contrastive learning objective based dot-product similarity and employ the InfoNCE loss (van den Oord et al. 2018) defined as:
exp(sim(q,
Gonntmat - log
exp(sim(q.ftt) + E, exp(sim(q.)) 07
where 5i is the updated embedding of the positive passage, {Øz } are the embeddings of the negative passages, and sim(a, b)
denotes dot-product similarity.
We intentionally use the unmodified query embedding g. instead of the query's updated embedding, maintain consistent reference point across candidate sets. This design choice avoids query- specific drift caused by the passage context and ensures that the passage representations are directly optimized to align with the static semantic anchor of the query. We validate this design in Ap- pendix A.I where compare the training stability between fixed and updated query embeddings.
Inference. At inference time, compute the relevance scores between the query and each candi- date passage based on their final embeddings. Given the updated passage embeddings {pI,.... P and the static query embedding y, compute 8; sim (g. P.), for i 1, ...., k. The final passage ranking is obtained by sorting the scores {sI....... } in descending order. This process is efficient and deterministie, leveraging the contextualized passage representations produced by EBCAR.
EXPERIMENTS
We conduct comprehensive experiments to evaluate the effectiveness and efficiency our proposed EBCAR reranker. Our study is designed to answer the following research questions: Q1: Does EBCAR achieve competitive or superior performance compared to existing methods on datasets that require passs-e inference? Q2: Is EBCAR more efficient inference time than existing SOTA rerankers? Q3: Are EBCAR's architectural components, namely the passage position encoding and the hybrid attention mechanism, individually beneficial? Q4: How does EBCAR perform datasets

---

Published as a conference paper at ICLR 2026
that require less cross-passage inference, particularly in out-tf--istributiin settings? Q5: How does EBCAR perform when the number of retrieved passages scales up? To answer these questions, the remainder of this section is organized as follows: we first describe the datasets and evaluation protocol, followed by a summary of the haseline methods. We then present our main experimental results and provide analysis of effectiveness and inference efficiency. Finally, we conduct ablation studies. In Appendix [A,6 Appendix A..7, we evaluate the generalization ability of EBCAR on datasets that do not emphasize cross-passage inference and the impact of retrieved candidate size.
4.1 DATASETS AND EVALUATION PROTOCOL
We conduct our evaluation on ConTEE Conti et al. 2025), recently introduced benchmark de- signed to test whether retrieval models can leverage document- wide context when encoding individ- ual passages. We use ConTEB as challenging testbed for evaluating rerankers' abilities to perform context-aware reranking with cross poos-passage inference. Many of the benchmark datasets comprise long documents where relevant information is dispersed across chunks or coreferences and structural cues are essential for resolving ambiguity. Our experiments include both in in-distribution evaluations, where test queries originate from the same domains the training data, and out-of-distribution evaluations, where test queries are drawn from entirely different domains than training.
Training Procedure. We train EBCAR on the ConTEB training splits: MLDR, SQuAD, and Nar- rativeQA. These datasets emphasize different forms of cross- passage inference, including entity- level disambiguation, fine-graines span matching, and narrative-level context comprehension. The training corpus consists of 311,337 passages and 362,142 training queries. For validation, we ran- domly sample 2,000 queries from the full set of 21,370 queries provided in the ConTEB validation split. To construct training examples, we use Contriever (Izacard et al.) 2021) to retrieve the top-20 passages for each query. Since the gold passage is not guaranteed to appear among the retrieved can- didates, we check its presence and, if absent, replace the 20th passage with the gold-positive one. This ensures that each training instance contains exactly one positive and up to nineteen nega- tive passages. To avoid bias from retrieval rank, we randomly shuffle the passage order within each candidate set before feeding it to the model. It is worth mentioning that during inference and evalu- ation, follow the same retrieval pipeline but do not enforce the inclusion of the gold passage; the top-20 candidates are taken directly from Contriever without any oracle intervention.
Test Datasets. We evaluate EBCAR on eight datasets from the ConTEB benchmark, covering both in-distribution and -it-of-distiibutton scenarios. (i) MLDR, (ii) SQuAD, and (iii) Narrative (NarQAA) are used for both training and evaluation, enabling assessment of in-distribution perfor- mance for EBCAR and baselines trained on these three datasets. MLDR and NarrativeQA consist of encyclopedic and literary documents, respectively, where relevant information may scattered across long passages. (iv) COVID-QA (COVID) is biomedical QA dataset based on scientific research documents, which similarly requires reasoning over long and dense texts. (v) ESG Re- ports are long-form corporate documents sourced from the ViDoRe benchmark, re-annotated for text-based QA to test document-wide contextual understanding capabilities. (vi) Football and (vii) Geography (Geog) are constructed from rephrased Wikipedia summaries in which explicit entity mentions have been replaced by pronouns via GPT-4o. This setup introduces referential ambiguity and demands cross-passage entity disambiguation and coreference resolution to answer correctly. (viii) Insurance (Insur) comprises statistical reports on European Union countries, where country names often appear only in section headers, and therefore requires knowledge of passage's struc- tural position within the document to disambiguate country-specific information.
Evaluation Metries. We report nDCG@ 10 as our main effectiveness metric in Section 4.4 with MRR@ @ 10 results provided in Appendix A.2 Both metrics evaluate ranking quality and precision. We additionally measure inference efficiency using throughput, defined as the number of queries processed per second single A100 GPU with batch size 1. This dual-axis evaluation allows us to assess each reranker's practicality in latency-sensitive deployment settings.

---

Published as a conference paper at ICLR 2026
Table I: nDCG @ 10 across datasets. Orange denotes in-distribution tests, and Green denotes out- of-distribution tests. Throughput is the number of queries processed per second single A100 GPU. The best and second best results are bolded and underlined, respectively.
Method
Training
Size
MLDR
SQuAD
NarrQA
COVID
ESG
Football
Geog
Insur
Avg
Thronghput
BM25
65.42
45.07
36.20
39.79
10.10
1.69
23.66
0.00
28. 12
263.16
Contriever
60.23
34.63
66.97
31.022
15.69
5.95
46.39
2.73
35.45
29.67
monoBERT' (base)
ConTEB Train 110M
40.44
27.94
36.63
20.52
11.67
4.01
29.23
2.92
21.67
4.24
monoTS (base)
ConTEB Train 223M
26.76
19.75
41.21
24.56
10:37
5.69
36.70
4.40
21.18
1.69
monoTS (baxe)
MS MARCO
223M
70.47
67.00
58.80
41.51
18.61
9.68
54.47
2.20
40.37
3.61
duo75 (base)
MS MARCO
223M
42.34
68.63
21.46
18.19
12.69
8.84
47.18
2.57
27.74
0.18
Rank Vicuna
MS-MARCO
7FF
70.30
66.59
79.48
51.87
22.97
11.46
71.08
4.27
48.38
0.17
RankZophyr
MS-MARCO
7/B
82.34
69.06
81.18
53.15
26.42
11:63
72.91
3.51
50.03
0.17
FIRST
MS-MARCO
7B
74.78
61.55
78. 02.62
41.17
20.06
7.88
00.92
3.21
43.52
1.73
FirstMistral
MS-MARCO
7B
74.41
62.19
78.69
11.35
20.26
7.78
G1.79
3 4.1
13.
1.78
LiT5Distil
MS-MARCO
218M
60.33
3/..7
66. 9-1
31.22
16.07
5.98
16.66
2.78
35.61
0.66
PE-Rank
MS-MARCO
8B
52,58
45.00
51,58
333 10
19.92
6,29
46.25
1.37
32,05
0.81
RankGPT (Mistral)
7B
65.75
36.61
668
353.23
15.04
7.67
51.75
3.05
37.67
0.12
RankGPT (Liama)
8B
76.80
61.16
74.74
47.
20.49
11,10
71.33
4.76
46.03
0.13
ICR (Mistral)
7.8
79.49
67.41
79.50
52.54
25.65
10.57
71.44
4.00
48.83
0.18
ICR (Llama)
8/83
83.93
69.09
79.96
53.412
28.36
10.91
73.10
4.16
50.15
0.19
EBCAR (ours)
CosTEB Train 126M
75.26
71.62
73.21
59.80
37.20
80.19
81.30
40.74
64.92
29.33
4.2
COMPARISON METHODS
We compare EBCAR against baselines spanning four categories: retriever-only, pointwise, pairwise, and listwise rerankers. All methods rerank the same set of 20 candidate passages retrieved by Con- triever, without any oracle modification. (a) Retriever-only baselines: (i) BM25 is classical sparse retriever based on lexical matching. (ii) Contriever (Izacard al. 2021) a dense retriever trained using unsupervised contrastive learning. (b) Pointwise rerankers: (iii) MonoBERT (Nogueira) et al., 2019) independently scores each query-passage pair using a binary classification head. (iv) mono T5 NNgueira et al. 2020) casts reranking a text generation task by producing the to- ken "True" or "False." " (v) monoT5 (MS-MARCO) refers to the same model trained on the MS- MARCO (Bjjaj et al.) 2016) dataset. (c) Pairwise rerankers: (vi) duoT5 (Pradeep al. 2021)
compares passage pairs to estimate their relative preference with respect to the query. (d) Listwise rerankers: (vii) Rank Vicuna (Pradeep et al. (0233),, (viii) RankZephyr (Pradeep al. 2023b), and (ix) LiTSDistill (Tamber et al.] (003) are distilled listwise rerankers supervised by ranked outputs from GPT-style models.
(x) FIRST (Gangi Reddy et al. 2024) and (xi) FirstMistral (Chen al. 2024 use the logits of the first generated token to derive passage-level ranking scores. (xii) PE- Rank (Liu al. 2025) encodes each passage into a single embedding for efficient listwise ranking. (xiii) RankGPT (Sun et al.) (2023) prompts a language model to return a ranked list. (xiv) ICR (Chen) jtt al. 2025) computes passage relevance using attention- basedd signals from in-context calibrated prompts over LLM representations. Due to limitations of computational resources, we only train monoBERT and m5noT5 on ConTEB and evaluate other baselines with their provided checkpoints.
4.3 IMPLEMENTATION DETAILS
For training our EBCAR, we use the Adam optimizer (Kingma & Ba 2015) with a fixed learning rate of 1 x 10-3 and no weight decay. The model is trained for up to 20 epochs, with early stopping based on validation loss using a patience of 5 epochs. We set the batch size to 256. The default EBCAR configuration uses 16 layers, i.e., M 16, hidden size of 768, 8 attention heads. We employ these configurations to obtain similar amount of parameters as BERT base model. The implementation details of baseline methods are discussed in Appendix A.3 All experiments are run on a single A100 GPU.
4.4 RESULTS AND ANALYSIS
Effectiveness. We first compare the overall ranking quality of EBCAR with a broad set of base- lines. As shown in Table I EBCAR achieves the highest average nDCG@ 10 across all test sets, outperforming all baselines. Our method shows particularly strong gains on tasks that require fine- grained entity disambiguation and coreference resolution, such as Football and Geography, where
8

---

Published as a conference paper at ICLR 2026
reasoning across multiple passages and resolving entity mentions is essential. EBCAR also excels on Insurance, a task that heavily relies on positional information and structured document layouts; in this case, architecture's s ability to incorporate explicit passage position embeddings proves ben- eficial. These targeted improvements reflect the strengths of EBCAR in handling tasks that benefit from structured passage representation and alignment. We illustrate one sample from the football dataset and one from the insurance dataset in Appendix A to highlight the underlying challenges.
On the remaining datasets, including both in-distribution datasets like MLDR, SQuAD, and Narra- tiveQA, and out-of-distribution datasets such as ESG and COVID, EBCAR maintains competitive performance. While does not always surpass the most powerful LLM-based rerankers, e.g.. ICR, we hypothesize that this is partly due to EBCAR's 's embedding bassed architecture: since each passage compressed into a fixed-size embedding, some fine-gran fine-grained information may be lost, a limitation not shared by LLM rerankers that directly process the full text of each passage.
Despite this potential information bottleneck, EBCAR achieves strong and stable performance across domains. As shown in FigureBaa our model achieves the lowest average gap to the best method across all tasks, with narrow interquartile ranges, indicating stable performance. These results demonstrate that embedding- based approaches, when combined with architectural enhance- ments, can offer a robust and efficient alternative to heavyweight LLM-based rerankers.
Efficiency. We also evaluate the inference speed of each method, measured as the number of queries processed per second on single A100 GPU. As shown in Table I EBCAR achieves a throughput of 29.33 33 queries per second, substantially outperforming all reranking baselines in terms of efficiency, while maintaining strong ranking quality. While models such as monoBERT and monoT5 have comparable parameter sizes to EBCAR, they process each query-passage pair inde- pendently, requiring k separate forward passes for & candidate passages per query. This leads to slower inference, especially in the case of pairwise models like duoT5, which compare passages in pairs and further increase the computational burden. At the other extreme, large LLM-based rerankers such as ICR or RankGPT are significantly slower due to both their large model sizes and their need to process the full textual content of all candidate passages. Even when some models like FIRST and FirstMistral do not explicitly generate output sequences, e.g., listwise orders like [3] [1] > [21, their long input contexts still cause substantial latency.
In contrast, EBCAR operates entirely in the embedding space, allowing all candidate passages to be encoded once independently. The final selection is performed through lightweight alignment com- putations, without the need for autoregressive generation or full-sequence processing. This design yields an order-of-mann magnitude speedup while maintaining strong performance, clearly positioning EBCAR on the Pareto frontier of ranking quality versus speed, as obviously shown in Figure (b).
Summary. Together, these results highlight that EBCAR offers a highly favorable integration of competitive ranking quality as well as fast and scalable inference.
Interestingly, we observe that monoBERT and monoT5 trained on ConTEB perform worse than Contriever when trained MS- MARMO. We hypothesize this stems from the fact that during training these pointwise models are forced to separate semantically similar passages from the gold passages while unable to utilize the document-wise context, which is the actual signal distinguish the passages. This resulted in poor generalization. In contrast, EBCAR can leverage global context and structural signals across the candidate set, enabling more robust and context-aware ranking.
4.5 ABLATION STUDIES
We analyze the contri- bution of key compo- nents in EBCAR via ab-
Table 2: Ablation studies on two core components of EBCAR.
Method
MLDR SQuAD NarrQA COVID ESG
Football Geog Insur
lation studies. Remov- ing positional informa- tion (w/o Pos), which in- cludes document ID em- beddings and passage po- sition encodings, leads to notable performance degradation, particularly on datasets like Insurance, where positional reasoning and document structure are critical. Ablating the hybrid attention mech-
Pos
60.72
60.87
64.18
43.71
23.36
42.88
62.44 34.16
w/o Hybrid
74.55
47.52
66.32
42.93
34.44
41.93
60.34 36.00
w/o Both
43.46
40.13
51.80
29.63
15.77
5.28
43.70 2.88
EBCAR
75.26
71.62
73.21
59.80
37.20
80.19
81.30 40.74

---

Published as a conference paper at ICLR 2026
(a)
BM25
Effectiveness (Average nDCG@10
monoBEKT
Concrever
Contraxar
mono T5
monaTS (MS)
Our EBCAR
manali (PS)
duaTs
RankYicuua
RarkZaphyr
RarkZostt
FRRST
FRST
FirtMarrra
FruPPorr
PERank
RankGPT
KankGPT (Uame)
20
RaniGart (Merrs)
Ranocaat (uamo)
ICX (Mizra)
ICR (Lams)
ICR (lams)
ESCAR
EBCAR
10
15
40
45
Absolute Gap to Best
Throughput (Queries/Second)
Figure 3: (a) For each method, we compute the gap in nDCG@ 10 from the best-performing method on each dataset. The boxes represent the interquartile range (25th 75th percentile), vertical lines and black triangles indicate the median and mean, respectively. (b) An overall effectiveness-and efficiency comparison across all methods. The effectiveness measured by the average nDCG@ 10 across 8 test datasets. The efficiency is quantified by the throughput.
anism (w/o Hybrid) also results in performance drops on several datasets such SQuAD. Interest- ingly, w/o Hybrid outperforms w/o Pos on datasets such as MLDR, NarQA, and ESG, while the opposite trend is observed on SQuAD, COVID, and Football. These variations in direction reflect -ptesct-specific characteristics. Tasks that rely heavily on structural or positional cues (c.g.. Insur- ance) more affected removing positional signals (w/o Pos), whereas datasets emphasizing semantic reconciliation across passages (e.g., Football and Geography) depend more on the hybrid attention for effective cross-passage reasoning (w/o Hybrid). This suggests that the two proposed components contribute complementarily to the overall performance of EBCAR. When both compo- nents are removed (w/o Both), performance deteriorates drastically across all datasets, underscoring the synergy between structural signals and the hybrid attention mechanism. These results validate the effectiveness of each component.
In Appendix A.5 we further analyze the hybrid attention mechanism through qualitative visualiza- tions of its attention heatmaps. Appendix Appendix/A..fand and Appendix/A.7] we evaluate EBCAR on datasets that do not emphasize cross- cross-passage inference and the impact of retrieved candidate size.
4.6 ROBUSTNESS TO OTHER RETRIEVER
To examine the robustness of EBCAR to different embedding models, we replace the Contriever retriever and embeddings with E5 (Wang et al. 2024b), a stronger dense retriever. As shown by @DCG@1 10 scores, EBCAR maintains consistent performance improvements across datasets and further benefits from the stronger embedding space provided by E5. Specifically, the retriever-only E5 achieves 62.78 MLDR, 55.86 SQuAD, 67.12 NarQAA, 33.7 74 on COVID, 15.38 on ESG, 6.05 on Football, 48.85 on Geography, and 4.02 on Insurance. When combined with EBCAR. the E5-based variant achieves 77.36 MLDR. 72.68 SQuAD, 75.04 on NarQA, 61.07 on COVID. 36.88 on ESG, 80.64 on Football, 82.47 on Geography, and 41.24 on Insurance. These results confirm that EBCAR regroicer-agnostic and seamlessly adapt stronger embedding models while consistently improving overall retrieval quality.
5 CONCLUSION AND DISCUSSION
In this work, we introduce EBCAR, a novel embedding reranking framework designed to model context across multiple retrieved passages, by leveraging their structural signals and hybrid attention mechanism. Extensive experiments on the ConTEB benchmark demonstrate that EBCAR outperforms strong baselines, especially on challenging datasets requiring entity disambiguation, co-reference resolution, and structural understanding.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
For reproducibility, we provide implementation details of our method in Section 4.3 and those of the baseline methods in Appendix A.3 The full codebase is released publicly httgs///stttbb.commoooollllilllf
LLM USAGE
We employed large language models exclusively to assist with polishing the writing of this paper. Specifically, LLMs were used to refine wording and verify grammar for clarity and readability, No aspects of method design, implementation, or experimental analysis involved LLM assistance.

---

Published as a conference paper at ICLR 2026
REFERENCES
Omar Adjali.
Exploring retrieval augmented generation for real-world claim verification. In Pro- ceedings of the Seventh Fact Extraction and VERification Workshop (FEVER), Miami, Florida, USA, 2024.
Manoj Ghuhan Arivazhagan, Lan Liu, Peng Qi, Xinchi Chen, William Yang Wang, and Zhiheng Huang. Hybrid hierarchical retrieval for open-domain question answering. In Findings of the Association for Computational Linguistics: ACI. 2023, Toronto, Canada, 2023.
Payal Bajaj. Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Ma- jumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. Ms marco: A human generated machine reading comprehension dataset, 2016.
Nicolas Boizard, Hippolyte
Duarte M. Alves, André Martins, Ayoub Hammal, Caio Corro, Céline Hudelot, Emmanuel Malherbe, Etienne Malaboeuf, Fanny Jourdan, Gabriel Hautreux, Jo ao Alves, Kevin El-Haddad, Manuel Faysse, Maxime Peyrard, Nuno M. Guerreiro, Patrick Fernandes, Ricardo Rei, and Pierre Colombo. Eurobert: Scaling multilingual encoders for curopean languages, 2025.
Shijie Chen, Bernal Jimenez Gutierrez, and Yu Su. Attention in large language models yields effi- cient zero-shot re-rankers. In The Thirleenth International Conference on Learning Representa- lions, 2025.
Zijian Chen, Ronak Pradeep, and Jimmy Lin. An early first reproduction and improvements to single-token decoding for fast listwise reranking, 2024.
Jaeyoung Choe, Jihoon Kim, and Woohwan Jung. Hierarchical retrieval with evidence curation for open-domain financial question answering standardized documents. In Findings of the Association for Computational Linguistics: ACL 2025, Vienna, Austria, 2025.
Max Conti, Manuel Faysse, Gautier Viaud, Antoine Bosselut, Céline Hudelot, and Pierre Colombo.
Context is gold to find the gold passage: Evaluating and training contextual document embed- dings, 2025.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. Overview of the tree 2019 deep learning track, 2020.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. Overview of the tree 2020 deep learning track, 2021.
Gabriel de Souza P. Moreira, Ronay Ak, Benedikt Schifferer, Mengyao Xu, Radek Osmulski, and Even Oldridge. Enhancing q&a text retrieval with ranking models: Benchmarking, fine-tuning and deploying rerankers for rag, 2024.
Aaron Grattafiori al. The llama herd of models, 2024.
Revanth Gangi Reddy, JaeHyeok Doo, Yifei Xu, Md Arafat Sultan, Deevya Swain, Avirup Sil, and Heng Ji. FIRST: Faster improved listwise reranking with single token decoding. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, USA, 2024.
Michael Glass, Gaetano Rossicllo, Md Faisal Mahbub Chowdhury, Ankita Naik, Pengshan Cai, and Alñio Gliozzo. Re2G: Retrieve, rerank, generate. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, United States, 2022.
Michael Günther, Isabelle Mohr, Daniel James Williams, Bo Wang, and Han Xiao. Late chunking: Contextual chunk embeddings using long-context embedding models, 2025.
Qiushi Huang. Shuai Fu, Xubo Liu, Wenwu Wang, Tom Ko, Yu Zhang, and Lilian Tang. Learning retrieval augmentation for personalized dialogue generation. In Proceedings of the 2023 Confer- on Empirical Methods in Natural Language Processing, Singapore, 2023.
12

---

Published as a conference paper at ICLR 2026
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning, 2021.
Mathew Jacob, Erik Lindgren, Matei Zaharia, Michael Carbin, Omar Khattab, and Andrew Drozdov. Drowning in documents: Consequences of scaling reranker inference, 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie- Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
Ziyan Jiang, Xueguang Ma, and Wenhu Chen. Longrag: Enhancing retrieval-augmente augmented generation with long-context Ilms, 2024.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. aeureeeel-augmeeted generation for showledge-intensivy NLP tasks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Pro- cessing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Qi Liu, Bo Wang, Nan Wang, and Jiaxin Mao. Leveraging passage embeddings for efficient listwise reranking with large language models. In The Web Conference 2025, 2025.
Sean MacAvaney, Andrew Yates, Sergey Feldman, Doug Downey, Arman Cohan, and Nazli Gohar- ian. Simplified data wrangling with ir da_aatssss. In SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, 2021.
Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. Generation-augmennte retrieval for open-domain question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th Inter- national Joint Conference on Natural Language Processing (Volume Long Papers), Online, 2021.
Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. Multi-stage document ranking with bert, 2019.
Rodrigo Nogueira, Zhiying Jiang, Ronak Pradcep, and Jimmy Lin. Document ranking with pre- trained sequence-to torecce-tt--suunneeee model. In Findings of the Association for Computational Linguis- fics: EMNLP 2020, Online, 2020.
Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. The sspando-mono-dae design pattern for text ranking with pretrained
models, 2021.
Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. Rankvicuna: Zero-shot listwise docu- ment reranking with open-source large language models, 2023a.
Ronak Pradeep. Sahel Sharifymoghaddam, and Jimmy Lin. Rankzephyr: Effective and robust zero- shot listwise reranking is breeze!, 2023b.
Renyi Qu, Ruixuan Tu, and Forrest Sheng Bao. Is semantic chunking worth the computational cost? In Findings of the Association for Computational Linguistics: NAACL 2025, Albuquerque, New Mexico, 2025.
Sahel Sharifymoghaddamm, Ronak Pradeep. Andre Slavescu, Ryan Nguyen, Andrew Xu, Zijian Chen, Yilin Zhang, Yidi Chen, Jasper Xian, and Jimmy Lin. Rankllm: A python package for reranking with llms. In Proceedings of the 48th International ACM SIGIR Conference on Re- search and Development in Information Retrieval, Sigir '25, New York, NY, USA, 2025.
13

---

Published as a conference paper at ICLR 2026
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is ChatGPT good at search? investigating large language models as re-ranking agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, Singapore, 2023.
Manveer Singh Tamber, Ronak Pradeep, and Jimmy Lin. Scaling down, litting up: Efficient zero- shot listwise reranking with seq2seq encoder-decode models, 2023.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models, 2021.
Nandan Thakur, Jimmy Lin, Sam Havens, Michael Carbin, Omar Khattab, and Andrew Drozdov. Freshstack: Building realistic benchmarks for evaluating retrieval on technical documents, 2025.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, 2018.
Hongru Wang. Wenyu Huang, Yang Deng, Rui Wang, Zezhong Wang, Yufei Wang, Fei Mi, Jeff Z. Pan, and Kam-Fai Wong. Unims-rag: A unified multi-source retrieval-augmented generation for personalized dialogue systems, 2024a.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training, 2024b.
Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Griffin Thomas Adams, Jeremy Howard, and lacopo Poli. Smarter, better, faster, longer: modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers), Vi- enna, Austria, 2025.
Shangyu Wu, Ying Xiong, Yufei Cui, Haolun Wu, Can Chen, Ye Yuan, Lianming Huang, Xue Liu, Tei-Wei Kuo, Nan Guan, and Chun Jason Xue. Retrievallaugmented generation for natural language processing: A survey, 2024.
Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024a.
Zhentao Xu, Mark Jerome Cruz, Matthew Guevara, Tie Wang, Manasi Deshpande, Xiaofeng Wang, and Zheng Li. Ruurievel-uugmented generation with knowledge graphs for customer service ques- tion answering. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024, 2024b.
Zhenrui Yue, Huimin Zeng. Lanyu Shang, Yifan Liu, Yang Zhang, and Dong Wang. Retrieval aug- mented fact verification by synthesizing contrastive arguments. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), Bangkok, Thailand, 2024.
Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li, and Min Zhang. mGTE: Generalized long-context text representation and reranking models for multilingual text retrieval. In Proceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, Miami, Florida, US, 2024.
Yang Zhou, Hongyi Liu, Zhuoming Chen, Yuandong Tian, and Beidi Chen. Gsm-infinite: How do your Ilms behave over infinitely increasing context length and reasoning complexity?, 2025.
Dawei Zhu, Liang Wang, Nan Yang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. LongEm- bed: Extending embedding models for long context retrieval. Proceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Processing, Miami, Florida, USA, 2024.
14

---

Published as a conference paper at ICLR 2026
50000
3.0
Trained with Updated Query Embedding
Query Embedding
40000
2.5
Training Loss in Eq.2
Eq.
Training Loss In
10000
8 Epochs
10
12
Epochs
Figure 4: Training loss comparison between using updated query embedding (left) and fixed query embedding (right) in Eq. 2 Updating the query embedding leads to large and unstable loss spikes, while keeping it fixed produces smooth and monotonic convergence. This supports our design choice to use a fixed query embedding.
Table 3: MRR @ 10 across datasets. Orange denotes in-distribution tests, and Green denotes out- of-distribution tests. Throughput is the number of queries processed per second on single A100 GPU. The best and second best results are bolded and underlined, respectively.
Method
Training
MLDD SQuAD
NarrQA
Geng
Avg
Throughput
BM25
61.35
42.90
36.18 18
37.35
8.56
3.72
23.46
0.00
26.09
263.16
Contriever
52.00
49.94
60. 10
25.25
11.63
4.48
37.87
1.72
30.37
29.67
monoBBRT (buse)
CosTEB Train 110M
31.50
20.00
25.87
14.63
7.96
2.73
21.96
2.31
15.87
4.24
monoT'S (basc)
CoaTEB Train
223M
16.97
13.26
30.39
17.87
6.31
1.36
28.07
3.35
15.07
1.69
monoT'S (base)
MS-MARCO
223M
66.39
61.91
51.44
38.39
14.83
8.63
47.82
1.35
36.75
3.61
duo'55 (base)
MS-MARCU
223M
35.25
66,89
16.14
14.88
10.75
7.68
42.25
1.69
24.44
0.18
RankViguna
MS-MARCO
7B
77.13
64.45
77.13
50.62
19.60
10.71
67.71
3.21
46.33
0.17
RankZophyr
MS-MARCO
7.
80.77
67.51
79.27
52.41
23.67
10.95
70.13
2 2.58
48.44
0.17
FIRST
MS-MARCO
7B
T1.43
39.16
76. 10
38.73
17.39
7.08
57.20
2.11
19.
1.73
FirstMistml
MS-MARCO
7/
70.93
60.03
76.24
33.04
17.64
6.95
58.37
2.83
41.50
1.78
LiTSDistil
MS-MARCO
248M
19. 10
50. 18
60. 10
25.46
12.01
4.53
38.15
1.75
30.54
0.66
PE-Rank
MS-MARCO
8B
41.32
35.93
39.3 39.34
25.42
16.04
4.48
35.34
0.52
24.80
0.81
RankGPT (Mistral)
7.B
58.93
52.20
61.97
27.80
10.07
6.44
44.34
1.81
32.95
0.12
RankGPT (Llama)
8.3
73.45
57.64
70.70
45.35
17.23
68.40
4.13
43.41
0.13
ICR (Mistral)
7.B
77.01
65.61
78.63
52.53
23.21
9.76
09.27
2.85
17.36
0.18
ICR (Llama)
8B
82.93
67.83
79.18
53.64
25.38
10.18
71.33
2.87
19.17
0.19
EBCAR (ours)
ConTEB Train
126M
68.34
64.50
67.44
51.44
26.04
74.73
75.61
20.54
$6.08
29.33
A APPENDIX
A. TRAINING WITH UPDATED QUERY EMBEDDING
To examine the effect of fixing the query embedding in the contrastive objective in Eq. com- pare two variants EBCAR: one that uses the fixed input query embedding and another that uses the updated query representation produced by the encoder to compute the loss. As shown in Fig- u training becomes highly unstable when the query embedding is updated. Loss values oscillate drastically and occasionally explode after few epochs. In contrast, the fixed-query variant con- verges smoothly and yields stable optimization. We attribute this instability to context drift, where the query representation becomes entangled with passage-specife context, thus breaking its role as a consistent semantic anchor across candidate sets. These results empirically validate our design choice of using the unmodified query embedding Eq.
A.2 SUPPLEMENTAL MRR @ 10 RESULTS
We provide full MRR @ 10 results in Table The overall trends consistent with our nDCG@ 10 results. EBCAR achieves the highest average score across all datasets. Notably, EBCAR excels on datasets such as Football, Geog, and Insurance, demonstrating its strong ability in handling tasks that
15

---

Published as a conference paper at ICLR 2026
benefit from structured passage representation and alignment. These findings reinforce the motiva- tions behind EBCAR and validate its robustness across varying cross posss-a reasoning challenges.
A.3 SUPPLEMENTAL IMPLEMENTATION DETAILS
For monoBERT and monoTS trained on ConTEB, we implement these two models from scratch, For training, we use the Adam optimizer (Kingma & Ba 2015 with a fixed learning rate of x 10 and 5 10 respectively,
suggested their original implementation details (Nogueira 2020 These models are trained for up to 20 epochs, with early stopping based on validation loss using patience of 5 epochs. We set the batch size to 32. All other baseline implementations are based public code releases: monoTS (MS-MARCO), duo'SS, Vicuna, RankZephyr, First- Mistral, and LiTSDistill are from the RankLLM library (Shariyymoghaddam et al. 2025 Follow- open-source models Mistral (Jiang al. 2023 and Llama- 3.1 (ct al. 2024 for the RankGPT method, instead GPT3.5/4 due to cost constraints. FIRST PE-Rank RankGPT and ICK" follow their respective original implementations. All experiments are run on a single A100 GPU.
2019
ing ICR (Chen et al. 2025).
A.4 EXAMPLES OF FOOTBALL AND INSURANCE
Example (Football).
Query: What was Andrew Henry Robertson's 's first goal for Hull City?
Gold passage: Despite the departure of several other first-team players, he chose remain at City. His inaugural goal for the club was scored on November 2015 in an away match against Brentford, where he initiated the scoring in 2-0 victory that placed Hull the top of the Championship table on goal difference. He participated in the 2016 Championship play-off finul against Sheffield Wednesday, which Hull won 1-0 achieve promotion the Premier League. However. the team lasted only one season in the top division before being relegated again.
All pronouns in this dataset are anonymized, so identifying that "he" refers to "Andrew Henry Robertson" requires reasoning across multiple passages mentioning the player and the club. EBCAR's hybrid attention enables such cross-passage coreference resolution effectively.
Example (Insurance).
Query: What is the amount of (Re)insurance GWP in million in Austria?
Gold passage: { ('General data': : Amounts Share total EEA
Population (in 1000) 8.979 2.00%
(issannuee GWP (in million) - 20,815.873 1.5%
Number of (re iresanuueeeee undertakings - 33 1.9%
Number of registered insurance intermediaries - 17,999 2. 2.1% -3
Without positional or structural cues, is difficult to associate this numerical information with the correct country (Austria), which only appears in the section title. EBCAR's positional encodings provide this structural grounding, allowing accurate passage ranking.
A.S HYBRID ATTENTION ANALYSIS
To examine how the hybrid attention mechanism routes information, conduct qualitative study across all datasets. For each dataset, we randomly sample 20 test queries and, for every example, render an aggregated attention heatmap averaged over heads. We focus our analysis on the final
htte:://ggthhb.cc//astttttoaa// 11m
-nnssrrutttt
16

---

Published as a conference paper at ICLR 2026
Table 4: nDCG@ 10 of selected baselines and EBCAR on TREC21, TREC22.
Method
Training
Size
TREC2021
TREC2022
Average
Throughput
BM25
54.10
48.66
51.38
263.16
Contriever
64.19
61.36
62.78
29.67
mon615
COnTEB Train
223.77
52.37
46.66
49.62
1.69
monoT5
MS-MARCO
223M
85.04
76.92
80.98
3.61
RankZephyr
MS-MARCO
TB
88.05
79.87
83.96
0.17
ICR (Llama)
8.
75.28
67.47
71.38
0.19
EBCAR (ours)
COITEB Train
126M
72.77
66.65
69.71
29.33
layer (Layer 15) where the model's 's decision is consolidated. Figure(Jjshows the shared full atten- tion at Layer 15. Although weights vary across cells, attention is broadly distributed across passages, indicating global context exchange among all candidates. In contrast, Figure [ shows the dedicated masked attention at Layer 15: information flow is sharply concentrated within passages that origi- nate from the same document (block-diagonal patterns) and between the query and those passages. While the figures present one representative case, we observe the same qualitative pattern consis- tently across the 20 sampled examples every dataset: the shared full attention supports global cross-passage reconciliation, whereas the dedicated masked attention strengthens intra-document consolidation. This matches EBCAR's design intent and explains the complementary gains seen in the ablations.
A.6 PERFORMANCE ON MS-MARCO V2
To further evaluate the generalization ability of EBCAR, we compare it against selected rerankers on the MS-MARCO v2 dataset, using test queries from TREC 2021 and 2022. We select ICR (Llama) because it is the most recent and best-performing baseline the ConTEB Test set, and thus serves as a strong comparator this evaluation. RankZephyr and monoT5 are selected because they are the best- best-performing methods among LLM-supervised distillation methods and point- methods, respectively.
Compared to MS-MARCO v1, version 2 provides additional structural annotations such docu- ment IDs and passage spans, making it suitable testhed for evaluating EBCAR'S s ability to leverage structural signals (Bajaj et al. 2016). We utilize the ir tr_datasets library (MacAvaney et al., 2021 to access the judged queries from the TREC Deep Learning Tracks (2021 and 2022).
EBCAR achieves comparable ranking performance ICR (Llama), despite being significantly smaller in model size. Moreover, EBCAR exhibits a dramatically higher through- put, highlighting its practical advantage scenarios that demand both efficiency and effectiveness.
As shown in Table 4
While EBCAR performs competitively against ICR, it does not surpass monoT5 (MS-MARCO) or RankZephyr, both of which are pretrained and fine-tuned directly on MS-MARCO v1. We hy- pothesize two main reasons for this gap: (1) MS-MARCO v2 contains multiple golden passages per query. However, EBCAR trained with the setting where there are only one golden passage among all candidate passages. This mismatch makes it harder for EBCAR to adapt to multi-relevant retrieval settings. (2) Both monoT5 (MS-MARCO) and RankZephyr are trained specifically on the MS-MARCO v1 dataset, which has overlaps with MS-MARCO v2 queries and documents, giving them an implicit domain advantage.
A.7 IMPACT OF RETRIEVED CANDIDATE SIZE
In this section, we study how the performance of EBCAR varies as the number of retrieved can- didate passages increases during inference. By default, EBCAR is trained with & 20 passages, which defines the positional range of its document ID embeddings and sinusoidal position encod- ings. During inference, however, increasing à beyond this value may result in extrapolation issues due to embedding mismatches or index overflows.
To study this issue, train additional variant of EBCAR using i 50 passages and compare its performance against the default k 20 variant, as well as against ICR (Llama), across varying numbers of retrieved passages from 10 to 50.
17

---

Published as a conference paper at ICLR 2026
100
EBCAR traind with Top
90
EBCAR trained with Top50
ICR (Liama)
80
Average nDCG@1
70
60
50
40
30
20
10
10
20
30
40
50
Number Retrieved Passages (k)
Figure 5: Impact of retrieved candidate size.
As shown in Figure we observe that: (i) The default EBCAR (trained with Top20) degrades as increases after k 20, likely due to extrapolation effects from unseen positional indices. (ii) The EBCAR trained with Top50 exhibits more stable performance across larger values and consistently outperforms EBCAR trained with Top20 variant beyond & 30, validating hypothesis about training-inferene mismatch. (iii) Interestingly, both EBCAR variants exhibit slight performance drop at large k. which aligns with prior findings (Jacob et al. 2024), where scaling up candidate size introduces noisy or distractive information that hampers reranking. (iv) Compared to ICR (Llama), both EBCAR variants outperform it across most & values. These results suggest that it is better to train the EBCAR model with larger size of candidate passages during the training phase to mitigate the extrapolation issue for inference.
18

---

Published as a conference paper at ICLR 2026
Query
DO P0 00000000000000000000000000000000
D1 P20 000-0000000000002222700000000000 411.1000112222010000000000000010
D0 01400000002000011100000101111010
D2.F1 000-4400000000411777044042282220 0
03 012111100009002222088880077..220 Ttt.://0...................0.0.0
from
D4, P15 000.0000000001100000000000000000 CLC4M 0480 3C 2190 www..000000.00.00
Ue
DS_F1 00000000011000000000000000000000 9.19 0100 1C
DO 0.10:0062002240000...10000 042 0370 034 (30 000.0.11400044433000000000000000 010.0000000.....0...0000000.0000
D7 4000700055004330000000.66 (,49 01455 0490 C O. DL A0t.://000.........//...........
DR 179-000000 -10-0088000770000000000000
0-90 9110 0390 0110 000 0' 0.0 0390 AIC 0390
D4 07200-08000001111330000000000000 8488 0480 044 0390 04550 0.000.00 D40 038 www.030000000..000 0TT.000000000.000000000000000000
000700057004440008880600000000
0440
2420
02
D12 P13 406680057006300066000066 4 0480 0440 (40 1410 09.666 0-4 D (4
Pt0.S-...00000300000000 0.00000000000000 2.
0900 0150
0.0000
D34_P4
0850 0410
0.0
Quay
P1
go. PM
D2 Pl
P22
D2 Pg
Pi
08 Pi0
P16
Pg
P16
010,0 PM
011.1 Pg
D12.00 P13
D13 Pg
D14 pa
D8
D9
p9.00
D1.
Keys nAing Tby
Figure 6: Shared full attention at Layer 15, averaged over all heads. The green box highlights the gold passage. Attention is more broadly distributed across passages, indicating global context exchange among all candidates.
19

---

Published as a conference paper at ICLR 2026
Query -ww.0.....8...22220000
D0 P0 42418.00000000 020.0090 wtt.0/0000//..///// 2 CUDOO 6000 0000 ww0..00...000000 LODO 0000 115 (1 0000 www...o....ooooooooooooooooo GIC
D1 69.
0.69
0.000003000000...... 0,000 0000 000C
www.0000.....000 0000, w0w.00000.000...... w.w.00000.00.....000 0.000 0000 0000 000..00 0000, 0,00 000
200..716
D0
0220::000.009022332220000 D. 0030 0000 00D.0000C 0000 0000 2.00.2244.000 000.00000000030000 000 OCOC 0000 OCO 0.0 www.000000000000 0900.00
D2.P1 02.F1-0.4000000 wwt.0.000..000000000000000 0.0020000000000000000000 www.0000.00CC.C0.00 wWw..0.C.........0O COOO
wwt../........00.0 W00.000..000 0.000 0.03 000 OCOC CCO OCOO.00 O www...0...00000
03. P22 0.6800 00w.000000000000000000000000 2.0031144 0000 000C W00.0.000000
0.
0.000, 0.000.00
D2 02_P3-M 133-447 007.00000000000000000000000
Queriezz (Altending from
04.P15-150000
0000--11000000000000000000 00000000000000000000000000000000 W00. DUDON 2000 200...0.C00000 0. 000..000 D00 OUCOLD 000 0.00.2840 24RCC CODO OCO DOOO DWO.CC BICOUORO
Ue
D5JP1 221-0666000000000
www.00000000000000000000000
D00
OOOG 0000,
D6.P10-0.635
D0.P3 02200:060.000.233322000 0.0 0.0000. 000.0000...00 0000 0000 20...21002220..0000 2.0000000.0000000000000000000000 0.0030 000 OCOC 000 OC 0.0 0.000,, 0000.00
07 0.655 www.0000000000000..... 000 DO00 0000 0ODC
WOO.C00CCCCCOOO CODO WW0.000...000 00.000 0.345 0L0O0 0OO OCOG DOG WWO.OO.OOOOOOOOO www.SCL.OOOOOOO
DOOG,
D9 P0.00-1 008-064-0000 00000 w00.000000000000000 0000 000 0000 0000 000C CODO. 3000 0O0 0000 0030 35 OC0G 0000 OCO 0000 0000 083
D4JP18 0.50 0.000.000.0000000000000 D DO30 0000 C.0000, 000.0000000000 DOG DOCO 0470 00000 0C0 0.00 000.00000...000000 DDCQ.OD3
0.6300
000000000000000000000000
W00.00000000 000000000000000 0.0000 0.000 3610 OCO 00000.00
02
D11_PE
www.0000000000000......
00000000
3560 00DO
D12 P13 085 00000000000000000000000000000000 000 0000 0000 000C JO0 w00.000..00 000 000 0030 0000 OCOG 2000 OCO 35000000.0000
01305-0000 000000000000000000000...
000C 3000 000 0000
0000.5500.000
014-44-0619 w00..00000000000000
0000
0.0
Quay
D1.
00, PM
D2 Pl
P22
D2 Pg
Pi
08 Pi0
P16
Pg
P16
010,6 PM
011 Pg
D13 Pg
D14 pa
D8
D9
p0.00
D7
D12
Keys (Atte noing Tay
Figure 7: Dedicated masked attention Layer 15, averaged over all heads. The green box high- lights the gold passage. Attention concentrates within passages from the same document (block- diagonal structure) and between the query and those passages, reflecting document-awwe awarre locality.
20


Reviews:
[{"review_id": "A4IuGaMsla", "reviewer": "Reviewer_VaSb", "summary": "EBCAR is a novel reranking method to solve the critical challenge of cross-passage inference, ignored by other methods. Specifically, it incorporates the structural information of passages via a hybrid attention mechanism to capture both high-level inter-document and low-level intra-document interactions. Combining evidences from multiple passages, the performance improvement on benchmark datasets shows its effectiveness and efficiency.", "questions": "-The proposed embedding based rerank method is highly dependent on the embedding model, such as BERT, Mistral and Llama. It is unfair to compare reranking models with different embedding models listed in both Table 1 and 3.\n\n-The performance comparison between EBCAR and ICR in Table 1 under the evaluation of NDCG@10 is inconsistent with their comparison result in Table 3 under the evaluation of MRR@10. The underlying reasons are to be explored.\n\n-How the reranked results’ influence on the generation results should be further explored.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "--Cross-passage inference, the core problem studied in this paper, is critical for complex reasoning tasks.\n\n--NDCG@10 of the proposed method is consistently better than state-of-the-art baselines on benchmark datasets.\n\n--It is more efficient compared with rerankers based on large and high-cost language models.", "weaknesses": "--Both the hybrid attention and the positional encoding are two key components for EBCAR according to the ablation study in Table 2. However, how the hybrid attention mechanism improves the cross-passage understanding is still a question to be explored and explained further. Maybe a detailed analysis of the hybrid attention should be provided and also an intuitive example is used to illustrate their relationship.\n\n--Compared with the efficiency of Contriever, the advantage of the proposed method EBCAR is not obvious. Its time complexity needs to be analyzed for complementary. \n\n--The performance improvement of the proposed method in Table 1 is 8~10 times of state-of-the-art baseline methods on Football and Insure, while the performance improvement is not so large on other datasets. The underlying reason should be clarified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gADNn6bl3x", "reviewer": "Reviewer_dequ", "summary": "The paper proposes Embedding-based Context-Aware Reranker (EBCAR), a reranking model designed to address the challenges of efficiency and cross-passage inference in passage reranking tasks. To improve efficiency, EBCAR operates directly on the embeddings of the query and passages, rather than their raw text representations. To enhance cross-passage reasoning, it augments each passage embedding with two additional components: a document ID embedding and a passage position embedding, enabling the model to better capture contextual and structural information across passages. Furthermore, the paper introduces a hybrid attention mechanism to refine the passage embeddings, which are then used to produce the final ranking.", "questions": "1. How scalable is EBCAR to larger or dynamic document collections, given that the document ID embedding matrix has a fixed size? \n2. Can the authors provide insights into why EBCAR achieves such large gains on the Football and Insur datasets? Are there characteristics of these datasets that particularly favor the proposed method?\n3. Has the model been tested with retrieval results from stronger retrievers like E5 or BGE? If not, how confident are the authors that EBCAR would generalize to different retrieval backbones?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a novel embedding-based context-aware reranker that improves both efficiency and cross-passage inference. \n2. Experimental results on eight datasets show that the proposed approach achieves the best average score. \n3. The paper is well-written and easy to follow.", "weaknesses": "1. EBCAR requires learning a fixed-size document ID embedding matrix of size k × d, which limits scalability. Expanding to a larger document set would require retraining the model to accommodate new document IDs, making it less flexible in dynamic or large-scale retrieval settings.\n2. As shown in Table 1, the proposed model achieves exceptionally high performance on the Football and Insur datasets (e.g., 80.19 vs. 11.63, and 40.74 vs. 4.76 compared to the best baseline), which contributes significantly to the overall average improvement. However, the paper lacks insight into why the model performs so well on these datasets. Moreover, the lack of detailed descriptions of each dataset makes it difficult to understand and analyze the differences in model performance.\n3. The reranking experiments are conducted only on passages retrieved by Contriever, making it unclear whether the proposed method can generalize to results retrieved by stronger retrievers such as E5 or BGE. Evaluating on diverse retrieval backbones would strengthen the validity and robustness of the proposed approach.\n4. The paper lacks discussion and comparison with document-level retrievers or rerankers, which are relevant baselines given the cross-passage inference objective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HPgD5lZ8Kt", "reviewer": "Reviewer_zHMo", "summary": "This paper presents EBCAR: a new reranker model that works directly on the retrieved embeddings of the passages.\n\nEBCAR concatenates the query and retrieved passages into one sequence, then calculates refined contextual embeddings for these passages in order to score them.  The refined embeddings are calculated by a  modified transformer architecture: a new dedicated masked attention module is added, where the attention is computed within passages of the same documents only. This document-masked attention is added along with the standard full-attention before the add+norm layer in the transformer encoder.\n\nThe paper has a comprehensive ablation and evaluation sections with multiple datasets covering various domains, showing uplift against strong baselines, and nearly a 7x throughput increase over the fastest neural reranker baseline in the experiments.", "questions": "- Figure 2 is very helpful, but it does not clearly show how the two attention modules are combined. Is the add an norm applied to all three sets of embeddings (raw, with shared full attention, and masked attention)", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The paper is easy to read and the contribution is clear.\n\n-  The dedicated masked attention module is a novel idea that seems to work well in capturing intra-document coherence and semantics.\n\n- The EBCAR model shows strong performance despite a very small size (126M parameters), showing significant uplift compared to much larger models.\n\n- The throughput of EBCAR is impressive, this might be in part due to its smaller size considered in the experiments, but also due to working on the embeddings directly.\n\n- The ablation studies in the appendix are comprehensive and justify the choices and architecture changes.", "weaknesses": "- Having a document ID embedding makes adding new documents difficult since a new embedding will need to be learned for it. It would be interesting to see experiments where the document embedding is computed from the token or passage embeddings to make adding new documents easier.\n\n- Furthermore, learning a specific embedding per document makes me concerned about potential overfitting to these documents. It would be interesting to further analyze the content of these embeddings to see what concepts they captured that isn't already included in the tokens.\n\n- I think a scaling study is lacking for this paper and would make it a lot stronger. Since the model is performing well on smaller scale, it would be interesting to see how the performance changes as we scale the model from 126M to 8B and from one GPU to 8 GPUs or even more, it would also be useful to understand how much of the impressive throughput is due to the architecture and embedding caching vs. the small number of parameters.\n\n- Even though the paper focuses mostly on LLM-based reranker, but including models like BERT in the baseline drove me to wonder about the performance of a multi-vector ranking and retrieval methods like ColBERT in the same setup. Having this datapoint would be excellent for a fuller picture of how this model performs, especially to understand the impact of the information-bottleneck mentioned in the paper, since the multi-vector methods do not suffer from such information-bottleneck.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HESm7WQ2IY", "reviewer": "Reviewer_AjAm", "summary": "The paper proposes EBCAR (Embedding-Based Context-Aware Reranker), a lightweight reranking framework that operates entirely in the embedding space. Experiments on the ConTEB benchmark demonstrate that EBCAR achieves strong performance in tasks requiring cross-passage entity disambiguation, coreference reasoning, and structural dependency modeling, while maintaining a favorable trade-off between accuracy and efficiency.", "questions": "1. Although the ablation study includes w/o Pos, w/o Hybrid, and w/o Both variants, the paper does not explain why the effects of these components differ in direction across certain datasets.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The method operates directly in the embedding space, avoiding repeated LLM-based text processing and improving inference speed.\n2. The experiments are conducted on the ConTEB benchmark, which emphasizes cross-passage reasoning, and include both in-distribution and out-of-distribution evaluations, with clear and comprehensive results.", "weaknesses": "1. Since embeddings represent compressed information, fine-grained textual details may be lost. The paper does not conduct a quantitative analysis of which types of information are most susceptible to loss.\n2. The model keeps the query embedding unchanged, yet no comparison is provided with an alternative setting where the query representation is updated. An analysis could clarify whether contextualizing the query improves or destabilizes semantic interactions.\n3. The paper would benefit from qualitative analyses. Specifically, visualizing hybrid attention heatmaps for typical success and failure cases to illustrate how the model routes information within and across documents, thereby improving interpretability.\n4. Most SOTA rerankers (e.g., RankVicuna, RankZephyr) are fine-tuned on MS-MARCO rather than ConTEB, whereas EBCAR is trained specifically on ConTEB, raising potential concerns about experimental fairness and comparability.\n5. The terminology is inconsistent: document-aware masked attention (line 238) and dedicated masked attention (line 94) are used interchangeably and should be unified for clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)