Beyond Sequential Reranking: Reranker-Guided Search Improves Reasoning Intensive Retrieval
Authors: Haike Xu
Tong Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Vector Similarity Search
Information Retrieval
LLM reranker
Primary area: optimization

Abstract: The widely used retrieve-and-rerank pipeline faces two critical limitations: they are constrained by the initial retrieval quality of the top-k documents, and the growing computational demands of LLM-based rerankers restrict the number of documents that can be effectively processed. We introduce Reranker-Guided-Search (RGS), a novel approach that bypasses these limitations by directly retrieving documents according to reranker preferences rather than following the traditional sequential reranking method. Our method uses a greedy search on proximity graphs generated by approximate nearest neighbor algorithms, strategically prioritizing promising documents for reranking based on document similarity. Experimental results demonstrate substantial performance improvements across multiple benchmarks: 3.5 points on BRIGHT, 2.9 on FollowIR, and 5.1 on M-BEIR, all within a constrained reranker budget of 100 documents. Our analysis suggests that, given a fixed pair of embedding and reranker models, strategically selecting documents to rerank can significantly improve retrieval accuracy under limited reranker budget.

Published as a conference paper at ICLR 2026
BEYOND SEQUENTIAL RERANKING: RERANKER- GUIDED SEARCH IMPROVES REASONING INTENSIVE RETRIEVAL
Haike Xu Massachusetts Institute of Technology ha maikexu@mit.enn
Tong Chen
University of Washington
chentong@cs. washington.c edu
ABSTRACT
The widely used retrieve-and-rerank pipeline faces two critical limitations: they are constrained by the initial retrieval quality the top-k documents, and the growing computational demands of LLM-based rerankers restrict the number of documents that can be effectively processed. We introduce Reranker- Brranker-Guided-e Suuded-SSearch (RGS), a novel approach that bypasses these limitations by directly retrieving documents according to reranker preferences rather than following the traditional sequen- tial reranking method. Our method uses greedy search on proximity graphs generated by approximate nearest neighbor algorithms,, strategically prioritizing promising documents for reranking based on document similarity. Experimental results demonstrate substantial performance improvements across multiple bench- marks: 3.5 points on BRIGHT, 2.9 on FollowIR, and 5.1 on M-BEIR, all within a constrained reranker budget of 100 documents. Our analysis suggests that, given a fixed pair of embedding and reranker models, strategically selecting documents rerank can significantly improve retrieval accuracy under limited reranker budge
INTRODUCTION
Retrieval involving complex query-document relationships has recently received significant research
attention across scientific reasoning, instruction following, and multi-modal contexts (Su et al. (2024);
Weller et al. 20024) Wei et al. (2024): Oh et al. (2024).. While embedding- based retrieval methods
Gao al. (2021); Karpukhin al. (202)) Formal al. (202T) Xiao al.) (2023); Meng al. (2024)
enable efficient search over large corpora, they often struggle with complex tasks due to inherent capacity limitations. The retrieve-then-reeank pipeline Nogueira & Cho (2019) addresses this by using computationally intensive rerankers to assess the top-k search results, with recent large language model (LLM)- based rerankers Sun et al. (2023); Pradeep al. (2023): Qin al. (2023): Weller et al. (2025); ; Shao et al. (2025) substantially improving relevance rankings by jointly analyzing query-document pairs.
While LLMM-based rerankers greatly improve relevance ranking, their high computational cost limits the number documents that can be reranked. As result, accuracy is bottlenecked by the initial retrieval stage. This leads to key question: Given fixed reranker budget, how we select documents for reranking to maximize accuracy? this work, we move beyond the common sequential scan of retrieval results and propose selecting documents for reranking using search algorithm over document-dou -dccumett similarity.
Inspired by the recent popular graph- based algorithm for nearest neighbor search in the bi-metric setting Xu et al. (2024), we propose Reranker- See--GGuddddrrreeee to bypass sequential reranking. Based on the clustering hypothesis Jardine & Rijsbergen| (1971) that similar passages have similar relevancy to the same query, we run a greedy search on the proximity graph built from Approximate Jayyaaam Nearest Neighbor Search (ANNS) algorithm Jayaram Subramanya et al. (2019); Malkov & Yashunin (2018): Fu et al. (2019) on the document embedding. Specifically, given a search query, we first retrieve a list of seed documents close to the query in the embedding space and rerank them. After
Our code is available h/
ke/ Guided- Search

---

Published as a conference paper at ICLR 2026
?
Retrieve-and-Reree Baankerr-Guied--Seaae
amplitude: appreach
calculete sin(2x)/x (1+x) when
D1
Retrieve
Properties
Simple Mctian//mplltude
Sine Function
D1
02 Derivative
D2
D2
Exponential
D3
Expanded by D1
D8
D3
Expanded by D2
L'Höpital's Rule
Only rerank neighborhood
Figure 1:
An example from TheoremQA-T illustrating how our Ceranker-Guided-SSeele works. We show the summarized version of the query and document titles on the left.
that, expand the document candidates by including the neighborhood of those documents favored by the reranker, and rerank those documents again. We iteratively repeat the expand and rerank process until reach the reranking budget.
Our method enhances retrieval accuracy by strategically allocating the reranking budget. We first rerank small neighborhood of the query, then avoid spending resources low-ranked neighbor- hoods. As illustrated in Figure while query Q and document C1/D1 share surface-level similarities (both and D1 mention sine functions, Q and CI mention amplitude), the reranker assigns C1 lower rank because it fails to provide the correct mathematical relationship. It then expands DI to get D2, and D2 to get D3, because document D2, despite minimal word overlap with Q, contains the appropriate method to solve the problem--a a quality captured by the reranker. Consequently, our approach finds the correct document D3 by exploring D2's neighborhood while reranking fewer than 500 documents. RGS saves reranker budget by skipping documents similar to the lower-ranked Cl. In this case, document D3 was initially ranked 281 12th the embedding space and wouldn's ' have been found using sequential reranking.
We empirically test our RGS algorithm on three different benchmarks: a reasoning intensive bench- mark BRIGHT (Su et al. (2024), multi-modality benchmark M-BEIR Wei et al. (2024), and an instruction following benchmark FollowIR Weller et al. (2024). When the reranker budget is fixed to at most 100 documents,
RGS method improves the NDCG@ @ 10 score from 25.3 to 28.8 on BRIGHT, from 60.4 to 63.3 on FollowIR, and from 25.9 to 31.0 on M-BEIR, compared to the standard sequential reranking method.
Our analysis shows that in the regime of high reranker budget (e.g. 500), RGS exhibits a unique property that the final retrieval accuracy less dependent on the embedding model capacity but more decided by the alignment between the reranker model and the groundtruth label. Furthermore, our RGS robust query embedding perturbance. still manages get reasonable retrieval accuracy even if the similarity between document embedding and query embedding provides little information about their relevancy, which makes RGS suitable for reasoning inteensive retrieval tasks.
In summary, this work makes the following contributions:
We propose novel retrieval pipeline RGS, which leverages the similarity of documents to prioritize reranking of more promising documents and improve performance retrieving complex query- document relationships.
We propose rarely discussed evaluation setup- Reranker@kk-ar assess the efficiency of different methods in applying rerankers and to evaluate their performance on complex query-document matching tasks, including BRIGHT, FollowIR, and M-BEIR.
We show that, given a pair of embedding and reranker models, strategically selecting documents for reranking can significantly improve accuracy within fixed reranker budget.
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORK
Graph-based reranking Recently, several graph-based reranking methods have been proposed to bypass the sequential reranking bottleneck. GAR MacAvaney et al. (2022) builds KNN graph over the corpus and maintains a graph frontier of reranked documents. It alterntes between reranking documents from the shortlist returned by vector similarity search and from the graph frontier. RAR Frayling et al. (2024) extends GAR by incorporating the query point and constructing a bipartite graph between documents and queries. SlideGAR (Rathee et al. 2025) further extends GAR to accommodate listwise rerankers. From a practical perspective, all prior works (GAR, RAR, SlideGAR) evaluate their methods only on the classical TREC dataset (Craswell et al. 2020). which does not capture the complexity of modern user queries. In contrast, our analysis shows that RGS exhibits strong performance on more complex retrieval tasks.
: Malkov & Yashunin 2018): Fu et al. (2019) are family of heuristic-based methods that have recently gained increased attention in the ANN community. These algorithms typically build a proximity graph over nigh-dimensional vectors and then perform a greedy search to traverse toward the query's ' nearest neighbors. Although Indyk & Xu (2023) has shown that graph-based methods lack worst-case theoretical guarantees, they have demonstrated strong empirical performance. In addition to excelling at standard ANN tasks, graph-based algorithms offer other advantages. For example, Xu et al. (2024) proposes bi-metric framework that leverages proximity graph built using the DisKANN algorithm under one distance metric to search for nearest neighbors according to different distance metric. This idea inspired to design Geranker-Guided-Seee for reranking.
Graph-based ANNS algorithms Graph-based ANN algorithms Jayaram Subramanya al. (2023),
METHOD
3.1 PRELIMINARIES
Given a document corpus C' = {d1, dn) and a query q. we have access to embedding model E(-) that maps queries and documents to a shared embedding space, and a reranker model D that evaluates the relevance between a query and document. An ideal embedding model E' should satisfy two conditions:
document-win similarity: Given two documents dj and d2, the more similar they are, the larger the inner product between E(d1) and E(d2).
query-document relevance: Given query q and document d, the more relevant the document is to the query, the larger the inner product between F(()) and F(()).
The goal of retrieval task is to return the most relevant document to the query in the most efficient way. the scope of our paper, we focus on how to better utilize the reranker model. Our objective is to find the most relevant document while under fixed reranker calls budget.
3.2 BASELINE
We mostly compare our RGS method with two other baselines: Retrieve-and- and SlideGAR. Here, briefly describe SlideGAR, introduced in recent work Rathee et al. (2025). SlideGAR performs reranking in rounds until its reranker call quota is exhausted. In each round, it asks the reranker to rerank w documents and retains the top w/2 among them. The w candidates are composed of two parts: one half comes from the top w/2 documents selected in the previous round, and the other half is alternately sampled from either the initial short list sorted by vector similarity or from the KNN graph neighbors of the w documents reranked in the previous round.
Although both SlideGAR and our RGS explores the graph frontier on the corpus graph, we highlight key algorithmic differences between them and our RGS method. First, our RGS maintains an up- to-date document order by the reranker's preference. RGS explores all the neighbors of the most promising document and inserts them to the sorted list. In contrast, SlideGAR only picks w/2 documents from the graph frontier with the top vector similarity score, discarding the rest them. This means that even if relevant document appears in the graph frontier, it may not be ranked
3

---

Published as a conference paper at ICLR 2026
immediately if its embedding is not close the query. Thus, view SlideGAR more like augmented linear rerank with the addition of the graph neighbors from the current candidates, while RGS tries simulate standard beam search used in graph-based approximate nearest neighbor search algorithms.
3.3 RERANKER SERANKER-GUIDED
Inspired by the clustering hypothesis Jardine & Rijsbergen 1971), we believe that if document is judged to be relevant by reranker, then its similar documents are likely to be relevant as well. We thus prioritize rerank those documents whose similar documents have been assigned high scores by the reranker. We combine this idea with the bi-metric techniques from Xu et al. (2024) to design Reranker- Guided-Searreeeee
Our RGS algorithm first builds proximity graph over documents by running the DiskANN algorithm Jayaram Subramanya al. (2019) on document embeddings. DiskANN incrementally constructs the proximity graph by inserting new points into the graph data structure. For each new point it first performs a greedy search on p to obtain the set of visited points V. Then it creates bi-directional edges between p and points in V. Finally, robust pruning procedure is applied to any points with more than edges. After preprocessing, DiskANN produces a proximity graph in which each point has most R outgoing neighbors (w). (Please refer to Jayaram Subramanya al. (2019) for details)
Given query q. RGS performs two-stage search locate the relevant document. In stage one, run DiskANN to find the document whose embedding is closest to that of the query. In stage two, run greedy search guided by the reranker, starting from document s. Specifically, RGS maintains an ordered list A of the most relevant documents found so far. Initially, the list only contains one document s. At each step, we pick the first unexpanded document in the list, explore neighbor documents Noue(t) in the graph, and append them (at most R documents) to the end of the list. We then use a listwise reranker to reorder the newly added documents from the end toward the front using a sliding window. This expand and rerank process is repeated until we reach the reranker call budget. Finally, the first document list is returned as the most relevant document to the query. Please refer to Algorithm I for details.
In our implementation, we focus on listwise reranker model D, as (unn al. (2023) suggests performs better than pointwise reranker when applied to LLMs. Our RGS algorithm can be easily modified to accommodate a pointwise reranker by replacing the list with a priority queue sorted by reranker scores.
Algorithm 1 Reranker- Saaeeeer-GGiidd-Seearhhee
1: Input: Graph index G, listwise-reranker D, query search list size L, sliding window size w.
2: Output: the most relevant document for query g
3: + Use DiskANN algorithm to search for the closest vector from query embedding
4: A A+h {s} [s}
5: U U+0 Ø
6: while A A U Ø do
the first vertex in A 8: U+U
7:
Append (o)( \ A to the end of A 10: for = |A| down to 0 step size w 22/2 do
9:
Nou(()): (u): out neighbors of node in G
Use reranker D reorder Ali-w i]
if |A L then
13:
A + the first L vertices in A 14: return the first element in
3.4 ANALYSIS
We first analyze the running time of our RGS algorithm. Since the time bottleneck lies in calling the reranker model, we focus on bounding the number of reranker calls. Empirically, the number of expanded nodes usually bounded by O(T)), where 7. is the search list size. For each expansion,
4

---

Published as a conference paper at ICLR 2026
rum the reranker D to reorder the search list using sliding window with step size w/2, which involves O(L/w) reranker calls. Overall, the estimated number of reranker calls is O L
For serrive-anddrenee and SlideGAR, since they are budget- budget-based algorithms, suppose the reranker budget is Q and the window size is w. Then the estimated number of reranker calls is O(Q/w).
Regarding accuracy, we can invoke Theorem 1. from Xu al. (024). which shows that as long as the vector similarity approximates the reranker score (e.g., for a pointwise reranker) within constant factor, our RGS algorithm is able to find the most relevant documents according to the reranker score.
4 EXPERIMENTS
To evaluate the performance of our RGS method, we compare RGS with other two methods (Retrieve- and-Rerank, and SlideGAR (Rathee et al., 2025)) and test them on three benchmarks: BRIGHT (Su et al., (024) (reasoning intensive), M-BEIR (Wei et al. 2024) (multi-modality), and FollowIR (Weller et al.] (2024) (instruction following). We believe that these benchmarks represent the most complicated retrieval tasks available, which can't 'tt be solved by simple semantic similarity, and serve as a better testbed for evaluating the efficiency of different high-performing retrieval methods.
4.1 EXPERIMENTAL. SETUP
Dataset Details BRIGHT Su et al. 2024) is a reasoning-intennivee benchmark composed of naturally occurring human data from 12 different domains. Relevance in this dataset goes beyond simple lexical or semantic similarity; the authors constructed hard negatives that are topically related to the query but do not meet its specific requirements. M-BEIR Wei al. (202)) is multimodal retrieval benchmark built from eight different query-corphr modality subtask. For each subtask, the query can text qt. an image Qi. or combination of both. Each candidate can be a text cr, an image c, or a combination (c,, Gt). It poses a challenge for retrieval methods, as models must integrate knowledge from multiple modalities to interpret user queries and return answers the specified modality. FollowIR Weller al. (2024) is an instruction-followiii
retrieval benchmark curated from three TREC datasets, in which annotators create different instructions for each query to test a model's ability to adjust retrieval results based nuanced differences in user intent. We believe these three benchmarks all demand a non-trivial degree of reasoning compared to classical IR tasks and are therefore more suitable for evaluating the capabilities of retrieval methods to capture complicated query-document relationships.
Metrics We propose a rarely discussed evaluation setup @eranner@@,, which limits the reranker to scan at most documents. In our experiment, set & 100/300/50 and measure NDCG@ 10 under different reranker budget. Note that our setup allows each document to be reranked multiple times by a listwise reranker. We also consider the number of token usage, API calls in Appendix A For M- M-BEIR, we randomly sample a subset of 100 queries to test and we remove those groundtruth labels not appearing in the corpus for accurate NDCG@ @ 10 evaluation.
Baselines and Algorithm Details We compare our method with two baselines shown as follows:
Retrieve-and-Renan For different reranker budget 100/300/500 it uses DiskANN algorithm to retrieve the top l documents closest to the query embedding and rerank them in a sliding-windon way (Sun et al. (2023) with window size w 10.
SlideGAR: We reimplement the algorithm in Rathee et al. (2055) First, SlideGAR uses the DiskANN algorithm to retrieve a shortlist of the top l documents closest to the query embedding. Next, it builds a KNN graph based on document-document similarity. It then alternates between reranking documents from either the retrieved shortlist or the graph frontier until the reranker budget is reached. Following Rathee et al. (2025), we set the window size to w 20.
Reranker-Guided. -Satechh- In practice, we make the following adaptions to Algorithm [ The search list size Ls is tuned based on the reranker budget is (Ls initialize our search with k/5 start points returned by the first stage vector similarity search. The algorithm is forced to immediately return the best documents seen far once the reranker budget k
10/30/50 for l
000/300/500) We
5

---

Published as a conference paper at ICLR 2026
is exhausted We set window size to 10 for our RGS method and return the top-10 10 documents in list for evaluating NDCG@ I0.
Simbedding/Rerankee Model Details We choose BGE-Large Xiao et al. (2023) the embedding model for BRIGHT and FollowIR, and CLIP Radford al. (2021) embeddings for M-BEIR. We prompt Gemini-2.0-Flashh to be reranker model. Please refer to Appendix [ for our prompt. For the multi-mon modality benchmark M-BEIR, when either the query or a candidate in the corpus contains both image and text information, use CLIP's 's text and image encoders separately encode each modality, and then apply element-wise addition to produce the final embedding.
C
4.2 RESULTS
Scientific and Mathmatical Reasoning The evaluation results on BRIGHT are presented in Table I] Using dense retrieval with BGE-Large as the encoder yields an average NDCG@ 10 of 13.7 across all datasets. When applying retrieval-and-rerankk pipeline, sequential reranking of the t0p-100 documents improves the score to 25.3, and scanning the top-500 documents yields 27.7. This suggests that dense retrieval alone struggles to capture complex query-document relationships, which rerankers can help improve.
SlideGAR achieves 25.4 at Reranker@ 100 and 26.9 at Reranker 550, showing minimal improvement over sequential scanning. In contrast, our method, RGS, achieves 28.8 (+14%) at Reranker @ 100 and 33.0 (+ (+19%) at Reranker@500, demonstrating a substantial improvement in accuracy under the same reranker budget.
We also observe that RGS provides the largest gains over sequential scanning in scientific reasoning tasks, whereas the improvement is minimal AoPS and LeetCode. We speculate that reranking list math or code examples requires intensive reasoning ability, which remains challenging even when using Gemini-Flash-2.0 the reranker. In these cases, no reranking method is able improve upon the first-stage retrieval.
Instruction Following The evaluation results FollowIR are presented in Table 2
Using dense retrieval with BGE-Large as the encoder yields an average NDCG@ 10 49.9 across all datasets. Applying retrieve-and-rerant pipeline, sequentially reranking the top- 100 documents improves the score to 60.4, while scanning the top-500 documents yields 61.6. This suggests that sequential reranking provides limited benefit in capturing user instructions.
SlideGAR achieves 62.5 at Reranker@ 100 and 59.8 at Reranker@ 500, showing slight decrease as the reranker budget increases. In contrast, our method, RGS, achieves 63.3 (+5%) at Reranker@ 100 and 64.2 (+4%) ) at Reranker 500, demonstrating a stable improvement in accuracy under the same reranker budget.
Compared with the results on BRIGHT, the performance gap between different methods is smaller on FollowIR. We observe that FollowIR, over 87% of the ground-truth answers are located within the top 100 documents ranked by query-document vector similarity, whereas the number is only 31 % on BRIGHT. This indicates that the marginal benefit of searching beyond the top 100 limited. Since most of the newly discovered documents beyond the top 100 are likely to be noise-and the LLM- based reranker doesn't always align with ground-truth labels-this explains the slight performance drop observed in some methods the reranker budget increases.
Multi-modality retrieval The evaluation results on M-BEIR are presented in Table 3 Using dense retrieval with clip- as the encoder yields an average NDCG@ 10 of 15.3 across all datasets. When applying a retrieval-and-rerank pipeline, sequential reranking of the top-100 documents improves the score to 25.9, and scanning the top-500 documents yiclds 30. This suggests that dense retrieval alone struggles to capture cross-modality retricval tasks, which rerankers can help improve.
SlideGAR achieves 28.6 6 at Reranker@ 100 and 29.9 at Reranker@ 550, showing minimal improvement sequential scanning. In contrast, method, RGS, achieves 31.0 (+20%) at Reranker @ 100 and 38.3 (+27%) at Reranker@ 500, demonstrating a substantial improvement in accuracy under the same reranker budget.

---

Published as a conference paper at ICLR 2026
We notice that the improvement is significant certain tasks. For example, on task 6, while the standard CLIP embedding scored less than even after reranking the top-500 candidates, our RGS achieves NDCG@ @ 10 score of 30.7 and 15.6 using the same reranker budget. In Wei et al. (2024), the authors mention that they use the 'Wikipedia page title' as text input rather than using the 'TO0 tokens from the Wikipedia page' the candidate because it provides better 'zero-shot performance." This indicates that they are aware of the limited ability of embedding models to encode long context, which we believe is ideal
demonstrate the advantage of our reranking method.
No reranker BGE- BGE-Large
Reranker@ 100 SlideGAR
Reranker@ 300
Reranker@t 500
Dataset
RR
RGS
RR
SlideGAR
RGS
RR
SlideGAR
RGS
Biology
11.7
31.1
33.9
37.0
37.6
37.0
42.8
38.9
40.7
46.8
Earth Science
24.6
42.2
43.7
45.3
43.0
43.2
47.1
38.5
43.5
49.2
Economics
16.6
23.0
26.0
26.2
27.4
29.1
28.5
28.8
28.8
30.4
Psychology
17.5
33.1
39.5
38.5
45.4
40.4
49.2
Robotics
11.7
24.5
24.6
26.5
22.9
28.0
29.5
25.0
21.9
29.8
Stack Overflow
10.6
27.2
26.2
29.9
27.5
26.7
29.1
27.5
24.2
27.7
Sustainable Living
13.1
30.8
31.4
35,5
24.7
35.5
36.2
26.8
30.8
37.1
LeeiCode
26.7
26.0
17.4
25.3
24.5
17.0
23.5
25.8
11.2
22.1
Pony
5.7
24.5
21.6
22.5
21.1
19.8
21.9
23.7
20.8
25.1
AoPS
6.0
6.7
6.9
7.1
7.5
4.0
7.8
6.7
3.5
7.4
TheoremQA-Q
13.0
20.1
17.7
21.2
22.3
20.6
28.6
25.8
23.7
30.2
TheoremQA-T
6.9
15.8
21.9
28.2
20.8
29.6
39.3
24.5
33.0
40.7
Avg
13.7
25.3
25.4
28.8
26.6
27.4
31.6
27.7
26.9
33.0
Table 1: BRIGHT evaluation with varying reranker budgets for RR (retrieve-amd-rerambk) SlideGAR
RRthhee al. 2025) and our RGS (Reranker-Gii Suiee--GGiidddd--lleeee
No reranker BGF--Large
Reranker@ 100 SlideGiAR
Reranker 300
Reranker@ 500
Dataset
RR
RGS
RR
SlideGAR
RGS
RR
Slide(iAR
RGS
Robust04
47.2
58.7
59,2
61.6
59.6
56.6
63.3
61.0
55.9
65.0
News21
52.6
57.4
58.5
58.8
58.7
57.3
58.6
58.6
56.5
58.2
Corel7
69.7
66.7
Avg
49.9
60.4
62.5
63.3
60.2
64.1
Table 2: FollowIR evaluation with varying reranker budgets for RR (retrieve-and-rerank) SlideGAR (Rathee et al. 2025 and our RGS Guided-Searchh
No reranker CLIP
Reranker & 100 SlideGAR
Rerankerff 30D
Reranker 50D
Task
Dataset
RR
RGS
RR
SlideGAR
RGS
RR
SlideGAR
RGS
VisualNows
24.4
36.4
36.9
37.4
35.1
39.8
39.9
35.8
41.2
36.5
MSCOCO
41.1
59.4
55.6
55.8
58.7
55.3
57.6
60.3
54.6
55,1
Fashion200K
2.4
7.4
5.9
5.6
10.3
6.8
8.6
8.9
9.9
7.7
g: -
WebQA
g. (c.
EDIS
24.5
43.2
46.6
52.6
45,2
46.7
$3.6
49.0
2
55.3
WebQA
VisuaNNews
24.0
30.4
33.3
27.3
31.1
30.9
32.3
30.3
29.5
33.3
4. -
MSCOCO
35.8
44.3
44.5
46.5
44.5
44.6
44.7
44.9
43.8
45.3
Fashicn200K
1.6
4.7
5.0
5.3
5.9
8.0
9.5
9.2
6.9
8.7
9. +
NIGHTS
24.4
26.7
24.0
28.9
29.8
26.0
26.0
31.0
26.2
25.4
6. + (g)a))
2.9
3.7
InfoSeek
0.1
1.0
2.1
3,9
1.3
4.1
6.8
0.7
4.2
15.6
7. a + (g)
FashionIQ
2.7
8.1
12.2
11.8
11.6
11.0
9.8
15.3
CIRR
12.8
31.8
33.3
33.6
40.8
35.6
47.5
41.6
34.9
49.2
8. (gi, g) + (z)
OVEN
14.8
33.2
41.4
15.7
36.8
57.9
29.9
36.8
59.5
InfoSeek
18.8
25.2
40.4
28.8
20.0
47.0
28.4
27.2
52.6
Avg
15.3
25.9
28.6
31.0
28.5
29.3
36.6
30.1
29.9
38.3
Table M-BEIR evaluation with varying reranker budgets for RR (retrieve-and-rerankk) SlideGAR (Rathee et al. 2025), and our RGS (Reranker- Srrankke--Giied-ee
7

---

Published as a conference paper at ICLR 2026
5 ANALYSIS
In this section, we investigate how the quality of the embedding and reranker models, as well as noise in the embeddings, affect retrieval accuracy. We perform an error analysis identify the source of missing ground-trus touun results and conduct ablation studies on the impact of first-stage retrieval and various graph types.
5.1 RETRIEVAL RESULTS WITH DIFFERENT EMBEDDING / RERANKER MODELS
We study the extent to which embedding quality affects final retrieval accuracy. Our experiments conducted on selected subset of datasets -iitastts-Bilogy.. Psychology, Sustainable Livings, and TheoremQA--T-ee T--from the BRIGHT benchmark. We choose these four datasets because they exhibit the largest performance gaps between our RGS method and RR, making the algorithmic behavior of our method more pronounced and easier to analyze. To examine the impact of embedding quality, we apply four embedding models of varying sizes: SFR-Mistral (7B), LaE-Large (300M), BGE-Bas Baase (100M), and BGE- -Micro (17M).
As shown in Figure 2 for the retrieve-and-rerank method, observe consistent performance gap between stronger and weaker embedding models. In cases where the reranker model has significantly higher capacity than the embedding model, the final retrieval accuracy is largely constrained by how many ground-truth answers are retrieved during the first-stage ANN search.
For our method RGS, although more powerful embedding models achieve higher retrieval accuracy at earlier stages, all models converge to similar accuracy under the Reranker @ 500 setting. This supports our claim that the embedding model primarily determines how many reranker calls are needed to reach a given retrieval accuracy, while the final performance is ultimately bounded by the reranker's capability.
Figure 2: Average NDCG@ 10
Reranker
Retrieve
result over a selected subset of datasets from BRIGHT. The left figure is our method RGS, the right figure is retrieve-and- rerank. Experiments involve 4 different embedding models with varying embedding quality.
40
3Bc0@10
30
80
Micro
Micro BGE Base
20
BCE: Base
20
BGE
Large
MiNtMistral
SFR.Mistral
10
10
200
300
400
500
100
300
400
500
Beranker Budget
Reranker Budget
We also study how our method performs across different reranker models. In Table5]rrom Appendix A we present experimental results subset of BRIGHT datasets using "gpt-4.1-minn" We observe that our RGS method consistently achieves the highest retrieval accuracy across various reranker budgets and datasets, although the final accuracy varies compared to Table [ This demonstrates the generalization ability of our RGS method, and highlights that different reranker models have their own specialties.
5.2 DIFFERENT ROLES OF DOCUMENT-DOCUMEN SIMILARITY AND QUERY-DOCUMENT RELEVANCE IN THE RETRIEVAL PROCESS
We perform fine-gra fine-grained ablation study to examine how embedding quality affects the retrieval accuracy of the three methods: RR, SlideGAR, and RGS. As mentioned in Section 3 we believe that an ideal embedding should satisfy two conditions: preserving document-wise similarity and approximating qucruedo relevance. In the following. we investigate the role of each by introducing different levels of perturbation to either the query or document embeddings.
Perturbation on query embeddings Fix ratio [0, 1J. we mix each query embedding with another random query embedding according to the ratio - w w. This simulates the situation where the embedding model fails to distinguish subtle differences between queries. In this case, the embedding model's ability to measure qucry-do relevance is compromised, while the document-wiss similarity structure remains intact, as the document embeddings are unchanged.
8

---

Published as a conference paper at ICLR 2026
Document
50
50
SlideGAR RR
Figure 3: For different levels of perturbation added on query (right) document(left) embed- dings, report the retrieval accuracy for different retrieval methods
30
ND00990
30
20
10
10
0.2
0.4
0.6
0.5
6
0.2
0.4
0.6
0.8
Perburbation Level
Porturbation Level
Perturbation on document embeddings Fix a ratio w [0, 1], we mix each document embedding with another random document embedding according to the ratio 1 w. This simulates the scenario where the embedding model fails to distinguish subtle differences between documents. In this case, both key functions of the embedding model-mmasuuring query-document relevance and preserving document-wise similarity- are compromised.
Please Figure for retrieval result the Psychology dataset from BRIGHT after applying two types of perturbations to the query and document embeddings. Note that w = 0 corresponds to the original, unaltered embeddings, while w represents the case where the embedding model completely confuses one document/query with another.
We observe that for RR, its performance remains consistent regardless of whether the perturbation is applied to the query or the document, as adding noise to either is symmetric under the sequential reranking process. Its retrieval accuracy drops to nearly zero w increases from 0 to
For SlideGAR, retrieval accuracy degrades more slowly when the perturbation is applied to the query embedding. This is because, even if completely incorrect query embedding results meaningless first-stage shortlist, SlideGAR can still leverage secument-document similarity to approach the correct answer.
For our RGS method, retrieval accuracy declines only slightly even when the query embedding contains no useful information (i.e., when w 1). This is because the query embedding used solely to initialize the starting point for RGS's second-stage search. A random starting point may slow down the search process but has limited impact on the final retrieval result.
5.3 ERROR ANALYSIS
perform an error analysis to understand why the performance of RGS stops improving as reranker budget continues grow. We categorize the ground-truth answers into three classes: (1) answers returned by the retrieval algorithm (orange), (2) answers seen during the retrieval process but not selected by the reranker (light orange), and (3) answers not seen at all during the retrieval process (gray). We plot the percentage of each class as the reranker budget increases for both RGS and the retrieve-and-rerad
method (see Figure 4 We observe that although the percentage of returned answers does not increase-and and sometimes even slightly decreases the percentage seen answers consistently grows with the reranker budget. This suggests that the final retrieval accuracy limited not by the efficiency of our retrieval algorithm, but by the fact that the reranker model's preferences do not always align with labeled relevance.
Reranker-Guided
For different Reranker budgets, each stacked bar shows the percentage of ground-truth passages that were retrieved (or- ange), examined but not selected (light orange), never exam- ined (grey). Results are re- ported for running RGS (left) and RR (right) methods on BRIGHT/Biology dataset.
Figure 4:
13.5%
11.8%
21.4%
33.2%
41.8%
49.8%
34.3%
34.4%
30.5%
15.0%
17.9%
23.9%
27.8%
14.7%
8.2%
43.3%
48 48.6%%
48.1%
52.2%
53.8%
31.99%
35.5%
39.0%
500

---

Published as a conference paper at ICLR 2026
5.4 IMPACT OF FIRST STAGE RETRIEVAL
Inspired by the observation that our RGS method still performs reasonably well even when the query embedding is completely incorrect, we question whether the first-stage nearest neighbor search based on the query embedding is necessary. To investigate this, we experiment with different initialization strategies for RGS (see Figure 5 When we initialize RGS with a noisy starting point (c.g.. the 1000th closest vector to the query) or use the default start point of the DiskANN algorithm, our method still achieves comparable retrieval accuracy- albeit with increased reranker usage. Notably, the default start point strategy does not require a first-stage search via vector similarity, nor does require the query embedding at all. This indicates that the initial vector similarity search is not essential for our method. In fact, when the first-stage search is entirely removed, the query embedding becomes unnecessary; all we need from the document embeddings is their encoding of document similarity.
BRIGHT//sychology
BRIGHT//Sychology
BHNAAN
0.50
+ Rasson
0.45
NDCC.810
NDCC.110
30-
0.40
0.35
20-
0.30
10
Savc Stitt
Mart
0.25
100
300
400
600
800
100
500
Reranker Budget
Beranker Budget
Figure 5: Impact of choosing different start points to initialize our second stage search on retrieval accuracy. Exact start means we start at the closest document to the query in the embedding space. Noisy start means we start at the 1000th closest document to the query. No start means we start at the default start point of the ANNS data structure 5.5
Figure 6: Impact of running our second-stage search on different graph data structures for re- trieval accuracy. DiskANN: proximity graph gen- erated by the DiskANN algorithm. KNN: each document connected to its k nearest neighbors in the embedding space. Random: a randomly connected graph with degree k.
IMPACT OF DIFFERENT GRAPH TYPES
Our RGS method relies heavily on the underlying graph data structure which reflects docuemnt-wise similarity. Here, investigate whether our RGS works on different graph structures. Besides the proximity graph built by the DiskANN algorithm, we also consider random graphs and KNN graphs, which are common choices in recent graph-based methods in the literature (Rathee et al. (025): ;
MacAwaney et al. (2022)) ). Please see Figure 6 We initialize RGS on the 50 documents closest to the query and observe how quickly the greedy search approaches the true answer. RGS still manages to improve on KNN graphs but the process is much slower, and it shows no improvement on random graphs. We hypothesize that in KNN graphs, edges exist only between nearby documents, while connections between distant documents are missing, preventing the search from reaching the correct neighborhood. This also explains why graph-based ANN algorithms (c.g., HNSW, DiskANN, NSG) do not perform greedy search on KNN graphs. For random graphs, it is expected that RGS shows no improvement because the search is equivalent to randomly scanning documents in the corpus.
6 CONCLUSION
In this paper, we propose Reranker- Reranker-Guided-ee Guuded-SSeachh to bypass the sequential reranking bottleneck. Our method leverages a proximity graph built on document- wise similarity and performs a greedy search guided by the reranker to traverse toward the correct answer. We conduct extensive experiments to demonstrate the effectiveness of our method on three reasoning-intensiv. retrieval benchmarks. Our ablation studies show that the quality of the embedding model affects the reranker budget required to achieve certain level of retrieval accuracy, while the final retrieval accuracy is primarily determined by the reranker model. A limitation of our RGS method is that adaptively choosing documents to rerank may hinder parallelization, which we leave for future work.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. Overview of the tree 2019 deep learning track. arXiv preprint arXiv:2003.07820. 2020.
Thibault Formal, Benjamin Piwowarski. and Stéphane Clinchant. Splade: Sparse lexical and expansion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference Research and Development in Information Retrieval, pp. 2288-2292, 2021.
Erlend Frayling, Sean MacAvaney, Craig Macdonald, and Iadh Ounis. Effective adhoc retrieval through traversal of query every-document graph. In European Conference Information Retrieval, pp. 89-104. Springer, 2024.
Cong Fu, Chao Xiang, Changxu Wang, and Deng Cai. Nsg Navigating spread-out graph for approximate nearest neighbor search. pattps://githane com/ /Jtm///Jeeeeaann/sas.. 2019.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104. 088208 2021.
Piotr Indyk and Haike Xu. Worst-case performance of popular approximate nearest neighbor search implementations: Guarantees and limitations. Advances in Neural Information Processing Systems, 2023.
Nick Jardine and Comelis Joost van Rijsbergen. The use of hierarchic clustering in information retrieval. Information storage and retrieval, 7(5):217-240, 1971.
Suhas Jayaram Subramanya, Fnu Devvrit, Harsha Vardhan Simhadri, Ravishankar Krishnawamy, and Rohan Kadekodi. Diskann: Fast accurate billion-point nearest neighbor search on a single node. Advances in neural information processing Systems, 32, 2019.
Suhas Jayaram Subramanya, Fnu Devvrit, Harsha Vardhan Simhadri, Ravishankar Krishnawamy, and
Rohan Kadekodi. Diskann.
2023.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis. Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP (1), pp. 6769-6781, 2020.
Sean MacAvaney, Nicola Tonellotto, and Craig Macdonald. Adaptive re-ranking with a corpus graph. In Proceedings of the st ACM International Conference Information & Knowledge Management, pp. 1491-1500, 2022.
Yu A Malkov and Dmitry A Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence, 42(4):824 836, 2018.
Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfr- embeediige-mistallenaacc text retrieval with transfer learning. Salesforce AI Research Blog, 2024. research. com/ sfr embedded mistral/
com/sir-embedded
Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with bert. arXiv preprint
190.009901.04085, 2019.
Hanseok Oh, Hyunji Lee, Seonghyeon Ye, Haebin Shin, Hansol Jang, Changwook Jun, and Minjoon Seo. Instructir: A benchmark for instruction following of information retrieval models. arXiv preprint arXiv:2402. 11.2.1443 14334, 2024.
Ronak Pradeep. Sahel Sharifymoghacdamm, and Jimmy Lin. Rankvicuna: Zero-shot listwise document reranking with open-souree large language models. arXiv preprint arXiv:2309.1508, 2023.
Zhen Qin, Rolf Jagerman. Kai Hui. Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, et al. Large language models are effective text rankers with pairwise ranking prompting. arXiv preprint arXiv:2306. /7563, 2023.

---

Published as a conference paper at ICLR 2026
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 9748-8763. PmLR, 2021.
Mandeep Rathee, Sean MacAvaney, and Avishek Anand. Guiding retrieval using llm-based listwise rankers. In European Conference on Information Retrieval, pp. 230-246. Springer, 2025.
Rulin Shao, Rui Qiao, Varsha Kishore, Niklas Muennighoff, Xi Victoria Lin, Daniela Rus, Bryan Kian Hsiang Low, Sewon Min, Wen-tau Yih, Pang Wei Koh, et al. Reasonir: Training retrievers for reasoning tasks. arXiv preprint urXiv:2504.2059) 2025.
Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Haisu Liu, Quan Shi, Zachary S Siegel, Michael Tang, et al. Bright: A realistic and challenging benchmark for reasoning-intensive retrieval. arXiv preprint arXiv:2407. 12880,, 2024.
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is chatgpt good at search? investigating large language models as re-ranking agents. arXiv preprint PxXiv:2304009422, 2023.
Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers. In European Conference Computer Vision, pp. 387-404. Springer, 2024.
Orion Weller, Benjamin Chang, Sean MacAvaney, Kyle Lo, Arman Cohan, Benjamin Van Durme, Dawn Lawrie, and Luca Soldaini. Followir: Evaluating and teaching information retrieval models to follow instructions. arXiv preprint arXiv:2403. 15246, 2024.
Orion Weller, Kathryn Ricci, Eugene Yang, Andrew Yates, Dawn Lawrie, and Benjamin Van Durme. Rank1: Test-time compute for reranking in information retrieval. arXiv preprint arXiv:2502. 18418, 2025.
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.
Haike Xu, Sandeep Silwal, and Piotr Indyk. A bi-metric framework for fast similarity search, arXiv preprint orXiv:2406.02891, 2024.
12

---

Published as a conference paper at ICLR 2026
A
FULL EXPERIMENTAL DETAILS AND RESULTS
Prompt for Gemini-2.0-Flash Here, we use a similar prompt from Sun et al. (2023) to ask Gemini-2.0-Flash to rerank the documents.
System
You RankGPT, an intelligent assistant that can rank answers based on their relevance to the query. I will provide you with 10 passages. each indicated by number identifier []. Rank the answers based on their relevance query: (query).
Instruction
[1] Passsgee 1) [2] PPsssage 2}
Messages
[10] {Passage 10}
Query: [query]. Rank the 10 passages above based their relevance the
query. The passages should be listed in descending order using identifiers. The most relevant passages should be listed first. The output format should be like [1] >[2] >[10]. Only response the ranking results, do not say any word or explain.
Table 4: Prompt for "Gemini-22 2.0-Flash""" serve reranker
Computational resource to reproduce experiments Our experiments Xeon(R) Platinum 848IC CPU with 44 cores and we use one NVDIA A A100 GPU to generate the embeddings. The estimated cost spent Gemini-2.0-Flashh reproduce Table T. Table and Table is $600.
Intel(R)
More metries Previously, we defined the reranker budget the number of documents seen by the reranker. However, this may not accurately reflect the total computational resources consumed during retrieval, as each document can be seen multiple times by a listwise reranker. Here, we plot the average number of tokens or API calls sent to the LLM-based reranker per query versus retrieval accuracy. Please see Figure for the results. We observe that RGS achieves the highest retrieval accuracy on BRIGHT under fixed budget, whether measured by token usage or API calls.
BRIGHT
BRIGHT
0.33
RGS
0.33
RGS
BA
0.32
-4-
0.32
0.31
0.31
NOC6810
0.30
NOCC@10
0.30
0.29
0.29
0.28
0.28
6.27
0.1
0.4
0.5 tokens)
50
125
175
200
Usage
Figure 7: Average NDCG@ 10 versus number of tokens API calls sent to the LLM-based reranker on the BRIGHT benchmark
Experimenting with "gpt-4.1-mini" We experiment with "gpt-4. 1-mini" on subset of the BRIGHT datascts. We observe that our RGS method consistently achieves the highest retrieval accuracy across various reranker budgets and datasets, although the final accuracy varies compared to the results Table | This demonstrates the generalization ability of our RGS method and highlights that different reranker models have their own strengths.
An example from TheoremQA-T Below we show an detailed example from TheoremQA-T, where our RGS algorithm able traverse the corpus graph and gradually approach the ground-truth
13

---

Published as a conference paper at ICLR 2026
No reranker BGE-Large
Reranker@ 100
Reranker@ 300
Reranker@: 500
Dataset
RR
SlideGAR
RGS
RR
SlideGARR
RGS
RR
SlideGAR
RGS
Biology
11.7
34.4
42.6
46.3
44.5
52.3
56.6
46.0
53.0
57.8
Psychology
17.5
32.3
34.7
42.8
43.2
42.2
50.7
41.9
42.3
49.4
Sustainable Living
13.1
33.3
38.5
41.1
36.7
42.3
42.9
36.7
43.2
TheoremQA-T
35.8
42.4
Table subset of the BRIGHT evaluation on "gpt--.1-mini" with varying reranker budgets for RR Satrieve-and-Rerante SlideGAR (Rathee et al. 2025), and our RGS Serraker-GGuied-Seee
answer, even when it is ranked low under cosine similarity. The first row is the query, and the second row is the starting point of our RGS algorithm. Each subsequent is neighbor of the previous document in the corpus graph. Following RGS, we eventually reach Document I 11592, which is ranked 2812 by cosine similarity but is in fact the ground-truth answer for the query.
B USAGE OF LARGE LANGUAGE MODELS
We use LLMs guide our search algorithm in our experiments. We also LLMs polish writing.
14

---

Published as a conference paper at ICLR 2026
Type
ID
Embedding Rank
Content
Query
Imagine you're 'r trying to tune a guitar. For each tuning setting æ, the amplitude is sin (xx)
1+
What does the amplitude approach + 02
Document
19940
22
Properties of the Real Sine Function. Let x E R. Properties of sin æ include continuity, absolute convergence of its defining series, sin 0 0, and odd symmetry sin(-z) sin Ø.
Document
18909
138
Power Series Expansion for Sine Function. The sine function has the Taylor expansion
T 2n+1
3
sin
Σ
(-1)
(2n 1)!
3!
===0
valid for all real r.
Document
17171
325
Boundedness of sin n/a. For all real x,
sin 2
S
Document
11216
126
Limit of x/ sin at Zero.
lim 1.
sin
Document
15227
2204
Derivative of Exponential at Zero. Let exp x: be the exponential
function. Then exp(x) 1 1. lim
Document
15118
581
Derivative of Logarithm at One. Let In x be the natural loga- rithm for > Then
In + x)
lim 2-50
Answer
11592
2812
L' L'Hôpital's Rule. Let ff and g be real functions continuous (a, bj and differentiable on (a, b), with g((xr) * 0 and s(a)
g(a) 0. Then
lim f(x) lim f( a b0 g(x)
x-aa g (x)
Table 6: An example from TheoremQA-T
15


Reviews:
[{"review_id": "86MG6LJ5lU", "reviewer": "Reviewer_nmG8", "summary": "The paper presents reranker guided search (RGS). The technique is summarized in Algorithm 1. Essentially, the approach iteratively decides which documents to retrieve next by (1) reranking the current set of retrieved docs, (2) using the top ranked doc (after reranking) to then decide which new docs to include, and repeating at step (1) until the budget is fulfilled. The results on multiple text and multi-modal datasets are strong.", "questions": "How does Recall@R compare using embedding-only vs embedding + proximity graph? Basically, what is the recall before listwise reranker applied?\n\n\"We believe that these benchmarks represent the most complicated retrieval tasks available, which can’t be solved by simple semantic similarity, and serve as a better testbed for evaluating the efficiency of different high-performing retrieval methods.\" There are a range of other tasks worth considering beyond this set. CRUMB (https://arxiv.org/abs/2509.07253) provides a good overview.\n\nIs it necessary to run the reranker each time at step 12 in the algo? Could the reranking simply be done at the end?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "S1. Retrieve and rerank is becoming dated. Listwise is certainly an improvement over cross-encoder, but the design space for search is massive yet very under-explored for papers that report results on the top datasets, including the ones mentioned in this paper. It is great to see new creative works like RGS being applied, and with seemingly good results. That being said, it's hard to calibrate the strength of the results without more baselines.\n\nS2. The paper includes extensive results and thoughtful analysis. I include some questions about new analysis, and questions about baselines, but the work is already very informative without these.\n\nS3. The results with perturbation are particularly fascinating. The field at large has started to question the need for vector search given that reranking can be used to filter down an initial candidate set, and this is a result in that same spirit.", "weaknesses": "W1. It is a major weakness that ReasonIR is not included as a baseline, nor other reranker alternatives like JudgeRank or Rank-R1. Basically, the results in Table 1-3 are very limited in their baselines, only using BGE embedding and not looking at many rerank alternatives.\n\nW2. Listwise rerankers are know to be very sensitive to order, for example see https://arxiv.org/abs/2306.17563 which shows that listwise rerankers in reverse or when re-run multiple times can have very different results. For this reason, it would be very helpful to show Recall@R in Tables 1-3. We can not tell otherwise if it is recall that is helping or a different ordering of results fed to the reranker. The text has \"the number [recall] is only 31% on BRIGHT [at top-100]\", which does suggest RGS is helping with recall, but it's not clear which datasets this holds for. Using Recall@R can even be used to compare all the methods without applying any reranking.\n\nW3. Intuitively, it's not clear why RGS should help with instruction following.\n\nW4. The benefits of RGS decline rapidly after budget hits 500. I actually wonder if it will start to get worse as the budget increases further. More importantly, it's not clear so clear whether 100 budget is needed, or much of the benefit can be achieved with only 50 or even 20.\n\nW5. Although time complexity of RGS is reported, there is no cost analysis between the different methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mSvaqfiNSA", "reviewer": "Reviewer_zg7B", "summary": "This paper proposes Reranker-Guided-Search (RGS), a graph-based retrieval pipeline that departs from the standard retrieve-then-sequential-rerank paradigm by using a greedy search on a proximity graph of document embeddings and letting an expensive LLM-based reranker steer which graph neighborhoods to expand. The method initializes from a small set of embedding-nearest seeds, iteratively expands the neighbors of high-ranked documents, and uses a sliding-window listwise reranker to keep the most promising candidates until a fixed reranker budget is exhausted. Empirically the authors evaluate RGS under a Reranker@𝑘 protocol on three challenging benchmarks, BRIGHT, FollowIR, and M-BEIR, and report consistent improvements over retrieve-and-rerank and a recent SlideGAR baseline, especially in low-reranker-budget regimes. The paper also presents ablations on embedding quality, perturbations to embeddings, and an analysis suggesting final performance is often limited by reranker alignment rather than search coverage.", "questions": "See Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses an important practical bottleneck: how to get more value out of a small number of expensive reranker calls. The idea of combining graph-based ANNS traversal with reranker guidance is natural but underexplored, and the RGS design is a clean, well-motivated instantiation of that idea: it leverages the clustering hypothesis and the bi-metric intuition to trade cheap embedding similarity for targeted reranker effort. The experiments are broad and focused on modern, hard benchmarks that stress reasoning and multi-modality rather than only classical TREC-style tasks; showing gains across BRIGHT, FollowIR, and M-BEIR strengthens the claim of generality. The Reranker@𝑘 evaluation protocol is also a useful contribution because it isolates the real-world constraint of costly reranker usage and makes comparisons fairer.", "weaknesses": "First, practical costs and latency tradeoffs are not fully quantified: report wall-clock latency, token usage, and parallelization implications (RGS is sequential by design) so practitioners can judge deployment feasibility. \nSecond,  novelty relative to GAR/SlideGAR is claimed but the differences are somewhat qualitative.\nThird, statistical significance and variance across random seeds or query subsets are not reported; adding confidence intervals would strengthen the empirical claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4977jlpln5", "reviewer": "Reviewer_BB8G", "summary": "The proposed method is Reranker-Guided-Search (RGS), a novel technique that directly retrieves documents based on a reranker's preferences using a greedy search on a proximity graph, rather than following the traditional sequential retrieve-then-rerank pipeline.", "questions": "-The paper focuses on the quality of initial top-k ranked results, and proposes a corresponding searching method to tackle the tradeoff between effectiveness and efficiency. The proposed metric rerank@k is used for evaluation in the experiments. But we need some traditional metric like ndcg@k for comparison in Table 1 and 2. Though NDCG@10 is also provided, the correlation between rerank@k and ndcg@k leaves unknown. Moreover, the reasonability of rerank@k is needed to be verified.\n\n-The paper must be self-contained. Some notations like q_t\\rightarrow c_i in Table 2 need to be explained.\n\n-A detailed analysis should be provided to explain why the proposed method RGS performs less better on multi-modality retrieval datasets than scientific and math reasoning datasets.\n\n-Time complexity of baseline methods retrieve-and-rerank and SlideGAR needs to be analyzed like section3.4.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-The proposed method aims at a critical retrieval bottleneck.\n\n-For effective reranking with limited budget, RGS strategically selects the most promising documents to rerank, achieving significantly better performance (as evidenced by the reported gains of 3.5-5.1 points) without increasing computational cost. \n\n-The method is empirically validated across multiple diverse benchmarks BRIGHT, FollowIR, M-BEIR, demonstrating its robustness and generalizability.", "weaknesses": "-The proposed method is highly dependent on the embedding model. Compared with Retrieve-and-Rerank in Figure2, the proposed method is less sensitive to the choice of embedding model with budget more than 200. However, the highly dependency on the embedding model is still similar to the Retrieve-and-Rerank method.\n\n-Why the proposed method RGS is better for reasoning intensive search tasks? Which componets of RGS play a critical role for this advantage?\n\n-RGS claimed that the novel introduction of documentation similarity will prioritize more promising documents and improving complexity query-document relationships. However, SlideGAR has used such information to improve search performance. The difference between them needs to be clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CZhLkmdsIG", "reviewer": "Reviewer_Y7xP", "summary": "This paper proposes a novel method Reranker-Guided Search (RGS) to address two key limitations of the traditional “retrieval-reranking” pipeline: 1) Its performance is constrained by the quality of the Top-K documents returned by the initial retrieval stage; 2) The computational cost of large language model (LLM)-based rerankers is prohibitively high, limiting the number of documents that can be processed. RGS bypasses sequential reranking by performing a greedy search directly on the document proximity graph generated by an approximate nearest neighbor (ANN) algorithm. This allows for strategic prioritization of more promising documents for reranking based on the reranker's preferences. Experiments on BRIGHT, FollowIR, and M-BEIR demonstrate that RGS achieves significant performance gains over sequential reordering methods.", "questions": "1. Given that RGS requires multiple steps to return its final search results, time delay must be considered as a key metric for evaluating RGS methods.\n2. Case studies can be used to highlight improvements and enhancements to RGS, as well as whether it has a corrective effect on irrelevant retrieval results.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.  Compared to the Retrieve-and-Rerank approach, reranking results are imperceptible to the retrieval process. In RGS, reranking results can help improve the retrieval process, thereby achieving performance optimization.\n 2. On three challenging benchmarks: BRIGHT, FollowIR, and M-BEIR, RGS achieved significant performance improvements over both the traditional method and SlideGAR. This demonstrates its effectiveness.\n 3. Experiments demonstrate that RGS is robust to noisy query embeddings. Even when query embeddings are of low quality, they can still identify relevant documents by leveraging the local structure of the document graph and guidance from the reordering module.", "weaknesses": "1.  RGS requires less computational effort, but each step's result depends on the previous one, resulting in longer processing delays compared to traditional reranking methods. In retrieval systems, this may be unacceptable to users. And RGS requires pre-constructing a proximity graph for the document collection. For large-scale document repositories, building and storing such a graph structure incurs additional preprocessing costs and memory overhead.\n2. For an initial retrieval result that is completely irrelevant and isolated in the embedding space, RGS struggles to correct it through re-ranking into a completely different, highly relevant region. Does RGS possess a certain degree of corrective capability?\n3. Although the effectiveness of RGS was validated across three datasets, these datasets are relatively small compared to real-world datasets such as MSMARCO. RGS requires validation of its effectiveness on a larger scale and should supplement its efficiency assessment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)