Retro*: Optimizing LLMs for Reasoning-Intensive Document Retrieval
Authors: Junwei Lan
Jianlyu Chen
Zheng Liu
Chaofan Li
Siqi Bao
Defu Lian
Venue: ICLR 2026 Poster (2026)
Keywords: Information Retrieval
LLM Reasoning
Reinforcement Learning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: With the growing popularity of LLM agents and RAG, it has become increasingly important to retrieve documents that are essential for solving a task, even when their connection to the task is indirect or implicit. Addressing this problem requires fine-grained reasoning to accurately assess the relevance between the task and each candidate document. This capability, however, poses a significant challenge for existing IR techniques. Despite recent progress in reasoning-enhanced IR, existing approaches still face significant challenges in applicability, scalability, and efficiency. In this work, we propose **Retro\***, a novel approach for reasoning-intensive document retrieval. Our method introduces a rubric-based relevance **scoring mechanism**, enabling the model to reason about the relationship between a task and a document based on explicitly defined criteria, whereby producing a fine-grained, interpretable relevance score. Retro\* also supports **test-time scaling** by combining multiple reasoning trajectories via score integration, which produces more reliable relevance estimates. To optimize Retro\*'s reasoning capabilities, we introduce a novel **reinforcement learning** algorithm tailored for its relevance scoring mechanism, which employs two composite rewards to fully exploit the trajectories of each training sample. Our experiments show that Retro\* outperforms existing document retrieval methods with notable advantages, leading to **state-of-the-art** performance on the BRIGHT benchmark.

Published as a conference paper at ICLR 2026
RETRO* * OPTIMIZING LLMs FOR REASONING-
INTENSIVE DOCUMENT RETRIEVAL
Junwei Lan 1.24 Jianlyu Chen 1.244 1. Zheng 25 Chaofan Li Siqi Bao Defu Lian 1:4
1 University of Science and Technology of China 2 Beijing Academy of Artificial Intelligence
3 Beijing University of Posts and Telecommunications
4 State Key Laboratory of Cognitive Intelligence Hong Kong Polytechnic University
6 Hong Kong University of Science and Technology
1 jiw1 jiv33mmiil ustu. edu.cn
(eengggill226@mmmer com
LiandefuRustc.eci
ABSTRACT
With the growing popularity of LLM agents and RAG, it has become increas- ingly important to retrieve documents that are essential for solving a task, even when their connection to the task indirect or implicit. Addressing this prob- lem requires fine-grained reasoning to accurately assess the relevance between the task and each candidate document. This capability, however, poses a signif- icant challenge for existing IR techniques. Despite recent progress in reasoning- enhanced IR, existing approaches still face significant challenges in applicability, scalability, and efficiency. In this work, propose Retro*, novel approach for reasoning-intensive intensive document retrieval. Our method introduces a rubric-base relevance scoring mechanism, enabling the model to reason about the relation- ship between task and a document based on explicitly defined criteria, whereby producing fine-grained, interpretable relevance score. Retro* also supports test- time scaling by combining multiple reasoning trajectories via score integration, which produces more reliable relevance estimates. To optimize Retro*'s reason- ing capabilities, we introduce a novel reinforcement learning algorithm tailored for its relevance scoring mechanism, which employs two composite rewards fully exploit the trajectories of each training sample. Our experiments show that Retro outperforms existing document retrieval methods with notable advantages, leading sate-offthhe-rt performance on the BRIGHT benchmark.
INTRODUCTION
Large language model (LLM) agents have become increasingly important for tackling complex tasks such as software engineering, mathematics, and scientific research (Chan et al.) 2024 Jin et al.) 2025 Wei et al.) 2025 Phan et al., 2025). In these applications, attrevallauugment
generation (RAG) Lewis al. 2020 Gao et al. 2023) plays crucial role, access to external knowledge is often necessary to produce high-quality solutions. However, in many scenarios, retrieval models must identify useful documents, even when their connection the task is indirect implicit, which makes the retrieval process particularly challenging. For example, in software engineering, a re- trieval model may need locate programs that share similar design patterns with the target problem rather than matching exact code snippets (Jimenez et al., 2023). In mathematics, it might involve retrieving proofs derived from the same underlying theorem, even if they are expressed differently (Chen et al. (023). Solving such tasks requires fine-grained reasoning to bridge subtle connections between the task and candidate documents. However, existing retrieval models are primarily de- signed to capture straightforward semantic relationships, such as matching question-answer pairs or identifying paraphrases (Lee et al. 2019; Karpukhin al.) 2020). Consequently, they often struggle with the complex reasoming required to uncover these deeper, more abstract connections.
Enhancing the reasoning capabilities of LLMs for reatonin tasks is a central focus of current research. One promising strategy is test-time scaling, which guides or reinforces LLMs to
"Core Contributors.
"Correponning authors.

---

Published as a conference paper at ICLR 2026
generate long-form thoughts before arriving final answer inference time (Wei et al. 2022 Jaech et al.] 2024; Guo et al.) 2025). This approach also enables the LLMs to explore multiple reasoning paths, evaluate alterattves, and ultimately arrive at more accurate solutions for complex problems (Wagg et al. 2022). Inspired by these advances, recent information retrieval (IR) research has begun to leverage the reasoning capabilities of L.LMs for reasoning- inttenniiee document retrieval
tasks (Su et al. 2024 Niu et al. [024] ZZuang et al. 2025 Weller et al. (022). Current methods typ- ically follow one of two paths: some directly prompt general-purpose LLMs to perform fine- fine-grained relevance analysis, while others optimize these models with fine-tuning algorithms to elicit more structured and systematic reasoning behaviors. Despite recent progress, current approaches exhibit three key limitations in relevance-measuring functionality, test-time scalability, and parallelism:
Lack of relevance-measuring functionality. Many RAG applications require specific func- tionality: the direct and interpretable measurement of document relevance. However, existing methods primarily provide relative ranking orders, which cannot capture the absolute level of relevance needed by downstream tasks.
Inflexible test-time scalability. Existing methods mainly focus on generating a single, long- form thought to reach answer. However, they neglect the significant potential of exploring and integrating multiple reasoning paths to achieve more reliable performance.
Limited parallelism capability, Existing methods, which are primarily listwise (Sun et al. 2023) setwise (Zhuang et al. 2024), must sequentially process the entire candidate set to produce the final retrieval result. This inherently sequential design is prone to substantial latency, especially when handling large group of candidate documents.
In this work, we propose Retro (Retro-star), novel LLM-based retrieval model designed for reasoning-intensive IR tasks. Distinct from existing approaches, Retro* is built on two key designs:
Rubric-based relevance scoring. Retro introduces a fine-grained of relevance rubrics, which explicitly define the relevance scores and their interpretations. Based on these rubrics, Retro* performs pointwise reasoming on the relationship between query and its candidate documents, producing concrete relevance scores with clear, interpretable meanings. This design enables direct measurement of relevance, rather than merely providing a relative ordering of documents.
Test-time scaling by score integration. Building on its rubric-based scoring, Retro* supports test-time scaling by generating multiple trajectories for each query-document pair and integrating their individual scores based on score similarity, resulting in more reliable and stable estimate of document relevance.
With these designs, Retro* can serve both relevance measuring and re-ranking applications. Further- more, its architecture naturally supports both flexible test-time scaling and high parallelism, making it proficient performing effective and efficient reasoning for complex retrieval problems.
To bring these designs to their full potential, we introduce a novel reinforcement learning (RL) algorithm with two composite rewards further optimize Retro*'s capabilities. The composite wards are designed to fully exploit the trajectories of every training sample during RL training. The Intra-Document Reward guides the policy model assign accurate relevance scores for each indi- vidual document, whereas the Inter-Documee discriminate the relevant document from an irrelevant one. To stabilize the training process and pro- vide Retro* with an initial reasoning ability, we incorporate a warm-up supervised fine-tur theettnnng (SFT) stage. This stage not only equips the model with basic reasoning skills, but also shapes the model to generate concise and well-structured thoughts before the RI. stage.
Reward incentivizes the policy model to effectively
To evaluate the effectiveness of our training strategy and the overall performance of Retro* on geasoning-intensivy IR tasks, we conduct experiments on BRIGHT (Su et al.) 2024), compre- hensive benchmark encompassing 12 datasets across science, mathematics, and programming. Ex- perimental results demonstrate that Retro achieves significant improvements over strong baseline methods, with substantial performance gains from the proposed test- test-time scaling mechanism and reinforcement learning method. To facilitate future research in this area, all resources leased at https:: / / qithub..ubu. com htt.bb/ccm/eeetttpaaeeaa.ae hrtpa//aaa..apptteeeeeeaeeeee/ee
Ret.r star
2

---

Published as a conference paper at ICLR 2026
Here is the relevance definition... RELEVANCE, SELLEVANE__PLAEHHOE PLACEHOLDER
given query and document... your
to perform following aceps.
1. Query Analysis: Think zeason and cescribe what information would... Dosument Analysia: Discuss how information provided the document...
3. Relevance Annotation:
annotate an integer score Irom 0 to 100. ...following guide: illighly Relevant):
60-80
(Releveat]:
40-60 (Modarstely Relevant):
20-43 (Slightly Relevant):
20 (Izzelevant):
conclude your entire reaponse with the Final relevance score...
<score> [.... </score>
Query: Beginn queryyQQERY QUERY IMPOT Document: [Begin of Document JDOCUMENT. INPoT[End of Document]
Quary)
Figure I:
Relevance rubric for Retro*, The Relevance Placeholder allows users to specify the defi- nition of relevance, while a 5-level criteria ensures consistent and interpretable scoring result.
2 RELATED WORK
Reasoning Large Language Models. Enhancing the reasoning capabilities of LLMs is crucial for tackling complex tasks. Techniques such as oh-in-of- prompting (Wei et al. 2022), which guide LLMs to reason step by step before producing the final answer, have shown significant performance gains. More advanced sampling strategies, including self-consistency (Wang et al.)
2022), tree-of-thought (Yao et al.. 2023). and Monte Carlo tree search (Xie et al. 2024). further enhance their reasoning quality and reliability. To unlock the full potential of LLM reasoning, recent research has increasingly explored reinforcement learning, which directly optimizes the model to
generate high-quality reasoning trajectories (Jaech et al. 2024 Guo et al. 2025 Yang et al, 2025a).
Reasoning-enhance IR Methods. Inspired by the recent advances in reasoning-capable LLMs, growing body of research has begun adapting these paradigms to IR, with re-ranking tasks emerging as a primary focus. Preliminary studies have explored a variety of strategies, ranging from zero- shot prompting (Niu et al., 2024), to distilling reasoning trajectories from powerful reasoning L.MMs through supervised fine-tuning (Weller et al.) [025] Yang et al.) (0255), and even reinforcement learning with carefully designed re-ranking rewards (Zhuang et al. 2025 Liu et al. 2025). Despite the promising progress of these approaches, there remains a lack effective methods for directly estimating the strength of relevance between query and document. Moreover, existing methods face limitations in terms of test-time scalability and parallelism, which significantly hinder their overall accuracy and inference efficiency.
METHODOLOGY
In this section, we first introduce the basic framework of Retro*, including the rubric-based mecha- nism for relevance scoring and the score integration strategy for test-time scaling. We then describe how we optimize Retro*'s performance by reinforcement learning with tailored rewards.
3.1 RUBRIC- CUBRIC-BASED RELEVANCE SCORING
Retro is designed to both measure the absolute relevance of each document and re-rank set of candidate documents. To this end, we introduce a rubric-based scoring mechanism that guides the model perform reasoning the relationship between query and its candidate documents based on a well-defined relevance rubric, yielding relevance scores with interpretable meanings.
3.1.1 RELEVANCE RUBRIC
The relevance rubric (I)) consists of two parts. One the Relevance Definition, where the specific intent of the retrieval task is declared in the [Relevance Placeholder). For example, may specify an intent to retrieve proofs that are grounded the same underlying theorem as the input. The other one is the Scoring Criteria, which defines the scope and rules for assigning relevance scores. In
3

---

Published as a conference paper at ICLR 2026
Source Dataset
Stage
Relevance Defuition: Diven query (marh problens) document (musl problle solation), the document rélevunt to
Rubric
(1,
Be query if the thooreses used the docusent provide
Query
Sampling
(2'2 92)
Filtring
SFT Dataset
Arlpjut insights for cobvnn the problem the
Triplets Collection
Doe d
Teacher Model
X Times
Posltives
Negatives
Source Sample
Raw Trajectorics
profice solution
mooh prosiion solustion
Initial Policy
Supervsel
Fine-Tuning
Stuge
Reward Assgaament
Ruhrie r
Of
itss closest to 5, and constraints holds
Query
(F. (
(1 +
Rintr S, (1)
-1
if furthest to s, and constraints holds otherwise
Rolling Out
Positive Doe
(T. q.q.
X Trijeturics
Pollcy Model
Runat .21
rst
Negative
Training Samples
RGvs
Doc
GRPO
Rune Bf b/))
n Maj aat)
(1 Rame()))
Source Sample
Figure 2: An overview of the two-stage training. SFT: the model is warmed up with filtered data from powerful teacher model. RL: the model is reinforced with the tailored composite reward.
our work, the relevance score is represented as an integer between 0 and 100, where higher scores indicate stronger relevance. The scope is partitioned into multiple intervals, each with explicit, interpretable meaning. For instance, score within the range of 80-100 indicates that the candidate document is highly relevant and comprehensively addresses the information need of the query. overview of the rubrics is shown in Figure [ with the full rubrics provided in Appendix
Based on this well-defined rubric, R is prompted to reason about the relevance between given query (g) and a candidate document (d), generating reasoning trajectory (y) along with corresponding relevance score (s): Retro[T, 9, d) > (y, s). See Appendix for examples.
3.1.2 TEST-TIME SCALING VIA SCORE INTEGRATION
To enhance retrieval accuracy, Retro leverages test-time scaling by sampling K times for each query-document pair, resulting in of reasoning trajectories: mon approach to integrate these results is majority voting, where the most frequent score chosen as the final score. While this approach works well for highly discrete outputs, not appropriate in Retro*, as drawing a reliable result would require vast number of samples, making the process computationally expensive and impractical.
(yK, sk)). A com-
To this end, we propose a Score Integration strategy in simple yet effective way: Σx s// Σx Ws where s; is the relevance score from the i-th trajectory and w; is its associated weight. The weights can be set based on the generation likelihood of each trajectory. When likelihoods are unavailable or just for simplicity, uniform weighting scheme, w; 1/K, can be used.
3.2 TRAINING STRATEGY
To optimize Retro*'s reasoning capabilities for reasoning intensive document retrieval tasks, we propose two-stage training strategy. The process starts with supervised fine-tuning (SFT) stage to warm up the model, followed by reinforcement learning (RL) stage for further performance enhancement. The overview of the training strategy is illustrated in Figure
3.2.1 SUPERVISED FINE-TUNING
In the first stage, perform SFT to equip the model with an initial reasoning ability and shape the model to generate concise and well-structured thoughts. The key of this stage lies in our approach to Training Data Curation, which involves crucial steps:
4

---

Published as a conference paper at ICLR 2026
Data Sourcing. We begin with collection of query-document pairs {(g. d)1..... ... (q. d) }. For each pair, we leverage a powerful teacher model to reason about the relationship between q and d based on relevance rubric: T(I, q. d) - (y.s), s). Besides, the teacher model is also guided to keep its thoughts concise, with an explicit length control instruction.
Data Filtering. To obtain high-quality training data, filter the trajectories with score inte- gration. Specifically, we sample the teacher model K times for each query-document pair, resulting in group of raw trajectories {(V1, si),..., (yK, sx)). We then integrate these scores to a score 8, as it serves as a reliable reference. Finally, the trajectory whose score is closest to s is selected, so that only one trajectory is used for each query-document pair. resulting in a curated dataset for SFT training: {(g, d, G. 8) s)... (g. d, 9. a) }. where (ú, 8) is the selected trajectory.
3.2.2 REINFORCEMENT LEARNING
In the second stage, we take the SFT model as the initial policy model and perform RL to further optimize its reasoning capability. As discussed carlier, Retro* is designed to achieve two key func- tionalities: accurately scoring the absolute relevance of individual documents and correctly ranking them by assigning higher scores to more relevant ones. To jointly optimize these functionalities, we introduce composite reward that fully exploits the trajectories of each training sample during RL.
Intra-Document Reward. Intra-document reward is designed improve the scoring accuracy for each individual document. It incentivizes the model to generate high-quality reasoning trajectories that produce stable and accurate relevance scores for the same query-document pair across multiple attempts. To this end, the reward is computed in two steps. 1) Rolling Out: For each query-document pair (q, d). the model rolls out N trajectories: {(11, s1)..... (yN, [ 2) Reward Assignment: The integrated score over all trajectories presents more reliable result than each individual score in expectation. As such, the trajectories with scores closer to & are preferred, while those deviating further from 8 are disfavored. For simplicity and robustness, we adopt a ternary function, where the trajectory with the closest score is assigned a reward +1, the furthest one is assigned -1, while the others are assigned 0. To prune the trivial cases where all trajectories already reach an agreement, we also introduce a threshold 7 requiring a minimum gap between 8 and the furthest deviate score. The above reward is formulated as the following function:
1
if is closest to s, and the threshold 7 constraints holds
Rintea (y, s)
-1 if is furthest from 8, and the threshold constraints holds otherwise
0
Diee--Douumett Reward. Inter-document reward focuses on improving the reranking performance for a group of candidate documents. It incentivizes the model to assign higher relevance scores to more relevant documents, thereby resulting in a correct ranking for a given task. In our work, we sample one positive (d't) and one negative (d document from the candidate set for each query, and the reward is computed via the following steps: 1) Rolling Out: For each document d E {dt }, the model rolls out N trajectories: [(vl sf),.... (vh s d E {dt, d }}. 2) Reward Assign- evaluated based whether their
ment: The trajectories of both positive and negative samples scores correctly reflect the ranking. For positive sample's 's trajectory (w,,, s;)), the reward is the proportion of negative ones whose scores are dominated by 57. Conversely, for a negative sample's trajectory (Uj , +5y ). the reward is the proportion of positive ones whose scores dominate 8,- These relationships are formally defined by the following equations:
N
N <
Rinter (MT,
N T(xT
Σ
> 45),
Rinter (yj .87)
1
N 1=1
1=1
where I(-) is the indicator function, and N is the number of trajectories for each q-d sample.
The intra-document reward and inter-document reward are combined with a parameter a E (0, 1) to obtain the composite reward: R(y. s) Rintra (y, 8) + (1-x:) -Rinter (v. 8), where Rintra guides the model to assign reliable relevance scores for each individual document, and Rinter incentivizes it to capture the relative rankings within group of candidate documents. The composite reward is then optimized using the Group Relative Policy Optimization (GRPO) algorithm (Shao et al. 2024). with further analysis of the hyperparameter a provided in Appendix F
5

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
In this section, we evaluate the overall performance of Retro* and the effectiveness of our detailed strategies. We focus on the following questions: 1) RQ1: How does Retro perform in reasoning- intensive scenarios? 2) RQ2: How does Retro*'s performance compare to other non-reasoning and reasoning-enhancer re-ranking methods? 3) RQ3: How scalable is Retro* with the size of the backbone model and the number of test-time samples? 4) RQ4: How does Retro*'s parallclism capacity impact its efficiency compared to listwise and setwise methods, particularly the number of candidate documents grows? 5) RQ5: Does Retro"s, 's rubric-based scoring mechanism provide a more reliable measurement of absolute relevance strength compared to those uncalibrated baselines? 6) RQ6: How do the key components of our proposed training strategy individually and collectively contribute the effectiveness of Retro*?
4.1 EXPERIMENTAL SETUP
Baselines and Backbone Models. Our comparison includes several leading re-ranking models, categorized as non-reasoning and reasoning-enhancee approaches. For the non-reasoning baselines, we choose the pointwise model RankLLaMA (Ma et al. 2024) and the listwise model RankZephyr (Pradeep et al., (023). In particular, RankLLaMA is built upon the Llama-2 (Touvron et al., (023), and RankZephyr is built upon the Mistral-vO.1 (Fiang et al.] 2023). For the reasoning-enhange baselines, compare against several powerful approaches built upon the Qwen2.5- Instrruuct fam- ily (Qwen et al.] 2025). These include the fine-tuned models Rank (Weller et al.) (025), Rank-R1 (Zhanng et al. 2025), and ReasonRank et al. (025), as well the zero-shot method JudgeRank
NNuu et al. 2024). The details for each model are provided in Appendix B To ensure fair com- parison, we align with the baselines by employing Qwen2.5-7B. 7-nn..5-BB-Isttruet and Qwen2.5-3255 -n2-strrett as our backbone models. In addition, we also provide results trained on other backbone models in Appendix G
Evaluation Settings. We evaluate all baselines and Retro* on the BRIGHT (Su et al. 2024 bench- mark, which is designed for reasoning-intensio benchmark and provide the detailed definitions in Appendix L Our Retro* evaluation is conducted using the FlagEmbeddiig (Xiao et al. 2 framework, and inference is accelerated with SGLang (Zeng et al. 2024a). For simplicity, tion strategy, which we denote as mean-score @ k. To provide a more challenging and comprehensive evaluation, use the powerful
IR. We define the relevance following the BRIGHT
employ uniform weighting scheme for our score integra-
the first- stage retriever, which is the current state-of-the-art embedding model on the BRIGHT benchmark. We utilize its publicly available search results, they provide more positive document candidates. In our evaluation, all methods re-rank the top-100 documents retrieved by this embedder with the original query, and we report nDCG@ @ 10 as the performance metric.
GGGE-Reasnne--ee Embedd--bbeeeeeeeee
(BGE- (BGE-Rcasoner-
Training Data. We construct our training data based on the AGE-Reasoner-De originally re- leased with the BGE--Reasoner-Embee model. BGE-Reasoner-Datt is synthetic, reasoning-centric retrieval dataset that covers all domains in the BRIGHT benchmark. Specifically, its queries are constructed using a documenn-tooqueey generation pipeline, where documents from each BRIGHT domain are used as source material to generate synthetic reasoning-intensive queries. These queries are then used to retrieve the candidate documents, which are finally annotated as positives or neg- atives using powerful LLMs. As result, the dataset provides high-quality synthetic queries and relevance annotations, suitable for training reasoning-intenss sample 500 queries per dataset for SFT and 1,000 queries per dataset for RL. Each query is asso- ciated with one positive and one negative document, yielding a total of 12,000 training samples for SFT and 24,000 for RL. For SFT, obtain the reasoning trajectories from a strong teacher model, Dwen3-235B-A228 (Yang et al. 2025a). For each training sample, we sample eight trajectories from the teacher model and train the model with only one selected trajectory. Also, we augment the original prompt template (shown in Figure g) with an explicit instruction asking the teacher model to keep its reasoning concise within 512 tokens as described in Section 3.2.I Generating all tra- jectories for SFT under this setting on 8 NVIDIA H100 GPUs takes less than 6 hours. We discuss
retrieval models. From this dataset,
1PCF--Ressooe--Fmmecd0031
com http:s///dttaooococccc/ooooeeees
matter/reeseerrc///RBeeeee Raa.kkeee_RReeeeeeee_

---

Published as a conference paper at ICLR 2026
Table 1: Performance on BRIGHT benchmark (nDCG@ 10), where all methods re-rank the top-100 documents retrieved by AGEE-Reasoner-mmle Retro" achieves the leading performance, with test- time scaling further pushing its edge.
StackExchange
Coding
Theoren--usee
Models
Methods
Avg.
Bio.
Earth.
Econ.
Psy.
Rob.
Stack.
Sus.
Leet.
Pony
AoPS
TheoQ.
Theo
HEEF-Reasnne--mmba
Retriever
32.5
42.6
42.6
27.8
37.3
26.4
29.6
30.6
36.9
25.7
34.9
46.1
Non-Romsoning Re-Ranking Baselines
Lan (7B)
Pointwise
19.7
20.8
31.7
11.1
19.1
18.2
10.9
20.0
16.7
57.9
6.3
10.3
13.4
RankLLaMA (14B)
Pointwise
21.3
26.4
34.1
19.1
24.4
19.9
15.6
22.0
17.2
41.8
11.8
16.6
RankZephyr (7B)
Listwise
20.8
27.1
22.7
19.9
21.9
13.2
22.3
22.1
50.6
21.3
12.4
Reasoning- Enhanced Re-Ranking Baselines
JudgeRtank (7B)
Pointwise
12.7
14.7
16.9
10.8
9.5
7.7
7.1
10.8
11.4
16.7
5.8
9.2
31.9
JudgeRank (32B)
Pointwise
17.4
21.4
26.0
13.1
12.9
19.7
19.9
16.4
31.5
Rankl (7B)
Pointwise
25.9
43.2
33.1
21.6
33.2
19.8
20.7
28.8
7.5
28.2
8.3
30.9
35.7
Rankl (32B)
Pointwise
29.7
46.0
31.5
27.2
33,9
19.5
22.8
33.1
17.2
36.2
15.8
28.2
44.6
Rank-RI R1 (7B)
Setwise
25.8
34.9
29.7
26.8
35.5
21.6
17.5
29.4
24.2
15.6
25.1
43.1
Rank-RI (148)
Serwise
31.8
46.6
43.3
28.9
42.2
30.3
26.5
38.0
18.5
18.5
10.9
33.9
44.0
ReasonRank
Listwise
33.5
44.4
37.1
ReasonRunk (32B)
Listwise
36.6
54,9
50.1
38.0
49.1
35.1
35.3
44.6
23.2
15.8
10.1
39.2
44.4
Our Models
Retro* (7B)
Pointwise 36.6
53.7
55.9
35.6
47.9
34.0
35.6
39.3
17.6
29.8
9.6
35.4
45.0
Retro* (32B)
Pointwise 38.5
59.4
58.3
41.8
48.3
38.4
13.9
Test. Scaling (Mean- Score
(Mean-Score@
Retro" (7B) Retro* (32B)
Pointwisc
38.7
58.4
59.2
35.0
49.3
33.9
37.7
41.1
18.8
33.5
10.7
40.2
46.7
Pointwise 40.6
61.4
61.3
41.7
50.8
37.8
41.9
46.8
13.5
32.0
12.2
39.7
47.8
Table 2: Performance (nDCG@ 10) on the BRIGHT benchmark using different first-stage retrievers: BM25, ReasonIR, where the top-100 documents re-ranked by Retro*
Models
BM25
Retro (7B)
Retro (32B)
ReasonIR
Retro (7B)
Retro* (32B)
@
@ 16
@1
@ 16
@ 16
@1
@
Avg. 27.0
35.3
37.0
36.6
38.5
30.6
36.8
38.4
37.4
39.5
the impact of this length control instruction Appendix available in Appendix
The training details of SFT and RL
4.2 MAIN RESULTS (FOR RQI AND RQ2)
Results in Table cleearl demonstrate that our Retro models achieve state-of-the-art performance on the BRIGHT benchmark. Specifically, our Retro* (7B) achieves an average nDCG@ 10 of 36.6, substantially outperforming all other 7B 7c-seale non-reasoning and reasoning enhanced baselines, and surpassing ReasonRank (7B) by 3.1 points. Scaling up to the 32B extends this lead, achieving a remarkable score of 38.5, and surpassing ReasonRank (32B) by 1.9 points.
reasoning-enhanced
Furthermore, with test-time scaling, the performance of our models further improved. Specifically, averaging over 16 sampling scores (mean-score @ 16) boosts the 7B model to 38.7, while enhancing the 32B model to 40.6. Notably, the scaled 7B model (38.7) 7) not only surpasses all baselines but also exceeds the performance our own standard 32B model (38.5), powerfully demonstrating both the effectiveness and potential of test-time scaling.
Additionally. we evaluate its performance on candidates from different first-stage retrievers, namely BM25 and ReasonIR (Shao et al.) 2025). As shown in Table 2 our model's performance remains consistent across different first-stage retrievers, which demonstrates the effectiveness and robustness of its ranking capabilities. More detailed results for BM25 and ReasonIR are provided in Appendix D To further assess the out-of-domain generalizability of our rubric, we evaluate Retro on R2MED eLi al. 2025), biomedical reasoning retrieval benchmark that lies outside the scientific, program- ming, and mathematical domains used during training. We provide detailed analysis these results in Appendix[] In addition to R effectiveness in reasoning -otouiinnnittnniveee intensive scenarios, also empir- ically validate its generalizability on traditional IR scenarios using the BEIR (Thakur et al. 202T),
with detailed results provided in Appendix E
7

---

Published as a conference paper at ICLR 2026
Retro
42.6
ReasonRanic
3000
Rank R1
mDCG
36.1
28.5
35.
Inference Time
96.8
25.3
Retro* (33)
Retro* (78)
Retro (325)
N. Samples
Candidale Documents
Figure 3: (Left); average performance (nDCG@ 10) on BRIGHT benchmark. Retro*'s re-ranking performance consistently improves with increased model scale and test-time samples. (Right): in- ference time on the TheoT. dataset from the BRIGHT benchmark. Retro exhibits a significantly lower time latency than other methods as the number of candidate documents increases.
4.3 PERFORMANCE OF SCALING (FOR RQ3)
To investigate the scalability of Retro*, we conduct experiments under scaling settings: scaling the model size and scaling the number of sampling trajectories at test-time.
Model Size Scaling. As shown in Figure 3 the performance of Retro* improves consistently as the model size increases. Specifically, under single-sample inference (N = 1). the nDCG@ 10 rises substantially from 32.4 for the 3B model 36.6 for the 7B, and further to 38.5 for the 32B. This scaling trend remains consistent across different numbers of sampling generations. These findings highlight the scaling benefits of larger LLMs, indicating that larger and more capable LLMSs exhibit stronger reasoning capabilities, enabling better comprehension of queries, more nuanced document analysis, and more accurate relevance estimates within our framework.
Test- Time Scaling. As shown in Figure 3 the performance of Retro improves consistently as the number of sampling trajectories increases. Specifically, for the 7B model, nDCG@ 10 improves from 36.6 (N= 1) to 38.7 (N 16), with consistent gains at intermediate sampling steps. This scaling trend remains consistent across all model sizes. These results underscore the effectiveness test- time scaling, demonstrating that integrating multiple reasoning trajectories leads to more accurate and reliable relevance scoring.
4.4 EFFICIENCY PARALLELISM (FOR RQ4)
Listwise and setwise re-ranking methods have limited parallelism capabilities, leading to significant latency when processing a large group of candidate documents. This inefficiency stems directly from their inherently sequential processing nature, which lacks the capability for effective document-level batching. In contrast, the pointwise approach evaluates each qucry-document pair independently. This property enables massive parallelism and is inherently more efficient terms of both inference time and throughput. To assess the practical efficiency gains from parallel processing, we compare the inference times of models with different re-ranking manners across varying numbers of candidate documents. This experiment is conducted on the TheoT. dataset using 4 NVIDIA H100 GPUs.
The results presented Dempriaemirr confirm the benefits this massive parallelism. Retro demonstrates a significantly slower increase in inference time as the number of candidate documents grows. In contrast, the setwise model Rank-RI exhibits extremely poor scalability, with inference time becoming prohibitively expensive wise model ReasonRank supports query-level batching, it still experiences a much steeper increase in latency. These findings demonstrate that the parallelism capability of Retro delivers significant efficiency, making it a far more practical and scalable solution for real-world IR applications.
for modest number of documents. Although the list-
4.5 SCORING RELEVANCE (FOR RQ5)
A primary limitation of existing models is their inability to provide direct and interpretable mea- surement of relevance. Although pointwise models like LankLLaMA and Rankl can produce ab- solute scores, their logit-based or probability- based scores lack clear and interpretable functionality
8

---

Published as a conference paper at ICLR 2026
Retro (7B)
RankLLaMA (7B)
Bankl (7B)
1.0
0.8
36
66
34
24
44
Barik
Figure 4: Score distributions from pointwise models on sample of positive and negative documents on the BRIGHT benchmark. The intensity of the color represents the density of scores, with darker hues indicating larger proportion of documents are assigned scores in that range.
Table 3: Ablation study on the key components of our training strategy.
Ftchanxe
Coding
Theorem-based
Models
Avg.
Bio.
Earth.
Econ.
Psy.
Rob.
Stack.
Sus.
Leet.
Pony
AoPS
Thea..
TheoT.
Qwen2.5-78 101225--B-Innsuue 18-Instruct
22.9
39.9
41.2
21.0
31.4
17.0
16.9
22.7
12.1
15.7
3.9
14.2
38.7
(Composite Reward)
only-SFT
30.1
46.9
51.3
29.1
37.4
24.4
28.4
35.0
15.7
20.6
7.9
27.9
36.6
only RL (Composite Reward)
35.1
56.7
52.9
33.2
45.1
28.1
32.5
35.8
21.2
36.3
8.6
29.8
41.0
(atra-Roward)
29.9
33.4
8.6
41.3
SIT RL (Inter Reward)
30.8
43.6
48.7
30.8
36.7
25.6
28.5
37.7
13.1
28.6
5.6
30.6
40.6
0.96
37
aD0G@10
10-
38
32
BCE Rossoner- RCC-RRessmmer-mmee (32.5)
Retro (78)
0.50
20
28
80
Training Steps
Te Training Steps
Figure 5: Training dynamics of Retro* (7B) during the RL stage. (Left): The training reward steadily increases over training steps. (Right): along with the improved reward, the retrieval accu- racy (nDCG@ 10) improves consistently with the training steps.
for measuring relevance. As shown in Figure this deficiency is evident. The score distributions of RanKLLaMA for positive and negative documents are heavily mixed, making it impossible to es- tablish meaningful threshold filter out the negative documents. Similarly, while Rankl's scores exhibit a binary tendency, a significant number of negative documents still receive high probability scores, undermining the reliability of its relevance scoring.
In contrast, Retro demonstrates a clean and consistent separation between positive and negative documents. Aligned with its explicit relevance rubric, positive documents consistently receive high scores (typically above 60), while negative documents are assigned low scores. This clear separation demonstrates that Retro" not only excels at relative ranking but also successfully provides the crucial functionality of relevance measurement, key advantage for downstream applications.
4.6 ABLATION STUDY (FOR RQ6)
We conduct ablation studies on the 7B model to validate the contribution of each component in training strategy. All variants follow the same training data and process as the main experiments, differing only in the component being ablated. Results shown in the Table As baseline, the backbone --nn..5-7BInntruuu achieves an average nDCG@ of 22.9 without any training. After two-stage training with the composite reward, the performance significantly improves to 36.6.

---

Published as a conference paper at ICLR 2026
The following experiments analyze the contribution of each component to this overall improvement of 13.7 points.
Effect of the Two-Stage Training. Training with SFT alone improves performance to 30.1 (+7.2), indicating that learning from the teacher model's ' reasoning trajectories provides the model with an initial ranking ability. In comparison, training with RL alone achieves higher score of 35.1 (+12.2), also outperforming other baselines of similar model size, demonstrating the importance of ranking- oriented rewards in enhancing the model's performance. Furthermore, training with both SFT and RL produces a better result of 36.6 (+13.7). This result indicates that combining the two stages helps the model perform more effective exploration and exploitation during the RL stage, while also SFT trajectories are not strictly required, as smaller models trained with RI. alone can already achieve competitive performance.
demonstrating that the teacher-genene
Effect of the Composite Rewards. We analyze the contribution of both intra-reward and inter- reward under the full two-stage training setup. Training with intra-reward improves performance from 30.1 to 33.2, while using only the inter-reward has limited improvement to 30.8. In contrast, combining both rewards results a final score of 36.6. These results demonstrate that without intra-reward, the model cannot generate reliable trajectory for given query-document, which introduces noisy learning signals and hinders effective learning. Without inter-reward, the model struggles to distinguish between the positive and negative documents. With composite reward, the model learns more steadily, as illustrated in Figure [ which shows that both the training reward and nDCG @1 10 the BRIGHT benchmark steadily increase throughout the training process.
CONCLUSION
This paper proposes Retro*, novel approach for reasoning-intensivy document retrieval. Retro* introduces a rubric-based relevance scoring mechanism, which enables the LLMs to reason about the relationship between task and a document based on well-defined criteria. Additionally, Retro* also supports test- time scaling by combining multiple reasoning trajectories via score integration, which produces more reliable relevance estimates. To optimize Retro*'s reasoning capabilities for document retrieval tasks, we propose a two-stage training strategy which includes SFT for warm up and RL with composite rewards for scoring and ranking functionalities. Empirical experiments on the BRIGHT benchmark demonstrate that Retro outperforms existing methods and achieves ofte--of-the-ar performance, with flexible test-time scalability and massive parallelism capability.
AUTHOR CONTRIBUTIONS
As the core contributors, Junwei Lan and Jianlyu Chen started this work collaboratively. Junwei was responsible for the model architecture, training algorithm, training and evaluation, experiments, and manuscript writing. Jianlyu contributed to the refinement and optimization of the model architecture, the model evaluation framework, and part of the manuscript writing.
ACKNOWLEDGMENTS
This work is supported by grants from Scientific Research Innovation Capability Support Project for Young Faculty and the National Natural Science Foundation of China (No. U24A20253).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
Payal Bajaj. Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Ma- jumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint 161 1.09268, 2016.
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherbum, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learn- ing agents machine learning engineering. arXiv preprint arXiv:2410.07095, 2024.
10

---

Published as a conference paper at ICLR 2026
Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. Theoremqa: theorem-driven question answering dataset. arXiv preprint 1rXii:2305512524. 2023.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10999 2(1), 2023.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16726. 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William Sayed. Mistral 7b, 2023. URL https:
7/arxiv. org
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 2023.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training lims to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09511 2025.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP (1), pp. 6769-6781, 2020.
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. arXiv preprint 100.::9900003000. 2019.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. oetrieval-augment gener- ation for knowledge-intensigg nlp tasks. Advances in neural information processing systems, 33: 9459-9474, 2020.
Lei Li, Xiao Zhou, and Zheng Liu. R2med: A benchmark for reasoning-driven medical retrieval. arXiv preprint arXiv:2505. 14558, 2025.
Jimmy Lin, Xueguang Ma, Sheng- Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, pp. 2356-2362, 2021.
Wenhan Liu, Xinyu Ma, Weiwei Sun, Yutao Zhu, Yuchen Li, Dawei Yin, and Zhicheng Dou. Reasonrank: Empowering passage ranking with strong reasoning ability. arXiv preprint M0Xiv:2508.070500 2025.
Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning llama for multi-stage text retrieval. In Proceedings of the 47th International ACM SIGIR Conference Research and Development Information Retrieval, pp. 2421-2425 2024.
Tong Niu, Shafiq Joty, Ye Liu, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Judgerank: Lever- aging large language models for reasoning-intensive intensive reranking. arXiv preprint arXiv:2411.0011 2024.
11

---

Published as a conference paper at ICLR 2026
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35:
27730-27744, 2022.
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity's last exam. arXiv preprint arXiv:2501.14249, 2025.
Ronak Pradeep. Sahel Sharifymoghaddam, and Jimmy Lin. Rankzephyr: Effective and robust zero- shot listwise reranking is breeze! arXiv preprint arXiv:2312.02777 2023.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang. Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
URL https: arxiv. org/ abs/ 2412. 15115
Rulin Shao, Rui Qiao, Varsha Kishore, Niklas Muennighoff, Xi Victoria Lin, Daniela Rus, Bryan Kian Hsiang Low, Sewon Min, Wen-tau Yih, Pang Wei Koh, et al. Reasonir: Training retrievers for reasoning tasks. arXiv preprint 25X45:2504.20555, 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepscekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang. Ru Zhang, Yanghua Peng. Haibin Lin, and Chuan Wu, Hybridflow: A flexible and efficient rIhf framework. In Proceedings of the Twentieth European Conference Computer Systems, pp. 1279-1297, 2025.
Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang. Haisu Liu, Quan Shi, Zachary S Siegel, Michael Tang, et al. Bright: realistic and challenging bench- mark for ceasonng-intensivv retrieval. arXiv preprint arXiv:2407. 12883, 2024.
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is chatgpt good at search? investigating large language models as re-ranking agents. arXiv preprint arXiv:2304.095: 2023.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2/04.08663 2021.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykow. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXivv2307.09928 2023.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improv- ing text embeddings with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11897-11916, 2024.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. 11171, 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. tffin-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Jason Wei, Zhiqing Sun. Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents, 2025. URL httttp://aaxxvv.orgg//////
12516
12

---

Published as a conference paper at ICLR 2026
Orion Weller, Kathryn Ricci, Eugene Yang, Andrew Yates, Dawn Lawrie, and Benjamin Van Durme. Rankl: Test-time compute for reranking information retrieval. arXiv preprint arXiv:2502.18418, 2025.
Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian- Nie. C-pack: Packed resources for general chinese embeddings. In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pp. 641-649, 2024.
Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min- Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint KKiv:2405.0045/11 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a.
Eugene Yang, Andrew Yates, Kathryn Ricci, Orion Weller, Vivek Chari, Benjamin Van Durme, and Dawn Lawrie Rank-k: Test-time reasoning for listwise reranking. arXiv preprint 24444:2505.14432, 2025b.
Shunyu Yao. Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in neural information processing systems, 36: 11111180111111 11822, 2023.
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs. Advances neural information processing systems, 37: 62557-62583 2024a.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory; Unified efficient fine-tuning 100+ language models. arXiv preprint arXiv:2403.13372 2024b.
Shengyao Zhuang, Honglei Zhuang, Bevan Koopman, and Guido Zuccon. setwise approach for effective and highly efficient zero-shot ranking with large language models. In Proceedings of the 47th International ACM SIGIR Conference Research and Development in Information Retrieval, pp. 38-47, 2024.
Shengyao Zhuang, Xueguang Ma, Bevan Koopman, Jimmy Lin, and Guido Zuccon. Rank-rl: En- hancing reasoning in Ilm-based document rerankers via reinforcement learning. arXiv preprint 25X0::2503.0033, 2025.
13

---

Published as a conference paper at ICLR 2026
OVERVIEW OF APPENDIX
Appendix
The Use of LLMs
Appendix
Baselines
Appendix
Training Details
Appendix
Detailed BM25 and ReasonIR Results BRIGHT
Appendix
Evaluation BEIR
Appendix
Analysis of the Hyperparameter a
Appendix
Different Backbone Models
Appendix
Analysis of the Length Control Instruction for Distillation
Appendix
Out-of-Domain Generalization on R2MED
Appendix
Rubrics of Rubrie-based Relevance Scoring
Appendix K
Analysis of the Necessity of Reasoning
Appendix
Task Relevance Definitions for Different Benchmarks
Appendix
M
Reasoning Examples
A
THE USE OF LLMS
We used the LLMs assist with writing. Specifically, their included grammar checking. rephrasing for clarity, and textual polishing. Additionally, we utilized the LLMs to help draft Python codes for plotting figures. The LLMs used for these purposes include Gemini 2.5 Pro ChatGPT DeepSeek-V33
B BASELINES
This section provides additional details for each of the baseline models used in our comparative analysis. As outlined in the experimental setup 4.1 we categorize these models into two groups: non-reasoning and reasoning-enhance approaches. For each baseline listed below, we describe its core methodology and implementation details:
RankLLaMA: A pointwise re-ranking model that jointly encodes the query and document, gen- erating a scalar relevance score by projecting the representation of the end-of-sequence token through linear layer. The model is trained with a contrastive loss.
RenkZephyr: A listwise re-ranking model that takes a query and set of candidate documents as input and outputs their relative ranking order. It employs a sliding window strategy to to handle large lists of candidates. The model trained on MS MARCO (Bajaj al. 2016) dataset knowledge distillation from GPT-3.5 (Ouyang et al. 2022) and GPT-4 (Achiam et al. 2023).
JungeRank: A zero-shot, reasoning enhanced pointwise re- re-ranking method. It utilizes carefully designed prompts to guide LLMs through explicit reasoning steps before producing the final rele- vance judgment ("yes" or "no").
A reasoning-enhaneed pointwise re-ranking model. For each query-document pair, the model generates reasoning process and outputs a binary relevance judgment ("yes" or "no"). Rank1 is trained on reasoning trajectories generated by DeepSeek-RL (Guo et al. (025 on the MS MARCO dataset.
Rankl:
Rank-R1: A reasoning-enhannn enhanced setwise (Zhuang et al. 2024 re-ranking model. Given query and candidate set, the model selects the most relevant document and applies a heap-sort pro- cedure to obtain top-k results, The model is trained on the MS MARCO dataset with the GRPO algorithm.
https::/qemini wwwp://mmiiii..o..eeo... com https://chat.qt... com
https//ccha.deepeekk
14

---

Published as a conference paper at ICLR 2026
Table 4: Hyperparameters for the SFT stage.
Learning
WarmUp Ratio
LR Scheduler
Batch Size
Gradient
Epochs
Per-Device
Accumulation
10
6
1
0.1
cosine
8
Table Hyperparameters for the RL stage with GRPO.
Learning Rate
Epoocss
Batch
Mini
Micro Batch Size
KLL Coefficient
Temperature
0.75
20
10-6
256
256
0.005
1.0
Table 6: Detailed nDCG@ 10 results for Retro on the BRIGHT benchmark. Retro re-ranks the top- 100 documents retrieved by either BM25 or ReasonIR. The retrieval stage utilized a GPT-4 reasoning query, while the re-ranking stage utilized the original query.
Models
Avg.
StackExchange
Coding
Theorem-bused
Bio.
Earth.
Econ.
Psy.
Rob.
Stack.
Sus.
Leet.
Pony
AoPS
TheoQ.
TheoT.
BM25
27.0
53.6
54.1
24.3
38.7
18.9
27.7
26.3
19.3
17.6
3.9
19.2
20.8
Retrot (7B)
35.3
56.4
56.1
30.8
46.4
31.6
36.7
39.8
22.7
28.2
7.3
27.6
40.4
(32B)
35.9
Test-Time Sealing (Mean-Score
16)
Retro* (7B)
37.0
60.7
59.0
31.9
48.2
32.1
34.9
40.8
25.2
33.2
29.8
40.1
Retro (328)
38.5
64.0
61.6
36.9
50.1
33.9
37.7
45.2
19.9
32.3
8.5
30.6
41.5
ReasonIR
30.6
43.4
43.0
33.1
20.9
31.0
27.0
31.6
19.5
7.4
33,9
36.9
Retro (7B)
36.8
54.7
54.5
33.5
47.7
32,5
40.1
41.2
22.7
28.9
8.3
34.1
43.7
(32B)
60.7
56.1
37.6
33.0
Teat-Time Scaling (Mean Score
Retro* (7B)
38.4
60.4
57.4
34.8
48.2
31.6
37.9
41.9
24.9
32.1
9.0
38.1
44.0
Retro* (32B)
39.5
64.7
57.5
38.5
49.7
31.7
42.6
47.3
19.4
30.2
10.0
37.8
44.8
A enhhancd listwise re-ranking model. It introduces an automated data synthesis framework to generate high-quality reasoning-intensive training data. The model is trained on this synthesized dataset in two stages: SFT with reasoning trajectories distilled from DeepSeek-RR followed by GRPO with multi-view re-ranking reward.
ReasonRank:
C TRAINING DETIALS
For the SFT stage, we use the LLaMA-Factor, framework (Zheng al.) 2024b). fine-tuning both the 7B and 32B models on single node equipped with 8 NVIDIA H100 GPUs. For the RL stage, we employ the GRPO algorithm through the VeRL. framework (Sheng al., 2025). In our main experiments, we set the hyperparameter a to 0.75 and to 20, as 20 represents a meaningful interval between relevance levels our rubrics 3.1.I An ablation study on the impact different a values presented in Appendix F The 7B model is trained on a single node, while the 32B model is trained across two nodes. Detailed hyperparameters for the SET and RL stages are provided in Table 4 and Table 5 respectively.
D DETAILED BM25 AND REASONIR RESULTS ON BRIGHT
This section provides the complete, per-dataset results the BRIGHT benchmark, corresponding to the summarized analysis the main body. detae the nDCG@ 10 scores of Retro when re-ranking the top- 100 documents retrieved by two different first-stage retrievers: BM25 and Rea- sonIR. The results highlight the model's 's robust and consistent performance across various domains, regardless of the initial retrieval method.
ReasonIR further introduces a zero-shot re-ranking method (QwenRerank) based on Qwen2.5- Instruct-32B, and conduct detailed comparison between this method and Retro* However, QwenRerank computes its final ranking scores by fusing normalized retriever and re-ranker outputs with 0.5 Steranker 00. Sretriever, meaning that its reported performance reflects the joint effect of
15

---

Published as a conference paper at ICLR 2026
Table 7: Comparison of Retro with QwenRerank from ReasonIR under two evaluation settings, where all methods re-rank the top-100, documents retrieved by ReasonIR. Retro consistently achieves higher performance both settings. Results marked with are reported by ReasonIR.
StackExchange
Coding
Theorem-based
Models
Avg.
Blo.
Earth.
Econ.
Psy.
Rob.
Stack,
Sus.
Lect.
Pony
AoPS
TheoQ.
Theo
Reason
30.6
43.4
43.0
33.1
39.6
20.9
31.0
27.0
31.6
19.5
7.4
33.9
36.9
Re-nanking
OwenRerank (32R)
30.3
50.2
47.5
23.6
35.8
24.7
28.1
29.9
30.3
26.5
5.5
20.7
40.8
Retro (7B)
36.8
54.7
54.5
33.5
47.7
32.5
40.1
41.2
22.7
28.9
8.3
34.1
43.7
Retro (32B)
37.6
38.6
27.3
9.0
Retro" (7B, Mean-Score
38.4
60.4
57.4
34,8
48.2
31.6
37.9
41.9
24.9
32.1
9.0
38.1
44,0
Retro" (32B, Mean @f 16)
39.5
64.7
57.5
38.5
49.7
31.7
42.6
47.3
19.4
30.2
10.0
37.8
44.8
Re-ranking w/ Score Fusion
QwenRerank (32B)
36.9
58.2
53.2
32,0
43.6
28.8
37.6
36.0
33.2
34.8
7.9
32.6
45.0
Retro (7B)
38.4
54.2
55.9
35.0
49.7
32.4
40.2
41.7
26.1
32.7
10.9
36.9
45.0
Retro" (32B)
39.6
60.0
58.2
38.4
51.1
31.4
39.2
45.3
25.0
33.4
10.5
37.0
45.4
Retro" Me
31.7
38.2
27.1
39.2
44.9
Retro* (32B, Meun Score O 16)
40.2
60.1
58.4
38.5
514
32.1
41.5
46.3
25.2
33.3
11.1
38.4
46.2
Table 8: nDCG@11 10 results on the BEIR benchmark, where all methods re-rank the top-100 docu- ments retrieved by BM25 provided by et al.) (2021). The retrieval stage and the re-ranking stage both utilize the original query. Results marked with are reported by ReasonRank.
Models
Methods
TREC-COVID DBPedia SciFact NFCorpus Signal-IM
Robust04 TREC-NEWS
BM25
39,5
Non-Reasoning Re-Ranking Baselines
RankZephyr (7B)
Listwise
82.9
44.4
75.4
38.3
53.7
52.8
Reasoning-Fnhuneee Ro-Ranking Baselinex
Rankl (7B)
Pointwise
79.0
35.8
73.3
47.7
Rank1' (32B)
Pointwise
Rank R1 (7B)
Setwise
53.6
83.7
42.3
72.2
38.9
33.1
54.5
50.6
Rank-RI" (14B)
Setwise
54.6
84.6
44.1
76.0
38.6
33.0
56.9
49.2
ReasonRank' (7B)
Listwise
54.4
82.0
46.0
75.6
39.6
31.4
55.4
50.5
ReasonRank (32B)
Listwisc
55.4
83.2
45.7
77.2
40.0
31.1
58.7
52.2
Retro* (7B)
Pointwise
55.8
84.7
45.9
77.1
37.1
31.2
64.7
49.7
Test-Time Scaling MMann-Score@:
Retro* (7B)
Pointwise
56.8
85.4
46.7
78.5
38.1
32.1
65.8
51.0
both retriever and re-ranker rather than its standalone re-ranking capability. To ensure a fair compar- ison, we compare both models under the same evaluation settings. As shown in Table Retro con- sistently outperforms QwenRerank across all evaluation settings, indicating substantially stronger re-ranking ability. These results suggest that most of the gains attributed to QwenRerank largely come from the strong retriever ReasonIR, whereas Retro achieves higher performance purely from its re-ranking capability.
E EVALUATION ON BEIR
To assess the generalizability of our model, we further evaluate its performance in traditional re- trieval scenarios. Following ReasonRank (Liu et al. (2025), we select seven datasets from the BEIR benchmark (Thakur et al.) 2021), for their relatively small number of queries. In the retrieval stage, BM25 is applied with the original query to obtain candidate documents, after which all re-ranking methods re-rank the top- 100 BM25 results. We report nDCG@ @ 10 the performance metric.
In this scenario, we utilize the MS MARCO dataset as our training data. Specifically, we construct 24,000 query-document pairs for both SFT and RL training. The RL stage is trained for 3 epochs, while the other training settings are consistent with Section
The evaluation results BEIR, shown in Table 8 demonstrate the strong generalizability of R to traditional re-ranking tasks. The Retro* (7B) model achieves an average nDCG@ 10 of 55.8, outperforming all non-reasoning and reasoning-enhanced baselines. These results confirm
16

---

Published as a conference paper at ICLR 2026
Table nDCG@ @ 10 results on the BRIGHT benchmark with models trained using varying a values.
StackExchange
Codlng
Theorem-hased
Models
Bio.
Earth.
Econ.
Psy.
Roh.
Stack.
Sus.
Leet.
Pony
AoPS
TheaQ.
Theo
SFT
30.1
46.9
51.3
29.1
37.4
24.4
28.4
35.0
15.7
20.6
7.9
27.9
36.6
SFT RI.(a
0.00)
30.8
43.6
48.7
30.8
36.7
25.6
28.5
37.7
13.1
28.6
5.6
30.6
40.6
SFT RL 1.00)
33.2
29,9
44.2
27.6
33.4
3644
20.1
23,3
8.6
32.0
41.3
0.25)
40.3
SFT RL (a 0.50)
35.8
51.3
54.3
32.7
47.4
31.7
35.0
39.3
20.7
29.2
8.2
34.6
44.6
SFT RI. (a 0.75)
36.6
53.7
55.9
35.6
47.9
34.0
35.6
39.3
17.6
29.8
9.6
35.4
45.0
that our casoning-enhannee pointwise approach is general-purpose and suitable for both reasoning- intensive and traditional retrieval scenarios. Moreover, by integrating over 16 sampling scores, the model achieves a final nDCG@ @ 10 of 56.8, indicating the effectiveness of test-time scaling even on traditional benchmarks, as illustrated in Figure
ANALYSIS OF THE HYPERPARAMETER Q
Table reports the performance of training with different weights a for the composite rewards. When 0 or a = 1, the model degenerates into using only one of the composite rewards, resulting in suboptimal performance. In particular, training with only the intra-document reward (a 1) clearly outperforms training with only the inter-document reward (a 0), suggesting that the intra-reward provides stronger optimization signal for model learning. This observation aligns with its role in stabilizing training by encouraging more consistent reasoning trajectories within document.
However, despite its dominance, the intra-reward alone still underperforms any weighted combina- tion of the two rewards. Across all a E (0, 1), we observe consistent performance gains over using either reward alone, demonstrating that the intra- and inter-document rewards serve complementary purposes. While the intra-reward promotes stable and learnable ranking behavior, the inter-reward introduces document-level discrimination that further enhances final ranking performance.
These results indicate that effective reward balancing enables the model to simultaneously exploit stable intra-document ranking signals and informative inter-document comparisons. Notably, per- formance peaks when a is biased toward the intra-reward (e.g., 0.75), highlighting that the intra-reward should dominate training dynamics, while the inter-reward as an essential comple- mentary signal that enhances ranking performance.
G DIFFERENT BACKBONE MODELS
To demonstrate the generalizability and effectiveness of our Retro*, we conducted experiments with several different backbone models. As shown Table 10, Retro* significantly improves the average nDCG@ @ 10 across all backbone models: Swen2.5-Inntrun. (7B) from 22.9 to 36.6, Llama2-chat (7B) from 8.2 to 30.9, Mistral-Instrrut-vo... (7B) from 8.6 to 30.1, Llama3. -IInstrcct (8B) from 17.3 to 34.4, and Qwen3 (8B) from 29.4 to 36.1. Furthermore, applying test-time scaling further improves performance consistently for all backbones as shown in Figure These results demonstrate that our approach consistently and substantially enhances ranking performance across diverse backbones, while test-time scaling remains effective with each backbone, further highlighting its robustness and broad applicability general retrieval model.
H ANALYSIS OF THE LENGTH CONTROL INSTRUCTION FOR DISTILLATION
In this section, we analyze the impact of the length control instruction used during the distillation from the teacher model. We augment the prompt template (shown in Figure S with an explicit instruction at the beginning of the task description. The modified prompt template is shown below:
sare is the erelevance definitions in retrieval task: trellvance_ddffintt...
17

---

Published as a conference paper at ICLR 2026
Table 10: nDCG@ @ 10 results on the BRIGHT benchmark across different backbone models. All backbone models re-rank the top- 100 documents retrieved by BGE-Reeasoner-mbee The retrieval stage and the re-ranking stage both utilize the original query.
StackExchange
Coding
Theoren-bused
Models
Avg.
Blo.
Earth.
Econ.
Psy.
Rob.
Stack.
Sus.
Leet.
Pony
AoPS
TheoQ.
Theo
AGE-Reasonee Embed
32.5
42.6
42.6
27.8
37.3
26.4
29.6
30.6
36.9
25.7
34.9
46.
Qwen2.5 (7B)
22,9
39,9
41.2
14.2
Retro" (7B)
36.6
53.7
55,9
35.6
47.9
34.0
35.6
39.3
17.6
29.8
9.6
35.4
45.0
Test-Time Sealing (Mean-Score
Retro* (7B)
38.7
58.4
59.2
35.0
49.3
33.9
37.7
41.1
18.8
33.5
10.7
40.2
46.7
Llama2-chat (7B)
8.2
13.4
14.6
11.2
7.0
10.6
7.6
4.4
3.2
Retro* (718)
25.4
43.7
40.1
22.0
35.8
20.7
26.8
29.7
18.1
19.4
6.6
22.0
19.5
Test-Time Sealing MMMan-Scoveeee
Retro (7B)
30,9
53.3
49.6
28.8
42.8
25.7
29.9
34.3
22.1
25.1
7.5
29.2
23.1
Instral-nsttree-vO.... (7B)
8.6
17.2
15.4
1166
7.7
8.3
10.6
6.0
2.5
3.8
Retro (7B)
27.8
49.1
42.4
24.4
38.0
23.7
29.7
31.9
15.5
20.7
9.3
24.5
24.5
Scaling (Mean
Retm (7B)
30.1
53.9
46.0
27.3
43.0
25.7
30.9
31.6
15.0
24.0
10.1
27.8
26.1
Llama3.1 1-----II-nt Innttruct (8B)
17.3
35.0
27.8
17.3
30.0
11.8
19.5
21.3
8.8
4.8
4.2
11.4
15.1
Retro (8B)
34.4
56.7
53.5
31.3
47.3
31.3
32.7
40.2
19.9
21.2
33.0
36.4
Scaling (Mean
Retro* (8B)
36.7
60.2
55.9
33.5
49.7
33.6
36.2
42.0
19.0
24.7
10.3
38.4
36.4
Qwen3 (8B)
29.4
53.5
53.8
28.3
34.1
26.3
29.2
33.1
14.1
8.4
20.6
44.4
Retro* (8B)
36.1
55.0
56.2
34,9
42.0
35.9
35.6
42.5
16.7
21.9
35.4
48.7
Time Scaling (Mean- Sannn-Scoreee
Retro (8B)
38.8
57.6
58.5
37.1
45.8
38.3
39.3
45.4
17.3
26.6
10.6
40.4
48.8
57.0
40
56.8
DDC0010 (BELR)
36.8
56.7
nDCG@10 (BRIGHT)
39
66.6
36.4
38
66.4
37
36.1
36.0
36
Owen2 55 5-Instruct
25.8
1lama3. H-ttttuct
55.6
34
Qwen3
16
Samples
Samples
Figure 6: Effectiveness of Test-Time Scaling for Retro (Left); Test-time scaling brings significant performance improvements even on the traditional BEIR benchmark. (Right): Test-time scaling is consistently effective across different backbone models.
Now giver a +.que ((quecy and vedocument.a (idoe_type) in this retrieva: task, your mission is to perform following steps (x>Please ensure your entire analysis and annotation across all steps does not exceed 512 tckenssA).
(The of the prompt template ramaing the same!
To evaluate the effect of this instruction on both the reasoning length and the model's performance, we construct two distillation datasets from the teacher model under two conditions: one with the instruction and one without it. We then train student models on these datasets, respectively. The impact on reasoning length illustrated in Figure 7 while the model performance is reported in Table [I
As shown Figure 7 the length control instruction substantially shortens the reasoning trajecto- ries, yielding an average trajectory length comparable to that of the backbone model. Meanwhile, Table shows that the model trained with this instruction achieves an average nDCG@ 10 of 36.6, essentially matching the 36.4 achieved without it. These results demonstrate that the instruction ef- fectively constrains the reasoning length for efficiency without compromising re- re-ranking accuracy. Furthermore, although the reasoning length of our trained model is close that of the backbone,
18

---

Published as a conference paper at ICLR 2026
wio Length Control Length Control
Pay
Backboae
Rob.
Eoon.
Stack
Earth.
.400
1100
Su
Bio.
T.
AdPS
Figure 7: Comparison of average completion tokens BRIGHT benchmark for different models. The model trained with the length control instruction (w/ Length Control) effectively reduces the response length compared to training without (w/o Length Control).
Table 11: nDCG@ 10 results on the BRIGHT benchmark for ablation study on the impact of the length control instruction.
StackExchange
Codlng
Theorem-based
Models
Avg.
Blo.
Earth.
Econ.
Psy.
Rob.
Stack.
Sus.
Leet.
Pony
AoPS
TheoQ.
TheoT.
BGE Reasoner Finhed Qwen2 5-Instruct (7B)
32,5
42.6
42.6
27.8
37,3
26.4
29.6
30.6
36.9
25.7
9.8
34,9
46.1
22.9
39.9
41.2
21.0
17.0
169
22.7
12.1
15.7
3.9
14.2
38.7
w/ Length Control
only-SFT
30.1
46.9
51.3
29.1
37.4
24.4
28.4
35.0
35.0
20.6
7.9
27.9
36.6
SFT RL (Composite Reward)
36.6
53.7
35.9
35.6
47.9
34.0
35.6
39.3
17.6
29.8
9.6
35.4
45.0
w/o Length Control
only-SFT
30.7
46.8
50.2
30.5
39.4
25.7
30.1
35.8
14.4
18.0
7.6
27.9
41.6
SFT (Composite Reward)
36.4
53.3
55.6
33.7
48.7
33.3
35.6
39.5
16.5
31.1
8.3
36.4
45.1
it exhibits stronger ranking performance, suggesting that the improvement does not arise from gen- erating longer reasoning trajectories. Instead, stems from our effective training strategy, which equips the model with powerful reasoning and ranking capabilities.
I OUT-OF-DOMAIN GENERALIZATION ON R2MED
To evaluate the out-of-domail generalizability of our rubric, we evaluate Retro* on R2MED ct al,, (2025), a biomedical reasoning retrieval benchmark that outside the scientific, programming. and mathematical domains used during training. By replacing only the task-specific relevance def- initions while keeping the task-agnostic scoring criteria, Retro is able to generalize effectively to this unseen domain. Following the setup of ReasonRank, 2024) as the first-stage retriever, re-rank the top- 00 retrieved candidates, and report nDCG 10. As summarized in Table 12 R outperforms all other re-ranking baselines, demonstrating both the robustness of its based scoring and its strong cross-domain generalization capability.
use E5-mistral-7bb - (Wang al.
J RUBRICS OF RUBRIC-BASED RELEVANCE SCORING
We provide the full relevance rubrics for our rubric- based relevance scoring mechanism in Figure The rubrics designed to guide the LLMs reason about the relevance between query and candidate document through three-step reasoning process (query analysis, document analysis, and relevance annotation) produce fine-grained relevance score.
19

---

Published as a conference paper at ICLR 2026
Here is the axzelevance definition in setrieval task: [ceelvanne__efinitiiio
Now given λаquarуи ((query_type) and vadocumentiv ([doe_type)) in this retrieva" task, your mission is to pezform the following steps.
1. Query Analysis: Think to resson and describe what information would be moat helpful In answering query.
2. Document Analysis: biscuss how the information provided by the documen: fulfills or fails the requi rements implied by query.
to
3. Re levance Annotation: Based on the relevance definition and the insights from the previous
wwo s.eps, clearly Justify your final zelevance sanoiation result and annotate an integer ses
103. Please
fowing guide:
xx80 100 (fliglly RelevanL] The document directly and comprebensively addieesses the coze authoritative answer.
query's intent.
+.60-8 (Relevant) :*. The document substantielly addressas the query's intent, providing key information, but might mias some minor details.
most
.+40-60 (Moderately Relevant) The document is on-topic and addreases part of the query's a intent, but it is not comprehensive answer.
+*20-40 (SLightly Relevant)
query,
nain topic is differeat. It offers very limited value.
-x0-20 (irrelevant) aff-topic.
does not address query's s
all is
After providing your detailed analysis and justification for all the stepa above, conclude your entire response with the final relevance score. The secre must be placed strictly between the <seere> tags. There should be other explanstion inside the taga:
<score>
[From
100, annotate degree relevance
query
</acoze>
Quezy (yquery.typel):
[Regin of Query] (query:
[End Query]
Document ([doc_Lype)):
[Begin of ]ocument] (doa)
[End becument
Figure 8: The full relevance rubrics for rubric-based relevance scoring.
Table 12: nDCG@ 10 results on the R2MED benchmark, where all methods re-rank the top-100 documents retrieved by 75-mistral-7b-inssruet, The retrieval stage and the re-ranking stage both utilize the original query. Results marked with are reported by ReasonRank.
Models
Methods
Avg.
Biolo.
Bioin. Med-Sci.
Med-Exam.
Med-Ding.
PMC-Treat. PMC-CL. IIYi-Cli
E5-mistral- 7b-instruct 75-mistral-7-innttet
Retriever
24.0
22.9
42.3
41.5
7.4
12.4
18.5
24.9
21.8
Gensoniin-Enhhance Re-Ranking Baselines
Rank11 (7B)
Pointwise
32.3
32.6
55.6
54.7
12.8
20.0
34.4
30.2
18.2
Rank1! (32B)
Pointwise
39.1
31.8
61.7
59.7
16.6
26.9
41.3
45.6
29.5
ReasonRank! (7B)
Listwise
39.5
46.8
59.7
60.1
16.5
24.9
39.2
39,1
29.9
ReasonRank (32B)
Listwise
42.9
45.6
67.7
63.5
18.9
30.6
41.1
46.1
29.4
Retro" (7B)
Pointwise
38.7
47.6
59.7
60.8
12.7
25.2
40.7
39.0
24.2
Retro" (32B)
Pointwise
44.5
68.0
67.3
16.6
34.2
43.3
45.4
32.6
Test-Time Scaling Scoren-)
Retro* (7B)
Pointwise
42.6
49.9
62.4
62.4
15.7
29.0
41.9
46.2
32.9
Retro" (32B)
Pointwise 46.2
51.9
67.8
67.2
19.2
35.7
43.8
48.7
35.6
K ANALYSIS OF THE NECESSITY OF REASONING
To investigate the necessity of reasoning for re-ranking performance, conduct ablation study by removing the structured reasoning steps from our relevance rubrics. the original rubrics, the model is prompted to analyze both the query and the candidate document before producing detailed relevance score. In the ablated variant, the model instead outputs a single score directly, without any reasoning trajectory rubric-guided analysis, as illustrated in Figure 9
20

---

Published as a conference paper at ICLR 2026
Table 13: Relevance definitions for each dataset in the BRIGHT benchmark.
Dataset
Relevance Definition
biology
Given query (biology post) and document (passage), the document is relevant to the query if the critical concepts or theories discussed in the document can provide references for domain experts to draft an answer to the query. Given a query (carth science post) and a document (passage), the document is relevant to the query if the critical concepts or thcories discussed in the document can provide references for domain experts draft an answer to the query. Given query (economies post) and document (passage), the document is rele- vant to the query if the critical concepts or theories discussed the document can provide references for domain experts to draft an answer to the query.
carth_science
economics
psychology
Given query (psychology post) and a document (passage), the document is rele- to the query if the critical concepts theories discussed in the document can provide references for domain experts draft an answer the query.
robotics
Given a query (robotics post) and a document (passage), the document is relevant to the query if the critical concepts or thcories discussed in the document can provide references for domain experts to draft an answer to the query.
stackoverflow
Given query (Stack Overflow post) and document (passage), the document is relevant to the query the critical concepts or theories discussed in the document can provide references for domain experts to draft an answer to the query.
sustainable_livinn
Given a query (sustainable living post) and a document (passage), the document is relevant to the query if the critical concepts or theories discussed in the document can provide references for domain experts draft an answer to the query.
leetcode
Given query (LeetCode problem) and document (coding problem solution). the document is relevant to the query the underlying algorithms or data struc- tures used the document can provide helpful insights for solving the problem the query.
pony
Given query (Pony coding instruction) and document (Pony documentation passage), the document is relevant the query if the Pony syntax described in the document is necessary for beginners with no prior knowledge of Pony to complete the coding instruction in the query.
aops
Given a query (math problem) and a document (math problem solution), the doc- ument is relevant to the query if the theorems used in the document can provide helpful insights for solving the problem in the query.
dieoremqa_questinn
Given query (math problem) and document (math problem solution). the doc- ument is relevant to the query if the theorems used in the document can provide helpful insights for solving the problem in the query.
theoremqa_theoree theorems
Given a query (math problem) and a document (math-related passage), the doc- ument is relevant to the query if the theorem described in the document can help solve the problem in the query.
This variant of R trained on the same dataset and with the same RL. algorithm as the original model. We focus the comparison specifically on the RI. stage to avoid introducing confounding factors, such as potential biases originating from the SFT teacher model. By keeping the learning signal consistent, this setup effectively isolates the impact of reasoning re-ranking performance. As shown in Table 16 removing the reasoning trajectory results in a substantial performance drop across all datasets. These results confirm that subric-guided reasoning enhances the fidelity of rele- vance estimation under RL training. By structuring the task into intermediate steps, the model can better capture how different aspects of relevance contribute to the final judgment. Without this rea- soning guidance, the model struggles to learn the underlying compositional logic of the relevance rubric, leading to the observed performance drop.
L TASK RELEVANCE DEFINITIONS FOR DIFFERENT BENCHMARKS
In this section, we provide the detailed relevance definitions used in evaluations the BRIGHT, BEIR and R2MED benchmarks. In reasoning-intenn intensive retrieval scenarios, the relevance definitions more complex, while in traditional scenarios, they more straightforward. The specific defi-
21

---

Published as a conference paper at ICLR 2026
Table 14; Relevance definitions for each dataset in the BEIR benchmark.
Dataset
Relevance Definition
TREC-COVID
Given a query (COVID-19 related query) and document (document), the document is relevant to the query if the document answers the query.
DBPedia
Given a query (query) and document (entity description from DBpedia), the document is relevant to the query if the entity described in the document matches the query. Given a query (scientific claim) and document (document), the document is relevant to the query if the document provides evidence supporting or refuting the scientific claim. Given a query (question) and document (document), the document relevant to the query if the document can best answer the question.
SciFact
NFCorpus
Signal-IM IM
Given a query (news event or topic) and a document (news headline or summary), the document is relevant to the query if it reports on, summarizes, or directly relates to the news event or topic described in the query.
Robust04
Given a query (information need) and document (news or government document), the document is relevant the query if contains information that satisfies the intent or topic described in the query, even if phrased differently.
TREC-NEWS
Given a query (contemporary news topic or event) and document (news article from The Washington Post), the document is relevant to the query if it discusses, explains, provides factual coverage of the specific event or topic mentioned the query.
Table 15: Relevance definitions for each dataset in the R2MED benchmark.
Dataset
Relevance Definition
Biology
Given query (biology post) and a document (passage), the document is relevant to the query if the document helps answer the query.
Medical-Sciences
Given query (medical science post) and a document (passage), the document relevant to the query if the document helps answer the query.
Bioinformatics
Given query (bioinformatics post) and document (passage), the document is relevant to the query the document helps answer the query.
MedXpertQAA-Exam
Given query (medical exam) and document (passage), the document is relevant to the query if the document helps answer the query.
MedQA-Diag
Given query (medical exam) and document (passage), the document is relevant to the query the document helps answer the query.
PMC-Treatment
Given query (clinical ease) and document (passage), the document is relevant to the query if the document helps answer the query.
PMC-Clinical
Given query (clinical case) and document (case), the document is relevant the query if it helps diagnose the query case.
IIYi-Clinccal
Given query (clinical case) and document (case), the document relevant the query if it helps diagnose the query case.
Table 16: Performance comparison the BRIGHT benchmark, showing that ennhancee-e re-ranking significantly outperforms models trained without reasoning.
Stachnxchange
Coding
Theorem-based
Models
Avg.
Bin.
Earth.
Econ.
Psy.
Rob.
Stack.
Sus.
Leet.
Pony
AoPS
TheaQ.
TheoT
BGE Reasoner Embeed
26.4
25.7
46.1
No Resson (only-RL)
24.8
38.4
40.2
22.3
32,4
20.5
25.6
26.0
19.6
16.9
4.7
17.7
33.8
Reason (only RL) 35.1
56.7
52.9
33.2
45.1
28.1
32.5
35.8
21.2
36.3
8.6
29.8
41.0
nitions for the BRIGHT benchmark are listed in Table 13
BEIR are listed in Table 14 and those for R2MED are listed in Table We use these relevance definitions for both training and evaluation.
M REASONING EXAMPLES
We provide some examples to present the reasoning process of Retro* in Table 17 (Suatainable Living), Table (P and Table 19 (TheoremQA Theorems).
22

---

Published as a conference paper at ICLR 2026
Here the +*rel dafinitionns in vetrieval task: vrelevanca_definttlie
given Axquery* ((quecy_type)
vidocumentra ([doc_type)) in this retrieva: task, your mission is to perform the following steps.
Baaed on the relevance definition, annotate integer score from scale of 100, Please use the follewing guids: 90-103 (Eighly Ralevant), 60-20 (Ralevant), 40-60 [Noderately Relevant), 20-40 (Slightly Relevant), 0-20 (Terelevant).
any explanation, directly providing your final relevance score. placad strictly between the <acare> tags. There should be other text explanation iraide tags:
must
escore?
[From a scale of 0 to 100, annozate the degree of relevance between the query and the document
</scoze>
Query
[Begin of Query]
(query) [End Query]
Document ([dee_type)):
[Regin [scument]
(doc)
[End Document]
Figure 9: The modified relevance rubrics for reasoning-fiee free relevance scoring.
23

---

Published as a conference paper at ICLR 2026
Table 17: Reasoning examples of Retro* (7B) on the Sustainable Living dataset of BRIGHT.
Relevance Definition: Given a query (sustainable living post) and a document (passage), the document is relevant to the query if the critical concepts theories discussed in the document provide references for domain experts draft query.
to the start depleting so are solar powered cars a good idea? How much power can they produce when is cloudy?
Query:
Are solar powered cars sustainable and are they good idea for future transportation "In 'nAs population rises, fossil fuels will
Positive Document:
"Photovoltuic system") ") in combination withinlatitude and climate, determines the annual energy outpat the system. For'nexample. solar panel with will produce'a 200 kWh/yr Standard Test Conditions if exposed to the Standard TestuCondition solar irradiance value of 1000 W/m for 2.74 hours a day. Usually'nsolar panels are exposed to sunlight for longer than this given day, butmith solar irradiance is less than 1000 Wm 2 for most of the day. A solar/apanel can produce more when the Sun is high in Earth's sky and will produceinless in cloudy conditions when the Sun low in the sky, usually the Sun'nis lower in the sky in the winter.
The efficiency of the solar cells used in a I photovoltuic systemin /Kriki/hotovoltii_syeeee
20% efficiency and an area of
Reasoning Process:
Query Analysis: The query seeks an evaluation of solar-powered cars terms of sustainability and their viability as future transportation. It specifically asks whether solar cars are sustainable and produce usable power under cloudy conditions. A helpful response would discuss solar car efficiency, energy capture, potential as a sustainable transport solution, and the impact of weather conditions system outpur.
2. Documentt Analysis: The passage explains how solar panel efficiency, climate, and solar patterns affect energy output. particularly noting reduced power under cloudy skies low-sunlight conditions. While validates solar panel performance under varying light conditions directly relevant solar car feasibility it does not mention solar- powered cars, automotivv application. transportation systems, or sustainability over time. However, it provides foundational data on power generation under cloudiness, addressing part the query.
3. Relevance Annotation: The document supplies critical information about solar panel energy output affected by cloudiness. which pertinent to evaluating solar-powered cars. However, it does not link this transponation solutions, sustainability assessment of solar cars, or holistic future outlook. offers technical underpinnings but lacks domain specific context regarding vehicles or long-term strategies. Thus, while informative for part of the query (especially on power availability in cloudy conditions), it omntts major integrative insights needed for comprehensive answer.
<score>>
</scorc>
Negative Decument:
The solar family car was in 2013. [16] Researchers at CaselnWestern Reserve University
It/wiki/Case. Westem__ Reserve__ Reserve_Unnversiig cun'nrocharge more quickly, due to materials used in the solar panels. [17]
Reserve University"). have also developed solar car which
Reasoning Process:
1. Query sislysis: The query seeks to evaluate the sustainability and practicality of solar powered cars, particularly in the context of future transportation. It emphasizes two key aspects: (1) whether they are sustainable (e.g. environmental impact, energy efficiency). and (2) whether they are good idea for addressing fossil fuel depletion and scalability, especially when sunlight availahility fluctuates under clouds. Domain experts would need information energy output, charging capacity, environmental henefits, and real- world feasihility.
2. Document Analysis:
The document mentions solar from 2013 and references reseurch improving charging speed with advanced materials. However, it lacks any discussion of sustainahility, energy prouction, energy cloud dependency, or broader transportation implications. It provides only a limited technical note on material advancements without addressing the core sustainability socictal questions raised.
While the document tangentially related mentioning solar powered cars- fails to address the critical concepts the query, such sustainability metries, energy output under cloudy conditions, long- viabiiiy. It does not offer references that would help draft comprehensive answer. The information too minimal and outdated to support expert judgment on the broader query.
3. Relevance Annotation:
<score>>
35
</scorc>
24

---

Published as a conference paper at ICLR 2026
Table 18: Reasoning examples of Retro (7B) on the Pony dataset of BRIGHT.
Relevance Definition: Given query (Pony coding instruction) and document (Pony documentation passage), the document is relevant to the query if the Pony syntax described in the document is necessary for heginners with no prior knowledge of Pony to complete the coding instruction in the query.
Query:
will use the programming language pany, lnooblem WnWiite a function to determine if string consisting of alphabets palindrome:'n 'aHere is the cude template:infun is palinddromo(st:: String): Bool =>>o... ...
Positive Document:
Functionsla IaPony functions are quite like functions (or methods) other languages. They can and O or return values. If the return type is omitted then the function will have return value of 'None' An ""ponyloclass Clafun add(x: U32, y: U32); U32 wolex + y'n Infum nop() =>lnadd (1. 2)// Pointless, we ignore the resultin "In The function parametens (if any) are specified in parentheses after function name. Functions that don't I take any parameters still need to have the parentheses in InFach parameter is given name and type. In our example function 'add' has 2 parameters, 'x' and "y". both of which are type 'U32: The values passed to function call (the 'I" and '2' in our example) are called arguments and when the call is made they are evaluated and assigned to the parumeters. Parameters may not be assigned to within the function they effectively declared "let' in InAfter the parameters comes the return type. If nothing will he returned this simply omitted. la lnAfter the return value, there's a '>>' and then finally the function body. The value returned is simply the value of the function body (remember that everything is an expression), which is simply the value of the last command in the function. Rensoning Process:
or more parameters
1. Query Analysis:
The query asks to write fiunction in Pony that determines whether a string is a palindroome. The key elements needed are: (1) a function taking a parameter of type "String" and returning 'Bool'. string handling in Pony. (3) string reversal/teration, and (4) comparison logic. However, for beginner with prior knowledge, the most critical information is understanding how to define a fimetion in fony-especially the syntax for defining a function with a parameter, return type, and body. This is foundational for structuring any function, including one for palindrome detection.
2. Document Analysis:
The dacument explains Pony function syntax depth, covering parameter specification, return types (including optional ones), function bodies, and return values using a basic example ('add"). It clarifies that functions use "fun", can have parameter lists with types, return statements via evaluation of the final expression, and can omit return types for 'None'. While does not mention string operations, it provides essential syntax needed to write any function, including ps__ddindr.mme
3. Relevance Annolation:
The document does not address handling strings or palindrome logic directly, but it fully covers the necessury syntactic strettrr for defining the function the paramtrrameteri functions with types, the 'fun" statement, and function bodies. This is foundational knowledge heginner must have to implement the requested function. While not comprehensive for the full task, supplies the core sy syntactic framework. Thus, it is highly relevant to the coding instruction in terms of necessary beginner syntax.
<score>>
</score>
Negative Document:
## Control Structures'a Wooo do real work in program have to be able make decisions, iterate through collections of items and perform actions repeatedly. For this, you need control structures. Pony has control structures that will be familiar to programmers who have used most languages, such as 'if", "while" and 'for', but in Pony, they work slightly differently.nn
Reasoning Process:
1. Query Analy sis: The query asks for function in Pony to check if a string is a palindrome. This requires understanding of string manipulation. comparison, and possibly loops or recursion. Key concepts for beginners include: defining a function with the correct signature (e.g g. 'String Bool'), accessing characters in string, reverse comparison, and possibly handling cuse sensitivity or ignoring non alphabetic characters. The critical syntax needed includes function definition, type parameters, string indexing. and comparison logic.
The document discusses general programming control structures ('if'. "while". 'For') and notes that these work similarly in Pony. However, it provides no specitic examples, syntax, or details on string handling, function syntax, or palindrome logic. It fails mention to manipulate strings, access characters, implement palindrome delection core needs for the query. The content is introductory and abstract, focusing on high-level structure rather than concrete, applicable code.
2. Document Analysis:
3. Relevance Annolation: The document touches on conditional and iterative constructs, which tangential palindrome logic, but does not provide the necessary syntax or examples for completing the task. It lacks any mention of string properties, indexing, or function definition relevant to the query. For heginner, this passage would offer minimal direct assistance. Therefore, only shares conceptual keywords (e.g., "if", "while") but misses all domain- specifcc syntax.
<score>
</score>
25

---

Published as a conference paper at ICLR 2026
Table 19: Reasoning examples of Retro (7B) the TheoremQA Theorems dataset of BRIGHT.
Relevance Definition: Given query (math problem) and document (math-rellted passage), the document is relevant to the query if the theorem described in the document can help solve the problem in the query.
Query:
In party, how many guests do you need to have to ensure that either four people all know each other or four people are all complete strangers to one another?
Positive Document:
Vsectton (Ransey' 's Theorem PuTTags: : Ramsey Theory, Named Theorems, Combinatoricsln la (begin((heorrm) [thorrmm] (nnel /nInn any coloring of the edges of sufficiently large complete graph, one will find monochromatic complete subgraphs, AnFo colors, Ramsey's is theorem states that for any pair of positive integers Sttuple sJS, there exists a least positive integer Siimap R (r. sIS such that for any complete graph on $'lmap (r. xJ$ vertices, whose edges are colored red or blue, there exists either Ss$ vertices which is entielly blue. Mnre
complete subgraph on StS vertices which entirely red, a complete subgraph
generally, for any given number of colors ScS, and any given integers Sn__ I. Widots, n_e5, there is number Silmap R (n_ I.
Widots, n_e|S such that:'n :f the edges of a complete graph of order Simmpp R |n_ Widots, n_e|S are colored with $c5 different colours, then some SiS between S15 and Sc5, must contain complete subgraph of order Sn_ Sn_i$ whose edges are color IiSis number Slnmap R I. Widots, n_c]S called the Ramsey number for $n I. Mldoss, The special case above has So 28 (and Sa_1 1= and So__ = sereee Simap R (t. s}S signifies an integer that depends on both SrS and SsS. It understood to represent the smallest integer for which the theorem holds in Wend (heorem) I'n in Voegin | proof |nnisstt we prove the theorem for the 2-color case, induction on Sr <S. Inll clear from the definition that'n :Woorll Van WN: Wmap (n, I] = Vkmaa & {I. n) 1Stbbecauee the complete graph on one node has no edges. la This is the base case. WeWe prove that SR Weft(rr s /WrightS exists by finding an explicit bound for it the inductive hypothesis, SWmap R (r- L. s and Simaap
1J5 exist.In Wend| proof/in In
Reasoning Process:
1. Query Analysis: The query asks for the minimum numher of guests (people) in party such that either people all know each other complete subgraph SK_
"knows" -graph) or 4 people are all strangers each other complete subgraph SK_4'S in a "strangers" -graph).. This is classic Ramsey number problem, specifically asking for SR(4,4)S, the smallest number of gnests such that in any 2-coloring of the edges (representing know-or-ste or-snnggg,rr there must he a monochromatic $K_4S. The key concept needed is Ramsey theory, particularly Ransey number SR(4,4)S.
2. Document Analysis: The document presents Ramsey's ' Theorem, defining the Ramsey number Stmpp R [rs]S as the smallest number such that any 2 2-clorring the edges complete graph on Samap R [ra}S verlices guarantees either all-red $K_s$ all-blue s$. It confirms the theorem for Sc=25, which directly applics to the query's 's 2-coloring framework. While it does not explicitly state SR(4,4)S or compute its value (which is 18), the theorem provides the theoreticul foundation to solve the query. From this, one could deduce that any graph on 18 or more vertices necessarily contains either a SK_4$ or Stoverline [K] L,,,, depending the coloring.
3. Relevance Annotation: The document is highly relevant because it introduces and explains Ramsey's 's Theorem, the core mathematical principle required to solve the query. Although it does provide the specific value of SR(4,4)8, it enables derive it via the theorem. It directly supports solving the problem by offering the theoretical framework.
<score>
95
</scorc>>
Negative Document:
Usection| | Infinite Ramsey's ' Theorem |inTags: : Ramsey Theory, Named Theoremsin In Whegin [heegi|(thenee theorem J' Sk, n Vlnn WNS. lnFor any set SSS, let SSaalarre a|$ denote the set Sisset ('sstt [s_1, lildots s_n]: Miextfeach s_i Win S)S cardinality Sn$ subsets of SSS InLet 3XS be an infinite shenn An for every partition SPS of SX* (X-(Parre n}S into Sk$ many components'n in :here is an infinite subset $Y laubsetey XS'ssuch thatkn each member of [Vgarea n)$ is in the same component of SPS.A Vend theorem in in libegin| [proof] In We will prove the theorem for fixed Sk3 by induction on SnS.An
proofff 'n
Reasoning Process:
1. Query Analysis: The query asks for the minimum number of guests (n) at a party such that either four people all know each other elique size 4 in social graph) or four mutually unknown people exist independent set of size 4). This
Ransey theory problem specifically involving the Ramsey number R(4,4), which is the smallest number such that any 2-coloring of the edges of a complete graph on vertices guarantces a monochromatic clique of size 4 or an independent set of size The key information needed is known result about R(4,4), preferably its value discussion of its combinatorial implications.
Document Analysis: document presents Infinite Ramsey Theorem, which states that any partition of the n-element subsets of infnite set into finitely many components, there exists infinite subset that monochrematic. This is much more general and abstruct result. While is situated in Ramsey theory, it deals with infinite sets and n-element subsets, and not address the finite Ramsey number R(4.4) or any finite Ramsey tyne problem. does not mention graph theory. social networks, cliques, or independent sets the context of a finite party problem. Thus, fails to provide any specific insight into the numerical or combinatorial condition required to solve the query.
3. Relevance Annotation: The document is thematically related (Ramsey Theory) but addresses fundamentally different
domain (infinite sets, infinite Ramsey Theorem) and offers no useful method or result for determining R(4.4) or solving the finite problem. cannot help directly in solving the given social puzzle. While both involve Ramsey theory, the connection is too abstract and generic without addressing the specific problem its mathematical tools.
<score>
</score>
26


Reviews:
[{"review_id": "6aMLs3sp8j", "reviewer": "Reviewer_9eA5", "summary": "This paper presents Retro*, a new method for training rerankers for reasoning intensive queries. Retro* is first trained with both SFT on traces from a large powerful model and then later trained with reinforcement learning. For the later stage, the authors design new reward functions specifically for improving reranking performance.", "questions": "Here are some questions:\n- I think there is a typo on line 214. There are four chosen trajectories and not just one, right?\n- For the only-SFT ablation how many training samples were used?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of the paper are as follows:\n- The RL training rewards proposed in this paper are novel and are distinct from other baseline methods. \n- The parallelism and inference time analysis is interesting.", "weaknesses": "The weaknesses of the paper are:\n- Lately, there has been a lot of debate whether reasoning is helpful for reranking. Is the reasoning trajectory actually necessary or can model be trained directly to just produce the score. The authors should ablate whether the reasoning trajectory are helpful and if so how helpful they are. \n- For the retro* results in table 1, the authors use first-stage candidates from BGE-Reasoner-Embed. What is the first stage retriever for the other baseline models in main table? If the first stage retriever is different then it is hard to compare numbers. \n- Along similar line, the authors should clarify that the baseline models are not build from the same backbone. For instance, RankLLaMA and RankZephyr are built from older llama models.\n- The performance gains seem limited when making comparisons in the same setting. For instance, ReasonIR+QwenRerank obtains a score of 36.9 on bright and this is with no additional reranker training (the qwen model instruct model is used as is). From table 2, we see that Retro* obtains a scores of 37.4 after SFT and RL training. \n- ReasonIR is mentioned in the paper but is not used as a baseline. I understand that ReasonIR is a retriever but the ReasonIR paper also presents numbers for ReasonIR+a zero-shot qwen based reranker.\n- Rubric-based relevance scoring is not novel. Table 10 of the Rank1 paper shows that data-specific prompts were used for BEIR and the non-stackoverflow BRIGHT subsets. ReasonIR did something similar as well. Also the name \"rubric-based scoring\" is a little mis-leading because it implied there are multiple rubric elements, but the rubric is just a definition of similarity for each task. \n- Minor issues: There are some typos in the paper. E.g. \"while positive documents are assigned low scores\", \"with zero RL achieves a higher score of 35.1.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JJxVUH4A0y", "reviewer": "Reviewer_d5S7", "summary": "This paper proposes Retro* a rubric-based relevance scoring mechanism for reasoning-intensive retrieval. Retro* leverage a two-stage SFT+RL training with composite intra-/inter-document rewards. The method achieves strong nDCG@10 on BRIGHT and shows consistent gains from model scale and test-time sampling.", "questions": "n/a", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Retro* uses rubric-based scoring methods, yields interpretable absolute scores rather than only relative ranks.\n2. Retro* propose intra/inter rewards for RL, and the paper ablates each components.\n3. The proposed methods achieve SOTA BRIGHT results.", "weaknesses": "1. The model uses two types of rewards (intra- and inter-document), but it remains unclear which dominates training dynamics or contributes more to final gains in run-time.\n2. As rubric-guided scoring is claimed to enhance interpretability, more qualitative or case-based evidence would strengthen this claim.\n3. The reliance on a strong teacher (Qwen3-235B) for SFT trajectory generation raises reproducibility and cost concerns, as smaller or open models may not reproduce the same reward quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2bSjpENwk5", "reviewer": "Reviewer_t3di", "summary": "The paper developed Retro*, a reasoning-based re-ranking method to score the relevance of the documents for reasoning-intensive retrieval. By leveraging distillation from a teacher model, then followed by reinforcement learning with composite reward evaluation, Retro* learns to score documents using verbal reasoning from the two-stage training strategy, and works together with test-time scaling strategies (multiple rollouts). As a result, Retro scores significant performance improvement over other re-ranking baselines on BRIGHT reasoning-intensive retrieval benchmark.", "questions": "1. The performance on Leetcode degrades significantly after reranking. I wonder if the authors have some understanding of why this is happening?\n2. Why test-time scaling by average can boost the performance by so much since the aggregation is average? I can understand that without averaging, there might be more randomness, but it is a bit surprising to see that this type of ensemble can lead to performance gain. \n3. Are there any potential data leakage in the experiments, especially when using the SFT model?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-written with clarity. \n2. Incorporating rubrics as part of the input and having both intra- and inter- document rewards for better learning are reasonable and original ideas. \n3. The empirical results are strong. The ablation studies show that the two-stage training is essential for performance improvements, with most of the gain coming from RL with composite reward. \n4. It’s interesting to see from the test-time scaling experiments that having an ensemble of scores from the same reranking model offers consistent performance improvement.", "weaknesses": "1. While the rubrics are an important part of the contribution, having them in the model may restrict the generalizability to other reasoning-intensive tasks that are unseen. \n2. The paper mentioned that the training data is based on BGE-Reasoner-Data, but it is unclear to the reader how BGE-Reasoner-Data is curated. It’s helpful to clarify that in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)