Rethinking Reasoning in Document Ranking: Why Chain-of-Thought Falls Short
Authors: Xuan Lu
Haohang Huang
Rui Meng
Yaohui Jin
Wenjun Zeng
Xiaoyu Shen
Venue: ICLR 2026 Poster (2026)
Keywords: information retrieval
reranking
reasoning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We conduct the first systematic study of explicit reasoning in document reranking, spanning pointwise and listwise paradigms, supervised and reinforcement learning, and diverse benchmarks including BRIGHT and BEIR.

Abstract: Document reranking is a key component in information retrieval (IR), aimed at refining initial retrieval results to improve ranking quality for downstream tasks. Recent studies—motivated by large reasoning models (LRMs)—have begun incorporating explicit chain-of-thought (CoT) reasoning into LLM-based rerankers. 
However, the effectiveness of such reasoning for ranking tasks remains underexplored.
In this work, we present the first systematic study of reasoning in reranking across both logits-based pointwise and listwise settings, under both supervised fine-tuning and reinforcement learning. Using diverse benchmarks, including reasoning-intensive datasets BRIGHT and standard IR benchmarks BEIR, we find that reasoning-augmented rerankers consistently underperform their direct counterparts that predict rankings without CoT, despite substantially higher inference costs. Our analysis reveals three core limitations: (i) in pointwise rerankers, reasoning breaks calibration and biases models toward the positive class, raising TPR but lowering TNR, which inflates false positives and degrades ranking in negative-dominant pools; (ii) in listwise rerankers, explicit reasoning improves the fit during training but leads to higher variance and fails to improve performance in both in-domain and out-of-domain evaluations, even when reinforcement learning shortens rationales; and (iii) overall, directly fine-tuned rerankers remain more stable, effective, and robust.  These findings challenge the assumption that explicit reasoning is universally beneficial for reranking. We conclude by highlighting future directions, including calibration-aware scoring for pointwise rerankers and the design of concise, targeted reasoning strategies to mitigate overfitting and overthinking in listwise rerankers.

Published as a conference paper at ICLR 2026
RETHINKING REASONING IN DOCUMENT RANKING: WHY
FALLS SHORT
Xuan Lu!22; Haohang Huang"; Rui Meng, Yaohui Jin', Wenjun Zeng"n, Xiaoyu Shen
'Shanghai Jiao Tong University
"Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative
'Institute of Digital Twin, Eastern Institute of Technology, Ningbo Lux199788 87 edu.cn xyshen@eitech. edu.cn
ABSTRACT
Document reranking is a key component in information retrieval (IR), aimed at refining initial retrieval results improve ranking quality for downstream tasks. Recent studess-mtiivate corporating explicit Chain-of-thought (CoT) reasoning into LLM-based rerankers. However, the effectiveness of such reasoning for ranking tasks remains underex- plored. In this work, we present the first systematic study of reasoning reranking across both logits-based based pointwise and listwise settings, under both supervised fine-tuning and reinforcement learning. Using diverse benchmarks, including reasoning-intensive datasets (BRIGHT) and standard IR benchmarks (BEIR), we find that aogoonnnnnggggeeeedllle rerankers consistently underperform their direct counterparts that predict rankings without CoT, despite substantially higher infer- ence costs. Our analysis reveals three core limitations: (i) in pointwise rerankers, reasoning breaks calibration and biases models toward the positive class, rais- ing TPR but lowering TNR, which inflates false positives and degrades ranking in negative-dominant pools; (ii) in listwise rerankers, explicit reasoning improves the fit during training but leads to higher variance and fails to improve performance in both in-domain and -fomof-domain evaluations, even when reinforcement learning shortens rationales; and (iii) overall, directly fine-tuned rerankers remain more stable, effective, and robust. These findings challenge the assumption that ex- plicit reasoning is universally beneficial for reranking. We conclude by highlight- ing future directions, including calibration-awan scoring for pointwise rerankers and the design of concise, targeted reasoning strategies to mitigate overfitting and overthinking in listwise rerankers.
by large reasoning models (LRMs)- have begun in-
INTRODUCTION
Document reranking crucial step in information retrieval (IR), aimed refining the coarse- grained results produced by first-stage retrieval methods. By reordering candidate documents, reranking improves precision and overall ranking quality, which is essential for downstream ap- plications such as retrieval-augmenter and recommendation (Ren et al.) 2024). The landscape of reranking is dominated by two primary paradigms: pointwise and listwise. Pointwise rerankers independently estimate the relevance score of each query-document pair and sort documents accordingly. Since each document is processed in isolation, pointwise rerankers allow parallel computation and efficiency. In contrast, listwise rerankers consider the entire candidate set jointly, asking the model to output a ranked list. While computationally more expensive, listwise rerankers often achieve more accurate rankings by lever- aging cross- doosmmd interactions and relative comparisons, which is fundamentally easier than assigning a precise relevance score each document in isolation.
generation (RAG) (Lewis et al. 2020 Lu et al.) 2025a)
"Equal contribution
Working at Google Cloud AT Research. & Corresponding author. We release our code httpt:://gtthhhhee wwt.://www............a/a..aa./e

---

Published as a conference paper at ICLR 2026
With the rise of large language models (LLMs), reranking performance has advanced substantially. By combining targeted prompts with task-speeific fine-tuning, LLM-based rerankers have achieved state-of-the-ar results on diverse benchmarks (Sun et al. 2023a). Recently, large reasoning models (LRMs), such as DeepSeek-R1 (Guo et al. 2025) and OpenAT ol (Jaech et al. 2024), have further drawn attention. Unlike typical L.MMs that directly produce answers, LRMs explicitly decode rea- soning chains before providing the final prediction. This process narrows the gap between input and output, smooths token-by-token transitions, and has been shown improve performance in many tasks. Motivated by these advances, recent studies have sought to extend test-time reasoning to reranking via supervised fine-tume ainettnnig Weller al. al. (2025); Ji et (2025); Yang et al. (2025) or using reinforcement learning Zhang et al. 2025); Zhuang et al. (2025) Liu et al. (2025).
Despite these developments, a fundamental question remains unresolved: Does explicit reasoning truly benefit reranking? Prior work often assumes that tothi-of-thouugt (Co'T) reasoning enhances reranking (Weller al.
2025 Yang et al.) 2025) Zhuang et al., (025), yet such claims are rarely
supported by fair comparisons against non-reasoning baselines (Weller et al. 2025 Yang et al. 2025 Zhuang et al., 2025 Liu et al. 2025). Moreover, emerging evidence suggests that reasoning-
augmented rerankers can suffer from overthinking and lengthy reasoning chains introduce noise that degrades performance (Jedidi et al. 2025; Fan et al.) (2025). However, these analyses are limited in scope, focusing narrowly on pointwise rerankers trained with supervised objectives, and fail to offer a systematic understanding of reasoning's 's actual role in reranking.
In this work, present the first comprehensive and fair study of reasoning in reranking- To ensure rigor and comparability, we adopt a unified experimental design: all rerankers are trained on the MS MARCO dataset, with
models using CoT chains generated by DeepSeek- R1. We cover both logits-based pointwise and listwise rerankers, both direct-oulput and reasoning- augmented variants, and both supervised tueettinng (SFT) and reinforcement learning (RL) training regimes. This setup eliminates inconsistencies across prior work and allows for clean, apples- to-apples comparison. We further evaluate models on two complementary benchmarks: BRIGHT, which emphasizes bessoning-intensive queries, and BEIR, a standard suite of retrieval datasets. The scale, diversity, and uniformity of this design ensure that our conclusions are not anecdotal but systematically validated. Our extensive experiments reveal a striking and consistent pattern: under- current training and inference setups, reasoning-buae -based rerankers underperform their direct-oupuu counterparts, even though they incur substantially higher inference costs. This finding holds across architectures, training strategies, and benchmarks, suggesting that explicit reasoning- -hich efits many other LLM tasks-does not translate into gains for reranking. Instead, reasoning intro- duces calibration errors, overthinking, and poor generalization, ultimately harming ranking quality. Our contributions can be summarized as follows:
which ben-
A rigorous, systematic study. We conduct the first large-scale, controlled comparison of rea- soning vs. direct reranking, covering pointwise and listwise paradigms, SFT and RI. training, and both reasoning-intensive and standard IR benchmarks.
Clear evidence against reasoning in reranking. Direct-output rerankers consistently outper- form reasoning- -uggennee variants, despite the latter's 's substantially higher inference cost.
Deeper insights into failure modes. Our analysis reveals that for pointwise rerankers, rea- soning does not improve calibrated relevance prediction; instead, it shifts the error distribu- coin-riiiing TPR while reducing ENR-which disrupts score calibration and introduces a bias toward false positives. Similarly, for listwise rerankers, reasoning leads to better training fit but increases variance and fails to yield gains on both in-domain and out-of-domain evaluations, even when rationales are shortened via GRPO.
Guidance for future research. Our findings suggest that reranking should prioritize effi- cient direct scoring rather than reasoning-heawe approaches. Promising directions include calibration-awant aware scoring for pointwise rerankers and designing concise, targeted reasoning strategies to mitigate overfitting and overthinking in listwise rerankers.
2

---

Published as a conference paper at ICLR 2026
Pointwise Ranking
Direct Ranking
Reasoning Ranking
W/Whe ore the kealth benefits grean reo!
Ronked list
GrWkker ere the heafth benefirs of green
Ranked list
velroar
DuBicorn SpScoee2
health
Dy-Scame DyrSoare 3
by Sruse two natted in China and
by Greas tao arignncted Cvino and
Scond
Q Whe benefirs green
of groon DuSeneerl atud auguatt that..
SySre
By Severdl atucias
Score
Scone
Listwise Ranking
Direct Ranking
Reasoning Ranking
gremn
th freae tws cartiine art eniiaddante then
by Green tea natted in China and now....
Ranked list
By dinas has oetheine onr suldinte that by dreen hos orignntedd China and il non...
vallory- stuii
Rankeed Dy
Groee
+D.
Dy Seserd stuckes suggest
D.Seward studles sugest that...
Figure I: Illustration of Pointwise and Listwise Reranking (Direct vs. Reasoning). In pointwise, each query- document pair is judged independently, with relevance scores computed the normalized probahility of the TRUE token over (TRUE, FALSE] logits. Listwise directly optimizes the ranking order over candidate sets, with or without explicit reasoning.
PRELIMINARIES
2.1
TASK SETUP
We consider the reranking task in information retrieval (IR), where the goal is to reorder an initial set of candidate documents that those most relevant query appear the top. Formally, given a query g, a retriever first returns a candidate set
C(q)= = {di, {di, d2.. d+)-
The reranker then takes (g, C(q)) as input and produces an improved ordering of the documents in C(q). This approach reflects the standard two-stage retrieve-and-rerank architecture used modern IR systems. First, an efficient retriever, optimized for recall, generates an initial, coarsely-ranked list of documents from a large corpus. Then, a reranker, typically a more expressive model optimized for precision, refines this list. The goal of this two-stage process is to maximize user satisfaction by placing the most relevant documents the very top of the results.
2.2 POINTWISE RERANKER
In the pointwise setting, each query docymet pair (g, di) is evaluated independently. Let E(g, di) denote the prompt that encodes the pair, and let the answer space A = {TRUE, FALSE), corre- sponding to tokens Tirue and Tralse- For candidate d,, the model produces logits & E IR(P) from which relevance score is derived the normalized probability of the TRUE token:
(xp(e[(TrRu))
exp(el(FrRue) + exp(C, [Twsee))
The final ranking is obtained by sorting { in descending order.
Optionally, pointwise reranker can be extended with explicit reasoning: before predicting the binary decision, the model generates an intermediate reasoning trace zι (e.g., a chain-of-thought),
pd(2 E(g, d4),,
8; = Pr(a P TRUE Elg. d:), =;);
where A and the probability is computed from the answer-token distribution conditioned on 24 In practice, multiple traces { (mn) LM Jm=1 may be sampled and aggregated (c.g., by averaging or voting). Suppressing 2; recovers the non-reasoning formulation above.
2.3 LISTWISE RERANKER
In the listwise setting, the model considers the entire candidate set C(g) jointly. Let C(g)) denote the encoding of the query and its candidate list. The model autoregressively generates
3

---

Published as a conference paper at ICLR 2026
permutation (Ti..... ................................ Tk) of indices:
pd क (lg. C(a))),
(l,...,kk /
At inference, one may decode the most likely permutation â (c.g., via greedy or beam search), compute ranking scores from partial sequence probabilities. When & exceeds the context win- dow, candidates can be processed in overlapping blocks (e.g., sliding windows), with local rankings merged into global order.
Optionally, listwise reranker can be extended with explicit reasoning. In this case, the model first generates global reasoning trace Z that captures cross-document comparisons:
pd Z plg. C(1))),
po(TT (gg. C(g)).
The trace may include pairwise judgments, list-level critiques, or structured deliberation, and can be produced either separate stage (generate Z then iT) interleaved with ranking. Suppressing recovers the standard listwise formulation described above.
3 EVALUATING THE IMPACT OF REASONING ON RERANKING
3.1 MODEL VARIANTS
We design four LLM-based rerankers, each corresponding to different combination of pointwise vs. listwise and reasoning vs. non-reasoning paradigms:
Direct-Point (Non-Reasoning Pointwise): the model directly outputs a binary relevance decision (TRUE/FALSE). We take the logits of the answer token and transform them into probability score, which is used for ranking.
Reason-Point (Reasoning Pointwise): the model first generates a reasoning trace describ- ing why the document may or may not relevant, and then produces the final binary decision. The relevance score is computed from the logits at the answer token position.
Direct-List (Non-Reasoning Listwise): the model takes the entire candidate list input and directly generates a permutation as the output ranking, e.g., [3] > 5 > [4] .....
Reason-List (Reasoning Listwise): the model first generates a reasoning sequence that compares and analyzes candidates, and then outputs the final ranking sequence.
3.2 TRAINING DETAILS
Backbone Models We adopt the Qwen3 series as the backbone for our rerankers, specifically Qwen3-4B and Qwen3-8B. This choice consistent with the prevailing practice in the reranking community All training experiments are conducted on two NVIDIA A100 (80 GPUs.
Pointwise Rerankers We study two pointwise variants: Direct-Point and Reason-Point. Both models trained the RANK corpus (Weller al. 2025) derived from MS MARCO, compris- ing 386k svery-passage pairs annotated by DeepSeek-Ri RI with chain-of -fain-of-thought rationale and binary answer (TRUE/FALSE). For Reason-Point we perform supervised fine-tuning on quadru- ples (query, passage, rationale, answer). For Direct-Point we ablate the rationale and fine-tuning on (query, passage, answer), training the model to emit single token in (TRUF, FALSE). Both variants minimize cross-entropy loss. We employ the ELaMA-Factory/ framework for supervised fine- tuning. All models are trained using LoRA with rank 32 and a = 64, learning rate of 1 x 10-4, and cross-entropy loss. Both DIRECT-POINT and REASON-POIN rerankers are trained for one epoch. At inference time, we compute the relevance score used for ranking as the probability assigned to TRUE via a two-way softmax the logits of {TREE, FALSE]; ties broken by the logit margin. Example prompts and data instances for both settings are provided in the Appendix C.T
complete list backbone models used in both our work prior reasoning-enhann rerankers provided in Appendix https:: thub. com/hs /hi thhhyoouaa//iia http://hooougaalaaaaaaaaaaa Pactoryyyr

---

Published as a conference paper at ICLR 2026
Listwise Rerankers We train listwise rerankers the REASONRANK training corpus (Luu al. 2023), which contains 1~13 candiccandi sets primarily derived from MS MARCO and related benchmarks, split evenly between SFT and GRPO (approximately 6.7k each). Each instance in- cludes a query, a candidate set, rationale produced with DeepSeek-R2 1e. and a gold ranking order. We consider variants: Direct-List, which generates ordering directly from the query and can- didate set, and Reason-List, which is prompted to first generate own rationale and then produce the final ordering.
We adopt two-stage training for traning Direct-List and Reason-List Stage I performs super- vised fine-tuning (SFT) teach the model output ranking sequences. To encourage struc- turally valid outputs, prompts require the model to produce reasoning segment demarcated by format.
<think>... Atthink>> and final ranking in <answer>[] > []] ...</answes Stage 2 refines the SFT model with Group Relative Policy Optimization (GRPO) (Guo et al. 2025 We follow the setting ReasonRank (Liu et al. 2025), using composite multi- view ranking reward that reflects position sensitivity, coverage, and list similarity:
Let Tat denote the predicted ranking sequence and s/ the gold ranking. We combine three signals:
Rm NDCG@ 10 v/) + Φ 10 v/) + y. (B-O .v'),
where é, weight coverage and overlap. Rank-Biased Overlap (RBO) (Webber et al. 2010) em- phasizes top ranks and is computed as
d-1 Viad Dili
RBO (1
(2)
d=1
with persistence parameter p 10, and yı:d the top-d prefix. Following REASONRANK, gate Rm with simple format validators to stabilize learning:
Rm,
both output and answer formats are valid,
R
0. only the output format is valid,
(3)
1,
otherwise.
where the oulput-formal check requires the presence of <think: nk> and <answer> tags, and the answer-format check verifies that the <answer> contains canonical listwise ordering.
With the gated multi-view reward in Eqs. I 2 we refine the SFT policy via GRPO (Guo al.) 2025). Given input v, group of samples G {ur} is drawn, sequence rewards R(x, vs) are converted to token-level advantages Åi,,, and the policy is updated by the elipped objective:
JaRRO
Σ min(r; (i))( clip(r,, - + e)A.,) D2I Tee)), (4)
where (0) is the importance ratio and uee the SFT reference. We implement GRPO with Ver for one epoch on the ReasonRank data, producing Direct-List and Reason-List models.
3.3 EXPERIMENTAL SETUP
All experiments are conducted first-stage candidate pool of 100 passages per query built with BM25. Unified system prompts are used across all datasets, as shown Appendix Dataset-level instructions strictly follow the original Rankl templates (Weller et al. 2025).
NVIDIA A100 GB) GPUs. Rerankers operate fixed
Baselines compare our four proposed rerankers--iieee PPinnt Reason-Point Direct-List, and Lissson-Lis-inclileee ablations across training stages (SFT only vs. SFT+GRPO), In addition, we LLM rerankers from prior work: Pointwise: Rank1-7B, Rank1-14B (Weller et al., (2025). [F-Rank-4B, TF-Rank-8B (Fan et al.
compare our models against ofth--fft prasoning-enhanee
2025 ; Listwise: Rank-R 1-BB, Rank-R1-145 (Zhuang et al. 2025 REARank-78 (Zhang et al,
2025
and ReasonRank-7B (Liu al. 2025).
https://githhb.
cengi ne/ hevee
5

---

Published as a conference paper at ICLR 2026
Benchmarks and Metrics We evaluate two retrieval benchmarks: BRIGHT, reasoning- intensive IR suite spanning diverse domains, and BEIR, a standard heterogeneous IR benchmark. Following common practice, the primary metric is NDCG@ 10, which captures both relevance and position sensitivity on the top-10 results.
3.4 MAIN RESULTS
Reasoning Does Not Improve Reranking Performance As shown Tables 1 and 2 a con- sistent and repeatable pattern emerges across all training settings (SFT and SFT+GRPO), model sizes (4B/8B), and benchmarks (BRIGHT and BEIR): direct rerankers consistently outperform their reasoning-augmentter
counterparts. For pointwise rerankers on BRIGHT, Direct-Point-43 exceeds Reason-Point-Pe by AN@10= + 9.0, and Strect-Point-a 8B by +6.1 on the original query, while the advantage remains +-4-6 points on the gpt4 reason query split on BEIR, the correspond- ing gaps are +5.33 and +4.3. For listwise rerankers, the advantage is smaller but remains stable: on BRIGHT, Direct- achieves +0.4++1.7 higher N@10 than Reason-List while on BEIR the mar- ranges from +0.3 to +.9, consistently observed under both SFT and SFT+GRPO. Moreover, GRPO provides further improvements for listwise rerankers compared to SFT alone, yet the su- periority of direct reranking persists regardless of training strategy. These results reveal clear trend: under current logits- based pointwise and generative listwise reranking setups, we observe that explicit reasoning does not improve ranking performance.
Comparison Results on BRIGHT and BEIR. Tables and report the performance our rerankers compared to eenhaneeg-enhaneed baselines BRIGHT and BEIR, respectively. On orig- inal query split of BRIGHT, our Direct- Liscr--- achieves the best overall score with N@10 = 27.1, followed closely by Direct-Point-83 at 26.8, both outperforming reasoning- baselines such as ReasonRank-7BB (26.4), TFRank-8B (22.6), and Rank-RI-14B (20.5). On the gpt4 reason split of BRIGHT, Direct-List-4B and Direct-List-SB (both N @110 35.3) likewise surpass all Reason-List baselines. On BEIR, the strongest model is Direct-Point-411 which obtains N@10 = 45.4, surpass- ing larger reasoning-enhannee enhanced listwise rerankers such Rank-RI-144 (43.8) and ReasonRank-7B (41.7). These results demonstrate that our direct rerankers not only outperform existing reasoning- based counterparts, but also highlight that explicit reasoning is unnecessary for achieving state-of- the-art effectiveness in LLM-based reranking.
4
ANALYSIS
1.0
1.0
Direct (ECE=0.105)
Reasoning (ECE=0.151)
0.6
0.6
Fraction of samples
Fraction of samples
0.8
0.8
Empirical accuracy
0.5 &
Empirical accuracy
Fraction of samples
Fraction gamp
0.6
0.4
0.6
0.3
0.4
0.4
0.2
0.2
0.2
0.2
0.1
0.1
0.0
0.0
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Predicted probability relevance
Predicted probability relevance
(a) Direct-Point-By
(b) Geason-Pointe
Figure 2: Calibration curves pointwise rerankers: predicted probabilities vs. empirical accuracies.
'Complete results for the gpt4 reason split reported in Appendix B

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison on BRIGHT across different reranker variants. We report results for Direct-Point, Reason-Point, Direct-List, and Reason-List under both SFT and GRPO training, together with representative pointwise and listwise baselines.
StuckExchange
Coding
Theorem-based
Model
Training
Bio. Earth.
Econ.
Psy.
Rob. Stack.
Sus.
Leet. Pony
AoPS TheoQ.
TheoT. Avg.
Pointwise
BM25
18.9
14.9
12.5
13.6
18.4
15.0
24.4
4.9
10.4
14.5
Rank1-7B
SET
31.4
18.3
25.4
13.8
17.6
24.8
9.5
9.5
11.6
18.5
Rank1-14B
SFT
29.6
34.8
172
24.3
18.6
16.2
24.5
14.4
9.2
10.7
TFRank-4B
SFT+GRPO
17.6
29.5
21.0
20.9
18.3
9.1
7.3
20.6
TFRank-BB
SFT+GRPO
33.7
46.2
23.7
26.0
24.1
20.1
23.6
28.8
12.5
10.8
11.4
9.7
22.6
Brason-Point-4B
SFT
23.6
29.0
15.0
23.7
16.7
12.2
18.3
12.4
11.0
9.4
16.5
Direct-Point-IB
SFT
34.9
45.1
23.3
318
26.6
23.6
30.7
18.5
35.4
13.6
15.2
25.5
SFT
24.9
34.6
17.5
26.2
25.9
22.4
19.7
11.9
36.6
9.3
6.5
12.6
20.7
Direct- Point-B
SFT
33.9
46.4
24.6
31.6
25.8
25.9
32.0
25.3
35.5
12.0
13.5
15.2
26.8
Listwise
Rank-R1-7B
GRPO
26.0
28.5
17.2
24.2
19.1
10.4
24.2
19.8
4.3
4.3
8.3
10.9
16.4
Rank-RI R1-14B
GRPO
31.2
38.5
21.2
26.4
22.6
18.9
27.5
20.2
9.2
9.7
9.2
11.9
20.5
REARANK-78
GRPO
23.4
27.4
18.5
24.2
17.4
16.3
25.1
27.0
8.0
7.4
7.9
9.5
17.7
RcasonRank-7B
SFT+GRPO
36.3
44.2
24.8
31.7
30.7
24.9
32.8
28.7
17.5
12.0
18.5
14.0
26.4
Reason-List-4B
SFT
30.7
18.7
27.7
27.9
19.8
28.5
28.1
13.7
13.9
13.3
Direct-List-4BB
SFT
32.7
38.6
20.0
28.4
28.6
20.5
31.2
30.9
15.1
17.8
15.6
Reason-List-8B
SFT
39.6
22.4
29.0
29.9
23.4
34.5
18.9
15.6
12.1
24.5
Direct-List-SB
SFT
32.6
38.4
21.3
28.9
31.9
22.6
31.8
28.9
16.9
18.5
15.4
24.9
Reason-List-4B
SFT+GRPO
33.6
21.6
28.0
33.3
26.0
29.3
13.3
16.5
15.4
25.0
Direct-List-4B
SFT+GRPO
33.8
41.5
23.4
29.3
34.0
23.9
34.2
33.4
13.7
11.9
17.1
14.6
25.9
Reason-T.ss-8B
SFT+GRPO
32.1
40.3
26.7
32.1
30.0
25.5
33.8
28.8
19.4
9.8
18.0
14.0
25.9
Direct-List--B
SFT+GRPO 35.2
42.7
23.1
30.6
34.0
27.6
33.9
29.2
22.9
12.1
17.9
15.8
27.1
4.1 POINTWISE: CALIBRATION FAILURE AND TRUE BIAS WITH REASONING
Reasoning breaks calibration of confidence and accuracy Calibration assesses whether pre- dicted probabilities match the true likelihood of relevance. In pointwise rerankers, the score signed to a candidate interpreted as the model's confidence that it is relevant. A perfectly cali- brated model satisfies, for example, that predictions of 0.9 correspond to roughly 90% truly relevant items; in reliability diagrams this appears as points along the diagonal y=z. To quantify devia- tions from perfect calibration,
the Expected Calibration Error (ECE) (Guo al. 2017). the weighted discrepancy between predicted confidence and empirical accuracy across M bins:
ECE = ΣM [Bml |acc(Bm)= (oon(Bm)) where Bm is the set of samples in bin |Bml its size, N the total number of samples, acc(Bm) the empirical accuracy, and conf [mm) the average predicted probability that bin. Smaller values indicate better calibration. Notably, we observe pronounced polarization in the logits: most predictions cluster near () or reflecting overconfident decision boundaries. As shown in Figure based on results from the BEIR, the direct pointwise thruuher not perfect-maintainn accuracy (ECE = 0.105). By contrast, the reasoning-enhance confidence with larger departures from the diagonal (ECE 0.151), indicating that adding rea- soning breaks confidence calibration in the pointwise setting. This miscalibration helps explain the observed degradation in ranking quality (e.g., lower NDCG).
clear monotonic relationship between confidence and
reranker exhibits systematic over-
Reasoning increases "True" proclivity Beyond aggregate calibration, class- condition analy- sis reveals a consistent shift toward predicting the positive class. The training data have a posi- tive:negative ratio of approximately 1:2. To match this prior, we construct evaluation pools with 100 positives and 200 negatives per query-bothh in-domain (MS MARCO 1L19/DL20 and out- of-domain (BRIGHT-Biology>--e
also include the teacher model (DeepSeek-RI), Letting the reranker judge relevance and decode the answer token, we report class-conditional performance in Table standard notation: TPR (true positive rate; recall positives) and TNR (true negative rate; specificity = 1-FPR). Under this matched prior, Reason models tend to achieve higher macro binary accuracy (the mean of TPR and TNR) than Direct models; however, the gains consistently

---

Published as a conference paper at ICLR 2026
Performance comparison on BEIR across different reranker variants. We report results for Direct-Point, Reason-Point, Direct-List, and Reason-List under both SFT and GRPO training, together with representative pointwise and listwise baselines
Table 2:
Model
Training
ArguA
ClimF
DBP FIQA NFCorp SeiDoe SciFact Touche TreeC. Avg.
Pointwise
BM25
/
39.7
16.5
31.8
23.6
33.8
14.9
67.9
44.2
59.5
36.9
Rank1-7B
SFT
26.4
16.2
37.7
38.4
37.9
16.5
76.1
24.5
79.5
39.2
Rank1-14B
SFT
32.2
15.6
34.2
36.6
35.1
73.8
25.9
38.7
TF-Rank-4B
SFT+GRPO
37.2
19.7
37.9
36.2
38.3
18.3
76.6
37.6
80.5
42.5
TF- IF-Rank-8B
SFT+GRPO
36.5
21.7
37.0
38.0
38.0
17.9
74.6
35.0
80.0
42.1
Reason-Point-4B
SFT
38.1
16.5
39.1
36.0
30.8
16.8
74.9
27.1
81.3
40.1
Direct-Point-4B
SFT
19.6
43.4
42.4
35.9
18.6
77.4
30.6
83.5
45,4
Reason-Point-SB
SET
40.1
13.8
37.5
35.6
32.5
18.2
74.1
21.6
80.3
39.3
Direct-Point-8l
SET
58.6
15.7
42.7
43.3
36.2
16.8
75..3
22.3
82.7
43.7
Listwise
Rank-RI-7B
GRPO
37.0
24.1
43.2
40.1
36.2
18.8
76.1
33.0
82.6
43.5
Rank-R1-14B
GRPO
34.4
24.2
44.0
43.0
37.9
19.7
77.5
29.6
83.9
43.8
REARANK-7B
GRPO
35.6
20.6
43.5
35.8
37.9
19.2
71.9
40.2
80.1
42.8
ReasonRank-7B
SFT+GRPO
33.3
20.0
44.7
38.2
36.6
19.7
72.8
30.4
79.6
41.7
Reason-List-4B
SFT
39.3
13.8
37.7
32.9
30.2
16.0
69.1
24.6
79.5
38.1
Direct-List-4B
SFT
41.7
14.1
36.5
33.5
72.4
22.9
78.1
39.2
Reason-List-8B
SFT
32.8
16.2
42.2
36.6
36.0
18.1
69.7
27.1
79.2
39.8
Direct-List-8B
SFT
28.9
16.4
42.3
37.7
35,9
18.9
73.6
27.0
80.1
40.1
Reason-List-4B
SIT+GRPO
30.8
15.7
43.3
36.1
36.5
18.2
73.1
26.4
78.0
39.8
Direct-List-4B
SFT+GRPO
36.7
19.4
43.8
36.4
18.2
69.8
29.4
40.6
Reason-List-SB
SFT+GRPO
28.6
19.9
43.5
35.5
37.2
18.8
72.5
24.9
77.8
39.9
Direct-List-8B
SFT+GRPO
36.4
19.4
45.1
38.2
36.9
18.7
71.0
31.6
78.7
41.8
Table 3: Class-conditional performance pointwise rerankers. We report TPR (%) and TN (%).
Table 4: Listwise (GRPO) performance on MS MARCO (NDCG@ 10).
Biology
MS MARCO
MS MARCO
Model
Avg.
Model
TPR
TNR
TPR
TNR
DL19
DL.20
DeepSeek-RI
52.4
96.1
40.8
85.1
68.6
Direct-List-4B
73.77
68.97
Reason-Point-
43.7
91.3
38.7
79.4
63.3
Reason-
70.76
68.71
Direct-Point-te
34.0
93.2
30.7
85.7
60.9
Meason-Point--B
50.5
98.1
35.9
85.5
67.5
Direct-List-8B
73.00
71.38
Direct-Point-93
31.1
100.0
25.5
94.2
62.7
Reason-List-8B
72.60
69.81
arise from higher TPR coupled with lower TNR (i.c., higher FPR). In reranking regimes where neg- atives dominate, this combination detrimental: elevated FPR promotes non-relevant documents into the head of the ranked list and, together with the calibration failure above, prevents the binary accuracy gains from translating into improved ranking metrics.
4.2 LISTWISE: REASONING IMPROVES TRAINING FIT BUT HURTS GENERALIZATION
Reasoning boosts training fit but hurts generalization Unlike pointwise rerankers that assign scores to qucry-do pairs and then sort, listwise objectives directly optimize the permutation a candidate set. We therefore ask whether exposing chain-of-thought (CoT) helps under listwise training. We evaluate four 8B variants on training split of 100 instances: Direct-List vs. Reason-List and Direct-List.GRPO vs. Reason- List.GRPO. As shown in Fig. reasoning attains higher mean NDCG@* 10 on the train- ing split but with markedly larger dispersion: Reason-Lit SFT 82.57 + 3.2 vs. Direct- List SFT 80.41 + 2.1 (A= + 2.16), and Reason-List GRPO 87.55 + 2.7 Direct-List GRPO
8

---

Published as a conference paper at ICLR 2026
86.93 + 1.6 (A= + 0.62). These patterns indicate that CoT can better fit the target permu- tations encountered during training, while simultaneously introducing instance- instability. On the in-domain MS MARCO Dev sets (DI.19/00), Direct-List consistently outperforms Reason-List across both 4B and 8B backbones (Table 4. Concretely, Direct-List-4B surpasses its reasoning counterpart by +3.01 (73.77 vs.
92.5
90.0
70.76) on DL19 and +0.26 (68.97 vs. 68.71) on DL20; Direct-List-8B leads by +0.40 (73.00
NDcog10
67.5
36.35.6
85.0
vs. 72.60) on DL19 and +1.57 (71.38 vs. 69.81)
82.57.32
02.5
on DL.00 Thus, the training-split advantage of Reason-List does not translate to stronger in- domain performance. The same trend holds on BRIGHT and BEIR: reasoning based listwise models lag behind their direct counterparts, rein- forcing that CoT's gains on the training split re- flect improved in-sample fitting rather than gen- uine out-of-domain generalization.
77.5
Mean
75.0
Direct_SFT
Reason_SFT
Direct_GPnO
Reason GRFC
Figure 3: Training-split listwise performance of four 8B variants. Reasoning improves mean NDCG@ 10 but increases variance.
GRPO improves performance and reduces overthinking. As shown in Table [ Table 2 and Fig. 3 GRPO yields substantial performance
improvements over SFT training. At the same time, GRP PO markedly shortens the rationales pro- duced by reasoning models. On the training split (Fig. 3}. the average rationale length decreases from 397.7 tokens in Reason-List SFT to 172.3 in Reaxon-List GRPO, reducing inference cost and mitigating extreme "overthinking," " yet achieving higher NDCG scores. This finding suggests that excessively long CoT rationales are not prerequisite for producing effective ranking orders. The compression effect follows directly from the GRPO reward design (Eq, 33, which incentivizes only output format validity and ranking quality rather than verbose reasoning. Although GRPO enhances stability and efficiency, Divect-List models still achieve stronger generalization on MS MARCO DL19/20 and BRIGHT/BEIR (Table 4. implying that shorter CoT mitigates overthinking but does not substitute for direct optimization of permutations. These observations point to a broader research direction: future work should explore how design concise and largeted reasoning strate- gies that balance interpretability, stability, and generalization, while avoiding overfitting and reliance on lengthy CoT outputs.
4.3 IMPLICATIONS FOR FUTURE RESEARCH
Our findings show that explicit reasoning does not inherently lead to performance gains in reranking, and therefore researchers should not assume that longer more elaborate reasoning will universally improve ranking quality. Instead, the results point to two concrete directions for future work.
For logits-based pointwise rerankers, the performance degradation is primarily driven by score miscalibration rather than insufficient reasoming capacity. This suggests that future progress is more likely to come from calibration-aware training objectives that preserve score monotonicity, rather than increasing the depth or length of reasoning traces. Beyond calibration, another promising direction explore token- scoring and ranking mechanisms (Fan et al., (2025). which operate at the token level rather than relying solely on scalar logit. Such methods can produce more stable and fine-gran fine-grained relevance signals, and therefore may complement calibration-oriente approaches in improving pointwise reranking.
For listwise reranking, we observe that the training data may contain unnecessarily long chain-of- thought traces, which increase inference cost without translating into better ranking quality. This suggests two promising directions: (i) developing more concise or adaptive reasoning strategies to reduce overthinking CheninnnkinnCnnnnnnnn et al. 2025). and (ii) designing ranking metrics, rather than relying on generic reasoning supervision.
directly aligned with
Overall, direct models remain more stable and effective across both in in-domain and OOD settings, highlighting calibration (for pointwise rerankers) and objective alignment (for listwise training) as the key bottlenecks for future research, rather than reasoning capacity itself.

---

Published as a conference paper at ICLR 2026
5 RELATED WORKS
LLMs for Ranking The use of Large Language Models (LLMs) for ranking has emerged as a dominant paradigm in information retrieval, with methods that can be broadly classified into two primary approaches: pointwise and listwise (Qin et al. 2024 Lu et al. 2025b). Pointwise rerankers evaluate the relevance of each query-document pair in isolation. This is typically achieved by fram- ing the task as a classification problem, where the model computes a relevance score from the output logits of binary tokens like "true" or "false". This approach, exemplified by influential models such as MonoTS (Nogueira et al. 2020). MonoBERT (Nogueira et al., 2019), and RankLLaMA (Ma) ee al. 2024), benefits from simplicity and computational efficiency, as each document can be scored independently. In contrast, the listwise paradigm is built on the principle of relative comparison, where multiple candidate documents are considered jointly to determine their final order. This cate- gory encompasses several implementation styles. The most direct application is generative listwise ranking, where models like RankGPT (Sun al., 2023b), Vicuna (Pradeep et al. 2023a). and RankZephyr (Pradeep et al. 2023b) leverage their generative capabilities to output fully sorted list of documents. This broader paradigm also includes pairwise methods, which learn relative preferences by predicting the more relevant document from a pair (Qin et al. (0244), and setwise ap- proaches that operate on group of candidates, for example by identifying the single most relevant document within the set (Zhuang et al.. 2024). To manage the long input sequences inherent to this approach, many listwise methods employ strategies like sliding windows or hierarchical ranking to improve efficiency (Sharitymoghaddam al. 2025).
LRMs for Ranking Inspired by the success of Large Reasoning Models (LRMs), a recent line of research has focused on incorporating explicit reasoning into the reranking process to handle complex queries and improve interpretability. This effort has largely followed two main technical strategies: supervised fine-tuning with distillation and reinforcement learning. The first strategy involves Supervised Fine-Tuning (SFT) and distillation, where reasoning capabilities are transferred smaller, more efficient rerankers. For instance, ReasoningRamk (Ji et al. 2025) and Rank (Weller et al.) 2025) distill pairwise or listwise comparative rationales into models such as the LLLMA-3 and Qwen2.5 series. This approach has also been extended to generative listwise settings, where reasoning tokens are directly integrated into the ranking sequence (Yang et al.] (025). A complementary strategy employs Reinforcement models. Works like Rank-R1 (Zhuang et al.) 2025) and REARANK (Zhang al. 2025) use RL directly optimize for ranking metries, while others such as TFRank (Fan et al. (025) and ReasonRank (Liu et al., 2025) adopt hybrid two- stage approach combining SFT with subsequent RL fine-tuning. Beyond these dominant paradigms, other methods have explored more structured formulations, such modeling reranking decision process to improve robustness (Lee et al., 2025 Niu et al.) 2024p. Despite these advances, the core premise that explicit reasoning beneficial is being called into question. Recent findings suggest that for pointwise rerankers, the addition of reasoning can be detrimental, leading to issues like overthinking (Jedidi et al. 2025 Fan al. 2025). This emerging evidence highlights that the utility of reasoning in reranking is far from settled, motivating the systematic investigation in our work.
from powerful teacher models (c.g., GPT-4, DeepSeck-RR
Learning (RL) to further refine these reasoning-awee
6 CONCLUSION
In this work, we systematically examined the role of explicit reasoning in document reranking across pointwise and listwise paradigms with SFT and GRPO training. Our findings are threefold: (i) in pointwise rerankers, reasoning breaks calibration, yielding overconfident scores and degraded ranking despite modest gains in binary accuracy; (ii) reasoning biases models toward the positive class, raising TPR but reducing TNR. which is harmful in negative-dominant candidate pools; (iii) in listwise rerankers, reasoning improves in-domain fit but increases variance and fails to generalize -fo-ain even when GRPO shortens rationales. Overall, direct models remain more stable and effective, pointing to the need for calibration-awan objectives in pointwise rerankers and more concise reasoning strategies in listwise rerankers to improve generalization.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work does not involve human subjects, personal data, sensitive information, and therefore raises no direct ethical concerns. Our research focuses on reranking methods for information re- trieval benchmarks, which do not include personally identifiable information sensitive content. We follow ICLR Code of Ethics and ensure that our methodology and results are presented with transparency and fairness.
REPRODUCIBILITY STATEMENT
We are committed to ensuring the reproducibility of our results. All datasets used in this paper are publicly available, and detailed descriptions of preprocessing steps are provided in the appendix. In addition, we will release our source code, trained models, and experiment configurations upon publication to facilitate replication of our experiments. This includes scripts for data preprocessing, training, and evaluation, ensuring that other researchers can reproduce our findings.
ACKNOWLEDGEMENTS
We thank EIT and IDT High Performance Computing Center for providing computational resources for this project. This work was supported by the 2035 Key Research and Development Program of Ningbo City under Grant N0.20257.
REFERENCES
Xinghao Chen, Anhao Zhao, Heming Xia, Xuan Lu, Hanlin Wang, Yanjun Chen, Wei Zhang, Jian Wang, Wenjie Li, and Xiaoyu Shen. Reasoning beyond language: A comprehensive survey on latent chain-of- thttuutt reasoning. arXiv preprint arXiv:2505. 16782, 2025.
Yongqi Fan, Xiaoyang Chen, Dezhi Ye, Jie Liu, Haijin Liang, Jin Ma, Ben He, Yingfei Sun, and Tong Ruan. Tfrank: Think-free reasoning enables practical pointwise 1lm ranking, 2025. URL
httts://aaxxt www.org//bbss/// 09539
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural http://///aaxi.../a/aaaaa/a////a wwg/ (aa.a///176..05500
networks, 2017. URL http::ttp////c /aap:i//aaivvvv.aa
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.11941 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Cameey, et al. Openai ol system card. arXiv preprint arXiv:2412.16722. 2024.
Nour Jedidi, Yung-Sung Chuang, James Glass, and Jimmy Lin. Don't "overthink" passage rerank- ing: Is reasoning truly necessary?, 2025. / http:///axxxiv.ooaa//////aa/ 6886
Yuelyu Ji, Zhuochun Li, Rui Meng, and Daqing He. Reasoningrank: Teaching student models to rank through reasoning- based knowledge distillation, 2025.
abs/2410.05m 05168
Dohyeon Lee, Yeonseok Jeong, and Seung won Hwang. From token to action: State machine reasoning to mitigate overthinking in information retrieval, 2025. URL /ttpps://axxv..
arg/aba/2505 23059
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmentt gener- ation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33: 9459-9472 2020.
11

---

Published as a conference paper at ICLR 2026
Wenhan Liu, Xinyu Ma, Weiwei Sun, Yutao Zhu, Yuchen Li, Dawei Yin, and Zhicheng Dou. Reasonrank: Empowering passage ranking with strong reasoning ability, 2025. URL https:
//axxiv.crg/ab//250000 07050
Xuan Lu, Haohang Huang, Rui Meng, Yaohui Jin, Wenjun Zeng, and Xiaoyu Shen. Tools are under- documented: Simple document expansion boosts tool retrieval. arXiv preprint arXiv:2510.22677
2025a.
Xuan Lu, Sifan Liu, Bochao Yin, Yongqi Li, Xinghao Chen, Hui Su, Yaohui Jin, Wenjun Zeng, and Xiaoyu Shen. MultiConIR: Towards multi-condition information retrieval. In Chris- tos Christodoulopoulls. Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 13471-13494 Suzhou, China, November 2025b. Association for Computational Linguistics. ISBN 979-8-89176-333-7 doi: 10. 10.18653/v1/2025 findings-emnll.725 URL https::/ 2wwp:aannttooooyyoo///222222
Eindings emmpp.g--emm.p..2220. 726/
Xueguang Ma, Liang Wang. Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning llama for multi-stage text retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2421-2425, 2024.
Tong Niu, Shafiq Joty, Ye Liu, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Judgerank: Leveraging large language models for reasoning-intensive reranking, 2024. URL https:
Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. Multi-stage document ranking with bert, 2019. URL httttt://aaxxiii/// wttpo////./...........
Rodrigo Frassetto Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. Document ranking with a pretrained sequence-to-sequeeee model. In EMNLP (Findings), 2020.
Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. Rankvicuna: Zero-shot listwise docu- ment reranking with open open-source large language models, 2023a. URL https://///a abs /309. 15088
Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. Rankzephyr: Effective and robust zero- shot listwise reranking is a breeze!, 2023b. URL /tttps://arxiv... wwwpor///bb////a// 02724
Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang. and Michael Bendersky. Large language mod- els are effective text rankers with pairwise ranking prompting. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp- 1504-1518, Mexico City, Mexico, June 2024. Association for Computational Lin- guistics. doi: 10. 18653/11/2024 1tt.3//////22..nninnngss.. naacl.... URL
oorg/
1024.findaa naw.ssnnacc......... 97/
Yuxin Ren. Qiya Yang. Yichun Wu, Wei Xu, Yalong Wang. and Zhiqiang Zhang. Non-autoregreesivv generative models for reranking recommendation. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5625-56344 2024.
Sahel Sharifymoghaddaam, Ronak Pradeep, Andre Slaveseu, Ryan Nguyen, Andrew Xu, Zijian Chen, Yilin Zhang, Yidi Chen, Jasper Xian, and Jimmy Lin. Rankllm: A python package for reranking with 1lm.. In Proceedings of the 48th International ACM SIGIR Conference on Re- search and Development in Information Retrieval, pp. 3681-3690, 2025.
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is chatgpt good at search? investigating large language models as re-ranking agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 14918811997, 2023a.
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is chatgpt good at search? investigating large language models as re-ranking agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 14918 14937, 2023b.
12

---

Published as a conference paper at ICLR 2026
William Webber, Alistair Moffat, and Justin Zobel. A similarity measure for indefinite rankings. ACM Transactions on Information Systems (TOIS), 28(4):1-38, 2010.
Orion Weller, Kathryn Ricci, Eugene Yang, Andrew Yates, Dawn Lawrie, and Benjamin Van
Durme. Rankl: Test-time compute for reranking in information retrieval, 2025. URL https:
//arxiv. wttp:or//abbb//./////////////// 18418
Eugene Yang, Andrew Yates, Kathryn Ricci, Orion Weller, Vivek Chari, Benjamin Van Durme, and Dawn Lawrie. Rank-k: Test-time reasoning for listwise reranking, 2025. URL https:
/aaxivv.rrg/bb//255555 14442
Le Zhang, Bo Wang, Xipeng Qiu, Siva Reddy, and Aishwarya Agrawal. Rearank: Reasoning ranking agent via reinforcement learning, 2025. URL 20046
Shengyao Zhuang, Honglei Zhuang, Bevan Koopman, and Guido Zuccon. setwise approach for effective and highly efficient zero-shot ranking with large language models. In Proceedings of the 47th International ACM SIGIR Conference Research and Development in Information Retrieval, pp. 38 47, 2024.
Shengyao Zhuang, Xueguang Ma. Bevan Koopman, Jimmy Lin, and Guido Zuccon. Rank-rl: Enhancing reasoning in I1m- base document rerankers via reinforcement learning, 2025. URL
https://arxiv. org/ abs//////////////// 2503. 06034
13

---

Published as a conference paper at ICLR 2026
A BACKBONES OF RERANKERS
Table summarizes the backbones and training strategies of both existing -eshannee-e rerankers and our proposed models. We observe that the Qwen family has become the mainstream backbone for LLM-based reranking. Our proposed Direct Rerankers and Reason Rerankers are built upon Qwen3-4B and Qwen3-8B, ensuring a fair comparison with recent eeasoing--ehaaneed baselines while highlighting the impact of reasoning versus direct decision-making.
Table 5: Overview of baseline and proposed rerankers.
Model
Training
Backbone
Type
BM25
Pointwise
Rankl-7B
SFT
Qwen2.5-7B
Pointwise
Rankl-14B
SFT
Qwen2.5-14B
Pointwise
TFRank-4B
SFT + GRPO
Qwen3-4B
Pointwise
TFRank-8B
SFT GRPO
Qwen3-8B
Pointwise
REARANK-7B
GRPO
Qwen2.5-7B
Listwise
Rank-RI-7B
GRPO
Qwen2.5-7B
Listwise
Rank-R1-14B
GRPO
Qwen2.5-14B
Listwise
ReasonRank-7B
SFT + GRPO
Qwen2.5-7B
Listwise
Direct- Point-4B
SFT
Qwen3-4B
Pointwise
Direct-Poine Point-P
SFT
Qwen3-8B
Pointwise
Reason-Point-4e
SFT
Qwen3-4B
Pointwise
Reason-Point-88
SFT
Qwen3-8B
Pointwise
Direct-List-4B
SIT GRPO
Qwen3-4B
Listwise
Direct-List-8B
SFT + GRPO
Qwen3-8B
Listwise
Deason-List-44
SFT + GRPO
Qwen3-4B
Listwise
Reason-List-8B
SFT + GRPO
Qwen3-8B BB
Listwise
B EXPERIMENTAL RESULTS FOR GPT4 REASON QUERY
Table reports the performance of Reason rerankers and Direct rerankers under different train- ing stages on BRIGHT with gpt4 reason queries. The results are consistent with the findings in Section 3.4 showing that non-CoT Direct rerankers consistently outperform their reasoning coun- reason
terparts. Table further presents the performance of Direct rerankers on BRIGHT with gpt4_reason queries. Among pointwise models, Ptrect-Point-4P achieves the best score of 33.3, followed by Direct-Point-8B with 32.0. For listwise models, both Direct-List-4B and Direct-List-8D obtain 35.3, outperforming reasoning-enhen does not lead to better reranking performance.
rerankers and providing further evidence that explicit reasoning
C PROMPTS FOR RERANKING
C. PROMPT FOR POINTWISE RERANKING
In the pointwise setting, the reranker judges each query-passage pair independently. The non- reasoning version (Direct-Point) directly outputs binary decision, as shown Figure 5 The reasoning version (Reason-Point) additionally generates rationale enclosed within <think> tags before giving the final decision, as shown in Figure 4
C.2 PROMPT FOR LISTWISE RERANKING
In the listwise setting, the reranker considers the entire candidate set and outputs a ranked order of passages. The non-reasoning version (Direct-List) directly produces the final ranking sequence, as shown in Figure 7 The reasoning version (Reason-List) first generates a reasoning trace in <think> tags and then outputs the final ranking within <answer> tags, shown in Figure 6
14

---

Published as a conference paper at ICLR 2026
Table 6: Performance of Direct-Point, Reason-Point Direct-List, and Reason-List under different training strategies on BRIGHT query).
StackExchange
Coding Leet. Pony
Theorem-based
Model
Training
Bio, Earth.
Econ. Psy. Rob. Stack. Sus.
AoPS
TheoQ.
TheoT. Avg.
Pointwise
Reason-Point-Bl
SFT
46.2
49.8
26.1
37.0
22.0
26.2
28.4
24.7
23.9
6.5
33.5
20.2
28.7
Direct-Point-4B
SFT
50.8
54.7
31.2
41.7
25.8
30.6
32.1
29.1
28.8
7.8
38.5
25.0
33,0
Reason-Point-St
SFT
42.0
44.5
27.0
36.4
23.1
28.7
30.2
28.1
18.9
7.5
24.5
19.5
27.5
Pirect-Point-Se
SFT
46.9
49.2
31.5
41.7
27.0
33.6
35.3
34.6
22.5
9.7
28.8
22.9
32.0
Listw ise
Reason-List-4B
SFT
50.9
45.5
27.5
38.0
28.5
28.8
34.0
20.1
21.5
6.9
24.0
31.0
29.7
Direct-List-4BB
SFT
53.5
48.3
29.8
39.9
31.2
30.5
36.9
23.0
25.0
8.2
26.3
34.2
32.2
Reason-List-8B
SFT
51.4
47.0
29.5
40.1
29.0
29.9
35.5
21.9
26.2
7.7
27.5
32.0
31.5
Direct-List-8U
SFT
54.0
49.6
30.1
42.0
31.6
31.8
37.1
24.1
28.0
8.4
28.1
35.0
33.3
Reason-List-4B
SFT+GRPO
55.1
49.0
28.7
42.7
31.1
33.1
36.6
21.8
23.8
7.8
27.7
37.2
32,9
Dirtct-List-4B
SFT+GRPO
58.4
51.8
31.3
41.6
34.4
33.4
41.0
25.2
27.9
9.8
29.5
39.2
35.3
Reason-List-8B
SFT+GRPO
54.5
49.4
30.8
44.4
29.9
32.1
38.7
22.6
28.0
8.9
31.4
36.0
33,9
Direct-List-8B
SFT+GRPO 56.1
52.2
30.3
44.8
32.5
36.4
38.8
25.0
30.9
8.5
29.6
38.9
35.3
Table 7: Performance of different rerankers BRIGHT datasets (gpt4.query
Theorem-base Bio. Earth. Econ. Psy. Rob. Stack. Sus. Leet. Pony AoPS TheoQ. TheoT.
StackExchange
Coding
Model
Training
Avg.
Pointwise
BM25
53.6
53.6
24.3
38.6
18.8
25.9
17.7
19.3
20.2
18.9
26.5
M-n0TS-3B
SFT
16.0
24.0
17.7
19.5
8.0
10.5
19.5
17.2
29.2
20.3
12.0
16.8
Ran-7.laMA-7B
SFT
17.5
15.5
13.1
13.6
17.9
6.9
16.9
8.4
46.8
2.2
4.5
3.5
13.9
RankLLaMA-13B
SFT
21.6
19.1
16.3
14.0
15.7
7.7
18.5
8.8
31.1
1.7
4.4
4.9
13.7
Rank1-7B
SFT
48.8
36.7
20.8
35.0
22.0
18.7
36.2
12.7
31.2
6.3
23.7
37.8
27.5
Rank1-14B
SFT
49.3
37.7
22.6
35.2
22.5
20.8
33.6
17.7
33.2
8,4
22.5
41.4
28.7
1-32R
SFT
49.7
35.8
22.0
37.5
22.5
21.7
35.0
18.8
32.5
10.8
22.9
43.7
29.4
Pirect-Point-4P
SFT
50.8
54.7
31.2
41.7
25.8
30.6
32.1
29.1
28.8
7.8
38.5
25.0
33.0
Direct-Point-8B
SFT
46.9
49.2
31.5
41.7
27.0
33.6
35.3
34.6
22.5
9.7
28.8
22.9
32.0
Listwise
RankZephyt-7B
SFT
44.1
31.0
17.9
28.4
27.0
21.6
18.9
15.9
12.7
Rank-K
SFT
50.4
46.2
30.6
46.7
32.4
32.2
28.3
26.6
33.3
ReasonRunk-7B
SFT+GRPO
56.4
51.2
28.4
43.4
23.0
29.9
39.1
32.4
Direct-List-4B
SFT+GRPO
58.4
51.8
31.3
41.6
34,4
33.4
41.0
25.2
27.9
9.8
29.5
39.2
35.3
Direct-List-SB
SFT+GRPO
56.1
52.2
30.3
44.8
32.5
36.4
38.8
25.0
30.9
8.5
29.6
38.9
35.3
D EXAMPLES FOR RANKING OUTPUT
D.1 EXAMPLES FOR POINTWISE RANKING
Direct-Poin rerankers perform single forward pass per input and predict a binary answer token (true/fal see); we report the logits at the answer position, so the model does not explicitly print the answer string. In contrast, Reason-Point first generates a natural-language rationale and then computes the logits at the answer position, thereby outputting the complete reasoning text. For clarity, Figure 8 presents an output instance, where we additionally decode the answer-position logits into the corresponding answer token.
D.2 EXAMPLES FOR LISTWISE RANKING
Figures and 10 illustrate concrete example of the listwise reranking setting. Figure shows the input prompt template provided to the model, where the query and candidate passages are listed. and the model is instructed to return complete ranking. Figure 10 presents the corresponding outputs
15

---

Published as a conference paper at ICLR 2026
<lin_start |>systen
Determine the following passage is relevant to the query. Answer only with 'true' or 'false'.
<lim_end|>
<lin_start
Query: ()
Passage: ()
<lim_and|>>
<|in_start paasssstant
Figure 4: Prompt template for pointwise relevance judgement.
<|in_start |>system
Determine if the following passage is relevant
to the query. Answer only with 'true! or false'.
<lim_end|>
<lim_start Jsuser
Query: ()
Passage:
<lim_end|>
/aassistan
<think> </think>
Figure 5: Prompt template for pointwise relevance judgement (non-reasoning version). The <think> tag kept empty to maintain consistency with reasoning prompts.
from different variants of our models. Non-reasoning models (Direct-List) directly produce the ranked sequence enclosed in <answer> tags, while reasoning models (Reason-List) first generate an explicit rationale enclosed in <think> tags before outputting the final ranking. This comparison highlights how reasoning influences the ranking process, providing intermediate explanations the cost of increased verbosity.
E REASONING ASSUMPTION IN RANKING AND VERIFICATION OF REASONING TRACE QUALITY
This section presents quantitative and qualitative analyses verify that the reasoning- based models generate valid and meaningful reasoning traces. The goal to ensure that the observed performance trends are not attributable poorly trained or defective reasoning behavior.
E. REASONING ASSUMPTION IN RANKING
Reranking with explicit reasoning implicitly relies on core assumption: the generated chain-of- thought should provide logicully valid, coherent, and decision--supporivee explanation that reflects the model's underlying relevance judgment.
This assumption has been adopted- cither explicitly or implicitly-in in recent reasoning-enhanced ranking systems, where the reasoning trace is expected to: (I) extract or reference query-document evidence relevant to the final decision; (2) maintain internal logical consistency; and (3) articulate the decision boundary in manner aligned with the teacher or supervision signal.
Under this assumption, if explicit reasoning improves the quality of the relevance decision. would expect the reasoning based reranker to produce accurate, coherent, and faithful reasoning traces, which subsequently translate into better ranking performance.
16

---

Published as a conference paper at ICLR 2026
You are RankLLM, an intelligent assistant that can rank passages based on their relevance to the query. Given query and passage list, you first think about the reasoning process in the mind and then provide the answer (i.e., the reranked passage list).
The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, 1,8., <think> reasoning process here </thinks <answer> answer here </answer>.
I will provide you with [num] passages, each indicated by numerical identifier []. Rank passages based on their relevance to the search qucry;
[1]: ([passage_1)} [2]: ([passage_2)} (more passages)...
Search Query: ((queryl). Rank the (num) passages above based their relevance the search query.
All passages should be included and listed using identifiers, in descending order of relevance. The format of the e.g., [2] [1].
should be [] (],
Figure 6: Prompt template for listwise reranking with explicit reasoning. The model produces reasoning trace within <think> tags and final ranking within <answer> tags.
Therefore, our analyses proceed from the following question: Do reasoning- basee rerankers indeed learn generate high-quality reasoning traces that support the relevance decision?
To answer this, we conduct both quantitative and qualitative verification reasoning trace quality in the following sections.
E.2 QUANTITATIVE VERIFICATION
To whether the reasoning model has successfully learned to produce teacher-level decisions, we evaluate its binary true/false classification accuracy without applying any ranking procedure. The reported results (Table compare the teacher model (DEEPSEEK-RI) RI), the reasoning baased student (REASON-POINT-8D] and the direct baseline (DIRECT-POINT-SD) across both in-domain (MS MARCO) and oft-of-domain (BRIGHT -BRGGHT-Biology) settings.
The REASON-POINT 8B model achieves an average accuracy of 67.5, closely matching the teacher model (68.6). This demonstrates that the student has successfully learned the teacher's decision behavior, indicating that the performance degradation observed in ranking cannot be attributed to failed reasoning acquisition.
E.3 QUALITATIVE VERIFICATION
We further assess the validity of generated reasoning traces through a targeted qualitative study. A total of 50 query- dncer--ocummet pairs were randomly sampled from MS MARCO and BRIGHT. For each pair, reasoning chains produced by DEEPSEEK-RL and evaluated by three annotators using three-dimensional rubric:
were independently
17

---

Published as a conference paper at ICLR 2026
You are RankLLM, an intelligent assistant that can rank passages based on their relevance to the query. Given query and passage list, directly provide the reranked passage list without generating any reasoning process.
I will provice you with [num) passages, each indi cated by numerical identi fier []. Rank the passages based their relevance to the search query:
[11: H|psssgee_l| 1|| [2]: Hpassage _21 (more passages)... ...
Search Qucry: ((query)). Rank the {num) passages above based on their relevance to the search query.
All passages should be included and listed using identifiers, in descending order of relevance. The format of the answer should be II > 1l, e.g., [2] |11.
Figure 7: Prompt template for listwise reranking without explicit reasoning inon-reasoning version). The <think> tag is kept empty to maintain format consistency with reasoning prompts.
Correctness: logical appropriateness of the conclusion,
Coherence: internal consistency of reasoning steps,
Faithfulness: structural alignment with the teacher's reasoning.
Majority- results summarized in Table g
Evaluation Dimension
DeepSeek-R1
Reason-Point-SB
Agreement (%)
Correctness
88%
86%
83%
Coherence
92%
88%
81 %
Faithfulness
83%
Table 8: Qualitative evaluation of reasoning trace quality.
The results show that REASON-POINT-8B produces reasoming traces that are largely correct (86%), coherent (88%), and structurally aligned with the teacher (83%). These findings confirm that the model generates valid and well-formed reasoning, and that the observed ranking behavior is not attributable to defective or low-quality reasoning traces.
F THE USE OF LARGE LANGUAGE MODELS (LLMS)
the preparation of this manuscript, we employed GPT-5 to assist with text refinement. Specif- ically, the model was used to improve the clarity, readability, and overall presentation of the paper by correcting grammatical errors, smoothing sentence structures, and enhancing stylistic consis- tency. Importantly, all core research ideas, experimental designs, and results validated by the authors; the role of the LLM was limited to linguistic polishing. This ensured that the scientific content remained entirely authored by the researchers, while benefiting from improved academic writing quality.
conceived and
18

---

Published as a conference paper at ICLR 2026
<lim_stare |>system
Determine if the following passage is relevant to the query. Answer only with 'true' or false'. <lim_end|>
<Iin_start
Query: Have governments ever defaulted en their domestic public debt treasury bonds due to hyperinflation political crises?
Passage: Argentina experienced a unique default when it suspenced all debt payments before a legal default was declared, with 60% of the debt held by Argentines. Rating agencies still classified as default. Russia defaulted on its demestic debt (GKOs) in 1998, which became one of the most significant financial crises of that decade. <lim end|>
<linsstart passissant
<think>
Okay, let's tackle this query. The user is asking if governments have defaulted their public domestic debt or treasury bends, specifically due to hyperinflation or political issues. The passage mentions Argentina and Russia, Argentina suspended all payments, considered default by agencies, and Russia defaulted CKOs in 1998. Hoth clear cases default domestic debt. While the passage does not explicitly state hyperinflation, these are examples that confirm the query.
</think> true
Figure 8: Example of a Leason-Pointwii output with explicit reasoning in the <think> block before producing the binary decision.
Search Query: where is a elephant habitat.
Passages:
[1] Elephant Habitat. Elephants make home in variety habitats including tropical and subtropical zones...
[2] Elephant Natural Habitat. Elephants are able to survive in variety of different locations...
[3] This is because the elephant is well known being intelligent and capable of experiencing...
[20] Elephant Habitat. Elephants make home in a variety of habitats including tropical and subtropical zones...
Rank the 20 passages above based on their relevance to the query. All the passages should be included and listed using identifiers, in descending order.
The format of the
should [] U, e.g., [2] [1].
Figure 9: Example listwise prompt for the query "where is elephant habitat" with 20 candidate passages.
19

---

Published as a conference paper at ICLR 2026
<think>
</think>
<answer> [8] [19] > [11] > [14] [3] > 14] </answer>
Reason-List-SB.SFT.
<think>
Okay, let's tackle this ranking problem. The user wants know where
elephants Live.
need to focus on passages that explicitly describe habitats ( savannas, forests, deserts, etc.).
Passage [1] lists tropical zones, dry ferests, savannas, semi-deserts,
rainforests.
[2] and [5] mention grasslands, deserts, forests, swamps. [8] mentions Africa and South Asia. [10] and [11] describe adaptability.
Passages [4], [6], [71, 19), [19] repeat similar information.
[13] talks about different species. [15] and [17] focus on conservation.
[18] and [20] similar to [1] and [16].
Less direct: [3], [14], [12].
So top: [1], [21, [5], [8], [10], [11,, [19], [13], [16], [20].
</think>
<answer> [1] [2] > [5] > [12] > [3] [14] </answer>
Diret-Liit-8B.GRPO
<think>
</think>
<answer> [8] [19] |1] ... [6] > [13] [14] </answer>
Reason-Lixt-BB.GRPO.
<think>
Okay, let's tackle this query about where elephants live. The user wants to know their habitats.
Passages that directly mention savannas, ferests, deserts, rainforests are key.
Looking at [1], [2], [5], [8], [10], [11), [16), [19], [20] all list habitats.
[8] and [19] mention Africa and South Asia, which is important.
Others talk about captivity or human impact, less relevant.
So the top ones are those listing habitats explicitly. </think>
<answer> [1] [2]
[5]
[17] [18] [14] </answer>
Figure 10: Examples of model outputs the elephant habitat ranking task. Non-reasoning models (NL) directly output rankings, while reasoning models (RL) generate intermediate rationales within <think> tags before the final answer.
20


Reviews:
[{"review_id": "a0OGuyMUB6", "reviewer": "Reviewer_QzJ4", "summary": "The authors of this paper investigate whether reasoning is helpful for point-wise and list-wise rerankers. Their experiments show that no reasoning methods outperform reasoning methods. They also conduct additional analysis in which they determine that reasoning methods are biased toward the positive class.", "questions": "Here are some questions I have:\n- Line 274 says \"both outperforming reasoning-based baselines such as Rank-R1-14B (25.9) and ReasonRank-7B (25.2),\" but these are not the numbers I see in table 1 or 2 for Rank-R1-14B and  ReasonRank-7B.\n- How much data was used for SFT and how much was used for GRPO?\n- Does the miscalibration explain all the observed degradation in ranking quality for reasoning models? Does breaking the ties by combining with the retriever score, as proposed in Shao et al., 2025, help improve calibration and performance for any of the methods?\n- What model is used as the base retriever? BM25?\n- Does reasoning bias the models toward the positive class always? Figure 2 in [2] seems to suggest that the proportion of samples in the negative class increases?\n\n[1] Rulin Shao, Rui Qiao, Varsha Kishore, Niklas Muennighoff, Xi Victoria Lin, Daniela Rus, Bryan Kian Hsiang Low, Sewon Min, Wen-tau Yih, Pang Wei Koh, et al. ReasonIR: Training Retrievers for Reasoning Tasks. arXiv preprint arXiv:2504.20595, 2025.\n[2] Nour Jedidi, Yung-Sung Chuang, James Glass, and Jimmy Lin. Don’t ”overthink” passage reranking: Is reasoning truly necessary?, 2025. URL https://arxiv.org/abs/2505.16886.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The strengths of the paper are:\n- The authors evaluate a comprehensive suite of models across various settings (pointwise/listwise and sft/GRPO).\n-", "weaknesses": "The weaknesses of the paper ares\"\n- The authors claim that \"reasoning-based rerankers underperform their direct-output counterparts, even though they incur substantially higher inference costs.\" While this seems to be true for current methods and datasets, it's possible their are other training methods or more challenging datasets that could benefit from reasoning.\n- All of the training is done with traces for MS Marco samples. MS Marco has questions that are much simpler than questions in datasets like Bright. Using traces obtained with more challenging questions might lead to improvements with reasoning. \n- In table 2, it look like  ReasonRank-7B is the best performing model. Reasoning helped for this dataset? \n- Some of the writing is misleading. Here are a couple examples:\n  - In the abstract, the authors state \"reasoning improves in-domain fit but increases variance and fails to generalize out-of-domain.\" The \"in-domain fit\" refers to performance on training samples and not on samples in an in-domain test set. This should be clarified. \n  - \"Our analysis reveals that for pointwise rerankers, reasoning improves relevance prediction at the cost of disrupting score calibration and creating a bias toward false positives, ultimately degrading ranking performance.\" Relevance prediction doesn't improves just because the true positive rate increases. If you predicted the positive class for every sample, then TPR will be 1 but this is not improvement in relevance prediction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s9AYqeRmPd", "reviewer": "Reviewer_uyHL", "summary": "The paper presents a comprehensive study on the effect of using reasoning during reranking for information retrieval. This paper compares direct rerankers with their reasoning-augmented counterparts across pointwise and listwise reranking paradigms, and finds that using reasoning often causes performance degradation in almost all scenarios. In addition, the paper analyzes the effect of using reasoning in terms of calibration, bias to positive class, and potential overfitting in domain.", "questions": "1. Typos: in line 208, GRPO should be Group-relative Policy Optimization. \n2. Strong claims: In line 257, the authors said, “Reasoning is unnecessary for reranking”. However, it might be premature to conclude this. These supporting results can at most say the existing reasoning-based reranking approaches underperform, rather than saying reasoning is not useful. \n3. Why is GRPO only used for Listwise rerankers but not pointwise rerankers?\n4. The experiments are only done for Qwen3 models, which natively support thinking/non-thinking modes. Can the same experiments be done for other models without these modes? Does including \"think step by step\" in the system prompt qualify as thinking/reasoning?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper presents a controversial point of view that reasoning can hurt ranking and offers a comprehensive analysis of the effect of incorporating reasoning into the reranking stage. \n2. The insights, such as reasoning affecting calibration and the tendency to predict positive class, are useful for the community. \n3. The experiments are comprehensive across standard and reasoning-intensive retrieval benchmarks, including BEIR and BRIGHT.\n4. The paper is well-written and easy to follow.", "weaknesses": "1. One issue from my point of view is the implicit assumption that the model can reason properly. Despite testing on multiple datasets, the paper neither checks the quality of reasoning via reasoning-related tasks nor performs any qualitative study to understand the reasoning traces. If the reasoning models were poorly trained and the reasoning traces are bad, it will definitely degrade the performance, and the blame should not be attributed to the use of \"reasoning\". \n2. Another issue is that LLMs are typically trained to output better tokens rather than probabilities. However, the scores from the reranker used here are based on **logits** instead of directly outputting scores as tokens. Many new LLM-based rerankers now directly extract the scores (in the form of tokens) from the output, and they can perform well [1]. So it might also be useful to examine whether reasoning works for those types of rerankers.\n3. Though the paper shows the limitations of reasoning, it does not provide enough insight into how to leverage these results for future directions. Should researchers/practitioners still pay more attention to reasoning for reranking? If so, which directions are more promising? \n\n[1] Shao, Rulin, et al. \"ReasonIR: Training Retrievers for Reasoning Tasks.\" *arXiv preprint arXiv:2504.20595* (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4yUQY2LLgW", "reviewer": "Reviewer_wkzz", "summary": "This paper looks at reasoning reranking models compared to non-reasoning reranking models trained with the same data. They show that reasoning reranking models do worse than direct models on various tasks. They show that this is because of worse calibration among reasoning models and a lack of out of domain generalization. \n\nOverall, although I think there may be other analyses that need to be done to fully convince me (and even if this holds, in general I am skeptical that reasoning can't improve reranking) this paper proposes an interesting negative result with the current methods available.  As with most analysis/negative results, a reviewer could always ask for more experiments: I think this is well-done and future work can continue the discussion.", "questions": "I think the paper is well written but in general am skeptical. However, that is not a reason to reject in my opinion, as that is good for science to have disagreement.\n\nIf it's helpful for the authors, my pushback would be that I would suspect some differences from these areas:\n\nA: The Rank1 results presented here are quite different than those in their paper (even for the GPT-4o experiments). That leads me to believe there is some difference in implementation which could affect the other results. Are the prompts used for each dataset the same? I think previous work gave custom prompts for unique datasets like BRIGHT.\n\nB: Given the wide range in performance (even in-domain as shown by Figure 3), I would guess that the exact prompt and/or model used (and perhaps even random seed in training) would cause a large difference.  Whereas for direct prompted model, I would suspect they don't fluctuate as much with other prompts (and may be too adapted to the one used in training). I would be curious to see the intermediate checkpoints results or another version trained with a different seed to see how varied performance is? Perhaps the higher variance leads to worse average-case performance but if carefully chosen, can do better.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The analysis is well chosen and timely and likely to be relevant to many in the retrieval-adjacent space\n- Many models at various sizes are shown as well as well-thought out experiments and ablations\n- The paper explores both listwise and pointwise models, as well as RL and SFT", "weaknesses": "1. There are some areas where the experimental details could be more clear, e.g. were there different prompts for the various datasets? Did the reasoning models overfit on the data or have large differences during intermediate steps? \n2. The paper claims the models do worse out of domain but I also think the results show they do worse in-domain no? So I'm not sure it's a \"they generalize worse\" problem but more of a \"they are worse all over\". This is relatively minor but I think the abstract/intro was leading me to believe it was a generalization problem. \n3. Minor: reading through it left me with some questions (see that section) about why this could be: however, I don't think they need to be done necessarily to improve the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8OYXJYqkxs", "reviewer": "Reviewer_sv2D", "summary": "This study investigates the practical impact of explicit reasoning in document reranking tasks. Using a unified experimental design, the authors compare models with and without explicit reasoning across two mainstream reranking paradigms: pointwise and listwise. The experiments encompass supervised fine-tuning (SFT) and reinforcement learning-based GRPO training methods, employing Qwen3 series models evaluated on multiple benchmarks including BRIGHT and BEIR. The results demonstrate that rerankers incorporating explicit reasoning generally underperform compared to corresponding models that directly output results, and the paper provides corresponding analyses.", "questions": "see Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The experiments are comprehensive, comparing both pointwise and listwise modeling approaches as well as investigating different training methods including SFT and GRPO.\n\nEffectively leveraging the reasoning capabilities of large language models to improve complex downstream tasks such as ranking is a cutting-edge and important research direction.\n\nThe paper is well-organized with a clear logical structure.", "weaknesses": "Questionable generalizability of conclusions:\n\nThe primary findings are based on Qwen3 series models (4B and 8B). However, many comparable baselines (e.g., Rank1, Rank-R1) were trained on the Qwen2.5 series. The authors do not verify whether the observed “harmful impact of reasoning” persists across other widely used base models such as Qwen2.5 or LLaMA, limiting the generality of the conclusions.\n\nAbsence of zero-shot baselines: The study lacks direct comparisons with zero-shot Qwen3 models. To substantiate claims made in Section 3.4 comparing Rank1 and Rank-R1 on BEIR, it is necessary to control for base model capability differences, for example by including Qwen3-based Rank-R1 results. Moreover, a zero-shot Qwen3 baseline on BRIGHT would clearly illustrate the relative improvements brought by different training methods (direct vs. reasoning), thereby strengthening the conclusions.\n\nResult volatility: Performance differences between 4B and 8B models, as well as between models with and without explicit CoT reasoning in the listwise experiments, are minimal. Further analysis is needed to exclude the possibility that these small gaps stem from normal fluctuations in model size or training randomness.\n\nLack of statistical significance testing: Given the small performance gaps, it is difficult to determine whether observed differences are genuine or due to experimental variability. The absence of significance testing reduces confidence in the results.\n\nInsufficient support for the use of the Expected Calibration Error (ECE) metric: The paper employs ECE to argue that reasoning degrades model calibration, which is cited as a factor contributing to ranking performance decline. However, no theoretical or empirical evidence is provided to support the core assumption that ECE negatively correlates with ranking quality metrics such as NDCG. Relevant literature citations or supplementary experiments are necessary. Additionally, reporting zero-shot Qwen3 ECE values as a baseline would provide a more complete understanding of calibration changes.\n\nLimitations related to training methodology: The pointwise models are fine-tuned using the parameter-efficient LoRA technique, which may affect model behavior differently than full fine-tuning. To enhance the generality of the findings, experiments with full fine-tuning are recommended.\n\nOther remarks:\n\nThe assertion in Section 3.4 that “explicit reasoning is not a prerequisite for achieving state-of-the-art performance” may be overly absolute. Current results suggest reasoning may underperform under specific training paradigms, but this likely reflects the lack of effective methods to exploit reasoning abilities rather than definitively ruling out the utility of explicit reasoning in ranking tasks. Ranking performance is influenced by multiple interacting factors, so this conclusion risks overgeneralization.\n\nNumerical inconsistencies between Figure 2 and Section 4.1 warrant further verification and alignment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)