Rank1: Test-Time Compute for Reranking in Information Retrieval
Authors: Orion Weller
Kathryn Ricci
Eugene Yang
Andrew Yates
Dawn Lawrie
Benjamin Van Durme
Venue: COLM 2025 (2025)
Keywords: retrieval
reranking
test-time compute
TL;DR: We train the first reranker using test-time compute in information retrieval

Abstract: We introduce Rank1, the first reranking model trained to take advantage of test-time compute. Rank1 demonstrates the applicability within retrieval of using a reasoning language model (i.e. OpenAI's o1, Deepseek's R1, etc.) for distillation in order to rapidly improve the performance of a smaller model. We gather and open-source a dataset of more than 600,000 examples of R1 reasoning traces from queries and passages in MS MARCO. Models trained on this dataset show: (1) state-of-the-art performance on advanced reasoning and instruction following datasets; (2) work remarkably well out of distribution due to the ability to respond to user-input prompts; and (3) have explainable reasoning chains that can be given to users or RAG-based systems. Further, we demonstrate that quantized versions of these models retain strong performance while using less compute/memory. Overall, Rank1 shows that test-time compute allows for a fundamentally new type of explainable and performant reranker model for search.

Published as a conference paper at COLM 2025
Rank1: Test-Time Compute for Reranking in Information Retrieval
Orion Weller
Kathryn Ricci
Eugene Yang
Andrew Yates
Dawn Lawrie
Benjamin Van Durme
Johns Hopkins University
owe wweller@cs.jhe jhu. edu
Abstract
We introduce RANK1, the first reranking model trained to take advantage of test-time compute. RANK1 demonstrates the applicability within retrieval of using a reasoning language model (i.e. OpenAl's o1, Deepseek's 's R1, etc.) for distillation in order to rapidly improve the performance of smaller model. We gather and open-source dataset of more than 600,000 examples R1 reasoning traces from queries and passages in MS MARCO. Models trained on this dataset show: (1) state-of-the-art performance on advanced reasoning and instruction following datasets; (2) work remarkably well out of distribution due the ability to respond to user-input prompts; and (3) have explainable reasoning chains that can be given to users or retrieval-augmented generation (RAG) systems. Further, we demonstrate that quantized versions of these models retain strong performance while using less compute/ /memory. Overall, RANk1 shows that test-time com- pute allows for a fundamentally new type of explainable and performant reranker model for search.
1 Introduction
Reasoning language models (LMs) like OpenAl's o1, Deepseek's R1, and Gemini's Flash- Thinking have shown improved reasoning abilities through the use of test-time compute, i.e. generating a reasoming chain of tokens that allow the model to "think" before giving the final answer. Another benefit to these style of models is that the reasoning chain easily be distilled into smaller models. As shown by Deepseek (Guo et al., 2025) smaller models learn incredibly well from simple supervised fine-tuning the larger model's reasoning chains.
The benefits that reasoning models bring to general text generation would also be valuable in an information retrieval (IR) context: allowing models additional time to reason why passage could be relevant, while also allowing an auditable reasoning process to give to the user RAG system. For this approach to be maximally effective, the model must be able to reason over both query and passage; if applied solely to the query, the reasoning model would not know the passage context and would have to try to infer it. This would be form of query-expansion (Nogueira et al., 2019) and limits the model's s ability to be precise. Thus, our work focuses on bringing test-time compute IR in a reranking setting, where the model needs to compute the relevance of an initial top-k candidates.
To accomplish this goal, we sample 635,000 examples of R1's s thought process on the MS MARCO dataset (Nguyen et al., 2016). We then fine-tune a suite of LMs on these reasoning chains and find that they show remarkable reasoning capabilities. Surprisingly, they also ex- hibit ability to be prompted despite training from the base LMs only (without instruction fine-tuning) and while having no instruction-base IR training data (only MS MARCO).
Models, code, and data are available at https: //ss::///tttt cww com/orionwe wrw.///aano//
1

---

Published as a conference paper at COLM 2025
what county is colton in
do snow leopards change color
Colton; California. Colton a city San Bernardino County, California, United States...
Snow leopards have long, thick fur, and their base color varies from smoky gray to yellowish tan, with whitish underparts
<think> The user is asking "what county is Colton in." So the first thing notice is that the passage directly mentions San Bernardino County as county where Colton is located.
is asking if snow leopards change color. The passage provided describes their color as varying from smoky to yellowish tan, with some patterns. Hmm. the key here is whether "varies" implies that they change color time, like seasonally or due to some factors.
<think>
The query specifically about the county, answer right there in the first sentence of the passage.
wait, "varies" might just individuals have different base colors, not that single leopard changes its color. The passage doesn't mention anything about changing color, it's just describing their appearance....
that different
But wait. maybe I should check if there's any possibility that the passage isn't relevant. The rest the passage talks but key information about San Berarddno County is present
Therefore, the passage isn't directly relevant to the question of whether they change color. So the answer should be false.
the answer should true because the passage answers the query directly.
</think> true
</think> false
Figure 1: Example reasoning traces from R1, used to train RANK1. Note the self-inquisitory reasoning (in blue) where the model questions if it has the correct answer.
This includes state-of-the-art performance on the BRIGHT benchmark for reasoning (Su et al., 2024), the NevIR benchmark on complex negation understanding, and the mFol- lowlR dataset on multilingual instruction-following in IR - despite having no non-English reasoning training data (Weller et al., 2025a).
We also conduct detailed analysis of performance on traditional IR benchmarks, such as TREC DL19 (Craswell et al., 2020) and BEIR (Thakur et al., 2021). We find that these datasets are likely model-saturated, documents (364% more than RankLLLMA--14B) We argue these benchmarks are no longer helpful for distinguishing between the best performing rerankers and that focus should be put on benchmarks that examine advanced reasoning, instruction-folowinng and have more modern (e-g- post-ChatGPT) annotations.
RANK1 surfaces extremely large number of unjudged
Overall, RANK1 shows the benefits that test-time compute can bring to the field of IR: explainable reasoning chains that can be audited by users used by agentic RAG systems, significantly improved reasoning performance, and adaptability from user-given prompts.
Model Training
2.1 Data Preparation
In order to distil from R1, we first need to gather data to use for prompting it. We use the MS MARCO collection (Nguyen et al., 2016) due to its diversity in topics and common use in previous work. We use kluster.ii as the APT service to access R1 using their batch mode with temperature of 0.3 and a maximum 1000 output tokens.
We generate data from equal number (25% of the data) from each of the (1) positive examples in MS MARCO, (2) sampled negatives from Tevatron (gathered from BM25 and CoCondenser), (3) rank 1-5 hard negatives from mT5-13B, and (4) rank 5-10 hard negatives from m15-13B. However, we found that Rl classified roughly 80% of the m155 hard negatives as positives. Thus we did another round of generation using only hard negatives from rank
2In initial testing we found that all reasoning chains were shorter than 1000 tokens. As each MS MARCO passage is typically around 100-200 token, doesn't 't need more than 1000 tokens to reason over it.
2

---

Published as a conference paper at COLM 2025
6-
5
Percent
4
3-
2
100
300
400
500
Number of Words
Figure 2: Distribution of word lengths of the reasoning chains generated from R1. It has slightly rightward skew but is generally normal shaped. Note that there is no noticeable difference in the distribution between passages that are predicted relevant vs non-relevant.
5-10 and easy negatives from Tevatron. As the mined hard negatives from m115 do not have an official label, likely that many of them were false negatives and that R1 was correct.
After all generation was done, our dataset has 635,264 examples of R1 generations, where R1 labeled 62.9% relevant and 37.1% non-relevant. We show a plot of these generation lengths in Figure 2 where we see a fairly normal distribution. Although we thought there may be length differences between these four subsets of data, we found that they all had the same rough distribution.
2.2 Data Mix and Quality Filtering
Since we had surplus of documents judged relevant, we tried various methods to arrive at our final data mix. We initially tried using all the data, after balancing for the labels. We found that this performed significantly worse than filtering based on the labels we were most sure about (i.e. the positives from MS MARCO and the negatives from Tevatron). However, even on those subsets, 15% of Rl's final prediction disagreed with the implied labels thus we filtered out these instances.
Beyond being labeled as positives, we found that a large number of the m15 mined hard negative samples were noisy.* To alleviate this, used model trained on the first mix to self-filter the data. 5 This filtered another 10% of the data, mostly false negatives. Since we still had surplus of "relevant" labeled instances, we took all positives from the official MS MARCO positives and all negatives from the self-filtered set. This left with training set of 386,336 high quality training samples: 136k from the original MS MARCO positives, 154k from the Tevatron negatives, and 96k from the mT5 negatives. We note that having more negatives than positives is standard: RankLLaMA (Ma et al., 2024) trained on a 15:1 ratio of negatives to positives for 7 million examples.
2.3 Training
We train three main models from the Qwen 2.5 family of models (Yang et al., 2024). These models have shown improved performance on recent LM benchmarks and have wide range of models, allowing to show the effects of scale. We use the 7B, 14B, and 32B parameter models.? We show that alternative base models also effective in Section 4. During initial experiments we found that the base models outperformed their instruction- tuned variants, so all models are trained from the base version (e.g. no instruction-trainingg)
4See Appendix Section where we found that 2/3rds the m negatives were actually positives. We used the RANK1 Mistral 24B version as it was the largest that fit on 1 GPU, filtering all instances where the model's prediction didn't agree with R1's prediction.
From the MS MARCO small triples file.
We show ablations on smaller Qwen models Section 3.8.
8This could be because instruct-versions are optimized for chat and math data, whereas have large amount of grranking-speciff data that doesn't benefit from chat/ hottmmahh-bsed instructions.
3

---

Published as a conference paper at COLM 2025
StackExchange
Coding
Theorem bassed
Avg.
Bio.
Earth.
Econ.
Psy.
Rob.
Stack.
Sus.
Leet.
Pony
Aol5
TheaQ.
TheoT.
BM25
19.2
27.1
14.9
12.5
13.5
16.5
15.2
24.4
7.9
6.0
13.0
6.9
14.8
BM25 GPT-4o CoT
53.6
53.6
24.3
38.6
18.8
22.7
25.9
19.3
17.7
3.9
18.9
20.2
26.5
MonoI5-3B
16.0
24.0
17.7
19.5
8.0
10.5
19.5
17.2
29.2
7.1
20.3
12.0
16.8
RankLLaMA-7B
17.5
15.5
13.1
13.6
17.9
6.9
16.9
8,4
46.8
2.2
4.5
3.5
13.9
KankLlaMA-13R
21.6
19.1
16.3
14.0
15.7
7.7
18.5
8.8
31.1
1.7
4.4
4.9
13.7
Rankl-7B
48.8
36.7
20.8
35.0
22.0
18.7
36.2
12.7
31.2
6.3
23.7
37.8
27.5
Rank1-14B
49.3
37.7
22.6
35.2
22,5
20,8
33.6
17.7
33.2
8.4
22.5
41.4
28.7
Rank1-32B
49.7
35,8
22.0
37,5
22.5
21.7
35.0
18.8
32.5
10.8
22.9
43.7
29,4
Table 1: The performance of retrieval models on BRIGHT. BM25 scores are taken from the official BRIGHT paper. All models rerank from the BM25 on GPT-4o CoT top 100 documents, but are not given the GPT-4o COT. We find large gap between similar sized rerankers and RANK1 models (sometimes 2x). Bold indicates the best score in the reranker section.
We train the models with LoRA using LLaMA A-MMttoryy (Zheng et al., 2024) for up two epochs or for up to three days. We found that there was increased learning for roughly the 1.5 epochs but then performance saturated. For more details and hyperparameters configurations see Appendix A.
Experiments
We show the capabilities of the RANK1 through evaluation on advanced reasoning, instructionfollooinng and semantic-understanding datasets. We also demonstrate per- formance on traditional benchmarks. We use mteb (Muennighoff et al., 2022; Enevoldsen et al., 2025) run all experiments except for DL19, which uses rankllm (Pradeep et al., 2023a). Inference is powered by vllm (Kwon et al., 2023) which makes it significantly faster than vanilla transformers (Wolf et al., 2020).
3.1 Baselines
We use BM25 (Robertson et al., 1994; Lu, 2024) and mE5-base (Wang et al., 2024) for first stage models. For reranking models we focus on other pointwise reranking models (e.g. models that output a score per document): MonoT5-3B (Nogueira et al., 2019), mT5-13B fine-tuned on MMARCO for multilingual tasks (Jeronymo et al., 2023), and RankLLaMA 7 and 13B (Ma et al., 2024). For instruction following tasks we also include FollowIR-7B (Weller et al., 2024a) which was trained solely for instruction-following
We show results for listwise models when those scores are available (Pradeep al., 2023b; Sun et al., 2023), but note that they are not comparable listwise models take an order of magnitude more time inference due to their sequential dependencies and have the advantage of seeing all documents in their context when reranking. We show them generally as a strong upper bound, as when state-of-the-art (SOTA) LMs (i.e. GPT-4o) are used.
3.2 Reasoning Capabilities
We show results on the reasoning intensive BRIGHT benchmark (Su et al., 2024) in Table 1. All reranker models judge the top 100 documents found using BM25 on the query plus
GPT-4o's Chain of Thought (CoT) reasoning, which performed significantly better than
BM25 without the query expansion (thus including more relevant documents in the top 100 to test rerankers with). However, at inference time, the rerankers are not given the CoT. We see large gap between RANK] and other models: in many cases near double the nDCG@10 score (e.g. 18.7 vs 7.7 for RANK1-7B vs KankLLaMA-13B on Stackoverflow).
These results are especially notable when you consider that RANK] models were trained on an order of magnitude less data than models like RankLLaMA (7 million vs 600k) while using the same training dataset (MS MARCO). We also find that performance scales with model size, with the 32B model outperforming the smaller models. Thus, RANK1 is SOTA when reasoning is needed.

---

Published as a conference paper at COLM 2025
Persian
Chinese
Russian
Average
Model
nDCG@20
p-MRR
nDCG@20
p-MRR
nDCG@20
p-MRR
nDCG@20 p-MRR
mE5-base
0.289
-3.9
0.316
+3.4
0.307
-2.1
0.304
-0,9
MonoT5-3B
0.118
-2.4
0.231
+5.0
0.240
+6.8
0.196
+3.1
FollowIR-7B
0.225
+1.8
0.375
+8.7
0.376
+0.4
0.325
+3.7
mT5-13B
0.453
-0.7
0.474
+2.3
0.505
-0.6
0.477
+0.4
RankLLaMMA
0.229
0.272
+1.1
0.248
+0.1
0.250
+0.7
RankLLaMA 13B
0.256
+0.8
0.287
+1.8
0.320
-0.6
0.288
+0.6
Rank1-7B
0.564
+7.0
0.582
+3.1
0.511
-0.0
0.552
+3.4
Rank1-14B
0.572
+11.9
0.611
+4.6
0.516
+5.4
0.567
+7.3
Rank1-32B
0.555
+3.9
0.598
+4.9
0.521
+6.6
0.558
+5.2
Table 3: mFollowiR Cross-Lingual scores across three language subsets. Bold indicates best
score.
Persian
Chinese
Russian
Average
Model
nDCG@20
p-MRR
nDCG@20
p-MRR
nDCG@20
p-MRR
nDCG@20
p-MRR
mE5-base
0.493
4.2
0.441
+0,3
0.417
-3.5
0.450
-2.5
MonoT5-3B
0.130
-3.8
0.233
+1.4
0.254
+2.2
0.206
-0.1
FollowIR-7B
0.163
-0.1
0.404
+6.6
0.379
+7.7
0.315
+4.8
m15-13B
0.498
+0.1
0,548
+4.2
0.506
+1.9
0,517
+2.0
RankLLaMA-7B
0.248
-0.9
0,397
+1.3
0.396
-0.1
0.347
+0.1
RankLLaMA-113
0.333
-2.0
0.484
+0.5
Rank1-7B
0.564
+2.4
0.665
+5.9
0.528
+4.7
0.586
Rank1-14B
0.605
+11.1
0.647
+0.8
0.530
-1.0
0.594
+3,6
Rank1-32B
0.619
+12.1
0.678
+10.2
0.535
+8.1
0.610
+10.1
Table 4: Results for mFollowIR multilingual across three language subsets (Persian, Chinese, Russian). All models rerank the top 100 docs found from the mE5-base model. We see wide gap between RANK1 and other models, despite it not having any multilingual reranking training data.
3.3 Semantic Understanding
We evaluate on NevIR which requires reasoning over negation in Table 2. Models rerank all documents, and we report scores for listwise models from van Elsen al. (2025).
Model
Score (%)
Listwise
RankGPT 4o-mini
64.1
RankGPT
70.1
RankGPT 03-mini
77.3
We again find that RANK1 performs extremely well, even matching GPT-4o and coming 15+ points higher than the next closest model. We see that even 03-mini in listwise setup only performs points higher.
RankL.ama 7B
31.6
RankLlama 13B
43.2
Pointwise
Mono'I5 base
34.9
MonoT5 3B
50.6
3.4 Instruction- Fortriion-FFlowiing
Rank1-7B
65.1
Rank1-14B
67.5
We show results on the mFollowiR dataset (Weller et al., 2025a) as it illustrates both instruction- following and multilingual capabilities. Table 3 shows results on the cross-lingual setup (En-XX) and Table the XX-XX task. We have all models rerank the top 100 scores of strong but small base model mF5-base. We find that RANK1 has much higher nDCG@20 scores (from 0.586 to 0.611 on the multilingual average) compared the next best model (mT5-13B trained Multilingual MS MARCO) with 0.517. Furthermore, when considering just instruction following metrics there is a wide gap, especially in the multilingual setting (+10.1 vs +4.8 p-MRR the custom instruction-traiee cross-lingual version, but there still remains notable gap between the best RANK1 and the closest other model (+7.3 +3.7 p-MRR).
Rank1-32B
70.1
Table 2: Pairwise acc. on NevIR
FollowIR-7B) Other models closer the
We again find this notable considering that mT5-13B a similar size and was trained on multilingual data. RANK1 significantly outperforms it solely with English reasoning data.

---

Published as a conference paper at COLM 2025
3.5 "Traditional" Benchmarks
DL19 We also evaluate DL19 scores in Table 5. All models rerank the top 100 passages found using RepLLaMA. We found that our models performed significantly worse the initial rel- evance judgments (i.e. qrels) and our analysis quickly showed this was due to the number of unjudged documents that our models ranks higher. As seen in Table the top 10 lists of existing pointwise rerankers is almost entirely judged passages (i.e. 96.1%), whereas RANK1 finds 10-15% less (down to 81.9% passages that are judged by humans).
Original
Fixed
Model
Judged@10
nDCG@10
nDCG@10
RankLlama 7B
96.1
76.2
76,9
RankLlama 13B
96.1
77.2
76.9
Mono 3B
91.2
72.0
74,8
Rank1-7B
83.5
66.1
78.6
Rank1-14B
82.3
64.8
77.6
Rank1-32B
81.9
66.0
80.1
Table 5: Results on TREC DL19
To remedy this, we manually annotated all top 10 documents that each model (baseline or RANKI) got wrong or were unjudged. We show examples of incorrectly or not labeled instances in Appendix D. We found that unjudged documents were mostly relevant, e.g. 70.6% of the unjudged documents for RankLLaMA 14B were relevant and 86.8% were relevant for Rank1-14b, etc." After this annotation fix RANK1 models are no longer penalized for finding models (e.g. 78.6 vs 76.8 nDCG for RANK1-7b vs RankLLaMA 7B).
documents and that performance is generally better than all other
Thus, it seems that the original DL19 benchmark is no longer suitable for discriminating between the top performing approaches. Although this was examined for some TREC. collections in 2022 by Voorhees et al. (2022), the largest model used in their experiments BERT. Thus, we would encourage the community to re-evaluate old TREC collections.
BEIR We show results on BEIR as a comparison Table 6, using the datasets with less than 2k queries. All models rerank the top 100 documents found using BM25s (Lù, 2024). We find comparable but worse performance with RANK1, Although out of scope for this work, found large number of similar issues in BEIR datasets we do in DL19. We discuss datasets individually in Appendix G. It appears that traditional reranking benchmarks like DL19 and BEIR although extremely successful at driving the field forward and still useful for weaker models are no longer as useful for distinguishing between the best rerankers.
3.6 Test-Time Scaling
Given the success of reasoning language models in scaling test time compute (i.e. getting better results with more tokens used), we also attempted several ways of using extra test-time compute to improve performance. However, using the simple budget-forcing method from sl (Muennighoff et al., 2025) did not improve performance and actually hurt performance in our limited initial experiments. We hypothesize this may be due to the lack of difficulty in the reranking task after all, reasoning over BRIGHT requires significantly less reasoning than the typical AIME problems used for evaluation. Alternatively, perhaps future work with new techniques are needed induce the desired results.
'e release our new judged qrels in the Github above to help facilitate future work. 10 note that ideally one would re-annotate the whole dataset with new pools from many models. This would allow for future models to be able to use the same evaluation. However, given the scale of annotations, this would be out of scope for this work. Thus, we include this experiment mainly to highlight the limitations of DL.19 and acknowledge that better evaluations will be needed in future work.

---

Published as a conference paper at COLM 2025
Model
ArguA
ClimF
DBP
FiQA
NFCorp SeiDoc
SciFact Touche TrecC
Avg.
BM255
47.2
18.6
32.0
25.4
34.3
16.5
69.1
34.7
68.8
38.5
MonoI5-3B
42.5
25.4
44.5
46.5
37.8
19.3
76.1
30.7
79.6
44.7
RankLLaMA-77B
54.4
23.2
43.7
42.1
27.0
16.6
71.1
41.4
80.2
44.4
RankLLaMA-13B
49.3
24.5
44.1
28.1
18.1
72.7
39.2
80.8
44.6
Rank1-7b
42.8
15.0
38.9
39.5
36.2
17.2
77.2
22.8
81.9
40.9
Rank1-14b
45.3
16.2
37.4
37.9
35.8
17.9
77.0
27.1
78.2
41.0
Rank1-32b
57.6
15.8
40.7
41.8
36.9
19.6
76.8
19.9
81.9
41.7
Table 6: nDCG@10 on the BEIR benchmark. Models rerank the top 100 documents from BM25S (Lù, 2024). Although not in scope for this work, we find that the way many of the datasets in BETR were constructed limits the ability to judge between the highest performing systems, similar to DL19 (see Appendix G for commentary individual datasets).
3.7 How much does the reasoning chain help?
To ablate how important the reasoning chain is for RANK1 we train an ablation on the 7B model where the reasoning chain omitted using the same data and base model. This allows us to isolate the benefits that the reasoning chain itself provides.
We find that our non-reasoning model scores an average of 17.5 on BRIGHT compared to 27.5 with Rank1-7b. This is an improvement over RankLLaMA-7B (13.9) but not the 2x improvement seen by the models with reasoning chains. Thus, we can that the reasoning chains are providing the expressive power that is needed for these complex reranking tasks.
Avg nDCG@] 10
0.5B
1.5B
3B
7B
14B
32B
Model Parameters
Figure 3: Average nDCG@10 across BRIGHT datasets for various sizes of RANK1. We see that with our training dataset there diminishing returns after 7B+ parameters and that smaller models still perform very well: Rank1-0.5b scores comparably to RankLLaMA 13B (13.9 average BRIGHT) despite having 25x less parameters and only slightly more than BERT-sized models.
3.8 Does this work at smaller sizes?
We have previously used 7B+ models, as it is the smallest size typically used for reasoning models applied to other tasks (e.g. AIME, etc.). However, for reranking, is it possible to have strong performance with smaller models?
We follow the same training recipe and train the rest of the Qwen 2.5 family of models (0.5B, 1.5B, 3B) and show the results with the main models in Figure We find that larger models do better (albeit with dimishing returns), but even smaller models do very well comparatively: the 13B RanALLaMA model (13.9 average) only scores slightly higher than the 0.5B version and less than the 1.5B version.
Thus, although test-time compute is typically used with larger models, we see that at least for reranking even small models show large benefits.
7

---

Published as a conference paper at COLM 2025
Model
Size (G)
SciFact NeviR Biology Eng-Zho
Rank1-7B
15.2
77.2
65.1
48.8
58.2
quantized
5.6
75.4
62.0
42.8
56.3
Rankl-Llama3-SB
16.1
73.2
61.9
45.8
58.1
quantized
5.8
72.9
57.1
41.5
54.8
Rank1-14B
29.6
77.0
67.5
49.3
61.1
quantized
10.0
75.6
66.4
44.6
58.8
RankI-Misttal-24BB
47.2
75.8
67.4
51.8
59.9
quantized
14.3
72.7
64.6
47.1
59.4
Rank1-32B
65.5
76.8
70.1
49.7
59.8
quantized
19.3
77.2
69.9
52.3
59.7
Table 7: Quantization results on subsets various tasks (SciFact from BEIR, Biology subset from BRIGHT, Eng-Zho mFollowwR-Crrssl..a tion into int4. We see that models retain most of their performance while being significantly smaller.
We use AutoAWQ for the quantiza-
4 Model Releases
Alternate Base Models Our main base models the Qwen 2.5 series due to their strong performance and varying parameter sizes. However, to show that this approach works on other base models, we train version with Llama 3.1 8B (AI@Meta, 2024) and Mistral Small 2501 24B. 11 Similar to the Qwen models, we start from the base versions, not the instruct versions. We show the results on selected benchmarks in Table 7 where we can see that our approach holds for other base models with performance generally increasing as they scale (i.e. 24B is better than 8B). We also see that Llama 3.1 8B slightly underperforms Qwen 7B.
Quantization IR models typically require a heavier inference workload compared many other LLM applications due to the number of passages to rerank. However, many common rerankers (MonoT5, RankLLaMA) do not take advantage of modern inference capabilities such as quantization.
We quantize each our models using AutoAWQ (Lin al., 2023) and compare perfor- mance before and after. We see that although performance drops slightly, the model size is significantly smaller, enabling all models (including the 32B version) to be run on one 24GB GPU. Despite the performance loss, quantized RANK1 models still significantly outperform the baselines on reasoning and instruction -oo.uutiioonoooooiiigg while being 1/3rd of the size.
5
Related Work
5.1 Advanced Reasoning and Instruction Following in IR
In the last couple years, retrieval systems have started to move beyond simple phrase-based semantic matching, to more complex information retrieval tasks. This has included a focus on new benchmarks: such reasoning retrieval (Su al., 2024; Zhao al., 2024; Xiao et al., 2024), instruction-following capabilities (Weller et al., 2024a; Oh et al., 2024), and retrieval for RAG systems (Lawrie al., 2024; Mayfield al., 2024).
On the modeling side, we have seen surge of interest in models that can understand the meaning behind the user's 's query rather than doing phrase-based matching: this includes models like Instructor (Su et al., 2022), TART (Asai et al., 2022), GritLM (Muennighoff et al., 2024), FollowIR (Weller et al., 2024a), Gecko (Lee al., 2024), and Promptriever (Weller et al.,
Mistral Small
can
be
found at
https://uuginngacce.cc///
Mistral- Smal1-24B Base-2501
8

---

Published as a conference paper at COLM 2025
2024b). These models typically use instruction-based data in their training data, that they learn to adapt to new user instructions. Other works have built multi-agent systems with LLMs that do reasoning (Niu et al., 2024; et al., 2024). However, for RANK1 we use single model and not provide any instruction-basee training data and fine-tune from the base (non-instruct) version of the LMs despite this model shows SOTA ability.
5.2 Reasoning Language Models
Reasoning language models were introduced by OpenAT with their ol model (Jaech et al., 2024). These models showed significantly improved performance on tasks that needed reasoning, such math, logic, and programming. Since their release, many others have trained similar style models, including Google's s Gemini Flash Thinking and Deepseek's s R1. Notably, R1 is the only reasoning model that provides reasoning chains through APIs and is the only open-weights model.
Other than the impressive performance gains of these models, one additional feature is that models can quickly learn to emulate stronger models through basic supervised fine-tuning, rather than the more complex reinforcement learning pipelines that are typically used. We take advantage of this capability to train RANK1 using simple training process.
There have also been flurry of works in the open-source space, both before and after ol on reasoning language models and systems, focused on reproduction (Snell et al., 2024; Muennighoff et al., 2025), calibration and confidence (Jurayj et al., 2025), agentic capabilities with explainable reasoning traces (Weir et al., 2022; 2024), and much more. We expect this line of work to continue and likely improve these models in retrieval as well.
Limitations and Future Work
Overthinking Like other reasoning models, RANk1 can make mistakes and it can be surprising to see the model's sss reasoning chain come close to a correct answer only to change its mind. We also found that RANK1 can be particularly stringent in marking passages as true. For example, on a TREC COVID query about the origins of COVID-19 it marked every single passage Wuhan. However, when given a prompt to assume that the user had no information about COVID-19, it was able to adapt better (although not perfectly). We observed this tendency to "overthink" when using the model interactively, as would already know the answer and was looking for very specific phrase. We expect that this could be reduced with data that specifically trains the model to calibrate this.
non-relevant, since none mentioned the specific wet market in
Inference Speed As a reasoning model using test-time compute RANK1 is slower than model with only classification head (i.e. RankLLaMA). In practice this can be somewhat mitigated by the usage of modern paged attention libraries like vLLM (Kwon et al., 2023) which we use, and also quantization techniques. Nonetheless, there is no getting around the fact that using test-time compute requires spending more compute than non-test-time compute models. Despite this additional compute usage, we that users willing to wait longer for quality search results, as illustrated by the popularity of the Deep Research products from Google and OpenAI.
Future Work RANK1 also brings many more exciting areas for future work (and concurrent work (Yang et al., 2025)). As highlighted in the experiments section, it does extremely well despite the relatively small and non-diverse training data. Some of the promising areas of future work are:
Fine-tuning with RL: although supervised fine-tuning works, it does not optimize for the final answer. It is likely that RL-based approaches would be able to add rewards/penalties additional rewards/ penalties that could better align the final prediction to the label.
and https: //prducts//
gemini/googe gewini- haaaai/ggooglleeiiiiddee----eeee

---

Published as a conference paper at COLM 2025
Listwise reasoning rerankers: although pointwise models are more efficient and more parallelizable, listwise rerankers are generally more performant since they can see many documents at once. Simply gathering new data should enable this to be very successful.
Multilingual and instruction-tuned versions: RANK1 was solely trained on English and non-instruct data and still showed strong results in these settings. Training a more curated set of datasets will likely significantly improve the performance on these tasks.
We expect to see many future applications of test-time compute applied retrieval with great success, as the recipe is both simple and effective.
7 Conclusion
We build the first reasoning reranker model that uses test-time compute, RANK1. We do so by collecting 600k+ examples from the reasoning language model R1, fine-tuning on its reasoning traces. Despite only using English MS MARCO data and training from base (non- instruct-tuned) language models, RANK1 shows state of the art reasoning and instruction following capabilities, even in multilingual settings. Overall, RANK1 introduces new category of reranking models that enable wide variety of more complex IR tasks.
Ethics Statement
RANK1 more generative form of many rerankers currently used in information retrieval. As such, it introduces the possibility of generating flawed text (just like LMs). This includes incorrect and hallucinated information, the potential for toxic language, biased output, and other such flaws. These same flaws may also be present in the R1-generated dataset used to train these models.
As improved reranker, is also increases the risk when used for illicit activities. As shown by (BehnamGhader et al., 2025) models that are better at ranking and following instructions can be used more effectively to retrieve information that leads to illicit purposes. However, most of the use cases of information retrieval are positive; such is the balance when using dual-use technology. We would encourage users to improved retrieval capabilities for positive purposes.
Acknowledgments
This work has been supported by both DARPA SciFy and the U.S. National Science Founda- tion under grant 2204926. Any opinions, findings, and conclusions or recommendations expressed in this article are those of the authors and do not necessarily reflect the views of the National Science Foundation or DARPA. OW is supported by an NSF GRFP fellowship.
References
Al@Meta. Llama 3 model card. main/MODEL_CARD md.
Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. Task-aware retrieval with instructions. arviv preprint ar Xxi::211100926: 2022.
Parishad Ghaner, Nicholas Meade, and Siva Reddy. Exploiting insriiggnnnsttuttttoooloooinigi retrievers for malicious information retrieval. ar Xiv preprint ar Xrxx:25033.0664, 2025.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. Overview of the TREC. 2019 deep learning track. arXiv preprint arXto:2003.07820, 2020.
10

---

Published as a conference paper at COLM 2025
Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzeminski, Genta Indra Winata, et al. Mmteb: Massive multilingual text embedding benchmark. arxiv preprint arXio:2502.13595, 2025,
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, ar preprint ar Xit::2511.11444 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. Xrv preprint ar arXiv:2412.1672) 2024.
Vitor Jeronymo, Roberto Lotufo, and Rodrigo Nogueira. NeuralMinddUNICAMMP at 2022 TREC NeuCLIR: Large boring rerankers for cross-lingual retrieval. arXiv preprint arXio:2303.16145, 2023.
Yuelyu Ji, Zhuochun Li, Rui Meng, and Daqing He. Reasoningrank: Teaching stu- dent models to rank through reasoning-basee knowledge distillation. arxio preprint #XXi:2410.001668 2024.
William Jurayj, Jeffrey Cheng, and Benjamin Van Durme. Is that your final answer? test-time scaling improves selective question answering. arXiv preprint afXi::2502.1396, 2025.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph F. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Dawn Lawrie, Sean MacAvaney, James Mayfield, Paul McNamee, Douglas W. Oard, Luca Soldaini, and Eugene Yang. Överview of the trec 2023 neuclir track, 2024.
Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, et al. Gecko: Versatile text embeddings distilled from large language models. Xiv preprint wiXiv:2403.20322 2024.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. arXiv, 2023.
Xing Han Lù. Bm25s: Orders of magnitude faster lexical search via eager sparse scoring. Xiv preprint arXim:2407.03611 033688 2024.
Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning llama for multi-stage text retrieval. In Proceedings of the 47th International ACM SIGIR Conference Research and Development in Information Retrieval, pp. 2421-2425, 2024.
Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. Www'18 open challenge: financial opinion mining and question answering. Companion proceedings of the the web conference 2018, pp. 1941-1942, 2018.
James Mayfield, Eugene Yang, Dawn Lawrie, Sean MacAvaney, Paul McNamee, Douglas W Oard, Luca Soldaini, Ian Soboroff, Orion Weller, Efsun Kayi, et al. On the evaluation of machine-generate reports. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1904-1915, 2024.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. arXiv preprint ar XrX::2210007311,, 2022.
Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning, arviv preprint 200::24002099066, 2024.
11

---

Published as a conference paper at COLM 2025
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Perey Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arxiv preprint ar Xri::2200.11393,, 2025.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: A human generated machine reading comprehension dataset. CoRR, abs/1611 .9266, 2016. URL http: / /ttx/vxivvvoo///////e/////a wwwiii.org//iiie a//////s/1111 /61./900.
Tong Niu, Shafiq Joty, Ye Liu, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Judgerank: Leveraging large language models for reasoning-intense reranking, arXiv preprint arXit:2411.00142 2024.
Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. Document expansion by query prediction. arviv preprint arXio:1904.08375, 2019.
Hanseok Oh, Hyunji Lee, Seonghyeon Ye, Haebin Shin, Hansol Jang, Changwook Jun, and Minjoon Seo. Instructir: A benchmark for instruction following of information retrieval models. arXiv preprint arXiv:2402.14334 2024.
Ronak Pradeep, Sahel Sharifynoghatddam, and Jimmy Lin. Rank Vanuna: Zero-shot listwise document reranking with open-source large language models. arXivo2309.15088 2023a.
Ronak Pradeep, Sahel Sharilymoghaddam, and Jimmy Lin. Rankzephyr: Effective and robust zero-shot listwise reranking is a breeze! arxiv preprint arXiv:2312.00774,, 2023b.
Stephen F. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and
Mike Gatford. Okapi at trec-3, In Text Retrieval Conference, 1994. URL https://api.
semanticscholar.c
httpr//sccccooooooo/orrrrrroooaa wwwporr/roooooo1//r1r1r 415639777
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling 11M test-time compute opti- mally can be more effective than scaling model parameters. arviv preprint arXiv:2408.03311, 2024.
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. One embedder, any task: Instruction- finetuned text embeddings. 2022. URL https: //arxiv////iivv//////// wttp.vvorr//bb///2..///444
Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Haisu Liu, Quan Shi, Zachary S Siegel, Michael Tang, et al. Bright: A realistic and challenging benchmark for massoning-intensin retrieval. arv preprint arXiv:2407.12883 2024.
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is chatgpt good at search? investigating large language models as re-ranking agents. Xiv preprint ar X0::2330..0994,, 2023.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint ar axXi::2104.08663, 2021.
Nandan Thakur, Luiz Bonifacio, Maik Fröbe, Alexander Bondarenko, Ehsan Kamalloo, Martin Potthast, Matthias Hagen, and Jimmy Lin. Systematic evaluation of neural retrieval models on the touché 2020 argument retrieval subset of beir. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1420-1430, 2024.
Coen Elsen, Francien Barkhof, Thijmen Nijdam, Simon Lupart, and Mohammad Al- liannejadi. Reproducing nevir: Negation in neural information retrieval. ar Xiv preprint arXit:2502.135566 2025.
Ellen M Voorhees, Ian Soboroff, and Jimmy Lin. Can old trec collections reliably evaluate modern neural retrieval models? arXiv preprint ar Xiv:2200 11086, 2022.
12

---

Published as a conference paper at COLM 2025
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. Fact or fiction: Verifying scientific claims. arv preprint (rXiv:2004.14974, 2020.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Multilingual e5 text embeddings: A technical report. arXiv preprint orXtv:2400.00672,2000
Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, et al. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. ar ar preprint ar XrX::24122.33663 2024.
Nathaniel Weir, Peter Clark, and Benjamin Van Durme. Nellie: A neuro-symbolic infer- ence engine for grounded, compositional, and explainable reasoning. arXiv preprint arXit:2209.07662, 2022.
Nathaniel Weir, Kate Sanders, Orion Weller, Shreya Sharma, Dongwei Jiang, Zhengping Jiang, Bhavana Dalvi Mishra, Oyvind Tafjord, Peter Jansen, Peter Clark, et al. Enhancing systematic decompositional natural language inference using informal logic. aiv preprint arXir:2402.14798 2024.
Orion Weller, Benjamin Chang, Sean MacAvaney, Kyle Lo, Arman Cohan, Benjamin Van Durme, Dawn Lawrie, and Luca Soldaini. FollowIR: Evaluating and teaching infor- mation retrieval models to follow instructions. Xrv preprint ar 4rXio:2403.15246, 2024a.
Orion Weller, Benjamin Van Durme, Dawn Lawrie, Ashwin Paranjape, Yuhao Zhang, and Jack Hessel. Promptriever: Instruction-trained retrievers can be prompted like language models. Xrviv preprint vrXiv:2409.11136 2024b.
Orion Weller, Benjamin Chang, Eugene Yang, Mahsa Yarmohammadi, Samuel Barham, Sean MacAvaney, Arman Cohan, Luca Soldaini, Benjamin Van Durme, and Dawn Lawrie. mFollowIR: A Multilingual Benchmark for Instruction Following in Retrieval, In European Conference on Information Retrieval, pp. 295-310, 2025a.
Orion Weller, Kathryn Ricci, Marc Marone, Antoine Chaffin, Dawn Lawrie, and Benjamin Van Durme. Seq vs seq: An open suite of paired encoders and decoders. Xiv preprint a/Xiv:250..1141, 2025b.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38- 45, Online, October 2020. Association for Computational Linguistics. URL https: / /wwp///wwww//ooo/ aclweb. org/ anthology/2e1 2020. .mmnlp emn.p-demos......
Chenghao Xiao, G Thomas Hudson, and Noura AI Moubayed. Rar-b: Reasoning retrieval benchmark, 2024. URL https://arxxvv. org/ arg/ 2466004777
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arxie preprint arXiv:2412.15115, 2024.
Eugene Yang. Andrew Yates, Kathryn Ricci, Orion Weller, Vivek Chari, Benjamin Van Durme, and Dawn Lawrie. Rank-k: Test-time reasoning for listwise reranking. arXiv preprint arXi::205.14432,, 2025.
Xinran Zhao, Tong Chen, Sihao Chen, Hongming Zhang, and Tongshuang Wu. Beyond relevance: Evaluate and improve retrievers on perspective awareness, 2024. URL https: 02714.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. ar Xiv preprint arXio:2403.1337, 2024.
13

---

Published as a conference paper at COLM 2025
Prompt for Inference
Determine if the following passage is relevant to the query. Answer only with 'true' or 'false'.
Query: query
Passage: document
<think>
Figure Prompt used to generate data with and also for inference with RANK1.
A Training and Hyperparameter Details
We use nodes of 4x80GB H100 machines for training. Models are trained for up to 2 epochs or until days of training (for the 32B model).
We fine-tune with LLaMA-Factory, using LoRA on all parameters with rank 32 and alpha 64. We use a learning rate of le-4 and an effective batch size of 128. We use early stopping based on the Bright Biology and NevIR scores. Models use prompts for BEIR and non- stackexchange cases so that they can understand the task (Appendix E).
Inference is done on 1 H100 80GB GPU. Baselines default hyperparameters for max length and fp16 (and there are no other parameters).
B Data Analysis from m m5 Negatives
We sample 20 of the instances from mT5 that were "hard negatives" and manually evaluated them. We found that 12 them were in fact positives and thus false negatives. This helps to explain why filtering these out made such large difference in the training data, as roughly 2/ /rds of these mT5 negatives were actually positive, making the training process more noisy.
C Prompt for R1
We use the following prompt (Figure 4) to generate data for R1 (and use the same template for RANK1).
D Examples of Unjudged and Incorrect DL19 Labels
We re-annotated the top 10 passages for each model that got an incorrect or unjudged label. This was 295 labels. We found that none of the labels changed from correct to incorrect, but some labels went from incorrect to correct (Table 8).
We also show some examples of these cases in Table 9, The incorrect labels affected all models roughly equally, as seen previous in Table at around 6% of the top 10 documents, while the unjudged documents were mostly found by RANK1.
E Prompts for specific datasets
Table 10 shows the -ata-specific prompts used for BEIR and the non-stackoverflow BRIGHT subsets.
14

---

Published as a conference paper at COLM 2025
Model
Unjudged Unjudged-RRelevae -
Changed Correct
MonoT5-3b
50.00% (19/38)
6.74% (29/430)
RankLLaMA-13B
70.59% (12/17)
6.28% (27/430)
RankLLAMA-7B
70.59% (12/17)
6.74% (29/430)
Rank1-7B
85.92% (61/71)
5.35% (23/430)
Rank1-14B
86.84% (66/76)
6.05% (26/430)
Rank1-32B
88.46% (69/78)
6.51% (28/ 430)
Table 8: Qrel changes in the DL19 annotations
Example
Query: what is physical description of spruce
Passage: Spruces are large trees, from about 20-60 metres (about 60-200 feet) tall when mature, and can be distinguished by their whorled branches and conical form. The needles, or leaves, of spruce trees are attached singly to the branches in spiral fashion, each needle on small peg-like structure.
Original label: (non-relevant)
New label: 3 (relevant)
Example
Query: causes of left ventricular hypertrophy
Passage: High blood pressure may also bring on heart failure by causing left ventricular hypertrophy, a thickening of the heart muscle that results in less effective muscle relaxation between heart beats. This makes it difficult for the heart to fill with enough blood to supply the body's organs, especially during exercise, leading your body to hold onto fluids and your heart rate to increase. Original label: unjudged
New label: 3 (relevant)
Example
Query: what are the social determinants of health
Passage: @ Zoltan Balogh. The social determinants of health (SDH) are the conditions
which people are born, grow, work, live, and age, and the wider set of forces and systems shaping the conditions of daily life. Original label: (on topic)
New label: 3 (relevant)
Example
Query: example of monotonic function
Passage: Overview of the exponential function. The exponential function is one of the most important functions in mathematics (though it would have admit that the linear function ranks even higher in importance). To form an exponential function, we let the independent variable be the exponent. A simple example the function fxx) = 2 Original label: unjudged New label: 3 (relevant)
Table 9: Examples of incorrect labels and unjudged documents in the DL19 annotations.
15

---

Published as a conference paper at COLM 2025
Dataset SciFact
Prompt
Claim: FILL QUERY HERE-roowlin>>-ceeinneAAAA relevant passage would provide evidence that either "supports"" **refutes** this claim. A passage with any information on any related subpart should be relevant.
ClimateFEVER
relevant passage would provide evidence that either "supports" or "*refutes* this claim. A passage with any information any related subpart should be relevant. the article answers any part of the question it is
Claim: FILL QUERY HERii-newline>-neeiinee
TRECCOVID
FIL__QERY relevant.
ArguAna
am looking to write find counterarguments eeent:eeeewineeeeeeeeeeeeeee _ wwweery_eERR--eeiiieeeeeee this passage have any counterargument or evidence that could be used to help me?
essay and need against
this state- Does
DBPedia
looking write essay on this topic and need much related background information to help me. The topic sennelline><newlineee
QUERY DEEREEnewline><neeliie the passage provides any background information that could be connected it is relevant.
F1QA2618
FILL SUERY_HERE Find a passage that would good answer from StackExchange.
NFCorpus
Topic: FILL_QUERT HERE<newline><cnee line the above topic, I need to learn about all aspects of It does not need to be directly relevant, only tangentially informational. Please mark as relevant any passages with even weak connections. I need to learn fast for my job, which means need to understand each part any connection means relevant even if indirect. So if it is not addressed, that okay -ikay-it does not need to explicitly<<newllin wnwwlinee<nnwwinne Find me passages with any type of connection, including weak connections!!!!
Again remember,
Touche2020
FILLQQUER HERE *any** arguments for or against
SCIDOCS
papers that could be cited in FIL.I_ _II__QUERY___ HERE. Anything with even indirect relevance should be relevant. This includes papers the same
broader field of science
BrightRetrieval
Find different
but similar math problems to FILL_QUER__T BiL_QUERR_EEEEeeeiinnee www.REeeewwnneeeeeeeeeeeee
aops
document is relevant if it uses the same class of functions and shares **any** overlapping tech- niques.
BrightRetrieval
Find passage which uses the same mathematical process as this one:
theoremqa
FILL FILL_QUERy_HEI
questions
BrightRetrieval
am looking to find different problems that share similar data struc- tures (of any kind) algorithms (e.g. DFS, DP, sorting, traversals, etc.). am looking for problems that share one or both of these similarities to besscnewwwie>--neiineeeee
leetcode
QUERY HUUFRY_HERE<eewliig WiIERE-neeliine-ceellnne this passage share any similarities? e.g. if there was a textbook on leetcode problems, this would be in the same book even though it could be in differ- ent chapter.
BrightRetrieval
Problem: to solve the problem above, I need to know things about pony. A passage is relevant if it contains docs that match any part (even basic parts) of the code I will have to write for the above program.
will use the programming language pony. SwweX__ERR--eewiiee-ecee
pony
BrightRetrieval
Can you find background information about the concepts used to answer the question<<newline>-ae lnnnlneeFii QUERY MeRRE<newline><eewlliine passage is relevant if it contains background information about **sub- concept that someone might cite/link to when answering the above question.
BrightRetrieval theoremqa theorems
Find a passage which uses the same mathematical process as this one: FILL QUEL__EERR__E HERE
Table 10: Dataset-specific prompts used in the BEIR and non-StackExchange subsets of BRIGHT.
16

---

Published as a conference paper at COLM 2025
F
Things we tried that didn't work
We tried a few other things that didn't work:
We tried calibrate the scores better by adding a ModemBERT model (Warner et al., 2024; Weller et al., 2025b) on top of the outputs of the reasoning chain. However, this ended up performing worse. is possible that with better curation this would be an effective approach however.
We tried adding extra loss to the last token of the next token prediction loss in LLaMA-Factory, However, this resulted in sub-par performance, likely because predicting the last true/ false token at the end of reasoning chain is fairly easy (as the model already states beforehand "the answer is X" leaving little room for doubt).
G Noise in BEIR benchmarks
We discuss here a few of the issues with individual BEIR datasets which cause them to be noisy and less accurate at judging between highly effective systems.
SciFact In the original SciFact work (Wadden al., 2020) the authors have three labels: support, refute, or not enough information (NEI). However, when incorporated into BEIR, the NEI queries are still included. As there was not evidence to support these queries (as determined by the original authors) these queries are effectively noise. In practice, what means that tests model's ability to find the top ranked BM25 document, which is guaranteed to not have enough information to either refute support the claim. The large number of these queries add a significant amount of noise.
FiQA2018 FiQA is scraped from Financial Stackexchange (Maia et al., 2018). However, during the original scrape the creators did not collect the post's details. Thus, the retrieval setup is to take the posts title and search for the top answer. Yet, in many cases the user clar- ified important details in the post that entirely changed the meaning of the query. Without this additional information for some queries it is impossible for someone to determine what the best answer is as the post details asked many other questions that were different from the title of the post.
DBPedia DBPedia has many partially relevant labels, that give credit for finding non- relevant information. For example, the entire passage with a relevance of "1" for the query "Eiffel Tower" is "The year 1989 in architecture involved some significant architectural events and new buildings." However, nothing about the Eiffel Tower is connected to 1989, After digging, the only connection is its the 100 year anniversary of it being built but the passage does not mention this. There are many such examples in the dataset (and more documents judged "1" than "2"), contributing to the noise.
Touche2020 To2che2020 has been well examined by Thakur et al. (2024). They created much cleaner version of the data, however, it is not the "standard" evaluation set in BEIR.
Other datasets with partial relevance Many of the other datasets give credit for partial relevance in a similar manner to DBPedia. This includes TREC COVID (which has so many real positives that it is a non-issue for the top-10), NFCorpus, and SciDocs. For many of these datasets, it is very difficult for even a human to match relevance: e.g. on NFCorpus you would have to guess any potential link that had been on that website and was even ancillary related to the title. An example is the query "How Fruits and Vegetables Can Treat Asthma" which matches a document with the title "Effect of a single high-fat meal on endothelial function in healthy subjects." and does not discuss asthma at all. The reranker would have to assume that any passage discussing food asthma separately in any context would be relevant, Although one intuition is that these documents should rank higher than completely non-relevant documents, for RANK1 it treats them the same (as they are both
17

---

Published as a conference paper at COLM 2025
equally non-relevant to the query). This may be suboptimal for some approaches, but for today's RAG use cases returning only actual relevant documents seems more useful.
18


Reviews:
[{"review_id": "IQ5orz03Wu", "reviewer": "Reviewer_Znd7", "summary": "The paper introduces Rank1, a model that leverages test-time compute for reranking in information retrieval (IR). Rank 1 distills reasoning traces from DeepSeek R1; in particular, the authors prompt DeepSeek R1 using the MS MARCO collection, gathering ~600k query-reasoning traces, which they use to finetune Qwen 2.5 using LoRA. Through a thorough experimental evaluation across scales, the authors present strong performance gains across a variety of benchmarks and baselines, gains that are maintained after quantization.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper tackles an interesting problem, demonstrating that it is possible to distill reasoning chains into ranking models, resulting in efficient IR models, that also provide reasoning.\n    \n- The paper has very strong empirical results; the authors present results across a wide variety of benchmarks, showcasing the advantages of Rank1. They have also included results across different scales (from 0.5B to 32B), observing that their method achieves better nDCG@10 performance across the BRIGHT dataset, with diminishing returns after 7B parameters. They also explore quantization, finding that it has minimal effect across many tasks.\n    \n- The authors also identified multiple mislabeled or entirely undjudged relevant documents in a subset of DL19 and BEIR, arguing that this may be indicative of why such traditional reranking benchmarks are cannot distinguish between current SOTA rerankers.", "reasons_to_reject": "- When evaluating on “traditional” benchmarks, the authors manually relabel the retrieved passages that were either wrongly labeled or previously unjudged. I believe this still inserts bias. Most unjudged or mislabeled hits are in Rank 1’s top-10, while the baselines’ top-10 lists were already almost fully judged. Therefore, the relabeling inflates Rank 1’s scores far more than theirs and makes the gain look larger than it really is.\n    \n- Although the current approach (supervised finetuning) yields significant gains and is simple, I think that, as the authors also point out in the future work section, exploring a RL based approach would -perhaps- result in significantly more important improvements.\n    \n- The paper assumes familiarity with IR jargon—terms like “qrels,” “DL19,” and “point-wise vs. pair-wise” appear without definition. I think that adding a quick one-line definition the first time each term appears (or a small abbreviation table) would help readers outside the IR sub-community.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bTV1wID2W9", "reviewer": "Reviewer_iKPU", "summary": "The paper focuses on reranking the retrieved passages based on their relevance to the question. To address this task, the authors prompt LLM to reason and verify whether a passage is relevant to the question, and construct a new dataset whose sample contains reasoning text and true/false answers. Models trained on the proposed dataset outperform previous ranking methods on recent and traditional benchmarks.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper provides a new dataset for reasoning on the relevance. The authors carefully generate and filter the data, enabling them to help further work in this field.", "reasons_to_reject": "- The compared method is too weak to validate the effectiveness. RankLLaMA is based on LLaMA-2, which is much weaker than Qwen2.5. The authors should compare with stronger models or use recent general LLMs as baselines.\n\n- The experiments neglect how the retrieval improves the final answering accuracy. The origin BRIGHT benchmarks evaluate Claude's answering accuracy with and without the reranked passage.", "questions_to_authors": "The authors can provide some output examples to present the reasoning process of the trained models.", "extra_scores": {}}, {"review_id": "ZTFDGtjuoR", "reviewer": "Reviewer_Lnne", "summary": "This work presents Rank1, a passage reranker that leverages inference-time compute and reasoning traces. The method uses the MS MARCO dataset as the training set and collect reasoning traces from a powerful reasoning LM (DeepSeek-R1). The paper considers different sources of hard negatives and found certain models work better than others. Then, the collected training dataset is distilled into smaller models, such as the Qwen-2.5 models. Finally, the experiments include comprehensive datasets ranging from traditional benchmarks such as BEIR to more recent reasoning-heavy tasks like BRIGHT. The experimental results are strong and the baselines are comprehensive.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-written and well-motivated as passage reranking is a popular and critical component of modern retrieval systems. Furthermore, the strong results are demonstrated on tasks on diverse tasks and benchmarks from semantic search to reasoning-heavy and multilingual tasks. \n- The experimental details are clear and convincing; the baselines are consisted of relevant recent models. \n- The produced artifacts, the Rank1 models, may be useful for the community in both future research and practical applications. \n- The paper also discusses the noise in traditional benchmarks that may be insightful to the community.", "reasons_to_reject": "- The paper is not particularly novel in terms of its technical contributions—there are many recent works that collect and distill reasoning traces into smaller models. This may not be a big issue as it’s applied in a different setting (reranking). \n- Although the paper includes many details and takeaways from preliminary experiments, it can benefit from additional explanations (perhaps in the appendix if there are not space in the main text for the curious readers). For example, what percentage of hard negatives from mT5 are actually false negatives, some human analysis may give more insight to the training data.", "questions_to_authors": "- Sec 3.6 mentions that “using the simple budget-forcing method” does not help, where are the results for this located?\n- What are the effects of quantization on the inference speed (Table 7)?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper leverages test-time compute for a reranking model (called Rank1) by distilling reasoning traces from DeepSeek R1 into smaller models. The reviewers gave consistent positive reviews. The work demonstrates clear technical merit with comprehensive evaluation across traditional and reasoning-heavy benchmarks, showing substantial improvements over existing rerankers, and contributes a valuable 600k query-reasoning trace dataset to the community. While concerns exist about potential bias from manual relabeling of unjudged documents and limited technical novelty beyond applying established distillation methods to retrieval, the fundamental contribution of achieving state-of-the-art performance with explainable reasoning represents meaningful progress for the retrieval community. As some reviewers pointed out, it would be better to include some details and takeaways from preliminary experiments in the final version.
Decision: Accept