ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability
Authors: name: Wenhan Liu; name: Xinyu Ma; name: Weiwei Sun; name: Yutao Zhu; name: Yuchen Li; name: Dawei Yin; name: Zhicheng Dou
Published: 2025-08-09 17:26:18

Abstract: Large Language Model (LLM) based listwise ranking has shown superior
performance in many passage ranking tasks. With the development of Large
Reasoning Models, many studies have demonstrated that step-by-step reasoning
during test-time helps improve listwise ranking performance. However, due to
the scarcity of reasoning-intensive training data, existing rerankers perform
poorly in many complex ranking scenarios and the ranking ability of
reasoning-intensive rerankers remains largely underdeveloped. In this paper, we
first propose an automated reasoning-intensive training data synthesis
framework, which sources training queries and passages from diverse domains and
applies DeepSeek-R1 to generate high-quality training labels. A
self-consistency data filtering mechanism is designed to ensure the data
quality. To empower the listwise reranker with strong reasoning ability, we
further propose a two-stage post-training approach, which includes a cold-start
supervised fine-tuning (SFT) stage for reasoning pattern learning and a
reinforcement learning (RL) stage for further ranking ability enhancement.
During the RL stage, based on the nature of listwise ranking, we design a
multi-view ranking reward, which is more effective than a ranking metric-based
reward. Extensive experiments demonstrate that our trained reasoning-intensive
reranker ReasonRank outperforms existing baselines significantly and
also achieves much lower latency than pointwise reranker Rank1. Through
further experiments, our ReasonRank has achieved state-of-the-art (SOTA)
performance 40.6 on the BRIGHT
leaderboard\footnote{https://brightbenchmark.github.io/.} Our codes are
available at https://github.com/8421BCD/ReasonRank.