Pairwise RM: Perform Best-of-N Sampling with Knockout Tournament
Authors: name: Yantao Liu; name: Zijun Yao; name: Rui Min; name: Yixin Cao; name: Lei Hou; name: Juanzi Li
Published: 2025-01-22 16:49:37

Abstract: Best-of-N (BoN) sampling, a common strategy for test-time scaling of Large
Language Models (LLMs), relies on reward models to select the best candidate
solution from multiple generations. However, traditional reward models often
assign arbitrary and inconsistent scores, limiting their effectiveness. To
address this, we propose a Pairwise Reward Model (Pairwise RM) combined with a
knockout tournament for BoN sampling. Instead of assigning absolute scores,
given one math problem, Pairwise RM evaluates two candidate solutions'
correctness simultaneously. This approach eliminates the need for arbitrary
scoring and enables cross-validation of solutions through parallel comparison.
In the knockout tournament, Pairwise RM conducts pairwise comparisons between
candidate solutions and eliminates the incorrect ones iteratively. We construct
\ourdataset, a large-scale dataset of 443K pairwise comparisons derived from
NumiaMath and annotated using gemini-1.5-flash, and train the Pairwise
RM via supervised fine-tuning. Experiments on MATH-500 and the Olympiad Bench
demonstrate significant improvements over traditional discriminative reward
models. And a 40\% to 60\% relative improvement is achieved on the top 50\%
challenging problems.