ICML 2025

AdaptiveStep: Automatically Dividing Reasoning Step through Model
Confidence
Yuliang Liu * 1 2 Junjie Lu * 3 Chaofeng Qu 4 Zhaoling Chen 1 Zefan Cai 5 Jason Klein Liu 4 Chonghan Liu 4
Yunhui Xia 4 Li Zhao 6 Jiang Bian 6 Chuheng Zhang 6 Wei Shen 4 Zhouhan Lin 2 7
Abstract
Current approaches for training Process Reward
Models (PRMs) often involve decomposing re-
sponses into multiple reasoning steps using rule-
based techniques, such as using predefined place-
holder tokens or setting the reasoning step’s
length to a fixed size. These approaches overlook
the fact that certain words don’t usually indicate
true decision points. To address this, we propose
AdaptiveStep, a method that divides reasoning
steps based on the model’s confidence in predict-
ing the next word, offering more information on
decision-making at each step, improving down-
stream tasks like reward model training. More-
over, our method requires no manual annotation.
Experiments with AdaptiveStep-trained PRMs in
mathematical reasoning and code generation show
that the outcome PRM achieves state-of-the-art
Best-of-N performance, surpassing greedy search
strategy with token-level value-guided decoding,
while also reducing construction costs by over
30% compared to existing open-source PRMs.
We also provide a thorough analysis and case
study on its performance, transferability, and gen-
eralization capabilities. We provide our code on
https://github.com/Lux0926/ASPRM.
1. Introduction
Large language models (LLMs) have demonstrated excep-
tional performance across various tasks. However, even
most advanced LLMs struggle to generate correct solutions
when facing complex reasoning problems, such as mathe-
*Equal contribution 1Nanjing University 2Shanghai Innovation
Institute
3University of Technology Sydney
4Independent
5UW-Madison 6MSRA 7Shanghai Jiaotong University. Corre-
spondence to: Chuheng Zhang <zhangchuheng123@live.com>,
Wei
Shen
<shenwei0917@126.com>,
Zhouhan
Lin
<lin.zhouhan@gmail.com>.
Proceedings...