AdaptiveStep: Automatically Dividing Reasoning Step through Model
  Confidence
Authors: name: Yuliang Liu; name: Junjie Lu; name: Zhaoling Chen; name: Chaofeng Qu; name: Jason Klein Liu; name: Chonghan Liu; name: Zefan Cai; name: Yunhui Xia; name: Li Zhao; name: Jiang Bian; name: Chuheng Zhang; name: Wei Shen; name: Zhouhan Lin
Published: 2025-02-19 18:35:55

Abstract: Current approaches for training Process Reward Models (PRMs) often involve
breaking down responses into multiple reasoning steps using rule-based
techniques, such as using predefined placeholder tokens or setting the
reasoning step's length into a fixed size. These approaches overlook the fact
that specific words do not typically mark true decision points in a text. To
address this, we propose AdaptiveStep, a method that divides reasoning steps
based on the model's confidence in predicting the next word. This division
method provides more decision-making information at each step, enhancing
downstream tasks, such as reward model learning. Moreover, our method does not
require manual annotation. We demonstrate its effectiveness through experiments
with AdaptiveStep-trained PRMs in mathematical reasoning and code generation
tasks. Experimental results indicate that the outcome PRM achieves
state-of-the-art Best-of-N performance, surpassing greedy search strategy with
token-level value-guided decoding, while also reducing construction costs by
over 30% compared to existing open-source PRMs. In addition, we provide a
thorough analysis and case study on the PRM's performance, transferability, and
generalization capabilities.