AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence
Authors: Yuliang Liu
Junjie Lu
Chaofeng Qu
Zhaoling Chen
Zefan Cai
Jason Klein Liu
Chonghan Liu
Yunhui Xia
Li Zhao
Jiang Bian
Chuheng Zhang
Wei Shen
Zhouhan Lin
Venue: ICML 2025 poster (2025)
Keywords: Process Reward Model
LLM Reasoning
Reasoning Step Dividing
Primary area: deep_learning->large_language_models

Abstract: Current approaches for training Process Reward Models (PRMs) often involve deconposing responses into multiple reasoning steps using rule-based techniques, such as using predefined placeholder tokens or setting the reasoning step's length to a fixed size.
These approaches overlook the fact that certain words don't usually indicate true decision points. To address this, we propose AdaptiveStep, a method that divides reasoning steps based on the model's confidence in predicting the next word, offering more information on decision-making at each step, improving downstream tasks like reward model training. Moreover, our method requires no manual annotation. 
Experiments with AdaptiveStep-trained PRMs in mathematical reasoning and code generation show that the outcome PRM achieves state-of-the-art Best-of-N performance, surpassing greedy search strategy with token-level value-guided decoding, while also reducing construction costs by over 30% compared to existing open-source PRMs. We also provide a thorough analysis and case study on its performance, transferability, and generalization capabilities. We provide our code on https://github.com/Lux0926/ASPRM.

AdaptiveStep:
Automatically Dividing Reasoning Step through Model Confidence
Yuliang Liu 12 Junjie La Chaofeng Qu
Zhaoling Chen Zefan Cai Jason Klein Liu' Chonghan Liu" Zhouhan Lin 27
Yunhui Xia* Li Z
Jiang Bian
Chuheng Zhang
Wei Shen
Abstract
Manually Divided Steps
Current approaches for training Process Reward Models (PRMs) often involve decomposing re- sponses into multiple reasoning steps using rule- based techniques, such as using predefined place- holder tokens or setting the reasoning step's 's length to a fixed size. These approaches overlook the fact that certain words don't usually indicate true decision points. To address this, we propose AdaptiveStep, method that divides reasoning steps based on the model's confidence in predict- ing the next word, offering more information on at each step, improving down- stream tasks like reward model training. More- over, our method requires no manual annotation. Experiments with AdaptiveStep-trann trained PRMs in mathematical reasoning and code generation show that the outcome PRM achieves sate-of-the-arr Best-of-N performance, surpassing greedy search strategy with token-level value- guided decoding, while also reducing construction costs by over 30% compared to existing open-source PRMs. We also provide a thorough analysis and case study on its performance, transferability, and gen- eralization capabilities. We provide code on https://gtt thub. com/Luxx925/ASPP..
Highly informativee Automatic
Rule-based Divided
Hightl informative
Automatic
General
AdaptiveStep
Highly informative Automatic
Informative
General
Automatic
Figure 1: Rule-based reasoning step dividing (e.g., using line breaks fixed number of tokens) is automated but results in low informativeness at the end of the step and is difficult to apply in domains that are hard to define rules. In contrast, manual step division provides high informativeness but is costly to scale and heavily reliant on the experts' do- main knowledge. AdaptiveStep. which divides steps based on model confidence, addresses these challenges by offering automation, efficiency, high informativeness, and applica- bility across various domains.
matical reasoning and code generation tasks (Huang et al. 2024 Ien et al. 2024 Mirzadeh et al.) 2024 Shen & Zhang 2024). To address these challenges using the step- wise Chain of Thought (CoT) approach (Wei et al. 2023 various strategies have been proposed by the research com-
1. Introduction
munity (Qin et al. 2024 DeepSeek-Al et al.) 2025 Team
et al. 2025 ) One promising method is training Process Reward Models (PRMs), which offer more fine-grained rewards at each reasoning step compared to Outcome Re- ward Models (ORMs), guiding the L.M to generate higher- quality responses than the original model output (Shao et al., 2024 Sessa et
Large language models (LLMs) have demonstrated excep- tional performance across various tasks. However, even most advanced LLMs struggle to generate correct solutions when facing complex reasoning problems, such as mathe-
"Equal contribution Nanjjng University 'Shanghai Innovation Institute 'University of Technology Sydney "Independent "UW-Madison "MSRA 'Shanghai Jiaotong University. Corre- spondence to: Chuheng Zhang < <zhangchuheng 12auuuhnnnl33@ii 233@iiv.com>>. Wei Shen <shenwei0917e @ 126.com², Zhouhan Lin <lin.zhouhan @ gmail.com>.
2024
Gao et 2024
However, as illustrated in Figure ! existing PRMs typically divide model's response into multiple reasoning steps ing rule-based methods, such as chopping with pre-defined symbol. This results in a series of coarse reasoning step divi- sions that lack decision - -maiing information at steps (Wangl et al. 2024a Lightman et al. 2023). Moreover, rule-based methods also face challenges when applied to tasks where
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

AdaptiveStep: yutomatically Dividing Reasoning Step through Model Confidence
the steps are difficult to define. Some studies have explored the application of PRMs the level of individual tokens or a fixed number of tokens (Lee et al. 2024, Luo al. [022]] nevertheless, balancing annotation costs with the granular- ity of division remains a challenge. Although studies have demonstrated the advantages of PRMs over ORMs, these limitations, along with the high building costs, continue to constrain the broader adoption of PRMs (DeepSeck-Al) et al. 2025).
vious open-source methods in BoN evaluation. In addition, compared to greedy decoding, TVD further improves the final performance by 3.15% and 14.4% on the GSM8K and MATH500 datasets, respectively, while incurring less than 70% of the training data construction costs compared to the open-source baselines. In code generation tasks, ASPRM shows superior performance and robustness in BoN evalua- tion compared to ORM. It outperforms greedy decoding by 6.54% and 3. 3.70% on the two datasets in TVD evaluation.
To address these issues, we aim to find an automatic step- dividing method to divide reasoning solutions into more informative steps, in contrast to the coarse division by rule- based methods. As suggested by Kahneman (2011), the cognitive cost of reasoning varies depending on the dif- ficulty of the decision or task. Additionally, a statistical analysis of common errors in reasoning tasks conducted by Roy & Roth (2016) revealed that many errors stem from incorrect numerical calculations or the misapplication of words, particularly verb misuse. This suggests that certain types of words or positions in the reasoning process require more attention.
Our main contributions are as follows:
1. We propose an automatic, efficient, general, and highly informative reasoning step-dividing method, Adap- tiveStep, along with its corresponding PRM imple- mentation.
2. Our results show that ASPRM is currently the state- of-the-art PRM, empirically simple and low-cost train- ing data construction. Furthermore, the PRM built using AdaptiveStep demonstrates stronger discrimina- tive power at the token level compared to greedy search and existing methods. Additionally, we analyze and explore several properties of ASPRM, including trans- ferability, domain generalization, and division features of the training data.
Therefore, our goal is to divide the reasoning responses at these key positions to ensure the valuable costs during infer- ence and training. We find that by pivoting on the prediction confidence, the model can automatically identify the critical breaking points in the reasoning process. Accordingly, we propose AdaptiveStep, a method that divides reasoning steps based on model confidence (Hills & Anadkat 2024). We conduct experiments on the PRM scenario, with the result- ing PRM named the AdaptiveStep Process Reward Model (ASPRM). This dividing method yields highly informative step divisions, enabling downstream tasks (e.g.. processing the reward model) to enhance performance.
3. We open-source a collection of competition-level cod- ing problems from LeerCode, along with test cases, and provide an easy-to-use sandbox. We also release the dataset, models, and our code.
2. Related Works
Step-wise methods for LLMs reasoning: Chain-of- Thought (CoT) (Wei et al. 2023) reasoning has become foundational approach LLM reasoning. Scaling the number of tokens and steps in test time tackle complex problems has become common practice (Team et al. 2025 DeepSeek-AI et al. 2025). In this paradigm, the model gen- erates an intermediate step-wise solution before providing a final answer. As expectations for model performance on more complex tasks increase, methods for step-wise ver- ification and alignment have also advanced rapidly, like PRM (Zhang et al.) 2024, Wang et al., 2024a Yuan et al.
In our experiments, we assess the effectiveness of ASPRM in mathematical reasoning and code generation tasks using the Best of N (BoN) evaluation. For the mathematical rea- soning task, we evaluate on GSM8k (Cobbe et al. (02T)) and MATH500 (Lightman et al.
2023) dataset. For the code gen- cration task, we collect a dataset named LeetCodeDataset containing 1, 1,940 problems from LeetCode, along with the corresponding Python solutions and test cases, which in- clude training and test splits to train and evaluate the PRM and further assess it on the LiveCodeBench (Jain et al. 2024
2024 and step-wise RLHF (Chen et al.) 2024 Lai al.
2024 Wang et al. 2024b). Inference time step-wise meth- ods also significantly enhance the model's 's reasoning capa- bilities, such as Monte Carlo methods (Feng et al. 2023). step-wise self-consistent (Zhao et al. 2024), step-wise beam heam search (Lee et al.) 2024), and flexible divide-and-conquer
Additionally, the most widely used PRM step-dividing method relies on fixed symbols, limiting the accuracy of the more fine-gre -eeereriindd judgment ability of PRMs. We find that ASPRM can provide precise rewards to perform Token-level Value-guided guided Decoding (TVD) for reasoning tasks, offering another evaluation method by integrating PRM directly into the model inference process.
methods ( Yao et al. 2023 Hao et al. 2023) for planning.
PRM for LLM reasoning and step-dividing methods: The importance of intermediate reasoning steps in LLMs
In mathematical reasoning tasks, ASPRM outperforms pre-
2

---

AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence
(a) ASPRM data construction pipeline
(b) Diff of Rule-based method and AdaptiveStep
Stap1: Samplo Calculate threshold
question
Sample
2% steps
Janet pays 540/hour for 3 hours week clarinet lessons 528hour for 5 hours a week of piano lessons. does she spend
BH I gutsrrlle Joharn
How much piano lessons than clarinet lessons in year? answer
Queriies
Confidence distribution
Calculate threshold
Dividing step Rollout
First, we need calculate her cost for the music lessons.
.....
prodict confilence
She spends 28 5 (40.6) 140 on
piano lessons per week.n
aasa genarated tokan
Ok, now we just need to compute
cost. in (89.3) Janel spends 120- 140
260 music (47.9) lessons per week.
Threshold
Step dividing
Rollout
answer 13520 (20.2).
Threshold T= 55.7
Unnecessary stop division
Figure 2: Method overview. a) ASPRM Training Data Construction Pipeline. Step Sample from the dataset of a given domain, collecting confidence scores and samples for the training data. Then, accumulate the confidence distribution of all samples and determine the threshold. Step 2: Divide reasoning steps based on the threshold and label the steps using rollout. b) The difference between Rule-based method and AdaptiveStep division. The Rule-based method divides the reasoning process using predefined symbols or fixed token counts (c.g., line breaks, as shown in the figure), while AdaptiveStep divides reasoning steps based on model confidence. We observe that the model tends to divide reasoning steps at key decision points, such as within mathematical expressions, at noun selections, and when determining the final answer. In contrast, we find that the confidence line breaks is particularly high.
for complex tasks was highlighted by Uesato et al. 2022)) which led to the development of Process Reward Models (PRMs) to enhance LLM reasoning by providing feedback at each step. Lightman et al. (2023) showed that step-by-step feedback improves reasoning reliability and reduces logical errors. Similarly, the OmegaPRM (Luo et al. 2024), utiliz- ing Monte Carlo Tree Search (MCTS), improves mathemat- ical reasoning performance by efficiently gathering process supervision data. Wang et al. 2024a) proposed heuristic annotation method, reducing PRM annotation costs. Step- level reward models (Ma et al. 2023) have demonstrated that feedback at each step helps guide L.LMs to more optimal solutions. Automated process verifiers (Setlur et al., 2024) further enable large-scale deployment of PRMs, improving LLM aligmment. Several works have explored PRM appli- cations in reasoning tasks (Xia al. 2024 Ma al. 2023 Luo et al. 2023 Snell et al. 2024). However, the predomi- nant step-dividing method used in PRMs or other step-wise methods remains rule-based, such as using pre-defined sym- bols, which results in sentence-
plored to improve performance and constrain text generation. Chaffin al. (2021) proposed incorporating value model into the LLM decoding process, using MCTS (Coulom
2006 to constrain output without fine-tuning. Liu et al.
2024) integrated the Proximal Policy Optimization (PPO)-
based value network with MCTS, enabling collaboration with the policy network during inference. In the code gen- eration domain, Stanning-Guidee Transformer Decoding (PGG-aD et al.) 2023) uses planning algorithms for lookahead search guide the transformer in producing more optimal code. Nie et al. (2024) employed a proxy code L.M to build an offline token-scoring model that real- locates token probabilities to guide decoding. Additionally, several works have applied value functions to guide token- level decoding (Dathathri et al. 2019: Choi et al. 2023 Xu et al. 2024 Krause al. 2020). In this paper, we use PRM as value function to directly guide the decoding process of large language models, aiming to validate the effective- ness of PRM and explore additional potential applications of PRM.
PRMs. Some works have developed token-level PRMs by dividing at fixed token intervals, but the high annotation cost remains a limita-
3. Methods
tion (Lee et al. 2024 Luo et al. 2024).
In this section, we first introduce how AdaptiveStep divides responses into reasoning steps, and then present how PRM can be trained on these data, as shown in Figure At last, we introduce Token-level balue- Decoding (TVD)
Guided decoding: Standard decoding in Large Language
Models (LLMs) typically involves sampling strategies to select the next token. Guided decoding has been widely ex-
3

---

AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence
Token confidence
PRM Score
Top-k candidates
Top-k candidates
2 91.2
44.5
threshold T=55.7
50.3
63.2
3
85.6
PRM
\n
71.1
PRM
6
60.4
77.3
73.2
71.1
59.7
65.5
88.3
46.1
72.9
67.7
87.7
22.3
22.3
Figure 3: We illustrate Token-level Value-guided Decoding (TVD) with simple example. The green token denotes the selected tokens, while the gray token indicates the tokens that were not selected. The question is (1 + 1) = and the correct output is 6. In this case, the model exhibits low confidence (where Cy < T) when calculating the result of 1+1, and subsequently determines which number to multiply by 3. The PRM should select the best token based on its judgment to arrive the correct final answer. As shown in the top-left corner, for each token, the middle box represents the token itself, the bottom box shows the predicted confidence, and the box the right displays the PRM score. The red confidence score indicates that the confidence of the Top-1 predicted candidate is lower than the threshold.
that can get better responses using the trained PRM.
reward for each reasoning step and then train a PRM that can predict the reward.
3.1. AdaptiveStep
estimate target reward, we mainly follow the heuris- rollout method proposed by Wang et al. (022a). We rollout the response generation process J/ times starting from each reasoning step, resulting in rollouts denoted as (p, ri,..., ..., rk, (5) kE[K],jE]] where Ij is the j-th trajectory starting from a partial response.
Given a question E Q, we can generate N responses with temperature-based random sampling using the lan- guage model π. We denote generated responses as {s², 82 2, .... | with s" E S. (For ease of notation, omit the dependence of the response s" q.) In this way, we obtain set of question-responss pairs (Q x S).
Then, we estimate the target reward of this step based on the correctness of any decoded solution. We use hard es- timation (HE) estimate the reward for the step rk- HE indicates whether any of the responses starting from the reach a correct answer. For current partial response
To divide the responses into reasoning steps, we use the probability of the sampled token as the metric for model confidence (Hills & Anadkat, 2024). Then we determine threshold T, which based on certain proportion of the token count, such that the tokens below this threshold become a breaking point.
our implementation, in the code generation tasks, we define correctness as whether the solution can pass all test cases; in the math reasoning tasks, we define correctness as whether the answer matches the ground truth. Formally, the target reward can be estimated as
Specifically, the model confidence can be written as
Car- = p(8" | y, 82,)
where we use of and s², to denote the i-th token and the tokens prior to the i-th token in the response, respectively. Low model confidence at the i-th token indicates that the model is hard to determine the token selection the th position, and therefore, this position may become the starting point of new reasoning step.
1, 0,
E (m...,, rks te, is correct otherwise
r
(2)
With the target rewards estimated based on the rollouts, we can train PRM using the following loss:
According to the above procedure, we divide the response H" into K reasoning steps 5" {ri, rr,,,,,..., rx) where the last token within each reasoning step is associated with below-the-threshol model confidence.
C C Σ
(rt log rg (1 log (r))), (3)
3.2. PRM Training
To train a PRM based on the question-respones pairs with divided reasoning steps, we first need estimate the target
where rf is the target reward and r = (2 (p,, .1.....7. denotes the reward predicted by the PRM R.

---

AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence
MetaMath-LLama GSM8k
MetaMath- GSMBK
MetaMatt-LLama MATH500
MetaMath- Mistral MATH500
48
Performance (36
Performance (%)
Performance (%)
Performancee
44
82
88
60
30
80
Bo
Bo 8
Bo 16 Valuus (a)
Bo 32
Bo 84
Bo
Bo 4
Bo 8
Bo 16 Values (b)
B2
Bo
84
Bo
Bo 4
Bo
Bo 16 Vallus
Bo 32
Bo 64
Bo
Bo.4
Bo
Bo 16 Bo Values (d)
Bo 32
Bo
Shepherd-M
ER-PRME
ASPRM-L
ASPRM-M
Figure 4: BoN results for the math tasks. We evaluate all PRMs on: (a) MetaMath- -lama generated GSM8K candidate solutions; (b) MetaMath-Mistee generated GSM8k candidates; (c) MetaMath- generated MATH500 candidates; and (d) SataaMath-Msstte generated MATH500 candidates. The "-L" and "-M" suffixes denote the base models (Llama and Mistral, respectively). We report the evaluation results based on the released versions of other works.
3.3. Token-level Value-guided Decoding
Deepseek-Coder-Base (Guo et al., 2024). which is called LCD-DS to serve as x in the code domain. To get the math PRM training data, we sample the MATH and GSM8k train- ing datasets using MetaMath-MMsttal and sample LeetCode- Dataset training data using LCD-DS to generate code PRM training data. For evaluation, we use MATH500, the GSM8K test set, the LeetCodeDatasee test set, and BenehodeBench- V4 (Jain et al., 2024). To align with previous work and conduct further analysis, we train two math PRMs: ASPRM- L (based on Meta-Llama-3.J 1-B (Grattafiori et al. 2024). which is called Llama in the following) and ASPRM-M (based on Mistral- 1), both with MetaMath- Mistral gen- erated training data, And one code PRM: ASPRM-D (based DeepSeek- Coder- with LCD-DS generated data.
The TVD strategy leverages the PRM to guide token selec- tion during language model decoding. Specifically, when the model encounters a low confidence score (Top-1 proba- bility cp < F) in decoding, it triggers the PRM to evaluate the tokens associated with the highest M probability given by the policy model π. si = { s!,
Among these candidates, the PRM selects the token it con- siders the best based on its learned reward estimation mech- anism:
8: arg max Re (p, o,)) "."
(4)
Where s; is the token selected as the optimal choice for the low-confidence decoding position, and R represents the score given by the PRM.
Baselines and metrics: There are several open-sourced PRMs in the math domain. We select Math Sheth-hhepherd (Wang al. 2024a) and ER-PRM (Zhang et al. 2024) as base- lines. For the code domain, due to the limited availability of open-source code PRMs with competitive construction costs, we trained a code ORM as baseline using the same data, with only the final rating position considered.
4. Experiments and Analysis
In this section, we first show experiment setup, including the dataset usage, model selection, baselines, metrics, and parameter setup. We then present the experimental results, followed by an analysis of the transfcrability, generalization, and features of the division.
For all tasks, we evaluate the PRMs' performance using the Best of N (BoN) metric and further assess model capabilities with TVD. In math reasoning tasks, evaluate whether the model's final answer matches the ground truth exactly, In the code tasks, we test the generated code by running it in a sandbox and checking if it passes all test cases. Follow- ing Wang et al. (2024a), we use the minimum PRM score across all scored steps as the PRM's final judgment for a given candidate in BoN
4.1. Experiments Setup
Datasets and models: We use MetaMathQA (Yu et al. 2023) to train V0. (Tiang et al.] (023) (Mistral), which is termed MetaMath-Mistral, serve as π in math domain, and use LeetCode training data to train
'To train ASPRM for code tasks, we collected 1,745 prob- lems from the LeetCode problems as our training set and 175 problems as the test set. The test cases for these data are manually collected from the LeetCode website (exclud- ing the test cases within the problem). The solutions gathered from GitHub open-sourced repositories, mainly from
Parameter Settings: We sample 30 times per data point and deduplicate the responses Step 1. For labeling the
https///ghuuh.com/dow/eeeee checked by GPT-4, and cross- verified with the test cases.
5

---

AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence
Table 1: Token-level Value-guided Decoding results. A/P@) refers to the inference model's greedy search performance, we use Accuracy @1 for math tasks and Pass@ for code tasks as the metrics. 1 and represent the performance improvement or decline compared to A/P@1. I.
Dataset
Inference Model
A/P@1 1
Math-Shepherd
ER-PRM
ASPRM-L/-M
ASPRM-D
MetaMath-M
77.10
75.66.
75.13
79.531
/77.331
GSM8k
MetaMath-L
81.80
81.73]
81.581
83.471 82.561
MetaMatt-M
25,00
27.601
27.801
28.607
26.801
MATH500
MetaMath-L.
38.80
41.00f
38.60J
42.00 41.20
LeetCodeDataset
LCD-DS 26.28
28.001
LiveCodeBench
LCD-DS 19.21
19.92
PRM training data, we perform 8 rollouts per step using the same model This process generates 388k PRM training samples. We MetaMath- MetaMatt-Miittal-geenn -iittrr--i---geereteddd data to train the math PRM. And we get 49k PRM samples for the code PRM. In our PRM training data, each sample includes a labeling point at the end of the response. We divide the responses by 2% The value set according to Kahneman (201T), which finds that deep thinking for humans accounts for 2% of the total thinking.
LestCodeDataset-DD LeetCodeDataset
LeetCodeDalaset-DS LiveCodeBench
Perfomance (%)
35
Performence (%)
30
20
25
18
20
Bo 16
64
Bo
Bo
Bo
Bo 16
Bo 32 Bo 64
Bo Valuess
(8)
(b)
ORM-D
ASPRM-D
4.2. Overall Results
Figure 5: BoN results for the code datasets, test ASPRM- D and Code-ORM (ORM-D) on (a) LCD-DS generated
BoN Results We report the BoN evaluation results for the math dataset in Figure and for the code dataset in Figure 5 respectively.
LeetCodeDataset BoN candidates; (b) LCD-DS generated LiveCodeBench BoN candidates.
In the math tasks, ASPRM-L performs best across Fig- ure bb), and d)) despite under more stringent con- ditions: the training data sources, and the construction costs and models. 1) For the training data sources, AS- PRM only utilizes the GSM8k and MATH training sets during training data construction, while both ER-PRM and Math- Math-Shepherd used the MATH test set (without using MATH500), which results in our performance being in- ferior to theirs on MATH500, 2) For the costs and models used in construction, the data construction costs for AS- PRM is less than 70% of that for the other two methods and only used single construct model. In addition to the above problems that lead ASPRM-M to poor performance in the MATH500 dataset, attribute its performance in Fig- ure at to the training dataset being constructed by a single model, constraining its test-time transferability.
Shepherd does not always demonstrate improvement, we hypothesize this is due that the inference models already perform well with the greedy search on GSM8k, needing a more precise score to provide better guidance.
The results further demonstrate the accuracy of the token- level judgment of ASPRM. In the code generation task, ASPRM has also achieved results surpassing greedy search by providing accurate judgment.
4.3. Transferability and Generalization Analysis
In this part, we investigate whether ASPRM demonstrates model transferability and rating position, in-domain, and cross-domain generalization capability, and the performance of mixed-domain data-trained PRM. In experiments, unless otherwise specified, the BoN candidate generator and TVD inference model is MetaMath-Misshe
In the code tasks results shown in Figure 5 ASPRM-D
demonstrates superior judgment ability. As N increases, the robustness of ASPRM-D outperforms that of ORM.
ASPRM exhibit model transferability: Since the quality of training data generated by rollout depends on the policy æ. we explore the transferability of training data of our method. We get 371k PRM training data generated by MetaMath- Ilaaaa and conduct the same process as MetaMath- Mistaaa
TVD Results We report TVD results in Table I In the math reasoning task, ASPRM has consistently shown an ability to enhance the reasoning capacity of the inference models. While the performance guided by ER-PRM and
6

---

AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence
In Table 2 we find that training Mistral-VO.J on data gen- erated by MetaMath-Lllama retains judgment ability, but its performance is weaker than that trained on data generated by the weak SaraMatt--Mistral This suggests that data generated through rollout has reasonable but limited trans- ferability. Using multiple models for data construction, as in the Math-Shepherd, may be an effective strategy to enhance transferability.
variables or sentences in the original GSM8k dataset, to test whether PRM can achieve in-domain generalization. We show our results in Table We find that ASPRM exhibits strong in-domain generalization as it achieves better results in TVD than greedy search, and selects the right samples in Bo64.
Table 4: In-domain generalization ability. The experiments are conducted on the GSM- GSM-Symbolic p2 dataset. 1 indicates the performance improvement compared to greedy search.
Table 2: Transferability of PRM training data: to M in- dicates training Mistral using PRM training data generated by MetaMath-Llana, and denote performance improve- ment or decline compared to ASPRM-M.
PRM Model
Base
Bo64 / TVD
ASPRM-I.
22.80
51.56 24.561
ASPRM-M
22.80
37.88/ 24.68T
Setup
Test Dataset
Bo64 / TVD
M-MATH500
34.201 10000/27.00 27.60
M-GSM8K
83.40./77.94 7
ASPRM exhibit cross-domain generalization: We as- sess the cross-domand generalizability of PRMs using two setups: evaluating the math PRM the code datasets and evaluating the code PRM in the math datasets. Our results are shown in Figure 5 We find that the ASPRM-L provides applicable guidance on code tasks and makes correct se- lections in BoN. However, ASPRM-D performs better on the more difficult MATH500 task but struggles on simple GSM8K. We hypothesize this is due to the long training data and long prompt in code PRM, as the GSM8k test data has a total length similar to the length of the prompt part of the code data on average, resulting in fewer low-confidence points for the model to learn.
to
L-MATH500
41.801
/41.40
L-GSM8K
87.87| / /7.479 82.494
ASPRM exhibit rating position generalization: eval- uate the rating position generalization of different PRMs and show the results in Table Three setups are employed in our experiments: confidence, random, and hard, we ex- plain the setting in the caption of Table The performance of ER-PRM-L. shows a significant difference between the hard and random setups, whereas the difference of ASPRM- L between the two setups is relatively small. Additionally, ASPRM-M performs better under the random setup than under the confidence setup, demonstrating its superior gen- eralization ability in the rating position. We attribute this advantage to the diversity of rating point types in the AS- PRM training data.
Table 5: Cross-domain generalization ability of the PRMs: Source represents the source domain and the corresponding model. Target represents the target dataset domain and the corresponding test data. f and I indicate performance improvements or declines compared to the A/P@ perfor- mance in Table
Table 3: Rating position generalization. In the confidence setup, rating points are the positions where confidence falls below the threshold. In the random setup, rating points are selected at five random positions. In the hard setup, rating points are line breaks.
PRM Model
Target
Bo64 TVD
Code-ICDD
34.291 / 28.00
ASPRM-L
Code-LCB
22.301 / 19.21-
GMth
75.13| / 75.228
Models
Scoring Setup
Bo64
ASPRM-D
Math MAth-M
30.001 / 26.00
confidence
90.45
ASPRM-L
random
90.22
confidence
85.82
Mixed data benefits downstream performance: Since both tasks are reasoning tasks, we explore whether mixing training data from different domains can enhance down- stream performance. To this end, we conduct two experi- ments: 1) training Mistral mixed math and code dataset, and evaluating it on MATH500 and GSM8k; 2) training DeepSeek on an equal amount of randomly sampled math and code data, and evaluating it on LeetCodeDataser and LiveCodeBench. The results are shown in Table [ We find that mixing data improves PRM performance on math
ASPRM-M
random
86.96
hard
86.50
MS-M
random
86.20
hard 88.70
ER-PRM-L
random
87.71
ASPRMexhibit in-domain generalization: We use
GSM-Symbolic (Mirzadeh et al. 2024), which modifies
7

---

AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence
datasets, while on code datasets, performance improves only in the TVD scenario LiveCodeBench. We hypothesize this outcome is due to the following reason: for the math PRM, mixing long code domain training data improves the PRM's judging ability. For the code PRM, code domain training data is more difficult to obtain. Adding new data doubles the dataset size but introduces shorter data. This results in decreasing the global rating ability relied upon by BoN while enhancing the local rating ability used by TVD.
Bo64 results on GSM8K
80
Porformance (N)
88
85 18
180
0.5
1.0 Throshold
1.5
2.0
1.5
2.0
Threshold
(8)
(b)
MetaMATT-Llmm776
MetaMATH-Laams
MotaMATH- -o-m
Ours
Bassline
Table 6: The test results of the PRMs trained with mixed training dataset. When the hase model is Mistral, the M+C training data consists of the MetaMATH-Mistra generated math dataset and full code training dataset. When the base model is Deepseek, the C+M training data includes all of the code dataset and an equal amount of randomly sampled math training data. 1 and & represent the performance im- provement or decline compared to the mixed data trained PRMs in the origin domain of test data.
Figure 6: Threshold analysis of BoN results on GSM8K dataset with multi-scale BoN candidates generators. (a) Ours represents using Mistral as the PRM base model, and baseline represents Math- Sath-Shepherd; (b) Ours represents using Llama as the PRM base model, and baseline represents ER-PRM.
Construction efficiency: The training data construction of ASPRM demonstrates superior efficiency in both do- mains. In the math domain, the training data for ASPRM is generated using only a single SitraMatt-Misttee model, with 30 samples per data point and 8 times rollouts per step. In contrast, ER-PRM performs fewer samples but conducts 16 times rollouts, while Math-Shepherd uses multiple mod- els for sampling and rollouts. The average number of steps per sample and sample counts for each method are presented in Appendix A.2 As a result, the data construction costs for ASPRM are less than 70% of those for the other two. In the code domain, there are 14.4 lines on average per answer for the LeetCode training set, whereas only 5.69 steps are required for our method on average.
Base Model
Train
Test
Bo64 / TVD
M+C M+C
GSM8k
86.351 / 7
Mistral
MATH500
35.401 / 29.00
C+M
LCD
37.71-/ 28.00-
Deepseek
C+M
LCB
24.96 / 20.331
4.4. Threshold Analysis
In this part, we show the impact of different thresholds in dividing steps of our PRM training data. We add the BoN re- sults of ASPRM models trained with the threshold of 0.5%, 1%, and 1.5% and test them on the GSM8k dataset with more task models, we show the results in Figure 6 How- ever, larger thresholds (more than 3%) mean more rollouts than the baselines, so we only do the experiments with threshold under 2%. We use multi-scale generators to test these PRMs.
Statistical features of the division: There are several fea- tures and findings in the AdaptiveStep division statistics. For brevity, we refer to low-confidence tokens as "deci- sion tokens" throughout this section. Taking the math PRM training data generated by Mistral as an example: 1) 3.85% tokens in mathematical expressions contribute 21.03% de- cision tokens; 2) only 2.7% decision tokens are newline tokens; 3) the inference model exhibits low confidence at semantic word points, particularly at Conjunction (29.00%). suggesting that continuous or transitional thinking is partic- ularly challenging for the model.
We find that the PRM judgment ability is enhanced with the increase of dividing points in Figure (a), but is not significantly improved in Figure 6 (b). This shows that for models with different abilities, the optimal threshold choice may not be 2%, and more powerful models need less training data. Additionally, we find that at nearly 50% of the data points, ASPRM that using single model to generate training data performs better than the baselines that use multiple training data generators.
For the code PRM training data: 1) the majority of decision points occur in the Code Comment type (80%), compared the Code type (20%), even though Code Comments tokens account for only 19% of the total tokens; 2) a detailed analysis reveals that the Code Comment samples primarily fall into two subtypes: explaining what previous lines do and planning what to do in the following lines. The first subtype accounts for 9% of the samples, while the second accounts
4.5. Feature Analysis
In this part, we discuss the features of the AdaptiveStep division used in ASPRM and its advantages.
8

---

AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence
for 91%. This indicates that the inference model triggers more during the planning process than during the writing process when generating code; 3) by further analyzing the Code type. we find that Logical Operators, Block Begin Keyword, Control Statements and Loop Statements occupy a high proportion of low confidence proportion with a small number of tokens. This suggests that, in addition to pre- planning in Comment, the model still requires assistance at certain logical decision points during the writing process.
Blockchain Special Project (No. 24BC3200100).
Impact Statement
AdaptiveStep is an automatic, highly informative, and effec- tive method for dividing reasoning steps. It can be easily applied to a wide range of complex tasks across various domains, such as code generation (as demonstrated in our paper) and AI-driven scientific reasoning. Furthermore, exploration of the properties of AdaptiveStep PRM and its training data features will contribute to advancing process reward assignment in LLMs, potentially shaping the devel- opment of more general PRMs.
The statistical information indicates that the inference model is prone to performing low confidence in the calculation process, semantic word selection in mathematics reasoning tasks, and the planning process in code generation tasks. The full statistical results are provided in Appendix A.1
References
Our results in indicate that PRM trained on mixed datasets can enhance downstream performance, making it possible to achieve better results in domains with hard-to- obtain data, such as code generation, at a lower cost. Based on the results and feature analysis in the code data, we hy- pothesize that the mutual enhancement arises from both tasks being reasoning problems. Similar to the text reason- ing process in mathematics, the Code Comments contain substantial content that outlines subsequent steps. There- fore, training on mixture of both datasets allows the model to achieve improved results.
Chaffin, A., Claveau, V., and Kijak, E. PPL- MCTS: Constrained Textual Generation Through Discriminator- Proceedings of the CtrlGen workshop, tual, United States, December 2021. URL https://hal wcw.eccehaal--044444000000
In Proceed- ings of the CtrlGen workshop, pp. 1-19, vir-
Decoding.
Chen, G., Liao, M., Li, C., and Fan, K. Step- level value preference optimization for mathemati- cal reasoning. In Al-Onaizan, Y., Bansal, M., and Chen, Y.NN. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 7889- 7903, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 16653//11//224.
5. Conclusion
In this paper, we propose a new reasoning step dividing method, AdaptiveStep. along with a corresponding Process Reward Model (PRM), ASPRM. We test the effectiveness of the PRM on mathematical reasoning and code generation tasks. To train the code PRM, we collect a function-level LeetCode dataset. We effectively integrate the PRM into the standard LLM inference process, achieving improvements over greedy search without additional inference overhead by token-level guidance. Our experiments on widely used datasets demonstrate robust performance with reduced com- putational costs. Furthermore, we analyze model transfer- ability and generalization, showing that ASPRM exhibits both rating position, in-domain, and cross-domain gener- alization. We also find that combining data from different domains further enhances PRM performance. Lastly, our feature analysis of the AdaptiveStep division confirms its effectiveness and informativeness.
URL https::/aclaatthl..g org/2024. findings- eewciinng-ennll... 463/
Choi, S., Fang, T., Wang, Z.,, and Song. Y. Kets: knowledge- constrained tree search decoding with token-level hal- lucination detection. arXiv preprint arXiv:2310.00 10..0944. 2023.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems, 2021. URL htttps:///axii.....
abs/2110. 14168
Coulom, R. Efficient selectivity and backup operators in monte-carlo tree search. In International conference on computers and games, pp. 72-83. Springer, 2006.
Acknowledgement
Dathathri, S., Madotto, A., Lan, J., Hung, J., Frank, E., Molino, P., Yosinski, J., and Liu, R. Plug and play lan- guage models: A simple approach to controlled text gen- eration. arxiv preprint arXiv:1912.022 1912,02164, 2019.
We sincerely thank Zilin Zhu for providing valuable sugges- tions on efficiency optimizations of our code and Di Yang, Xiaochen Zhu, and the reviewers for their advice during the completion and review of the work. This work is spon- sored by the Shanghai Science and Technology Commission
DeepSeck-AL Guo, D., Yang, D., Zhang, H., Song. J., Zhang, R., Xu, R., and et.al. Deepseek-r1: Incentivizing

---

AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence
reasoning capability in Ilms via reinforcement Jearnnnn learning, 2025. Utt.httpss//arra...rrg/ab///////
Lai, X., Tian, Z., Chen, Y., Yang. S., Peng, X., and Jia, J. Step-dpo: Step-wise preference optimization for long- chain reasoning of I1ms, 2024. URL httpps///axiv
12948
Feng. X., Wan, Z., Wen, M., McAleer, S. M., Wen, Y., Zhang. W., and Wang, J. Alphazero-lin tree-search can guide large language model decoding and training. arXiv preprint ArXiv:2309.177 17179, 2023.
org/abs/2406. 18629
Lee, H., Yang. J. Y., Heo, B., Han, D., and Yoo, K. M. Token-supervised value models for enhancing mathemati- cal reasoning capabilities of large language models, 2024. URL https://arxix httpss////xxvvoooaaaaaaee abg//bs//aa 2407. 12863
Gao, B., Cai, Z., Xu, R., Wang, P., Zheng, C., Lin, R., Lu, K., Liu, D., Zhou, C., Xiao, W., Hu, J., Liu, T., and Chang, B. Lim crities help catch bugs in mathematics: Towards better mathematical verifier with natural lan- guage feedback, 2024. URL htttps://arivv org/abs/ 2486. 14824
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I,, and
Cobbe, K. Let's verify step by step, 2023. URL h
//arxiv.org abs// 2205./ 20050
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A. Al-Dahle, A., and et.al. The llaaa 3 herd of models, 2024. URL JtLLhttps:///at //axxiv org/ abs 2407. 21783
Liu, J., Cohen, A., Pasunuru, R., Choi, Y., Hajishirzi, H., and Celikyilmaz, A. Don't 't throw away your value model! generating more preferable text with value- value-guided monte- carlo tree search decoding, 2024. Ittpssts::/aard
Guo, D.. Zhu, Q., Yang, D., Xie, Z.,, Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y., Luo, F., Xiong. Y., and Liang, W. Deepscek-coder: When the large language model meets programming the rise of code intelligence, 2024. Utth:tts::/
15 15028
Luo, L., Lin, Z., Liu, Y., Shu, L., Zhu, Y.. Shang. J., and Meng. L. Critique ability of large language models, 2023. URL httph:ttp////rae wwt.aaawwww...aaaaaaea/a.er..eee h/tg://b///// ab//////...2///////// 04815
org/ abs/ abeea//....12 14196
Hao, S., Gu, Y., Ma, H., Hong. J. J., Wang, Z., Wang, Z. and Hu, Z. Reasoning with language model is planning with world model, 2023. URL https://arxiv.or org/
Luo, L.., Liu, Y., Liu, R., Phatale, S., Lara, H., Li, Y., Shu, L., Zhu, Y., Meng. L.. Sun, J., and Rastogi. A. Improve mathematical reasoning in language models by automated process supervision, 2024. URL https:://ark: org/ abs/ 06592
abs/2305.
Hills, J. and Anadkat, S. Using logprobs. 2024. URL URLhttps: //cookbook.oppeee wtw. wammm/ss/ppsesinnggggge.. logprobs Accessed: 2024-12-10.
Ma. Q., Zhou, H., Liu, T. Yuan, J., Liu, P., You, Y., and Yang, H. Let's reward step by step: Step-level reward model as the navigators for reasoning. arXiv preprint arXiv:2310.1008: 2023.
Huang. J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., and Zhou, D. Large language models cannot self reasoning yet, 2024. URL /tL/hhtps://anr htrgabs/23111 01798
Mirzadeh, I., Alizadeh, K., Shahrokhi, H., Tuzel, O., Ben- gio, S., and Farajtabar, M. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large lan- guage models, 2024. URL 2410. 05229
Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Live- codebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint, 2024.
org/abs/
Nie, Y., Wang. C., Wang. K., Xu, G., Xu, G., and Wang, H. Decoding secret memorization in code llms through token-level characterization. arXiv preprint arXiv.2410.08858. 2024.
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L.,, Lavaud, L. R., Lachaux, M.- A., Stock, P., Scao, T. L., Lavril, T., Wang. T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023. URL https:
Qin, Y., Li, X., Zou, H., Liu, Y., Xia, S., Huang, Z., Ye, Y., Yuan, W., Liu, H., Li, Y., and Liu, P. O1 replication journey: A strategic progress report part 1, 2024. URL https: httpr/iww.org/bb//111e0/ 18982
//arxiv. .rg. abaaaa////33 068225
Kahneman, D. Thinking, fast and slow. Farrar, Straus and Giroux, 2011.
Roy, S. and Roth, D. Solving general arithmetic word problems, 2016. URL httts:://axxi..or//ass.
Krause, B., Gotmare, A. D., McCann, B., Keskar, N. S., Joty, S., Socher, R., and Rajani, N. F. Gedi: Generative discriminator guided sequence generation. arXiv preprint 2XXi:20009006667, 2020.
01413
Sessa, P. G., Dadashi, R., Hussenot, L., Ferret, J., Vieil- lard, N., Ramé, A., Shariari, B., Perrin, S., Friesen,
10

---

AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence
A., Cideron, G., Girgin, S., Stanczyk, P., Michi, A., Sinopalnikov, D., Ramos, S., Héliou, A., Severyn, A., Hoffman, M., Momchev, N., and Bachem, O. Bond: Aligning lims with oe distillation, 2024. URL attps://arxiv.
Wang, P., Li, L.., Shao, Z., Xu, R. X., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations, 2024a. URL
68935
14622
Wang, T., Chen, J., Han, x., and Bai, J. Cpl: Critical plan step learning boosts Ilm generalization in reasoning tasks, 2024b. URL httphttss///rriiii org/ abs/ 2489.08642
Setlur, A., Nagpal, C., Fisch, A., Geng, X., Eisenstein, J., Agarwal, R., Agarwal, A., Berant, J., and Kumar, A. Re- warding progress: Scaling automated process verifiers for Ilm reasoning, 2024. URL https:://ara org/abs/ 2410.08 08146
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. and Zhou, D. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL https: org/ www.orgg//////e///e//../eeaeeeee 2201 11903
Shao, Z., Wang. P., Zhu, Q. Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., and Guo, D. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models, 2024. URL
Xia, S., Li, X., Liu, Y., Wu, T., and Liu, P. Evaluating mathematical reasoning beyond accuracy, 2024. URL https://arxiw.org/aaa 2404 05692
https://arxiv.org/aaa www.:oor/abb//442.025g
Xu, Z., Jiang, F., Niu, L.., Jia, J., Lin, B. Y., and Poovendran, R. Safedecoding: Defending against jailbreak attacks via safety-awar decoding. arXiv preprint arXiv:2402.08983, 2024.
Shen, W. and Zhang, C. Policy filtration in rlhf to fine-tune IIm for code generation. arXiv preprint arXiv:4409.06957, 2024.
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L.,, Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models, 2023. URL https://arxiv wwwrxivv.orrroooo 2305. 10601
Snell. C., Lee, J., Xu, K., and Kumar, A. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters, 2024. URL https: org/abs/ 03314
Yu, L., Jiang, W., Shi, H., Yu, Liu, Z.,, Zhang, Y., Kwok, J. T., Li, Z., Weller, A., and Liu, W. Metamath: Boot- strap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 122844 2023.
Team, K., Du, A., Gao. B., Xing, B., Jiang, C., Chen, C., Li, C., Xiao, C., Du, C., Liao, C., Tang, C., Wang, C., Zhang, D., Yuan, E., Lu, E., Tang, F., Sung. F., Wei, G., Lai, G., Guo, H., Zhu, H., Ding, H., Hu, H., Yang. H., Zhang, H., Yao, H., Zhao, H., Lu, H., Li, H., Yu, H., Gao, H., Zheng, H. Yuan, H., Chen, J., Guo, J., Su, J., Wang. J., Zhao, J., Zhang, J., Liu, J., Yan, J., Wu, J., Shi, I.,, Ye, I.., Yu, L.., Dong, M., Zhang, N., Ma, N., Pan, Q. Gong. Q. Liu, S., Ma. S., Wei, S., Cao. S., Huang, S., Jiang, T., Gao, W., Xiong, W., He, W., Huang, W., Wu, W., He, W., Wei, X., Jia, X., Wu, X., Xu, X., Zu, X., Zhou, X., Pan, X., Charles, Y., Li, Y., Hu, Y., Liu, Y., Chen, Y., Wang, Y., Liu, Y., Qin, Y., Liu, Y., Yang, Y., Bao, Y., Du, Y., Wu, Y., Wang. Y. Zhou, Z., Wang, Z., Li, Z., Zhu, Z., Zhang, Z., Wang, Z., Yang. Z., Huang, Z., Huang, Z., Xu, Z., and Yang, Z. Kimi kl.5: Scaling reinforcement learning with llms, 2025. URL/httss://arxiw.orr. http://awwww...ora/////a////// 12599
Yuan, L., Li, W., Chen, H., Cui, G., Ding, N., Zhang, K., Zhou, B., Liu, Z., and Peng, H. Free process rewards without process labels, 2024. URL https
//axxvv httpl//.rg//b///////// 2412, 01981
Zhang, H., Wang, P., Diao, S., Lin, Y., Pan, R., Dong, H., Zhang, D., Molchanov, P., and Zhang, T. Entropy- regularized process reward model, 2024.
Zhang, S., Chen, 7.,, Shen, Y., Ding, M., Tenenbaum, J. B., and Gan, C. Plamning with large language models for code generation. arXiv preprint arXiv:2303.05511 2023.
Zhao, Z., Rong, Y., Guo, D.,, Gözlüklüi E., Gülboy, E., and Kasneci, E. Stepwise self-consistent mathematical reasoning with large language models, 2024. URL URL 7/arxiv. orggiv.orr//ee wtwg/rr/////ao/ae/oooooeeeere/// abs////s///22 17786
Tyen, G. Mansoor, H., Cärbune, V. Chen, P., and Mak, T. Lims cannot find reasoning errors, but can correct them given the error location, 2024. URL https://arxi: org/abs/2311 23816///1110
Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving, G., and Higgins, I. Solv- ing math word problems with process- and outcome- based feedback, 2022. URL bhttts:://arxttt //trxv//aviv.//.// org//bs// 2211. 14275

---

AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence
A. Appendix
A.1. feature statistic
In this part, we present the statistics of the decision token types in our dataset. Table 7 and Table shows the statistical information of the math data, and Table shows that of the code. We adopt en en coreecorrbbloeee _e__weeb_sn web_sm model from Spacy library as tokenizer and POS tagger make statistics. We show the cases for types of tokens in Appendix A.3
A.2. Dataset Information Statistic
In Figure 7, we report the statistical information of math training data for our dataset and ER-PRM, Math aath-Shepherd, PRM800K In Figure 8
we show the statistical information for our math BoN candidates.
A.3. Case Study
We show divided cases in Math (Table 11 and Code domain (Table 12
A.4. Traning Data Distribution
Since the training data is generated and divided by the model, we will worry about whether there are a large number of difficult questions divided into few steps. Therefore, we analyze the relationship between step division and correct answer rate in our Mistral-generated GSM8K training data, we show the results in Figure | We find that only few difficult questions (1.62%) is hard to answer by the training data generator with few divisions.
Table MetaMath-Mistiee generated data statistic results: percentage of tokens types and percentage of decision tokens types for math domain. Natural Sentence stands for piece of text separated by New line break or Punctuation like Period and Question Mark. Reasoning represents symbolic reasoning or Math Formula; Entity represents Noun like apple personal name; Semantics represents Conjunction, Verb and Determiner. We also find that there are few word level splits represented by Split Word; we retained these segmentation points to enhance the model's generalization at these points during PRM training.
Position
Categories
Subtypes
Token type proportion (78m)
Decision token proportion (1517k)
New line break
3,85%
2.70%
Natural Sentence
Punctuation
26.92%
4.61%
Symbolic Reasoning
15.39%
6.79%
Reasoning
Math Formula
3.85%
21.03%
Entity
Noun
15.38%
11.01% 01%
Conjunction
20.51 %
29.00%
Semantics
Verb
6.41%
5,34%
Determiner
7.69%
2.64%
"Same to
Wang et al. 2024a), We counted the number samples for PRMS0OK and a quarter Math-Shan Sath-Shepherd.
12

---

AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence
Table 8: Proportion of decision tokens in the original data of the same type for math domain generated by MetaMath-Llama.
Position
Categories
Subtypes
Token type proportion (81m)
Decision token proportion (1413k)
New line break
2.47%
6.69%
Natural Sentence
Punctuation
28.40%
14.91% %
Symbolic Reasoning
16.05%
5.66%
Reasoning
Math Formula
3.7%
20.24%
Entity
Noun
14.82%
7.35%
Conjunction
20.99%
23.48%
Semantics
Verb
6.17%
5.24%
Determiner
7.4%
2.99%
Table 9: Proportion of decision tokens in the original data of the same type for code domain
Position
Categories
Subtypes
Token type proportion (17m)
Decision token proportion (47k)
New line break
6.99%
11.79%
Syntax Symbol
Space Character
77.58%
1.60%
Numbers
Number
4.21%
0.84%
Boolean Operators
0.26%
3.21%
Logical Operators
Arithmetic Operators
2.04%
3.54%
Definition
Def Class
0.53%
1.82%
Import Statement
From Import
0.58%
0.76%
Type Defination
0, 0.16
0.48%
Function
Build-in Function
0.49%
0.77%
Instance Method
0.09%
0.26%
Control Statements
If Else Elif
0.64%
3.51%
Loop Statements
For While
0.62%
1.73%
Return
0.68%
0.58%
Others
Punctuation Mark
4.99%
6.52%
13

---

AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence
Table 10: Proportion of decision tokens in Code and Code Comment
Categories
Trigger type(234k)
Token type (17m)
Line number(1599k)
Code
47k (19.95%)
4m (26.15%)
1280k(80.02%)
Code comment
187k (80.05%)
13m (73.85%)
319k(19.98%))
Table 11: Samples of decision tokens for math domain.
Categories
Subtypes
Sample
New line break
works on 4 of them each day. nAAterr 5 days,
Sentence
Punctuation
If Billie has 18 crayons, and Bobbie has three times gives them 3 points. So in total, Joe's team has 3 = so
Text reasoning
Reasoning
Math formula
10 = 25
Entity
Noun
Ron gets to pick
book 1 out of 13
Conjunction
their ages is 34, so we can write the equation L (L + 4) 34.
Semanties
Verb
In 14 days, each dog will 250 grams/day we can round this to the nearest whole number.
Determiner
Table 12: Samples of decision tokens for code domain
Categories
Subtypes
Sample
New line break
\n i += num bytes
Syntax Symbol
Space Character
dp[i][j] dpli I][j] * (j k)
Numbers
Number
j (target x * 2) //2 c in count and с a: dp [[0] (n+1) for in range(n+1)) def is_valid(r, c): class Solution:
Boolcan Operators
Logical Operators
Arithmetic Operators
Def
Definition
Class
From
from collections import defaultdict
Import Statement
Import
import collections
Type Defination
for size in list(dp(curr sum]): if abs(next_coum) 1) 0:
Function
Build-in Function
Instance Method
self.count =
len(tokens)) i
Control Statements
Else
else:
Elif
elif level == 0 and expression[i] =
For
for i in range(len(fronts))
Loop Statements
While
while != self. self.parent[x]: return (merged[ n // digit-sum (11.val if 11 else 0) (12.val if 12 else 0)
Return
+ merged[n // 2]) / 2.0
Others
Punctuation Mark
14

---

AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence
x10
8
7.22
444655
6.23
4
388403
6
Sample numbers
Average step
5.17
4.90
4.31
3
273226
4
2
98731
Ours-M
Ours-L
ER-PRM
M-S
PRMB0OK
Ours-M
Ours-L
ER-PRM
M.S
PRMB00K
Dataset
Dataset
(a)
(b)
50
46.53
350
306.19 303.34
40
36.11
38.16
37.49
Tokens per sample
300
273.01
270.10
Tokens per step
33.57
249.87
250
30
200
20
150
100
10
50
0
Ours-M
Ours-L
ER-PRM
M-S
PRMB0OK
Ours-M -M
Ours-L
ER-PRM
M-S
PRM800K
Dataset
Dataset
(c)
(d)
Figure 7: Statistic Information of our math dataset, Ours-M represents data constructed by Mistral, and Ours-L represents data constructed by Ilama. ER-PRM, Math-Shepherd (M-S), PRM800K. (a): Average step; (b): Sample number; (c): Average tokens per reasoning step; (d): Sample length. We use Mistral tokenizer for statistics.
15

---

AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence
420
412
Total length Answer length
379
Total length
Answer length
343
350
318
280
Length
239
Length
224
205
210
210
194
338
321
269
140
250
175
160
70
147
136
M-MATH500 M-GSMBK L-MATH500 L-GSMBK
M-MATH500 M-GS MBK L-MATH500 L-GSMBK
Datasets
Datasets
(h)
Figure 8: Statistic Information of our BoN dataset (a): Statistic with Mistral tokenizer; (b): Statistic with Llama tokenizer.
Rollout 64 Train Dataset
Low Acc Low Step Ratio: 1.62%
Regression line
90th Percentile Line
2500
20
80th Percentile Line
70th Percentile Line
Low Acc & Low Step
2000
15
Density of points
Steps
10
1000
500
0.0
0.2
0.4
0.6
0.8
1.0
Accuracy
Figure 9: Training data distribution (division numbers and rollout accuracy).
16


Reviews:
[{"review_id": "w5EyB16SDl", "reviewer": "Reviewer_T2AZ", "summary": "Simple yet effective method for segmenting reasoning traces into individual steps, resulting in modest but significant improvements on relevant math and coding benchmarks.\nThe paper proposes to segment reasoning traces according to next-token confidence levels, instead of e.g. heuristics like new lines.\nThe experiments show that this segmentation method outperforms prior work in PRM training and ultimately results in higher performance on Math and Coding datasets.\nThe analysis is limited to coding and math problems, which however constitute highly relevant domains.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer Comments\n\n**Dear reviewer T2AZ:**\n\nWe sincerely appreciate your constructive feedback and the time you have devoted to reviewing our manuscript. We try to address your concerns point by point.\n\n>## W1: Discussion or empirical analysis for failure cases of AdaptiveStep\n\n**Response:**  Thank you for highlighting this important aspect. Though explicitly classifying \"failure cases\" in the model's segmentation is challenging as what is regarded as \"failure\" for humans indeed aligns with the model's confusion points, we still identified several instances of suboptimal segmentation results.\n\nWe understand that failure cases can be categorized from two perspectives. The first perspective involves words that should remain intact but are erroneously split into separate parts. The second perspective concerns instances where the model fails to segment at positions prone to errors. We illustrate the first type of failure cases through the following examples, while the second type is demonstrated through a figure.\n\n**Example 1:** \"We can rewrite the **quad** / **ratic** as $y = 3(x^2 + 2x) + 9$\"\n\n**Example 2:** \"**Sub** / **stituting** these values\" — where \"quadratic\" and \"substituting\" are incorrectly segmented despite being complete words\n\n**Example 3:** \"**Natal** / **ie** would need to trade\" — where \"Natalie,\" a proper noun referenced in the question, is inappropriately segmented\n\nWe conduct a statistical analysis of the aforementioned type where words are split, and find that about 2% of the segmentation belongs to this category. However, since this split is determined by the model itself, we retained these splits during the training process.\n\nAdditionally, in the very begining of our work, we observed that approximately 3% of split points occur at the beginning of solutions, which we classify as erroneous segmentations, we removed these split positions because they indicate that the solutions have not yet started to be generated. \n\nFor the second perspective, due to limitations of the base model, certain complex questions exhibit little segmentation in their solutions, as illustrated in the [figure](https://anonymous.4open.science/r/PIC-0969/fig1.png), we believe that the problems in the lower-left corner (the 1.62% with low accuracy after 64 generations and segmentations numbers lower than 5) represent problems that exceed the model's capabilities, and require additional supervisory information to assist.\n\nIn summary, aside from the examples that are beyond the model's capabilities to judge (which account for about 5% of the total questions), there are 5% of the splits that, from a human perspective, seem unreasonable. We have removed some of these (those at the beginning of the solutions) and retained others. We will explain this in more detail in the appendix. Thank you again for your highlighting.\n\n>## W2: Algorithm definition\n\n**Response:** Thank you for your suggestion. We will include the following algorithmic in out next version:\n\n## Algorithm: PRM Training from Confidence-Guided Rollouts\n\n**Inputs**:\n- `Q`: A sequence of questions  \n- `N`: Number of responses to generate per question  \n- `J`: Number of rollouts per split point  \n\n**Output**:\n- A trained process reward model (PRM)\n\n```text\nfor each question q in Q do\n    // Step 1: Generate N responses for the question\n    responses ← GENERATE_RESPONSES(q, N)\n\n    // Step 2: Compute confidence distribution for the responses by eqution (1)\n    confidence_distribution ← COMPUTE_CONFIDENCE(responses)\n\n    // Step 3: Determine threshold and find split points\n    threshold ← COMPUTE_THRESHOLD(confidence_distribution)\n    split_points ← FIND_SPLIT_POINTS(confidence_distribution, threshold)\n\n    for each split in split_points do\n        labels ← empty list\n\n        // Step 4: Perform J rollouts for each split point\n        for i from 1 to J do\n            label ← PERFORM_ROLLOUT(split)\n            APPEND(labels, label)\n        end for\n\n        // Step 5: Aggregate rollout labels into a hard estimate by eqution (2)\n        prm_label ← HARD_ESTIMATE(labels)\n\n        // Step 6: Train the PRM with the obtained labels by eqution (3)\n        TRAIN_PRM(prm_label)\n    end for\nend for\n```\n\n---\n\n**We welcome any additional feedback you may have regarding our manuscript or this response. We are committed to incorporating your suggestions to enhance the quality of our work. Thank you again for your valuable insights.**", "claims_and_evidence": "Claim: AdaptiveStep yields a SOTA PRM for math tasks.  \nEvidence: Evaluation on relevant benchmarks and for multiple models.", "methods_and_evaluation_criteria": "The method constitutes cutting generations at tokens that have low probability, training a PRM on the data, and using the PRM to train a Math/coding model. this is a straightforward procedure that follows relevant prior work.", "theoretical_claims": "NA.", "experimental_designs_or_analyses": "yes.", "relation_to_broader_scientific_literature": "The paper fits well with recent literature on step-by-step reasoning for math and coding problems.", "essential_references_not_discussed": "NA.", "other_strengths_and_weaknesses": "Strengths:\nSimple method and significant improvements on relevant benchmarks.\nPaper is well written and problem setting is relevant.\nCross-domain generlaisation results suggest robustness.\n\nWeaknesses:\n- Failure cases of proposed segmentation method are neither discussed nor empirically analysed. It would be nice if the authors could both provide examples of common failure cases, as well as a discussion of them\n- The Adaptive step method is only described in words, it would be great to have an algorithm definition in the methods section.", "other_comments_or_suggestions": "See weaknesses.", "questions_for_authors": "See weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l76IcaUj0V", "reviewer": "Reviewer_ZPdL", "summary": "1. In this paper, a novel step dividing method, AdaptiveStep is proposed. The method enables automatic step dividing while being more informative than rule-based step dividing method.\n2. By adapting AdaptiveStep, the ASPRM demonstrates stronger discriminative power at the token level than existed method at some extent.\n3. The author open-sourced a LeetCode dataset, which may benefit following works on PRM.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer Comments\n\n**Dear reviewer ZPdL:**\n\nThank you for your insightful comments and valuable feedback. Below, we address the concerns raised and provide clarifications and improvements.\n\n>## W1: Concerns about experiments results\n\n**R1:**  Thanks for your comment and we will revise our paper to clarify the setting/scope of the SOTA statement accordingly. We argue that the overall performance of ASPRM-M exceed that of the baselines. Specifically, ASPRM-M significantly outperforms baselines across all TVD experiments. In the BoN-Mistral generated data, ASPRM-M performs better than Math-Shepherd when N is small. In positions with few instances, such as Figure 4(b) for Bo64 and Figure 4(d) for Bo16 and above, ASPRM-M is slightly worse than Math-Shepherd. In subsequent experiments, such as those involving position generalization, ASPRM also performs well.\n\n>## W2: Llama compared with Mistral\n\n**R2:**  Thank you for bringing this up. The experiments involving Llama and Mistral were simply presented in the same figure for space efficiency. There was no intention to directly compare the performance of these base models. We likely would not have conducted the experiments with the Mistral-based ASPRM-M model if we want to make a direct comparison.\n\n>## W3: Reward Model Related Work\n\n**R3:**  We appreciate the suggestion to add related work on reward models. We will revise the manuscript to include relevant references and discussions on prior work in this area (e.g., the Bradley-Terry model, GenRM, Pairwise reward model, etc.).\n\n\n>## W4: RLHF and More Models\n \n**R4:**  \n**RLHF Methods:**  \nWhile RLHF methods are often used to evaluate reward models, they are not strictly necessary for this area. Several influential works on (process) reward models have not included RLHF experiments but use BoN or test-time scaling to evaluate the model’s capability, yet they remain impactful in the field [1, 2]. Our focus was on proposing and testing a novel reasoning step segmentation method. Given the substantial computational resources required for RLHF experiments, we are unable to include them in this study. We kindly hope that the reviewer takes this into consideration.\n\n**More Models:**  \nRegarding the inclusion of more model scales, we have conducted experiments on a wider range of model sizes and thresholds in [Table 1](https://anonymous.4open.science/r/PIC-0969/fig4.png) and [Table 2](https://anonymous.4open.science/r/PIC-0969/fig5.png), which we will incorporate into the updated manuscript, we observe that as the model size increases, the PRM’s ability to make judgments remains effective.\n\n\n[1] Let's Verify Step by Step  \n[2] Training Verifiers to Solve Math Word Problems\n\n---\n\n>## Q1: Our Contributions\n\nWe sincerely request that the reviewer reconsider the contributions of our work. Our primary contribution is the introduction of a novel reasoning step segmentation method AdaptiveStep, which we validate in the PRM senario. The PRM trained with AdaptiveStep can guide the task model in generation and reasoning at the token level, which previous PRMs have not been able to do in our experiments. Moreover, we have successfully extended PRM to domains where step segmentation is difficult using rules (such as code generation). And we use a single model for data construction and reduces construction cost by 30% in the math domain and 60% in the code generation domain compared to rule-based methods. \n\nIn the experimental setup, we have made every effort to maintain consistency with previous work, including model selection, PRM training methods, and evaluation techniques. This was done to establish a baseline for a fair comparison, which consequently limited the potential for introducing significant innovations in the training of the PRM or the testing of BoN/TVD.\n\n---\n\n>## Q2: Credit and License\n\nThank you for pointing this out. We use this dataset only for research purposes and will strictly regulate its usage context and license. We will do our best to align our open-source work with [LiveCodeBench](https://github.com/LiveCodeBench/LiveCodeBench), which collects data from LeetCode weekly contests. \n\n---\n\n>## General Rebuttal\n\nWe propose a novel reasoning step segmentation method and have evaluated it within the context of process reward models, supported by statistical analysis and extensive experiments. While RLHF methods are a reasonable evaluation approach, the heavy computational resources required for such experiments made it unfeasible for us to incorporate them, so we use the lightweight BoN and TVD method. As highlighted earlier, many impactful reward model studies have not employed RLHF experiments, yet they have been influential in the field.\n\n---\n\n**We greatly appreciate the your thoughtful feedback and your time for upgrading our work once again, and we hope that these clarifications will be satisfactory. We welcome any further constructive comments that can help improve the quality of our work.**", "claims_and_evidence": "The article claims that ASPRM is a SOTA PRM model in the introduction, but in the experiment, the ASPRM-M model is not as effective as Shepherd-M. In addition, the article does not show the effect of ASPRM in reinforcement learning training. In view of these circumstances, I think the conclusion that ASPRM is SOTA is problematic.", "methods_and_evaluation_criteria": "I think the proposed method make sense to me. Using confidence score as an indicator to identify difficult tasks and divide step based on it seems like a feasible solution to balance cost and informative of the annotated dataset.", "theoretical_claims": "This is primarily an experimental paper that focuses on empirical results rather than theoretical claims requiring formal proofs. Therefore, no formal proofs needed to be verified during my review.", "experimental_designs_or_analyses": "The paper lacks experiments and results on how PRM can improve the final performance of the model when used for reinforcement learning. In terms of experimental design, it is inappropriate to compare Llama based PRM with Mistral based PRM.", "relation_to_broader_scientific_literature": "### Innovation in Reasoning Step Segmentation\nExisting Process Reward Models (PRMs) typically use rule-based methods, such as predefined symbols or fixed-length reasoning steps, to segment reasoning steps. The AdaptiveStep method proposed in this paper automatically segments reasoning steps based on the model's confidence in predicting the next token. This aligns with the cognitive cost theory, which suggests that the cognitive cost of reasoning depends on task difficulty. Additionally, many reasoning errors stem from incorrect numerical calculations or misuse of words, further supporting the necessity of segmenting reasoning steps at critical points.\n\n### Improvement in Process Reward Models (PRMs)\nThe AdaptiveStep PRM (ASPRM) introduced in this paper demonstrates excellent performance in mathematical reasoning and code generation tasks, outperforming existing open-source PRMs. This is consistent with research emphasizing the importance of intermediate reasoning steps in complex tasks and demonstrating how step-by-step feedback enhances reasoning reliability and reduces logical errors.", "essential_references_not_discussed": "I believe essential references are discussed. However, it would be better to discuss in the related works part how this work differ from the previous works to help the reader gain a big picture of this work.", "other_strengths_and_weaknesses": "The weaknesses mainly on the experiment part. Results on more scenarios (performance lifting when leverage the proposed PRM on RL training), more based models (32B, 70B models) are needed to fully exhibit the effectiveness of the proposed method.", "other_comments_or_suggestions": "Regarding the presentation of TVD results across different methods, I suggest replacing the current large table containing many blank cells (where code models are not applicable to certain tasks) with a series of focused bar charts.", "questions_for_authors": "1. I noted that the RPM training and TVD methods described in this paper have similarities to previous work in the field. Could the authors clarify the specific novel contributions or advancements in these two areas? In particular, I would appreciate a more detailed explanation of how these approaches differ from or improve upon existing techniques, as this would help better position the work within the current literature and highlight its unique contributions.\n2. I appreciate the contribution of open-sourcing the LeetCode dataset, but I have some concerns about the ethical implications. Given that LeetCode problems are proprietary content, could the authors clarify the licensing arrangements or permissions obtained for redistributing this data? Additionally, it would be helpful to understand what steps were taken to ensure compliance with relevant terms of service and intellectual property rights when creating and sharing this dataset.\"", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5zRyK9iVBZ", "reviewer": "Reviewer_LC8A", "summary": "The paper addresses the challenge of training Process Reward Models (PRMs) for large language model reasoning by introducing a novel step segmentation method called AdaptiveStep. Instead of using fixed rules or token counts to break a model’s chain-of-thought into steps, AdaptiveStep dynamically segments the reasoning process based on the model’s own confidence in predicting the next token.\n\nThe authors sample solution paths from an LLM and compute the probability (confidence) for each generated token; tokens with unusually low confidence (below a learned threshold τ) are treated as decision boundaries, initiating a new reasoning step​. Using these segmented steps, the authors then train a PRM by simulating rollouts from each partial solution (each step) to see if a correct final answer can still be reached​.\nEach step is labeled positive if any continuation yields a correct answer, or negative if all continuations fail, following a heuristic “hard” reward assignment, similar to how PRM is usually trained. \n\nAdaptiveStep-augmented PRMs (termed ASPRM) are tested on complex reasoning tasks in mathematical problem solving (GSM8K, MATH dataset) and code generation (LeetCode-style programming problems, LiveCodeBench) and the approach achieves good results overall.\n\n## update after rebuttal\n\nI have given a weak accept to the paper and I wont mind seeing it get accepted.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer Comments\n\n**Dear Reviewer LC8A:**\n\nWe greatly appreciate your insightful comments and suggestions and thank you for your time. Due to the character limit, we have summarized your questions. If we have missed or misunderstood any points, please let us know.\n\n>## Related Work\n\nThank you for your valuable suggestions regarding related work. We will incorporate literature on reward models for LLMs in our revised manuscript, including BT model, GenRM from Zhang et al. (2024), Pairwise RM from Jiang et al. (2023), Cobbe et al. (2021), Li et al. (2022) and so on.\n\n\n>## Q1 and W1: Suboptimal Concerns\n\n**R1:** Thank you for expressing your concerns. We provide a [statistical results](https://anonymous.4open.science/r/PIC-0969/fig1.png) to show the relation between segmentation numbers and task difficulty. As shown in the figure, there are indeed some segmentation issues in the lower-left corner and on the right side, where few difficult problems are segmented into fewer steps, or certain easy problems are segmented into more steps. Although these cases are rare (fewer than 2%), the issue does exist. We believe that combining rule-based methods with AdaptiveStep would result in better divisions.\n\n>## Q2 and W2: Threshold Settings\n\n**R2:** Thank you for your insightful suggestion. We will include more percentage settings in our experimental evaluation to provide a more comprehensive analysis of our method's performance across various conditions. We used more models for generation to supplement our results. [Table 1](https://anonymous.4open.science/r/PIC-0969/fig4.png) and [Table 2](https://anonymous.4open.science/r/PIC-0969/fig5.png) shows the BoN results of ASPRM with threshold at 0.5%, 1.0%, and 1.5% compared with the baselines, and a 0.5% change means that each solution has one less step on average. Due to computational limitations, we are unable to scale up more rollouts in a short time, but we will conduct further analysis in subsequent versions, we hope for your understand. We find that although there is some fluctuation, a greater number of segments within the range of 0.5% to 2% generally means better evaluation capability.\n\n>## Q3, Q4 and W6: Implementation Details\n\n**R3:** Thank you for highlighting the missing methodological details. Regarding data preprocessing, we have provided training data specifications in Sec.4 (Parameter Setting), which documents 30 solutions per problem and 8 rollouts per step for labeling. In Sec.4.4 (Construction Efficiency), we present a comparison of computational costs with others.\n\nFor ASPRM training, we employed a batch size of 256 and learning rate of 1e-6 (Mistral), 2e-6 (Llama), 5e-6 (Deepseek), consistency with the Math-Shepherd parameters in the OpenRLHF script.\n\nFor baselines, we utilized the released models of other methods and rigorously adhered to their prescribed usage for evaluation on our dataset. At the 7B model scale, our reported Math-Shepherd BoN results exceed those in the original paper, demonstrating the fairness of our comparisons.\n\n>## Q5 and W4: OmegaPRM and RLHF\n\n**R4:** Thank you for raising these questions. OmegaPRM is a method that integrates binary search into MCTS to construct data, while ours is a fine-grained step segmentation method. The methods are orthogonal, and since this and data have not been released, we did not use it as a baseline for comparison. RLHF is indeed a great comparison approach, but the computational resources required for RLHF are currently beyond our capacity. Therefore, we opted for the more lightweight BoN and TVD methods for evaluation, which are commonly used in previous reward model works. \n\nRegarding the combination with these methods, we believe that fine-grained segmentation helps OmegaPRM more accurately identify errors, thereby improving the error detection efficiency. As for RLHF, positions with low confidence indicate that the model likely generate other branches. Compared to rule-based step-level RLHF, we believe that using AdaptiveStep allows the model to better learn from these branches, leading to good results. We will also validate these claims when sufficient computational resources are available.\n\n>## Q6 and W3: Commensense Domain\n\n**R5:** Utilize to commensense domains remains challenging for process reward models. Some researchers employ powerful models like GPT-4o or human judgment for evaluation [1]. We believe AdaptiveStep offers advantages in this context: our segmentation method identifies low-confidence model outputs, suggesting more choices and higher error probability, thereby improving labeling efficiency.\n\n**[1]** O1 Replication Journey: A Strategic Progress Report: Part I\n\n---\n\n**We sincerely thank you for your constructive feedback and insightful recommendations. These comments have significantly helped us identify areas for improvement in our manuscript. We look forward to incorporating more suggestions from you to enhance the quality and clarity of our research.**", "claims_and_evidence": "A lot of claims are made in the paper and are backed by empirical results.\n\n1. Dynamically segmenting reasoning steps by model confidence yields more “decision-making insights” per step than naive rule-based segmentation: This is supported qualitatively by examples and quantitatively by analysis of the segmentation output: they observe that AdaptiveStep tends to insert breaks at meaningful junctures (e.g. mid-formula or before a crucial choice in logic) rather than at arbitrary punctuation or fixed intervals​.\n\n2. The claim that no manual annotation is needed is right as the proposed method relies purely on the model’s probabilities and an automated rollout procedure. \n\nOther claims like lower cost and reduced number of samples are also verified in the paper.", "methods_and_evaluation_criteria": "The proposed method is very simple and using loprobs to estimate confidence is not new and has been explored a lot in the past. Applying this to PRM is unique but this approach makes intuitive sense – low confidence often indicates the model is choosing among multiple possibilities (e.g. figuring out the next step in a math proof or deciding on a coding approach), which is exactly where a new step and potentially a feedback signal would be most valuable.\n\nThe PRM training procedure – labeling each segmented step via rollouts – is a logical way to obtain supervision. It mirrors prior work (like Wang et al., 2024a’s heuristic rollout for Math-Shepherd) but importantly removes the need for manual identification of step boundaries, which is one of the main contributions. The training is the same as past works and the authors’ choice to use binary labels (hard estimation) for each step is reasonable and aligned with previous PRM approaches. \n\nThe evaluation criteria and benchmarks are appropriate and standard for the domain - math and code. In the code domain, since no established stepwise PRM existed publicly, they construct a baseline Outcome Reward Model (ORM) by training a reward model that only gives feedback at the final answer​. \n\nOverall, I would say that the benchmarks and metrics clearly align with the paper’s goals and is pretty standard.", "theoretical_claims": "The paper is mostly empirical and has no theoretical claims as such. \n\nOne interesting claim is the rationale behind the 2% threshold for low-confidence tokens. The authors justify setting the threshold so that roughly 2% of generated tokens are below it, citing Kahneman’s (2011) finding that about 2% of human thinking is “deep thinking”​. This is very interesting but hard to judge. The paper does not prove that 2% is the best choice; rather, it assumes this fraction yields a reasonable number of decision points. While the results with 2% are good, I would be interested in seeing a grid search over more possible values. \n\nFinally, the main claim of the paper is that “low model confidence indicates a potential decision point”. Intuitively, this claim is sound – if an LLM is uncertain about the next token, it likely means multiple continuations are plausible, implying a branch in reasoning. The paper’s empirical analysis supports this (confidence dips align with meaningful junctures), but theoretically one could question if low confidence always corresponds to an important decision. There might be cases where an LLM’s confidence is low simply because it’s generating an uncommon word or name, not because it’s at a logical decision point.  Therefore, this claim is plausible and supported by examples, but not theoretically guaranteed for all scenarios. \n\n\nOne theoretical aspect that might have been explored more is the calibration of model confidence. AdaptiveStep assumes the model’s token probability is a meaningful indicator of uncertainty. In theory, if a model’s confidence estimates are poorly calibrated, the threshold might not accurately reflect decision difficulty. The paper doesn’t delve into calibration theory or prove that the chosen LLMs have well-calibrated confidences in these domains. They proceed empirically, and indeed the approach works, implying the confidences were informative enough.", "experimental_designs_or_analyses": "The experiments are well designed and the authors clearly describe their experiment setup, including datasets, model choices, baselines, and metrics used to test the models. The baselines themselves are appropriate and the authors make an effort to ensure comparability. For math, they use published open-source PRMs (Math-Shepherd and ER-PRM). It’s a little unclear whether they recomputed those baselines’ performance on their setup or took numbers from the literature. Maybe I might have missed this in the paper. \n\nThe metrics and analysis of results are appropriate. They measure both Accuracy/Pass@1 (to see if the PRM hurts single-shot performance – it doesn’t; guided decoding often improves it) and Best-of-N accuracy (to demonstrate how well the PRM can identify a correct solution among many).\n\nOn thing missing in the paper is that the paper doesn’t report any significance tests. Improvements like +3% on GSM8K could be within error margin if not enough samples. I am not sure what % of improvement is good.", "relation_to_broader_scientific_literature": "The authors discuss the usefulness of PRMs and discuss prior works. There is a good coverage of recent literature on PRMs and stepwise reasoning alignment.\nOne interesting thing is that the authors used OpenAI cookbook ideas to assess logprobs, which is interesting. I wonder if the authors are the first to turn this idea into a full pipeline for training a reward model.", "essential_references_not_discussed": "I think the authors mentioned all PRM related past works. One area that wasn’t explicitly mentioned is the line of research on verifier models or consistency checks aside from PRMs. For example, works like Cobbe et al. (2021) or Li et al. (2022) on verifying chain-of-thought or Iterative Polishing might be related.", "other_strengths_and_weaknesses": "### Strengths\n- The core idea of using the model’s own confidence to segment reasoning steps is very clever. It addresses a clear limitation of prior methods, which relied on ad-hoc rules or costly annotations. \n- AdaptiveStep yields a more efficient data generation process for training the PRM. By segmenting only when needed, it produces far fewer total steps to label.\n- The experiments follow past works and results are provided on math and code. The authors provide insightful analysis of where the model places step breaks (e.g., highlighting that conjunctions and math operators often trigger low confidence​). \n\n### Weakness:\n- AdaptiveStep’s segmentation is only as good as the model’s confidence estimations. If an LLM has idiosyncrasies in its probability outputs (e.g., it might be overconfident in certain wrong steps or underconfident in trivial but rare phrasing), the segmentation could be suboptimal. \n- A notable weakness is the somewhat arbitrary choice of using the bottom 2% confidence as the segmentation threshold. While the authors cite a cognitive science justification, this parameter was not deeply examined. \n- [MINOR]: Interesting to see how the method would perform in domains without binary success criteria (e.g., open-ended logical reasoning or commonsense questions where “correctness” is fuzzy). The paper’s approach relies on having a ground truth check to label steps.\n- The baselines chosen were appropriate (other PRMs and an outcome model), but one could argue that the paper doesn’t compare against the strongest possible alternatives. For example, Reinforcement Learning from Human Feedback (RLHF) at the step level or even outcome level could improve reasoning – how would a model fine-tuned with RLHF on these tasks compare to using a PRM? Similarly, OmegaPRM (which uses MCTS and presumably a powerful orchestrator) is mentioned but not empirically compared.\n- [MINOR] A minor weakness is that some details of the method are not fully spelled out in the paper, potentially hindering replication. For example, the exact number of samples generated per question, the value of J (number of rollouts per step) used for labeling, or the training hyperparameters for the PRM model are not explicitly listed (at least not in the excerpt we see).", "other_comments_or_suggestions": "-", "questions_for_authors": "1. Your method assumes the model’s confidence (top-1 probability) is a reliable indicator of decision difficulty. Did you observe any cases where the model was overconfident in a wrong step (thus no break inserted) or underconfident in an easy step (inserting an unnecessary break)? In other words, how robust is AdaptiveStep to calibration errors in the base LLM’s probabilities?\n2. How sensitive are your results to the choice of the 2% confidence threshold for segmenting steps? Did you experiment with different percentages or adaptive thresholds per dataset/model? It would be useful to know if 2% is truly optimal or just one reasonable setting.\n3. Could you clarify the data generation process for PRM training in terms of scale? Specifically, how many solutions N did you sample per problem to determine the confidence distribution and threshold, and how many rollouts J per step were used to label each step? These numbers are important to understand the compute cost. \n4. For the math baselines (Math-Shepherd and ER-PRM), did you reimplement/retrain those methods on the same datasets and base models that you used for ASPRM, or are you citing their reported results? \n5. You cite OmegaPRM (which uses MCTS) and stepwise RLHF approaches. How do you expect AdaptiveStep to compare to those in practice? \n6. How would AdaptiveStep and PRM training apply to domains where checking correctness is non-trivial (for example, commonsense reasoning or legal question answering where there isn’t a single numeric answer or test cases)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1jJbJO3MdC", "reviewer": "Reviewer_NpJW", "summary": "This work proposes a method (Adaptive Step Process Reward Model - ASPRM) to split reasoning chains into reasoning steps based on the model confidence rather than pre-defined rules. For each token in the generated output, if the model’s probability for that token is below some threshold, then this token is treated as the start of a new reasoning step.\n\nThe authors train their ASPRM with rollouts and hard estimation of intermediate step values, and show that this adaptive step strategy yields higher-performing PRMs on math and coding benchmarks.\n\nThis work also experiments with token-level value guided decoding: during decoding, when the model generates a low-confidence token, the PRM can be used to rank the top M low-confidence tokens at that step. Experiments show that value-guided decoding performs best with ASPRM.\n\nEventually, multiple ablation studies are made in the experimental section to show the generalization capacity of the proposed approach.\n\n## update after rebuttal", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer Comments\n\n**Dear Reviewer NpJW:**\n\nWe would like to express our sincere gratitude for your time, your thorough review and valuable feedback on our manuscript. Your comments have provided us with important insights that will help improve the quality of our work.\n\n>## W1: Ablation Study on Threshold Decision\n\n**R1:** We appreciate your suggestion regarding the threshold decision process. Though our approach is grounded in cognitive theory, an ablation study is needed in the revised manuscript to demonstrate the impact of different thresholds. We add the BoN results of ASPRM models trained with threshold of 0.5%, 1%, and 1.5%. However, larger thresholds (3%, 5%, 10%) mean perform more rollouts. Due to computational resource constraints during rebuttal, we are unable to conduct extensive experiments with various models and higher threshold combinations, we will conduct further ablation studies when we have more computational resources available. Table 1 and Table 2 shows the BoN results of ASPRM trained with threshold at 0.5%, 1.0%, and 1.5%, compared with the baselines, and a 0.5% change means that each solution has one less step on average. We find that although there is some fluctuation, a greater number of segments within the range of 0.5% to 2% generally means better judging capability. We used more models (MetaMATH-Llama-7b / -13b - 70b) for generation to supplement the results. We hope for your understanding and approval due to the lack of additional results. \n\n**Table 1: Bo64 results on Math500.**\n\n| model | MetaMATH-Llama-7b | -13b | -70b |\n|:---:|:---:|:---:|:---:|\n| ASPRM-M-T0.5 | 25.00 | 28.80 | 32.60 |\n| ASPRM-M-T1.0 | 25.00 | 29.60 | 31.60 |\n| ASPRM-M-T1.5 | 27.80 | 28.80 | 31.40 |\n| ASPRM-M-T2.0 | 25.40 | 31.00 | 34.60 |\n| Math-shepherd | 28.40 | 31.00 | 33.00 |\n| ASPRM-L-T0.5 | 31.80 | 35.2 | 37.00 |\n| ASPRM-L-T1.0 | 31.60 | 34.80 | 36.20 |\n| ASPRM-L-T1.5 | 32.00 | 35.60 | 38.60 |\n| ASPRM-L-T2.0 | 33.40 | 37.80 | 40.00 |\n| ER-PRM | 33.20 | 37.40 | 38.80 |\n\n**Table 2: Bo64 results on GSM8k.**\n\n| model | MetaMATH-Llama-7b | -13b | -70b |\n|:---:|:---:|:---:|:---:|\n| ASPRM-M-T0.5 | 79.45 | 83.32 | 85.82 |\n| ASPRM-M-T1.0 | 81.04 | 83.55 | 88.48 |\n| ASPRM-M-T1.5 | 81.65 | 82.64 | 89.61 |\n| ASPRM-M-T2.0 | 81.27 | 85.80 | 89.23 |\n| Math-shepherd | 84.23 | 85.22 | 88.40 |\n| ASPRM-L-T0.5 | 86.84 | 86.85 | 91.38 |\n| ASPRM-L-T1.0 | 89.60 | 86.58 | 91.74 |\n| ASPRM-L-T1.5 | 86.20 | 90.00 | 90.02 |\n| ASPRM-L-T2.0 | 85.52 | 88.25 | 91.66 |\n| ER-PRM | 86.58 | 87.49 | 88.86 |\n\nBesides, we also investigated the influence of various thresholds on the performance of TVD. The results have been collated and presented in the following two figures: [Figure 1](https://anonymous.4open.science/r/PIC-0969/fig2.png) and [Figure 2](https://anonymous.4open.science/r/PIC-0969/fig3.png).\n\n>## W2 : Model Influence of Confidence Threshold\n\n**R2:** We use the pecentage of whole distribution of model confidence, instead of a fixed threshold, this means that different models will indeed have different threshold values, and different tasks will also have different threshold values. We use a table to illustrate this, the following Table 3 below shows the values at a 2% confidence threshold for different models and tasks. We find that for the same split ratio, models with greater capabilities and simpler tasks tend to have higher threshold values.\n\n**Table 3: 2% confidence threshold for different models and tasks.**\n\n| dataset | MATH500 | GSM8k | \n|:---:|:---:|:---:|\n| MetaMATH-7b | 25.51 | 34.26 |\n| 13b | 25.78 | 35.91 |\n| 70b | 33.41 | 39.51 |    \n\n\n\n>## W3 and T1: Figure, Table and Typographical Errors\n\n**R3:** Thank you for highlighting the issues with the table, the up and down arrows' color and the typographical errors in our manuscript. We will correct these inaccuracies in the revised version to improve clarity and readability.\n\n---\n\n**Thank you again for your time, thorough review and constructive feedback. Your insights have significantly helped us identify areas for improvement in our work. We welcome any additional suggestions that could further enhance the quality and rigor of our research.**", "claims_and_evidence": "Yes, claims are supported by convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed methods and benchmarks (Math500 & GSM8k & LeetCode problems) make sense for introducing a new type of PRM", "theoretical_claims": "No theoretical claims found.", "experimental_designs_or_analyses": "Experiments and their analysis seems reasonable and valid. A lot of experiments have been made which is nice. It can get a little complicated to follow sometimes because in PRMs there are 3 models interacting with one another:\n1. the model that generates trajectories and that is used with rollouts to generate data of the form `(partial trajectory, target score)`.\n2. the actual PRM that is trained on the data generated in (1).\n3. a policy model that is generating trajectories with the help of the PRM at inference time.\n\nFor each of these models the authors experimented with Mistral & Llama for math benchmarks, and deepspeed for LeetCode problems.", "relation_to_broader_scientific_literature": "Process Reward Models (PRM) became a popular research topic recently because of their advantages over Outcome Reward Models (ORM) in giving intermediate feedback while training LLMs. This particularly became important as modern LLMs output reasoning chains before their final answer. Cheking the validity of each reasoning step can boost their performance. The idea of splitting reasoning chains based on model confidence (rather than on every new line character which is the default strategy) is novel to the best of my knowledge and seems to perform well. This shows once again that letting the model decide is better than imposing human judgement.", "essential_references_not_discussed": "No critical related work missing to the best of my knowledge.", "other_strengths_and_weaknesses": "# Strengths\n\nThis paper presents a novel strategy to split reasoning chains into reasoning steps in the goal of training Process Reward Models. Throught numerous experiments the proposed method is shown to perform better than more traditional PRMs.\nIn addition, the method is tested as a value-guided decoding tool, which is also shown to perform well.\nEventually, numerous generalisation experiments and analysis is being presented at the end of the paper, showing rigorous evaluation. \n\n# Weaknesses\n\nOne weakness of the proposed approach is that we need to define a confidence threshold to define “low” confidence steps and thus break the reasoning chain into steps. The decision of this threshold could influence the performance of the proposed method. The authors set the threshold such that 2% of reasoning tokens would fit under. This feels arbitrary and could benefit from further motivation.\n\nIn addition, an ablation study on the decision of the threshold could also benefit this paper. How does 2% compares to 1%, 5%, 10%, 20%, etc…\n\nEventually, the model used to estimate the confidence could also influence this decision: maybe some models are more confident than others and their threshold should be higher. An informed discussion with experiments could boost the quality of this work.", "other_comments_or_suggestions": "Clarification suggestions:\n\n- In tables 2, 4, 5, 6 it would be clearer to put the performance of the method you compare against directly in the caption (or as an additional table row) so that the reader doesn’t have to search where this number is in another result table.\n- The up & down arrows are nice (though the colors could be inverted: red for decline & green for improvement).\n\nTypo:\n- in the last paragraph of section 4.4, “_ results in 4.3 and 4.3 indicate …_”", "questions_for_authors": "No additional questions for now.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper propose a simple method for training a process reward model (PRM), deemed AdaptiveStep. In the proposed method, the intermediate steps are determined based on the confidence of the model using a threshold that has been chosen to force 2% of the reasoning steps to become break points. This is in contrast to prior work that uses heuristics such as line breaks as their break points. The paper shows that the proposed PRM achieves better performance compared to existing methods. The experimental section of the paper as well as the interpretation of the results seems solid and appreciated by the reviewers as well. During rebuttal, reviewers asked for further ablations and evidences that were successfully provided by the authors. There are some concerns about the claims where the proposed method does not always lead to better performance compared to other methods. I think the authors have done a good job trying to understand and explain these issues. I think it would be better if the authors also tone down the claims in abstract and introduction a bit to reflect the results. There is also concerns about lack of empirical evidence in using the new PRMs to train RL; I agree with authors that using the new models with BON to see the scaling is sufficient, especially for this type of rewards and given that BON is already a very strong baseline for RLHF problem (Beirami et al 2024). Overall, I think this is a solid paper that should be accepted to the conference. Congratulations to the authors!

Beirami, Ahmad, et al. "Theoretical guarantees on the best-of-n alignment policy." arXiv preprint arXiv:2401.01879 (2024).
Decision: Accept (poster)