AutoPSV: Automated Process-Supervised Verifier
Authors: Jianqiao Lu
Zhiyang Dou
Hongru WANG
Zeyu Cao
Jianbo Dai
Yunlong Feng
Zhijiang Guo
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
Math Reasoning
Commonsense Reasoning
Automatic Process Annotation
Multi-step Reasoning
Primary area: natural_language_processing
TL;DR: AutoPSV enhances the reasoning capabilities of LLMs by automatically generating process annotations and confidence scores for reasoning steps, significantly improving performance in tasks involving complex reasoning.

Abstract: In this work, we propose a novel method named \textbf{Auto}mated \textbf{P}rocess-\textbf{S}upervised \textbf{V}erifier (\textbf{\textsc{AutoPSV}}) to enhance the reasoning capabilities of large language models (LLMs) by automatically annotating the reasoning steps.
\textsc{AutoPSV} begins by training a verification model on the correctness of final answers, enabling it to generate automatic process annotations. 
This verification model assigns a confidence score to each reasoning step, indicating the probability of arriving at the correct final answer from that point onward.
We detect relative changes in the verification's confidence scores across reasoning steps to automatically annotate the reasoning process, enabling error detection even in scenarios where ground truth answers are unavailable. 
This alleviates the need for numerous manual annotations or the high computational costs associated with model-induced annotation approaches.
We experimentally validate that the step-level confidence changes learned by the verification model trained on the final answer correctness can effectively identify errors in the reasoning steps.
We demonstrate that the verification model, when trained on process annotations generated by \textsc{AutoPSV}, exhibits improved performance in selecting correct answers from multiple LLM-generated outputs.
Notably, we achieve substantial improvements across five datasets in mathematics and commonsense reasoning.  The source code of \textsc{AutoPSV} is available at \url{https://github.com/rookie-joe/AutoPSV}.

AutoPSV:
Automated Process-Supervised Verifier
Jianqiao Lu', Zhiyang Dou', Hongru Wang", Zeyu Cao², Jianbo Dait, Yingjia Wan', Yunlong Feng", Zhijiang Guo 'he University of Hong Kong The Chinese University of Hong Kong University of Cambridge
*University of Edinburgh SIndependent
jqluêes hk, zg2830cam.ack
Abstract
In this work, propose novel method named Automated Process-Superviser Verifier (AUToPSV) to enhance the reasoning capabilities of large language mod- els (LLMs) by automatically annotating the reasoning steps. AUTOPSV begins by training verification model on the correctness of final answers, enabling it to gen- erate automatic process annotations. This verification model assigns confidence score to each reasoning step, indicating the probability of arriving at the correct final answer from that point onward. We detect relative changes in the verification's 's confidence scores across reasoning steps to automatically annotate the reasoning process, enabling еттог detection even scenarios where ground truth answers are unavailable. This alleviates the need for numerous manual annotations or the high computational costs associated with model-induced annotation approaches. We experimentally validate that the step-level confidence changes learned by the verification model trained on the final answer correctness can effectively iden- tify errors in the reasoning steps. We demonstrate that the verification model, when trained on process annotations generated by AUTOPSV, exhibits improved performance in selecting correct answers from multiple LLM- -LM-gennrated outputs. Notably, we achieve substantial improvements across five datasets in mathemat- ics and commonsense reasoning. The source code of AUTOPSV is available at attps://gitae htt.tttttttt/..///////gg wwwwroooieeeooeeeeooooooooooeeee /atrooooiejjjeeaeeeee AutooPV..
Introduction
Large language models (LLMs) have shown impressive performance various reasoning tasks [1- 4]. Prior efforts primarily focus on specific prompting techniques, such as few-shot prompting with intermediate steps and augmented demonstrations [5-8]. While these methods have shown promise, their effectiveness is often task-specific, and designing prompts can be labor-intensive, leading to inconsistent results [9, 10]. Another approach improve reasoning in LLMs is through instruction tuning or knowledge distillation [11-14]. These methods typically involve fine-tuning LLMs and require a large set of examples annotated with of-in-of-thoughts (CoT; [5]). However, these approaches can be eesource-intensive and may not always produce reliable results.
To address these challenges, verification techniques have emerged promising solution [15, 16]. Verification models are trained to evaluate and potentially correct the reasoning process generated by LLMs. This approach aims mitigate the risk of relying solely the top-1 result, which may not always be reliable [17, 18]. By reranking candidate responses, verification models can ensure higher accuracy and consistency in LLM outputs, and provide valuable feedback for improving LLMs [19, 20] further.
!Coresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Verification models generally fall into two training paradigms: outcome supervision and process supervision. In outcome supervision, the training annotations rely on the correctness of the final answer [21, 22], while in process supervision, the annotations are based on evaluations of each reasoning step [23, 19]. However, process supervision is demanding in terms of annotations. Typically, it relies on either expensive and highly skilled human evaluators [23, 16] or model-induced process annotations [18. 17] to estimate the future correctness of the current reasoning step using Monte Carlo tree search [24, 25]. In contrast, outcome supervision only requires annotations for the output, making more economical in terms of annotation effort but less effective. That being said when answers involve multiple reasoning paths, all aforementioned model-induced methods require numerous samples to ensure accurate estimations.
In this paper, we introduce Automated Process-Supervisee Verfier (AuToPSV), a novel approach that synergistically combines the strengths of both process supervision and output supervision. Our method begins by training an srrccmme-supervisee verification model using outcome supervision annotations. This model then assigns confidence scores to each intermediate reasoning step, estimating their likelihood of contributing to correct final answer. A distinguishing feature of AUTOPSV is its ability to automatically generate process annotations through relative step-level confidence change analysis, significantly reducing annotation effort while maintaining supervision quality without requiring ground truth answers. These automatically generated process annotations subsequently serve as training data for developing enhanced verification model that leverages both process-level and outcome-level supervision signals. The complete framework of AUTOPSV is illustrated in Figure 1. We conduct extensive experiments across five datasets, including mathematical reasoning benchmarks and commonsense reasoning tasks. The results demonstrate that our method effectively improves the reasoning capability of the model with our highly efficient labeling scheme for process supervision. Our contribution is summarized as follows:
We introduce AUTOPSV to automate the labeling of process data to enhance LLMs' reason- ing capabilities. By combining the strengths of output and process supervision, AUTOPSV effectively identifies variations in model confidence to annotate the correctness of intermedi- ate reasoning steps, enabling efficient automatic labeling for process supervision.
Comprehensive experiments demonstrate that AUTOPSV significantly improves the perfor- mance and scalability of verification models in mathematical and commonsense reasoning tasks. This approach greatly reduces the need for manual intervention and extensive compu- tational resources, making it a valuable tool for enhancing LLM capabilities.
AUToPSV's 's versatility is evident in its applicability to both labeled and unlabeled dataset settings after completing the training process. This flexibility and generalizability highlight the method's potential for widespread adoption in various LLM applications.
2 Related Works
To enhance the reasoning capabilities of LLMMs, prior research primarily focuses on specific prompting techniques [26]. Existing efforts include few-shot prompting with intermediate steps augmented demonstrations [5-7, 27] or zero-shot prompting with specific instructions [28, 29]. Although these methods have shown promising results, their effectiveness is often constrained by their task-specific nature and the labour-intensive process of designing prompts, leading to inconsistent outcomes across different tasks [9, 10]. Another strategy to facilitate reasoning involves instruction tuning or knowledge distillation, which elicits reasoning paths from LLMs without explicit prompting [1 [11-13, 30]. These approaches typically involve resource- intensive fine-tuning over LLMs and require a large set of examples annotated with chain-of siiin-ff-hhooo thoughts (CoT). Unlike methods that directly modify parameters or prompts, AUTOPSV focuses on training an additional verification model to select the desired output from the original model's 's output. This approach is further discussed in the context of process supervision in the following paragraph.
Improving Reasoning Abilities of LLMs
From Outcome to Process Supervision Recent efforts have focused on enhancing the reasoning capabilities of LLMs through the use of verifiers to select the best answer from multiple candidates. There are two main types of verifiers: the Siicome-SSuprrisee Verifier (OSV) and the Process- Supervised Verifier (PSV). The OSV is supervised with a signal based on the final answer [21, 22]. while the PSV is with detailed feedback which requires evaluating individual reasoning steps [15,
2

---

Problem:
money, ingrodlyy
Annotation
via Ground Truth
via Confidence Variation
Solution
Solution Stops:
Steps
Steps
(variation%)
Stap
Step
Step
0.27
0.35
This leaves her with 5x SSit Sholt Since Stucii - S7M, see cas solve for
2
Step
(+ (+-0.30)
0.08
Step 3
Step
Step
(-0.77)
Pentrome-Superseen Verifier
Precess- Supervised Verifier Supersiiio
Outcome Supervisio
Figure An overview of AUTOPSV. It utilizes an outcome-supervised verifier to automatically generate process annotations for each reasoning step by detecting its own confidence variations, without relying on ground truth annotations. AUTOPSV efficiently produces annotations serving as process supervision during the LLM training, which sidesteps costly annotations.
19, 16, 23]. Despite the time-consuming annotation cost, the PSV offers several advantages that make it preferable the OSV. It can provide fine-grained feedback by pinpointing the location of errors, which is valuable for reinforcement learning and automatic correction [16, 20J. To alleviate the extensive human annotation, recent approaches [17, 18] propose a machine annotation framework using Monte Carlo Tree Search [24, 25]. This process demands a lot of computing resources, potentially imposing a limitation on the usage. AUTOPSV is more efficient, supervised verification model to assign confidence to each reasoning step and calculate relative step-level confidence changes, eliminating the need for additional sampling or manual labeling.
utilizes an outcome-
AUTOPSV
In this section, we introduce the main problem that this paper focuses on in Section 3.1. We then discuss the motivation behind why we believe it is necessary to train a verification model in Section 3.2. Finally, describe how we accomplish the transition from outcome supervision process supervision during the training of the verification model in Section 3.3.
3.1 Problem Setting
Objective Our research addresses the challenge of response selection from multiple candidates generated by Large Language Model (LLM). Specifically, given an LLM acting as a response generator, we seek to develop an effective method for identifying the correct response among multiple generated solutions. Our primary goal is to maximize the probability of selecting an accurate solution from the available candidates.
Notation We define the input question presented to the model, and s(1:t) as the sequence of intermediate reasoning steps up to step for the i-th solution to question The binary correctness label for the i-th solution is denoted yt, and represents training question dataset. This formalization enables us to systematically approach the response selection problem while maintaining mathematical rigor in our methodology.
3.2 Motivation
Response selection methods can be broadly divided into two categories: models specifically fine-tuned for the selection task and those that employ various prompting strategies.
In our exploration, we initially investigate whether existing open-source L.LMs could serve as effective selection agents to evaluate model outputs and choose the correct response without fine-tuning. We
3

---

choose Dixtral-Instruce [31] as the response generator, and its response results are listed Table 1. Our objective focuses on identifying the correct response from five candidate solutions.
To establish robust conclusions, we evaluate selector models ranging from 7B to over 70B parameters, applying various prompting strategies. The results, tested the GSM8K test set [21], presented in Table 2. Notably, even models exceeding 70 billion parameters demonstrate suboptimal selection performance when relying solely on prompting without fine-tuning.
Table I: Performance of Mixtral-Instruce GSM8K. All results are reported in accuracy (%).
Response Generator
Model Size (Parameters)
Pass 1 (%)
Pass@5 (%)
Self-Consistency (%)
Mixtral-Instruct [31]
(MOE)
62.55
82.31
69.06
Table 2: Comparison of different selection methods across various model sizes for selecting a response from candidate responses generated by Mixtral-Instruut. All results are reported in accuracy (%)
Prompt Strategy
Selector
Model Size
Pairwise
Classification
Classification CoT
Scoring
Scoring CoT
Mistral-Instruct [32]
7B
60.73
61.18
64.82
61.49
69.75
Mixtral-Instruct [31]
8x7B
58.83
59.14
67.40
61.79
65.58
Llama2-chat [33]
70B
62.70
59.74
62,93
Qwen [34]
72B
59.14
66.64
69.52
61.86
65.88
Based on these findings, our research focuses on training dedicated verification models and enhancing their response selection capabilities.
3.3 Training Methodology
AUTOPSV integrates outcome supervision with process supervision create an effective training methodology. Below, we detail each component of our approach.
Outcome-Superee Suppeviiin We begin by training outcome-supervised verification (OSV) model, denoted as fe(-), where 0 represents the optimized parameters. The training utilizes Mean Squared Error (MSE) loss for each solution step:
C(S (1:1)
Ur:g) = (fo(s!l:)))) ys
(1)
The complete objective function across all training questions Q is:
Liotal (2) =
Σ QEQ
(fo(s (f) :))
Where represents solutions per question and m, denotes steps in the i-th solution. The OSV output approximates the expected correctness probability, as formalized as follows:
Theorem For a model trained with outcome supervision, fo. characterized by optimally tuned parameters 0, the assigned score for the sequence S(1t) is an estimation of the likelihood of ultimately deriving correct answer, denoted by based on the progression observed in ((sst) and the pertinent question q. This is mathematically represented as:
(1:t():);;)) ((a))(1:)) .q)
The proof follows from optimizing the MSE loss in equation (2), with details in [22].
Process-Suue Superss-S
We compute the relative confidence change between steps:
L.oon (0($)==11); (l(()):));))
(3)
(o(S(1:);

---

A conss represents the relative variation in the model's confidence score from step t to step t - 1. A negative value of yon signifies reduced confidence in achieving correct answer after incorporating information from the (t + 1)-th step. We denote the process label for the t-th step as y! and 0 as the variation threshold.
For process labeling, we employ the "first error location" strategy [15] with threshold 0:
.If con > 0: yf =
con
Otherwise: yf = 0 and Vi' > 4, yf = 0
The process-supervision loss function is:
( .2:0)) (0)
-1)-
(4)
Preliminary Findings
In this section, present findings aiming to validate two key aspects: In Section 4.1, present a comprehensive analysis of the OSV model, i.e., to validate that the initially trained OSV model is effective and robust. In Section 4.2, further introduce self-designed benchmark for process errors and calculate o to detect these errors, i.e., to demonstrate the effectiveness and reliability of relative step-level confidence change in the proposed method. The validation of these two components serves foundation for automatic process labeling via AUTOPSV, described in Section 3.3.
4.1 Experiment Outcome-Supervisee Verifier Performance
In this section, we validate the effectiveness and scalability of the OSV model. Initially, we fine-tune pretrained language model using ground truth data from the GSM8K dataset. Then, we use this fine-tuned model to generate multiple response samples for the GSM8K training prompts. We label these samples based the correctness of their final answers. After this, we train an OSV model using the method described in Eq. (I).
To evaluate the OSV, we measure its ability to select sample with the correct final answer from samples generated by various LLMs, denoted as the Response Generator. Specifically, our task involves selecting the correct candidate from five responses. We assess the effectiveness of outcome supervision on two models: Phi2 (OSV (Phi)) [35] and Mistral-7B (OSV (Mistral)) [32]. To explore the scalability of this outcome- suteoie-supervvsed verifier effect, we choose Response Generators of varying scales, ranging from 7B to 72B parameters, i.e., Mistral-7B-Instreu
(Mistral-Instruett) [32], Mixtral
8 7B (Mixtral-Instruet) [31] and Qwen-72B Twen-72B-Chat (Qwen) [34]. This allows to check the OSV's generalized ranking capability across different LLM scales.
Table 3: Performance of OSV models across different configurations.
Response Generator
Pass@1
P
SC
OSV (Mistral)
OSV (Phi)
Mistral Instruct
42.08
69.90
50.03
60.72
52.61
Mintral-lnstruct
62.55
82.31
69.06
74.07
69.37
Qwen
77.03
91.13
81.27
85.00
84.
The results demonstrate the effectiveness and scalability of the OSV model in selecting the correct response among multiple responses generated by different generators. Specifically, the OSV models, trained using either Mistral or Phi, consistently outperform the self-consistency (SC) baseline across all generator configurations. The results validate the effectiveness of the OSV model in enhancing model selection strategies, particularly when applied to larger and more accurate LLM generators.
We further analyze the performance discrepancy between the two OSV models:
The performance disparity among the verifiers can be attributed primarily to variations model sizes and the quality of their training data. It's important to note that the OSV model is continuously trained from the GSM8K fine-tuned model with the
Performance Analysis of Different OSVs
5

---

addition of a value head. This means that the training data for each OSV model is generated from its corresponding tine-tuned base model. Specifically, the training data for OSV (Mistral) is generated from the fine-tuned Mistral model, while the training data for OSV (Phi) is generated from the fine-tuned Phi model.
Table 4: Model sizes and training data accuracy for training OSVs.
Training Data
Verifier
Size
Quality (acc.%)
Quantity (per question)
OSV (Mistral)
7B
0.9914
100
OSV (Phi)
2.7B
0.9605
100
Table 4 presents consolidated view of the model sizes along with the precision metries of their outcome supervision training data.
It is worth noting that while both models are trained on the same quantity of data per question (100 samples), the quality this data differs due to the capabilities of their respective base models. The Mistral model, being larger and potentially more capable, generates higher quality training data for its OSV, which turn leads to better performance. In our content, we select the OSV (Mistral) model as the OSV model among other experiment settings due to its superior performance, as demonstrated Table 3.
4.2 Detecting Calculation Error During Math Reasoning
In this section, we verify the effectiveness and reliability of our method AUTOPSV, Specifically, We calculate o to identify inaccuracies in the process, as outlined in Eq. (3).
In Section 4.2.1, we introduce the concept of math calculation error and establish a preliminary benchmark. In Section 4.2.2, we assess the performance of calculating 2 to detect calculation errors against our established benchmark.
4.2.1 Math Calculation Error
We outline a method for identifying instances of math calculation error, which we define occur- rences where the numerical values on either side of an equals sign within mathematical expression do not align. This misalignment indicates breakdown in logical reasoning, categorizing the instance as a calculation error in the context of mathematical problem-solving. This process establishes benchmark for math calculation error detection with more details in Appendix E.
Math Calculation Error Detection To identify calculation errors in mathematical reasoning, we monitor the relative step-level confidence changes between the intermediate steps defined in Eq. (3). A Lunn < 0, we view the step as "incorrect". We also provide a detection example in Figure 2 for better understanding.
4.2.2 Quantitative Results
We introduce three metrics for thorough evaluation of math calculation error detection: Precision (Prec.), which calculates the proportion of samples with correct final answers but exhibiting hallu- cinatory errors during the reasoning process; Recall, which determines the proportion of samples with math calculation emrors that the OSV model successfully identifies through step-level confidence changes; Fi-score, which gauges the verifier's overall efficacy. In Table 5, we explore how different threshold (0) values affect the precision, recall, and Fl-score for math calculation error detection.
The results in Table 5 demonstrate that our method using step-level confidence change effectively detects calculation errors across threshold values from - 0.5 to - 0.9. As the threshold becomes more negative (stricter for labeling errors), the precision increases, indicating higher precision identifying true errors. However, the recall decreases, meaning fewer actual errors are caught. Importantly, the F1- score, balaneing precision and recall, remains relatively stable across thresholds. This demonstrates that our method strikes good balance between detecting real errors and minimizing incorrect

---

Table 5: Process Calculation Error Detection Performance with Varying Threshold (0) Values.
Threshold (0) Value
Metric
0.5
0.6
0.7
0.8
0.9
Prec. 0.85
0.88
0.91
0.93
0.94
Recall
0.90
0.89
0.86
0.83
0.80
F1-Score
0.88
0.89
0.88
0.88
0.86
flagging of valid calculations. Overall, our detection method is effective and robust, performing well over range of thresholds without significantly compromising overall detection quality.
We note that setting 0 0.5 in our detection methods helps maintain balance between precision and recall, which can ensure balanced distribution of labeled "incorrect" and "correct" responses.
Validation and Foundation for AUTOPSV Our empirical validation of the OSV model encom- passes two key aspects: its efficacy in response selection (Section 4.1) and its capability in detecting calculation errors (Section 4.2). These experimental results provide robust foundation for au- tomating process annotations using AUTOPSV. Furthermore, Theorem establishes the theoretical framework for utilizing OSV to estimate the probability of reaching correct final answers from any given intermediate reasoning step. This convergence of theoretical guarantees and empirical evidence provides the methodological groundwork for applying AUTOPSV to generate process-supervee training data in our subsequent experiments.
5 Experiment
In this section, we first introduce the experimental setup a subsection, which includes the response generator LLMs and evaluation settings in Section 5.1. We then present the main result of process supervision-enhanced verification model on both mathematical and commonsense reasoning benchmarks, as described in Section 5.2.
5.1 Experimental Setup
We selected three instruction-tume LLMs of varying sizes, ranging from 7 billion to over 70 billion parameters, serve the response genenator. Specifically, we used Mistral- Instrat-Instruct-71 (Mistral-Instruct), Saeiral--xxB--nstuucte (Mixtral-Instruct), and Qwen-72B-Che (Qwen).
Models:
Our evaluation encompasses five benchmarks across two domains: For mathematical rea- soning, we include GSM8K [21], containing math word problems requiring multi-step reasoning, and MATH [36], composed of high school-level competition problems covering a range of math subjects. For commonsense reasoning. we use HellaSwag [37], a dataset for physically situated commonsense reasoning, Winogrande [38], fill-nn-hheblland problems requiring commonsense pronoun resolution and ANLI [39], a dataset for natural language understanding and reasoning.
Datasets:
For evaluation, follow the methodology outlined in [40] ensure consistency across benchmarks. Our protocol involves:
Evaluation:
(i) Our object is to select the correct answer from five candidate responses.
(ii) For OSV models, use the final value assigned the whole solution for evaluation, while for PSV models, we utilize the product of step-level scores as the aggregation function.
(iii) To obtain more reliable pass @k results, implement the estimation method described in [41]. This involves generating n samples per task (where > k) and assessing the number of correct samples that pass unit tests. We then calculate the unbiased estimator for pass@k. k. For self-consistency (Self- Cons.) and verifier results, we randomly select k out samples and perform separate calculations. All results reported with accuracy of +0.1 95% confidence level.
Additional details regarding generation hyperparameters and different aggregation functions are provided in Appendix F.2.

---

5.2 Enhanced LLMs Reasoning via Process Supervision
To evaluate the efficacy and scalability of our proposed approach (detailed in Section 5.2), we conducted comprehensive experiments across mathematics and commonsense reasoning tasks using five diverse datasets.
Our experimental framework employs an autonomous process-Supervision data annotation method where we calculate Denn based on model confidence from OSV, using a threshold of 0 = -0.5. This annotated data is then utilized to continuously fine-tune the OSV model, resulting in our enhanced OSV PSV model.
We evaluate three distinet approaches: (i) Coff-CCnstteecy (Self-Cons.): Our baseline approach (ïi) Outcome-supervisee verifier (OSV): Our initial verification model (ii) Process spceess-uperrvied en- hanced verifier (OSV + PSV): Our proposed enhancement. For each approach, we assess Pass @ performance, which represents the upper limit of achievable performance on these benchmarks.
Mathematies Reasoning: As shown in Table 6, the process-supervised enhanced verifier demon- strates superior performance over the outcome-supervise verifier and Self-Coonistency models for all evaluated response generators GSM8K. For the MATH benchmark, the process-supervised enhanced verifier outperforms the other two approaches for Mistral-Instruct and Mixtral-Instruce but it is slightly less effective than the Self-Consistency model when applied to Qwen-72h.
Table 6: Results on mathematics benchmarks.
GSM8K
MATH
Response Generator
P
Self-Cons.
OSV
OSV PSV
P
Self-Cons.
OSV
OSV + PSV
Mistral-Instruct
69.90
50.03
61.18
61.41
7.7
1.64
5.10
5.30
Mixtral-Instrue 82.30
69.06
74.91
76.04
22.80
10.66
15.2
16.92
Qwen 91.13
81.27
84.91
85.15
56.10
40.10
38.94
39.36
Commonsense Reasoning: According to Table 7, OSV PSV again leads to the best results among the three methods for each response generator tested on HellaSwag. For Winogrande, Mistral-Instruct paired with OSV PSV achieves the highest performance, whereas, for Mixtral-Instruct and Qwen- 72b, the original OSV without process supervision has a marginal advantage. When looking at the results of the ANLI benchmark, OSV + PSV is the fighest-performing method for Mistral-Instruc Despite this, for Qwen-72b, the OSV model alone falls slightly behind the integrated OSV PSV.
and
Table 7: Results commonsense reasoning benchmarks.
Respouse Generator
Pasuc 5
Setr-Cas:
HellaSwag
OSV
OSV PSV
Pi
Winogrande Self-Coos.
OSV
QSV PSV
PreeR
Self-Cons.
ANLI
OSV
0SV PSV
Mistral-Innn Instruct
76.84
40.30
73.81
74.45
91.16
58.64
79.16
79.98
73.4
45.6
59.8
59,3
Mixtral Instuct
86.05
73.67
82.83
83.62
68.75
68.4
64.0
Qwen 72b
95.28
85.44
93.08
93.99
38.63
12.21
$0.34
79.32
82.4
71.4
Conclusion: Our experimental results indicate that the process--upervised enhanced verifier (OSV PSV) consistently outperforms matches the baseline models across mathematical and commonsense reasoning tasks. By leveraging automatic process annotations, our approach enhances the model's 's capacity verify reasoning processes, resulting in improved accuracy and robustness across a wide range of benchmarks and response generators.
Analysis
In Section 6.1,, we compare our process annotation method, AUTOPSV with two other model- induced annotation methods to showcase the effectiveness and efficiency our proposed approach. In Section 6.2, we validate the data quality constructed via AUTOPSV as described in Section 5.2.

---

6.1 Advantages of AutoPSV in Labeled and Unlabeled Settings
Aside from the labeling method defined by Eq. (3) in AUTOPSV, another labeling strategy is the Monte Carlo Tree sampling estimation (MCTS), as described in [17, 18]. To better demonstrate the effect of our method, we make a comparison with this approach and conduct experiments on mathematical benchmarks across both labeled and unlabeled settings (i.e., whether the ground-truth answers to the questions are provided).
Performance in Labeled Settings We first evaluate the performance of AutoPSV in labeled settings. We follow the experimental settings described [17, 18] to ensure a fair comparison. More implementation details are provided in Appendix F.3. Table 8 presents a comparison of process labeling methods across different response generators on the GSM8K and MATH datasets.
Table 8: Comparison of process labeling methods' performance across different response generators on GSM8K and MATH datasets. The table evaluates the @ 5, Colf-Consistency (Self-Cons.), and response selection performance of models fine-tuned using process annotations labeled by MCTS and AUTOPSV.
Response Generator
P
Self-Cons.
GSMBK Process (MCTS)
Process (AUTOPSV)
PaseS
Self-Cons.
MATH Process (MCTS)
Process (AUTOPSV)
Inntal-Inntrcctt Mixtral-Instten Instrrct
1.64
72.12
Qwen-72h
91.13
81.27
87.17
82.83
56,10
40.10
36.88
37,10
The experimental results shown in Table 8 suggest that proposed method for process labeling, which relies on detecting changes in model confidence, performs competitively with the MCTS method from [18, 17]. In some cases, our method even outperforms the MCTS method, especially the more challenging MATH benchmark.
A key advantage of AutoPSV is its computational efficiency. As shown in Table 9, our method requires significantly fewer tokens for process labeling compared to MCTS-based methods.
Table 9: Comparison of annotation costs between MCTS and AUTOPSV for process labeling on the GSMSK and MATH datasets. Annotation cost represents the processed tokens into a model when generating process annotations, encompassing both input and output tokens.
Dataset
#Questions
ASteps(Ang.)
#Seeall(
ASolution Statistical
(oolees(AAgg))
(vveels(Overall)
Process (MCTS) Process (AUTOPSV)
Annotation Cost
GSMBK
7,473
4.47
334,358
126
9,379.258
2,808
127
MATH
7,498
16.00
1,200.
272
1,621,515.894
21,626
273
This efficiency stems from AutoPSV's ability to generate process annotations without requiring multiple samples for each reasoning step, making it particularly suitable for large-scale applications scenarios with limited computational resources.
To further demonstrate the flexibility of AutoPSV, we evaluate its performance in unlabeled settings. We generated additional dataset of 7,000 unlabeled math problems using the Evol-Instruct method from WizardI.M [42]. These problems, created by LLMs without accompanying gold solutions, represent challenging scenario for traditional supervision methods. We then conducted an experiment incorporating these unlabeled questions alongside the GSM8K dataset. Table 10 compares various methods across different response generators. In this context, OSV+PSV (GSM8K) refers to the original AutoPSV setting, while OSV+PSV (GSM8K+Wizardll includes process annotations sourced from both GSM8K and WizardLM unlabeled questions. Notably, both MCTS and OS VSVon training cannot leverage these unlabeled data, highlighting another key advantage of AutoPSV.
Performance in Unlabeled Settings
Table 10: Performance enhancement of our proposed AutoPSV method in unlabeled settings, where both MCTS and OS training are unable to utilize unlabeled data.
Response Generator
P5
Self-Coes.
OSV (GSM8K)
MCTS (GSMBK)
OSV+PSV (GSMBK)
OSV+PSV (GSMBKK+WizardL..0
Mistral-Instroc Instrect
69,90
50.03
61.18
60.82
61.4]
63.11
Mistral-Instruct
78.15
Qwen
91.13
81,27
84,91
84.85
85.15
86.77

---

The results demonstrate that the addition of unlabeled data leads to noticeable improvements across all response generators. For instance, the performance of Mistral-Instruct improves from 61.18 (OSV) to 63.11 (OSV+PSV with 4 These results further underscore the value of the AutoPSV approach, particularly its ability to effectively utilize unlabeled data for enhanced performance.
In summary, AutoPSV offers several key advantages over MCTS -CTT--bssd methods:
Consistent Performance with Computational Efficiency: AutoPSV demonstrates robust, consistent improvements across different response generators and datasets. Its ability to efficiently generate process unnotations-witene without the need for extensive sampling like MCTS makes particularly well-suited for large-scale applications and environments with limited computational resources.
Leveraging Unlabeled Data for Enhanced Performance: Unlike MCTS, which relies on ground truth labels, AutoPSV can effectively utilize unlabeled data. This capability not only enhances model performance real-world settings but also offers scalability in scenarios where labeled data scarce. The flexibility to harness unlabeled data ensures that V can drive significant improvements even data-constrained situations.
6.2 Outcome-Supervised Verification vs. Process-Supervise Verification
We apply the OSV to relabel the process-supervised training data as in Section 5.2. We then retrain model using this relabeled data. This experiment highlights the performance gap between suteove-supervised and supervs-e training.
Table 11: Experimental results showing the performance of OSV models across different configura- tions tested on GSM8K test sets.
Response Generator
Pass@1 1
Pass@5
SC
OSV
PSV
Mistral-Instruct 42.08
69.90
50.03
60.72
59.14
Dixtral-Instrued Qwen-722
62.55
82.31
69.06
74.07
71.39
77.03
91.13
81.27
85.00
83.70
The experimental results in Table 11 reveal that retraining the model with process supervision from AUTOPSV still yields better performance than self-consistency across three different response generators. We also noticed a small performance gap between the PSV and OSV. It is worth noting that our PSV was trained using data from the OSV. The small performance gap between the PSV and OSV models demonstrates that relabeled process-supervisee training method successfully inherits information from the outcome-supervise model without requiring ground truth annotations. This ablation study further provides quality assurance for automatic process labeling via AUTOPSV. Moreover, OSV training is limited to labeled datasets, while AUTOPSV demonstrates superior performance by utilizing both labeled and unlabeled data as shown in Table 10. This comparison further highlights the versatility and effectiveness of AUTOPSV in real-world scenarios where ground truth annotations may be scarce or unavailable.
Conclusion
In conclusion, we propose novel method for automatic process labeling in LLMs by detecting relative changes in model confidence. Our experimental results demonstrate that AUTOPSV significantly enhances the precision and scalability of the verifier models various reasoning tasks, ranging from mathematical to commonsense reasoning. AUTOPSV therefore has the potential to considerably enhance existing LLMs' performance while drastically reducing the need for intensive computation and manual intervention. For future work, we aim to utilize the automatically constructed PSV to supervise the generator using step-wise proximal policy optimization, to enhance the accuracy the generator's 's output during greedy decoding without the need for subsequent reranking. This avenue of research could lead to even more advancements in the capabilities of L.MMs and their application in reasoning tasks. The limitations and broader impact of the paper are discussed in Appendix A and B.
10

---

References
[1] OpenAl. GPT-3.5 Turbo, 2023.
[2] OpenAl. GPT-4 technical report. CoRR, abs/2303.08774, 2023.
[3] Mistral AI. Au large, 2023.
[4] Anthropic. Introducing the next generation of claude, 2023.
[5] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia. Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28: December 9, 2022, 2022.
[6] Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity- prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[7] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[8] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[9] Xi Ye and Greg Durrett. The unreliability of explanations in few-shot prompting for textual reasoning. In Sanmi Koyejo, S. Mohamed, Agarwal, Danielle Belgrave, Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
(10) Yongchao Zhou, Andrei Toan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[11] Hyung Won Chung, Le Hou, Shayne Longpre. Barret Zoph. Yi Tay, William Fedus, Eric Li. Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling custruction-finetumee language models. CoRR, abs/2210. 111111101111.....111111111111111 2022.
[12] Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of GPT-4. CoRR, abs/2300.00207. 2023.
[13] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tau- man Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need. CoRR, abs/2306. 1644, 2023.
[14] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. CoRR, abs/2308 0000000998,,,,08 2023,
11

---

[15] Jonathan Uesato, Nate Kushman, Ramana Kumar, H. Francis Song, Noah Y. Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback. CoRR, abs/2211.144777 14275,, 2022.
[16] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Iya Sutskever, and Karl Cobbe. Let's verify step by step. CoRR, 0bs/2305.20050 2023.
[17] Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. CoRR, abs/2312.08923 2023.
[18] Zihan Wang, Yunxuan Li, Yuexin Wu, Liangchen Luo, Le Hou, Hongkun Yu, and Jingbo Shang. Multi-step problem solving through a verifier: An empirical analysis on model-induced process supervision. CoRR, abs/2402.02658,, 2024.
[19] Yifei Li, Zeqi Lin, Shizhuo Zhang. Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making language models better reasoners with step-aware verifier. In Anna Rogers, Jordan L. Boyd-Graber, Graber, and Naoaki Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 5315-5333, Association for Computational Linguistics, 2023.
[20] Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans. LA, USA, December 10 16, 2023, 2023.
[21] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.14160 2021.
[22] Fei Yu, Anningzhe Gao, and Benyou Wang. Outcome-supervised verifiers for planning in thematical reasoning. CORR, abss2331.009722 2023.
[23] Qianli Ma, Haotian Zhou, Tingkai Liu, Jianbo Yuan, Pengfei Liu, Yang You, and Hongxia Yang. Let's reward step by step: Step-level reward model as the navigators for reasoning. CoRR, abs/2310. 10080, 2023.
[24] Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In H. Jaap den Herik, Paolo Ciancarini, and H. H. L. M. Donkers, editors, Computers and Games, Sth International Conference, CG 2006, Turin, Italy. May 29-31, 2006. Revised Papers, volume 4630 of Lecture Notes in Computer Science, pages 72 83. Springer, 2006.
[25] David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George den Driessche, Julian Schrittwieser, loannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukeuoglu, Thore Graepel, and Demis Hassabis. Mas- tering the game of go with deep neural networks and tree search. Nature, 529(7587):484 2016.
[26] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan. Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, abs/2005.1410 14165, 2020.
[27] Jing Xiong, Zixuan Li, Chuanyang Zheng, Zhijiang Guo, Yichun Yin, Enze Xie, Zhicheng Yang. Qingxing Cao, Haiming Wang, Xiongwei Han, Jing Tang, Chengming Li, and Xiaodan Liang. Dy-lore: Dual queries with low rank approximation re-ranking for in-context learning. In The
12

---

Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
[28] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models zero-shot reasoners. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, IA, USA, November 28- December 9, 2022, 2022.
[29] Michihiro Yasunaga. Xinyun Chen, Yujia Li, Panupong Pasupat, Jure Leskovec, Percy Liang, Ed H. Chi, and Denny Zhou. Large language models as analogical reasoners. CoRR, 10./2330.00114,, 2023.
[30] Jianqiao Lu, Wanjun Zhong, Yufei Wang, Zhijiang Guo, Qi Zhu, Wenyong Huang, Yanlin Wang, Fei Mi, Baojun Wang, Yasheng Wang, Lifeng Shang, Xin Jiang, and Qun Liu. YODA: teacher-student progressive learning for language models. CoRR, abs/2401.15670, 2024.
[31] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts. CoRR, abs/2401.0400 2024.
[32] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample. Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/23 2023.
[33] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale. Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307. 0tts/2377.0928, 2023.
[34]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji. Mei Li. Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang. Wei Wang. Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang. Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[35] Yichen Zhu, Minjic Zhu, Ning Liu, Zhicai Ou, Xiaofeng Mou, and Jian Tang. Llava-phi: Efficient multi-moda assistant with small language model. CoRR, abs/2401.0233 2024.
[36] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the MATH dataset. CoRR, 0bbs/2103.03877 2021.
[37] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Anna Korhonen, David R. Traum, and Lluís Marquez,
13

---

editors, Proceedings of the 57th Conference of the Association Computational Linguistics, ACL 2019. Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791-4800. Association for Computational Linguistics, 2019.
[38] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In The Thirly-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirly-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8732-8740. AAAI Press, 2020.
[39] Joe Stacey, Pasquale Minervini, Haim Dubossarsky, Sebastian Riedel, and Tim Rocktäschel. Avoiding the hypothesis-only bias in natural language inference via ensemble adversarial training. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 8281-8291. Association for Computational Linguisties. 2020.
[40]
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023.
[41] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray. Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert. Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang. Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.0333 2021.
[42] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qing- wei Lin, and Daxin Jiang. Wizardlm: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
[43] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information P Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28. December 9, 2022, 2022.
[44] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosiute, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemí Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton. Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield- Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: harmlessness from Al feedback. CoRR, abs/22 12. 0200000. 2022.
14

---

[45] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale. Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang. Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliaan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Rohert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307 htts/237.00928. 2023.
[46] Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. CoRR, abs/2308. 03 188, 2023.
[47] Patrick Fernandes, Aman Madaan, Emmy Liu, António Farinhas, Pedro Henrique Martins, Amanda Bertsch, José G. C. de Souza, Shuyan Zhou, Tongshuang Wu, Graham Neubig, and André F. T. Martins. Bridging the gap: A survey on integrating (human) feedback for natural language generation. CoRR, abs/2305.00955 2023.
[48] Kevin Yang. Yuandong Tian, Nanyun Peng, and Dan Klein. Re3: Generating longer stories with recursive reprompting and revision. CoRR, abs/2210.06774, 2022.
[49] Jianqiao Lu, Zhengying Liu, Yingjia Wan, Yinya Huang, Haiming Wang, Zhicheng Yang. Jing Tang, and Zhijiang Guo. Process-driven autoformalization in lean 4. CoRR, obs/2400.01194, 2024.
[50] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. CRITIC: large language models can self-correct with tool-interactive critiquing. CoRR, abs/2305.1 11738, 2023.
[51] Dong Huang, Jianbo Dai, Han Weng, Puzhen Wu, Yuhao Qing, Jie M. Zhang, Heming Cui, and Zhijiang Guo. SOAP: enhancing efficiency of generated code via self-optimization. CoRR, abs/2405. 15189, 2024.
[52] Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y. Zhao, Ni Lao, Hongrae Lec, Da-Cheng Juan, and Kelvin Guu. RARR: researching and revising what language models say, using language models. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 16477-16508. Association for Computational Linguisties, 2023.
[53] Wenhao Yu, Zhihan Zhang, Zhenwen Liang, Meng Jiang, and Ashish Sabharwal. Improving language models via plug-and -nd-play retrieval feedback. CoRR, abs/2305.14002, 2023.
[54] Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. Maieutic prompting: Logically consistent reasoning with recursive explanations. In Yoay Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 1266-1279, Association for Computational Linguistics, 2022.
[55] Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. In The Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
15

---

[56] Yuxuan Yao, Han Wu, Zhijiang Guo, Biyan Zhou, Jiahui Gao, Sichun Luo, Hanxu Hou, Xiaojin Fu, and Linqi Song. Learning from correctness without prompting makes LLM efficient reasoner. CoRR, abs/2403. 19094, 2024.
[57] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. CoRR, abs/2303. 2023.
[58] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforeement learning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[59] Alec Helbling, Mansi Phute, Matthew Hull, and Duen Horng Chau. LLM self defense: By self examination, Ilms know they are being tricked. CoRR, abs/2308. 07308, 2023.
[60] Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min- Yen Kan, Junxian He, and Michael Qizhe Xie. Self-evaluation guided beam search for reasoning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[61] Hao Yan, Saurabh Srivastava, Yintao Tai, Sida I. Wang, Wan-tau Yih, and Ziyu Yao. Learning to simulate natural language feedback for interactive semantic parsing. Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 3149-3170 Association for Computational Linguistics, 2023.
[62] Sconghyeon Ye, Yongrae Jo, Doyoung Kim, Sungdong Kim, Hyconbin Hwang, and Minjoon Seo. Selfee: Iterative self-revising Ilm empowered by self-feedback generation. Blog post, May 2023.
[63] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. CoRR, abs/2310.01798, 2023.
[64] Jianqiao Lu, Wanjun Zhong, Wenyong Huang, Yufei Wang, Fei Mi, Baojun Wang, Weichao Wang, Lifeng Shang, and Qun Liu. SELF: language-driven self-evolution for large language model. CoRR, abs/2310.00533, 2023.
A Limitations
AUTOPSV is a promising solution for enhancing the reasoning capabilities of LLMs. However, it important acknowledge several potential limitations of the method. Firstly, while AUTOPSV aims to reduce the need for manual intervention, there is still a risk of inaccurate annotations. The relative step-level confidence change used to produce process annotations an estimation and may not always accurately represent the actual correctness of reasoning step. This could compromise the quality of the annotations and, in turn, the effectiveness of the method. Secondly, the success of AUTOPSV is heavily dependent on the performance of the verifier. If the model is not accurate enough in its step-level scores, the quality of the process annotations generated by AUTOPS V could be compromised. Thirdly, AUTOPSV V is specifically designed to improve the reasoning capabilities of LLMs. Therefore, its applicability may limited to tasks that involve complex multi-step reasoning. It is unclear how well the method will scale or generalize to other tasks or domains that do not involve intensive reasoning. This is an important consideration for future research and development of the method.
16

---

B Broader Impact
Positive Societal Impacts The proposed AUTOPSV has the potential to bring about several positive societal impacts. By enhancing the reasoning capabilities LLMs, AUTOPS can lead to more accurate and reliable information, which in turn support better pecision-making in various sectors, including healthcare, education, and finance. Moreover, AUTOPSV combines the strengths of output supervision and process supervision automatically annotate reasoning steps, significantly reducing the time, effort, and cost associated with manual annotation. This makes the process of training LLMs more efficient and accessible. Additionally, the process supervision data generated by AUTOPSV can improve the performance and scalability of verification models, allowing for the development of more complex and sophisticated LLMs capable of handling wider range tasks and upplications.
Positive Societal Impacts However, AUTOPSV also presents several potential negative societal impacts. The automation of the annotation process could lead to job displacement for individuals currently employed in this role. There is also a risk that AUTOPSV V and the enhanced LLMs could be misused, for instance, to spread misinformation reliance on LLMs for decision-making could potentially result in a decrease in critical thinking and problem-solving skills among individuals. Furthermore, the of LLMs in various sectors could lead to privacy and security issues, as these models often require large amounts of data for training.
manipulate public opinion. The increased
C More Related Work
Learning From Feedback Improving LLMs through learning from feedback has become preva- lent strategy, notably through reinforcement learning from human feedback, which seeks to align LLMs with human values by refining their outputs based feedback [43-45]. However, this method faces challenges such as high costs due to manual labor and a lack real-time feedback capabilities [46, 47]. An alternative strategy involves using self-correcting LLMs, which rely automated feedback to iteratively adapt and understand the consequences of their actions without heavy reliance human intervention. This feedback can be derived from outside sources such as other models [48, 49), tools [50, 51], knowledge bases [52, 53], evaluation metrics [54, 55] or generation logits [56].
External feedback leverages external perspectives to identify errors and verify factual accuracy, offering insights that may not be recognized by the LLM alone. Conversely, feedback can also be internally generated, where the LLM evaluates and refines its output iteratively until a desired quality is achieved [57-60]. This self-improvement mechanism is particularly valuable in scenarios where external feedback is scarce or restricted [61, 62]. However, recent effort [63] suggests that LLMs struggle to independently identify and correct emrors through self-generated prompts.
D
Vanilla Evaluation Methods Description
Classification: For this method, the evaluator is presented with multiple answers for given question and is required to choose the best answer among them. The selection is made based on the evaluator's judgment of which answer most accurately addresses the question or provides the most relevant information.
Given the following question: '[question]', and these five answers:
[answer] [answer]
2.
3.
[answer]
4. [answer]
5. [answer]
Which answer the best? Please provide the number of the best answer. You should strictly follow the output format requirements and not output any other content.
Example: Answer [number] better. Let's Begin!
In Classification + COT, the evaluator must not only identify the best answer but also analyze and compare all provided answers before making their decision. This method
Classification COT:
17

---

requires deeper examination of the content and context of each answer to determine its quality and relevance the question.
Given the following question: 'qquestion]', and these five answers:
1. [answer]
2. [answer]
[answer]
4. [answer]
[answer]
Which answer is the best? Please analyze and compare the provided answers and then identify the number of the best answer. You should strictly follow the output format requirements and not output any other content.
Example: Comparison and Analysis: [analysis]. Best answer: [number]. Let's Begin!
Scoring: In the Scoring method, the evaluator assigns a numerical score to each answer based on its quality relevance the given question. The scores typically range from 10, with 10 representing the highest quality or most relevant answer.
Given the following question: '[question]', please score these five answers on scale from 1 to 10, where 10 the best:
1. [answer]
[answer]
3. [answer]
[answer]
5. [answer]
Please provide a score for each answer. You should strictly follow the output format require- ments and output any other content. Example: Answer [score]. Let's Begin!
Similar to Scoring, Scoring COT also involves assigning numerical scores each answer. However, in Scoring _ot, the evaluator is required to provide an analysis of each answer before assigning score. This analysis informs the scoring process and ensures a more informed evaluation of each answer.
Scoring COT:
Given the following question: '[question]; please score these five answers on scale from 1 10, where 10 is the best:
1. [answer] [answer]
3. [answer]
[answer]
5. [answer]
Please analyze each answer, and then provide a score for each answer. You should strictly follow the output format requirements and not output any other content. Example: Answer analysis: [analysis]. score: [score]. Let's 's Begin!
Pairwise Comparison: In this method, the evaluator is presented with pairs of answers for given question and tasked with determining which answer better in each pair. The evaluator compares the content or quality of each answer and selects the one they deem superior. The Pairwise Comparison method differs from other evaluation methods in that evaluates two candidate answers at time and chooses the winner to proceed to the next comparison with the next candidate answer. For set of candidates, this method conducts n-1 pairwise comparisons. order preference bias exhibited by LLMs, we adopt method similar to [64] which shuffles the positions of two answers during prompting. This ensures a fair evaluation process by eliminating any bias toward the position of the answers.
mitigate the potential
18

---

Given the following question: '[question]', compare each pair of answers and decide which one is better:
Compare 1. [answerl] with 2. [answer2]
For the comparison, indicate the better answer with its number. You should strictly follow the output format requirements and not output any other content.
Example: Answer better. Let's Begin!
By employing these different evaluation methods, we aim to comprehensively assess the quality and relevance the answers generated by our models for various questions. Each method offers unique perspective and contributes to more thorough evaluation process.
E Math Calculation Error Benchmark
Methodology We utilize the LlaMA2-chat model (LlaMA) for mathematical reasoning step genera- tion. Using regular expressions, we extract computational steps and evaluate expressions to the left of the "=" sign using Python's eval function to verify their correctness against right-hand side results. We term this process "Math Calculation Error Detection" and present the results in Table 12.
Table 12: Performance of the LlaMA model on GSM8K training data, evaluated through few-shot prompting strategy.
Model
Pass (%)
Self-Consisteng. (%)
Math Calculation Error Detection (%)
LlaMA
0.4791
0.2881
0.1824
Data Processing Non-computational expressions (e-g.. "x 2 that cannot be evaluated due to unsolvability incorrect formatting are excluded from the ground truth data. This ensures our ground truth accurately reflects only computational errors during reasoning. While DeepMind [15] employs human labelers for "trace errors" annotation, our automated approach using Python's eval function provides scalable alternative.
Error Detection Example To demonstrate our calculation of A con for math calculation error identification, we present an example in Figure 2. The red highlights indicate calculation errors in the reasoning process, detected through significant decreases in model confidence.
F
Experiment Details
F.1 Training Hyperparameters
Computing Infrastructure Our experiments were conducted using NVIDIA A100 GPUs, each with 40GB of memory. All models underwent full-parameter fine-tuning using the Adam W optimizer.
Verifier Training Configuration For both process-supervised and outcome-supervised methods, we maintain consistent training parameters as detailed Table 13. The training process spans epoch with batch size of 512 and learning rate of 2 x 10 incorporating 3% learning rate warmup period.
Table 13: Verifier Training Hyperparameters.
Hyperparameter
Global Batch Size
LR
Epo.
Max Length
Weight Decay
Warmup Ratio
Value
512
2 10 6
1
2048
0.03
SFT Model Prerequisites Prior to verifier training, we establish a supervised fine-tuning (SFT) model. This model serves to generate responses and facilitate outcome supervision labeling by validating final answer correctness. The SFT model's training parameters are specified in Table 14. For comprehensive details regarding OSV implementation, refer to [22].
19

---

Question:
Randy just turned 12 and started playing the piano. His friend Sheila told him about the 10,000- hour rule which says, after 10,000 hours of practice, you become an expert or master in your field. Randy wants to become a piano expert before he is 20, how many hours a day will he need to practice if he practices every day, Monday Friday, and takes two weeks off for vacation each year?
Solution:
If Randy become piano expert before he is 20, will need practice for 10, 000 hours in total. Since he is now, that means he will need to practice for 10, 10002020 500 hours a year. [0.2]
If practices every day, Monday Friday, that he will need practice for 500*52 500+52=260 260 weeks 0.6].
Since he takes weeks off for vacation each year, that he will need to practice for 260/52 hours a day [0.1].
The answer is 5.
Figure 2: A data example during math calculation error detection. We apply the OSV model to detect calculation errors by calculating the step-level confidence change each step, denoted inside the brackets [] for the step containing calculations. The ground truth location the math calculation error is marked in red.
Table 14: SFT Training Hyperparameters.
Hyperparameter
Global Batch Size
LR
Epo.
Max Length
Weight Decay
Warmup Ratio
Value 128
5
10-6
2
2048
0.03
F.2
Generation Settings
Parameter Settings We employ two distinct temperature configurations: temperature 0.0 for the greedy decoding strategy and temperature = 0.7 for the pass @k evaluation strategy.
Pass @k Evaluation Protocol To ensure unbiased estimation of pass@k metrics, we follow the methodology established in [41]. We generate n 20 samples per problem instance and evaluate each sample against unit tests to determine correctness. The unbiased estimator for pass @ is then computed based these results. This entire process is repeated times to establish 95% confidence intervals, as detailed in Section 5.1.
Score Aggregation Methodology Our primary aggregation mechanism employs a multiplicative approach for step-level scores. This method computes the product of confidence scores across all steps, yielding compound probability that represents the overall likelihood solution correctness.
Comparative Analysis of Aggregation Functions Following [16], we conducted a comparative analysis different aggregation strategies. While our primary approach utilizes the product of step-level scores, also investigate the minimum of step-level scores alternative approach. The comparative results for these aggregation functions on the GSM8K dataset are presented below:
As shown in Table 15, the choice of aggregation function can have slight impact on the results, with the product function generally performing comparably or slightly better than the minimum function across different response generators.
20

---

Table 15: Comparison of aggregation functions on GSM8K dataset.
Response Generator
Product
Minimum
Mistral-Instruce
60.72
61.17
Mixtral-Instruct
74.07
72.45
Qwen
85.00
84.28
F.3 Process-Supervision Labelling Strategy
Overview Our implementation encompasses two distinct labelling methodologies: proposed approach (Process Ours) and a comparative Monte Carlo Tree Search approach (Process MCTS).
Implementation Procedure The procedure consists of the following sequential stages:
1. Initial Generator Training: We conduct supervised fine-tuning of generator using combined dataset of approximately 15,000 problems from GSM8K and MATH.
2. Sample Generation: The trained generator produces 10 distinct solution attempts for each unlabeled prompt within the training datasets, Each solution receives binary label (0 or 1) based on the correctness of its final answer.
3. Speeff-Speeiffc Processing:
(a) Process Ours:
Train output-supervised verifier using the complete set of 150,000 generated samples (15,000 x 10)
Relabel samples using a relative confidence change criterion (as defined in Equation
3)
Retrain the verifier using these refined labels
(b) Process MCTS:
Decompose each of the 150,000 samples into constituent reasoning paths
For each incomplete reasoning path, generate eight complete path variations
Compute the accuracy for each complete path
Calculate the proportion of correct paths to determine the final label
Retrain the verifier using these MCTS -CTS-derived labels
Technical Note
While our main experimental results employ 50 samples per problem, restrict this implementation to samples due to computational constraints. The full MCTS labeling process for 50 x 15,000 samples would require approximately 10- 12 days of computation time.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the "abstract" and "introduction" sections accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We provide the limitations in the "Limitations" section attached in the Ap-
pendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer: [NA]
Justification:
The paper focuses on exploring a novel process approach to enhance verifiers, and such, it is primarily based experimental results rather than theoretical proofs. The paper does not introduce new theoretical results; instead, relies theorems that have already been proven in previously published literature. These refer- enced theorems are clearly cited the "Method" section of the paper, indicating that the authors have not provided new proofs but have utilized established results to support their experimental work.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper includes all the necessary information required to reproduce the main experimental results. This information detailed within the "Experiment" section and the "Appendix" of the paper, ensuring that readers have access to the methodologies, parameters, and data needed to validate the main claims and conclusions presented.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
23

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide the training scripts in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/ mublic/guides wwwpe//oooeSSaaaiiiiiiiiioiie
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips. "nippsccpppult wwtpccccppppccceecceccccccccecet moubicc/guides deeieeeee/oeeebmmiiiiee ssiooPolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
The paper provides comprehensive details regarding the training and testing in the "Appendix" due the page limit.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We mentioned in "Experiment" section that The results are presented with an accuracy of +0.1, calculated at a 95% confidence level.
Guidelines:
24

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We include sufficient information regarding the computer resources required reproduce the experiments "Experiment" and "Appendix".
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https http://enurrrsccceeee httpppubii//thiiiiiiiidaaaaeeeen
Answer: [Yes]
Justification: We conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
25

---

Justification: We discuss the societal impact of the work in the "Broader Impact" section attached in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymeen of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper does not present any high-risk data or models that could be prone to misuse, as it is focused on enhancing the performance of Language Model Support Systems (LLMS) within the field of Natural Language Processing (NLP).
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, all experiments conducted in the paper utilize open-source models. Proper references and citations are included in the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
26

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaeed, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide the code in the supplementary materials and include README file with details about data annotation, training. and limitations.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
27

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
28


Reviews:
[{"review_id": "kiornGCV3t", "reviewer": "Reviewer_kK32", "summary": "The authors propose AutoCV, a method for solving multi-step reasoning tasks with chain-of-thought prompting that involving automatically labeling each step of the multi-step process based on its likelihood of leading to the correct outcome, and combining an outcome supervised classifier (OSV) and process supervised classified (PSV) to come up with a more effective method. Experiments on 2 math and 3 reasoning datasets show that OSV + PSV outperforms self-consistency as well as just OSV across the board, albeit by a small margin in several cases.", "questions": "Please see weaknesses above.", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is carefully written and provides good background for readers (like myself) not very familiar with the approaches considered. The idea of using the difference in the confidence of an OSV model at step t and at t+1 is interesting; it's perhaps a bit surprising that it actually works well, as it's not often easy to learn how good an early intermediate step is towards achieving the final goal.\n\nThe findings in section 4 seem to lay out a good justification for the design choices behind AutoCV, although I found the discussion in 4.2 (hallucination in math reasoning) harder to follow.\n\nThe performance on 5 benchmark datasets, as noted above, is a strength of the paper, even though in many cases the absolute improvement is small.", "weaknesses": "Not having familiarity with the area, I wasn't able to judge the novelty of the work and the substance (i.e., whether there is enough new material to warrant publication). It seems to me that f_theta() being a good indicator of the probability of reaching the correct final outcome is something known from reference [22]. The new part here is looking at the delta of f_theta() between time steps t and t+1, as done in Eq (3).\n\nIn terms of naming, I found the use of the phrase \"confidence variation\" not intuitive for referring to the change in the confidence from t steps to t+1. To me, confidence variation suggests how the confidence changes as one varies some parameter or some sampling, not as one takes another step. In a similar vein, I found \"AutoCV\" to not be an informative name for the system -- it doesn't connect well with the task of multi-step reasoning or with chain-of-thought prompting. That said, this is a subjective choice and it's up to the authors to decide if they want to look for a more intuitive name.\n\nAs noted above, I found the discussion in section 4.2 somewhat confusing.\n\nLastly, as also noted above, while OSV + PSV generally shows consistent gains over the OSV, the gains are often very small. This somewhat lowers the value of the proposed technique, but consistent gains remain a net positive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and insightful comments. We hereby address your concerns below:\n\n**R W1  Novelty and Substance of Our Paper**\n\n1. **Theoretical Contribution:** While reference [22] demonstrates that $f_\\theta$ can indicate the probability of reaching the correct final outcome, our contribution extends this by exploring the change in $f_\\theta$, denoted as $\\Delta_{conf}^{t}$, between time steps $t$ and $t+1$. Unlike [22], which applies $f_\\theta$ for reranking during decoding, our AutoCV demonstrates the effectiveness and robustness of $\\Delta_{conf}^{t}$ for error detection, as detailed in Section 4.\n2. **Experimental Contribution:** We demonstrate the performance gains of applying $\\Delta_{conf}^{t}$ for process-supervision training in Sections 5 and 6. AutoCV combines the strengths of output supervision and process supervision to automatically annotate reasoning steps, thereby enhancing model performance.\n3. **Scalability:** Unlike [22] and other MCTS-based methods, which limit their experiments to math tasks, our method is generally applicable across both math and commonsense reasoning. Furthermore, unlike methods limited to settings where ground truth is available, AutoCV can leverage both labeled and unlabeled data, significantly expanding the dataset for training. This flexibility enhances the robustness and applicability of our approach. More details on this can be found in **R W4.**\n\n\n\n**R W2 Naming Issue**\n\nThank you for the suggestion. To better reflect the concept of changes in confidence from step t to step t+1, we propose using the term **\"Step-level Confidence Change**. This name more accurately conveys the idea of confidence change between consecutive steps.\n\nSimilarly, to reflect these aspects more clearly, we propose renaming the system to **\"AutoPSR: Automated Process-Supervised Reasoner\"**. This name highlights the focus on automated process labeling within the context of multi-step reasoning and chain-of-thought prompting. We use AutoCV during the rebuttal for consistency, but we will **revise the term in the revised manuscript**.\n\n**R W3 Discussion in Section 4.2:**\n\nTo address the confusion regarding Section 4.2, we provide further clarification.\n\nSection 4.2 aims to **automate** the evaluation of the effectiveness and robustness of detecting relative variations defined in AutoCV for generating process labels. \n\nIn Section 4.2.1, we automate the acquisition of ground truth by detecting errors specifically in mathematical calculations. **This approach eliminates the need for manual process data ground truth.** We successfully labeled mathematical calculation errors in the outputs of Llama2 for 7500 GSM8K training data samples. This process, as stated in line 195, \"establishes a benchmark for PCH detection.\" **To avoid potential misunderstandings, we will revise the term \"hallucination in math reasoning\" to \"math calculation error\" in the revised manuscript.**\n\nIn Section 4.2.2, we show that our method performs well in detecting calculation errors, as evidenced by precision, accuracy, and F1 scores. As noted in line 209, \"Table 5 demonstrates that our method using confidence variation effectively detects calculation errors.\" This validates our method as a reliable source of process supervision information, **establishing an experimental basis for automating process annotations in Section 5.**\n\nWe will include the clarification above in the revised manuscript for better presentation.\n\n\n\n**R  W4 Small Gains with OSV + PSV compaed with OSV only:**\n\nIt is important to highlight that these gains, while seemingly small, **are statistically significant**. We conducted statistical significance tests over five repetitions. Experimental results consistently produced p-values well below the significance level of 1e-6. For example, with Mixtral-Instruct, the t-statistic was -15.06 with a p-value of 1.09e-07 This statistical significance underscores that the improvements observed are not due to chance and validate the effectiveness of the proposed OSV + PSV approach.\n\nMoreover, it is essential to consider the broader applicability and additional benefits that OSV + PSV provides. **Unlike OSV-only, which is limited to settings with available ground truth, OSV + PSV can leverage both labeled and unlabeled data.** This capability allows for the utilization of a more extensive and diverse dataset, which is particularly advantageous in real-world scenarios where labeled data may be scarce.\n\nTo illustrate the benefits, we generated 7K math problems following the Evol-Instruct method from WizardLM[1], **which are generated by LLMs without gold solutions**. Both MCTS and OSV-only training **cannot** leverage these unlabeled data. The results are as follows:\n\n| **Response Generator** | Pass@5 | Self-Cons. | OSV (GSM8K) | MCTS(GSM8K) | OSV+PSV (GSM8K) | OSV+PSV (GSM8K+WizardLM) |\n| ---------------------- | ------ | ---------- | ----------- | ----------- | --------------- | ------------------------ |\n| Mistral-Instruct       | 69.90  | 50.03      | 61.18       | 60.82       | 61.41           | 63.11                    |\n| Mixtral-Instruct       | 82.30  | 69.06      | 74.91       | 75.10       | 76.04           | 78.15                    |\n| Qwen                   | 91.13  | 81.27      | 84.91       | 84.         | 85.15           | 86.77                    |\n\nIn this context, OSV+PSV (GSM8K) refers to the original AutoCV setting, while OSV+PSV (GSM8K+WizardLM) includes process annotations sourced from both GSM8K and WizardLM unlabeled questions. The addition of unlabeled data leads to noticeable improvements across all response generators. For instance, the performance of Mistral-Instruct improves from 61.18 (OSV) to 63.11 (OSV+PSV with GSM8K+WizardLM). These results further demonstrate the value of the AutoCV approach.\n\n[1] [WizardLM: Empowering Large Language Models to Follow Complex Instructions. ICLR 2024](https://arxiv.org/abs/2304.12244)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ys0aQUlPTV", "reviewer": "Reviewer_rE8z", "summary": "This paper proposes a new method (AutoCV) that bridges the gap in popular techniques for enhancing reasoning capabilities of LLMs. Prior work have proposed *verification models* (models that evaluate generated reasoning steps and rerank candidate responses) as a promising solution. Here the community has focused on two training paradigms for verification models: (1) outcome supervision (train on correctness of final answer; cheap but less effective) and (2) process supervision (train on labels for each reasoning step; expensive but effective). AutoCV attempts to bridge these two by first training an outcome-supervised model and then use it to annotate confidence for unannotated intermediate steps. Then, by calculating the relative confidence variation between steps, AutoCV generates process supervision data that can be used for training.", "questions": "Note: Added most questions in the Weaknesses section. Additional:\n\nCould the authors comment on how this would work with Agent based reasoning traces? Particularly, the procedure to generate process supervision labels (described in Section 3.3) would not work because it makes the following assumption: `any step, including the final result, following a step containing an error is considered incorrect`. This does not hold true for agentic approaches that use environment feedback to correct their mistakes (think code generation w/ execution feedback).", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "AutoCV tackles clear limitations in prior approaches with some novel ideas:\n1. Interesting use of how an outcome-supervised (OS) model (theoretically) implicitly learns process annotations:\n     - OS replicates final answer’s correctness label across all steps, causing the model to implicitly learn values for partial paths.\n     - Consequently, forces the optimal solution under MSE to be estimate of the probability of reaching the correct answer.\n\n2. Overcomes clear efficiency limitations of previous automated process annotation approaches like MathShepherd and MiPS that use MCTS. The MCTS soln is to sample multiple complete traces at each reasoning step, and using the outcome to score the potential of this step (say k/N lead to correct answer).", "weaknesses": "My concerns are mainly around the presentation, clarity, and significance of some results:\n\n**Presentation/Clarity**\n1. Throughout the paper, there are several experiments using verifier at inference to rerank generations from a *response generator*. For instance, Section 4.1 and Table 3. However, there is no mention or analysis of the #samples being reranked?\n\n2. Similarly, there is no mention (or analysis) of the aggregation function used for process-supervised models at inference time?\n\n3. Section 4.1 and Table 4 attribute the performance disparity among OSVs (for Phi2 and Mistral) to training data quality. Particularly confusing, was why the training data was different at all for each model? From the description, it seems like the training data was generated from _a single_ (GSM8K fine-tuned) pretrained LLM. If not, how does using a stronger model to generate data affect performance?\n\n**Significance of results.** Table 6-7 describe the performance of a PSV finetuned on self-generated confidence variations of an OSV. From that, it seems like there is *very minimal gains* from using the PS data generated to enhance (finetune) the OSV model. Furthermore, when the PS data is used to retrain a new PSV, the performance even drops a few points (Table 10).\n\n1. While one can argue that there is \"minimal loss\" in information from the OSV to a PSV, it concerns me that the quality of process labels might not be good. The generated labels add little-to-no value to overall performance? i.e., why use AutoCV to train a PSV at all if OSV is equally effective AND can provide process-annotations scores?\n\n2. With minimal gains, the only advantage of AutoCV is that it is more sample efficient than MCTS based approaches for PSV training. Are there contributions beyond efficiency?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and insightful comments. We hereby address your concerns below:\n\n**Response to W1 Presentation:**\n\n**1.1 No mention of the #samples being reranked**\n\nOur task involves selecting the correct candidate from **five** responses. Therefore, we compare our method to the Pass@5 performance in each table, including Table 3 in Section 4.1. Specifically, we noted on line 256 that \"Pass@5 represents the upper limit of performance.\"\n\nWe will add this detail to the table captions and the main context to enhance clarity. \n\n**1.2 Analysis of the aggregation function**\n\nWe utilize the **product of step-level scores as the aggregation function**. Additionally, we have analyzed different aggregation functions, specifically the **minimum** and the **product** of step-level scores, following the setting of [1]. Below, we present the results on GSM8K:\n\n| **Response Generator** | **Product** | **Minimum** |\n| ---------------------- | ----------- | ----------- |\n| Mistral-Instruct       | 60.72       | 61.17       |\n| Mixtral-Instruct       | 74.07       | 72.45       |\n| Qwen                   | 85.00       | 84.28       |\n\nThe product aggregation function multiplies the confidence scores of each step, leading to a compounded probability that represents the overall likelihood of a correct response. \n\n\n\n**1.3 Why the training data was different in Table 4**\n\nThe OSV model is **continuously** trained from the GSM8K fine-tuned model with the addition of a value head.\n\nTherefore, in Table 4, the training data for OSV (**Mistral**) is generated from the fine-tuned **Mistral** model, while the training data for OSV (**Phi**) is generated from the fine-tuned **Phi** model. This distinction in training data sources explains the performance disparity observed between OSV models for different response generators.\n\nTo avoid any confusion, we will add further clarification in the manuscript.\n\n\n\n**Response to W2 Significance of results:**\n\n**2.1 Quality of process labels**\n\nTable 10 shows that retraining the model with process labels only from AutoCV yields better performance than self-consistency, indicating that these labels are of good quality and successfully inherit information from the outcome-supervised model without requiring ground truth annotations.\n\nAdditionally, Table 5 in Section 4 (Preliminary Findings) further demonstrates the effectiveness and reliability of AutoCV's relative confidence variation in detecting errors during math reasoning.\n\n\n\n **2.2 Why train a PSV if using PS data drops performance (Table 10) and OSV is equally effective and provides process-annotation scores?**\n\nThe training data for the PSV model **does not utilize ground truth labels**; it uses process supervision data generated by detecting confidence changes between steps (as in Eq. 3). Therefore, in the OSV+PSV training phase, we can include both the GSM8K/MATH training set and unlabeled math problems.\n\nIn Table 10, the PSV model **maintains performance close to the OSV model. Notice that OSV relies on ground truth labels** to determine the correctness of each solution, but PSV does not. To further illustrate the no-ground-truth benefits, we generated 7K unlabeled math problems following the Evol-Instruct method from WizardLM [1], which are generated by LLMs without gold solutions.  We conducted an additional experiment on GSM8K by including these unlabeled questions. **Methods like MCTS and OSV-only training cannot effectively utilize these unlabeled questions.**\n\n| **Response Generator** | Pass@5 | Self-Cons. | OSV (GSM8K) | MCTS(GSM8K) | OSV+PSV (GSM8K) | OSV+PSV (GSM8K+WizardLM) |\n| ---------------------- | ------ | ---------- | ----------- | ----------- | --------------- | ------------------------ |\n| Mistral-Instruct       | 69.90  | 50.03      | 61.18       | 60.82       | 61.41           | 63.11                    |\n| Mixtral-Instruct       | 82.30  | 69.06      | 74.91       | 75.10       | 76.04           | 78.15                    |\n| Qwen                   | 91.13  | 81.27      | 84.91       | 84.         | 85.15           | 86.77                    |\n\nMCTS and OSV-only training is limited to GSM8K training data, our AutoCV extends to apply unlabeled data, contributing to a performance gain (e,g, 61.41 to 63.11 for Mistral-Instruct). Including unlabeled data further enhances the model's capabilities, demonstrating the value of the AutoCV approach.\n\n\n\n**2.3 Advantage of AutoCV compared with MCTS-based methods beyond efficiency.**\n\nThe advantage of AutoCV is not just efficiency over MCTS-based approaches but also the ability to leverage **unlabeled** data for enhanced performance. MCTS-based methods **require ground truth on the correctness of final answers**. In contrast, the process annotations in AutoCV are generated by detecting relative confidence variations without needing ground truth annotations. This flexibility allows AutoCV to utilize unlabeled data, improving the model's performance. The experiment result is provided in the response to **2.2.** \n\n**R Q1 How AutoCV work with Agent-based reasoning ?** \n\nIntegrating AutoCV into agent-based reasoning traces is an interesting direction to explore. We can dynamically update labels based on feedback at each step instead of marking all subsequent steps as incorrect after detecting an error:\n\n1. Detect the confidence variation $\\Delta_{conf}^{t}$ as defined in Eq (3). If $\\Delta_{conf}^{t}$ is below an error threshold, mark the step as incorrect.\n2. Continue labeling subsequent steps as incorrect until $\\Delta_{conf}^{t}$ exceeds a correct threshold.\n\nThe correct and error threshold settings may depend on experimental findings, similar to how we choose the threshold in AutoCV, as shown in Table 5.\n\n\n[1] [WizardLM: Empowering Large Language Models to Follow Complex Instructions. ICLR 2024](https://arxiv.org/abs/2304.12244)\n\n[2] [Let's Verify Step by Step. ICLR 2024](https://arxiv.org/abs/2305.20050)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8fukuIufVv", "reviewer": "Reviewer_hKWR", "summary": "This paper introduces AutoCV, a novel method to create process reward models (PRMs). The approach involves training an an outcome-supervised verification model based on (S_{(1:t)}, y) pairs where S_{(1:t)} is the 1st~t-th steps in the response and y is the final correctness label of the entire response. The outcome-supervised verification model then generates process annotations by detecting confidence variations across reasoning steps. AutoCV combines outcome and process supervision, reducing the need for manual annotations or computationally expensive methods. Experiments across five datasets in mathematics and commonsense reasoning show a significant advantage of the PRM for best-of-N sampling. The method outperforms self-consistency baselines and outcome-supervised verifiers while requiring fewer tokens for annotations than existing approaches.", "questions": "- It seems the verifiers are based on the Mistral series and in general its gain is less significant when the response generator is Qwen, especially on MATH where AutoCV best-of-5 underperforms self-consistency. Is it because the method lacks generalizability or does the method only work well when the response generator is weak?", "limitations": "See above", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This work proposes a novel method to obtain process annotation for free from outcome supervision. The method is elegant and derived from a theoretical perspective.\n- Unlike previous work on PRM that only focuses on the math domain, this work shows effectiveness on commonsense reasoning. and natural language inference.", "weaknesses": "Though the method is interesting, the experiments are not very convincing and the paper is not well presented. \n- I cannot get the point of section 4.2.1 since it's a bit confusing to me. Is there a reason why the method is specifically useful for \"process calculation hallucination detection\"? It seems to me that it's a general method to detect all faulty steps in a response, which is what a PRM is supposed to do. Also, I think the term \"hallucination\" may be overloaded, which originates from an open-ended generation. I suppose the setting in section 4.2.1 is simply \"calculation error detection\".\n- There is no baseline in the main experiment (Table 6&7). I'd suggest moving section 6.1 to the main exp, and results beyond the math domain as a separate subsection in section 5.\n- The setup part (sec 5.1) does not mention if the experiments are all in-domain. OOD evaluations are required to examine the generalizability.\n- From Table 8, AutoCV performs very comparable with MCTS-based methods with differences of <1% accuracy. The performance gain is marginal.\n- Table 9 may not be a fair comparison of efficiency between AutoCV and MCTS-based methods. Despite using fewer tokens, AutoCV requires training an ORM/OSV for annotation which could also be costly. Maybe the sum of GPU hours for training/inference would be fairer?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and insightful comments. We hereby address your concerns below:\n\n\n\n**R W1 Presentation of Section 4.2.1:**\n\nThanks for the suggestion. To avoid potential misunderstandings, we will change the term to \"calculation error detection.\"\n\nWe would like to provide the following clarification for Section 4.2.1. We introduce a method to **automate** the acquisition of ground truth by detecting errors in mathematical calculations. This approach provides an effective and robust automated framework for detecting math calculation errors **without requiring manual annotations**. Based on this framework, we further establish a PCH benchmark as described in Appendix E. This addresses the primary challenge of evaluating whether a model detects all faulty steps due to the lack of ground truth process annotations. \n\n\n\n**R W2 No baseline in the main experiment (Table 6 & 7)**\n\nWe will move the baselines from Section 6.1 to the main experiments (Tables 6 & 7) and separate the math and common reasoning tasks into different subsections for better presentation.\n\n\n\n**R W3 OOD evaluations are required to examine the generalizability**\n\nTo address your concern regarding the need for out-of-domain (OOD) evaluations, we utilized the trained OSV+PSV model from the original manuscript. We conducted additional OOD tests on MMLU, BBH, and ARC-C. As shown in the table below, our method consistently outperforms baselines on three benchmarks.\n\n|                        | MMLU   |           |      |         | BBH    |           |      |         | Arc-C  |           |      |         |\n| ---------------------- | ------ | --------- | ---- | ------- | ------ | --------- | ---- | ------- | ------ | --------- | ---- | ------- |\n| **Response Generator** | Pass@5 | Self-Cons | OSV  | OSV+PSV | Pass@5 | Self-Cons | OSV  | OSV+PSV | Pass@5 | Self-Cons | OSV  | OSV+PSV |\n| Mistral-Instruct       | 62.5   | 47.4      | 52.5 | 54.3    | 40.5   | 30.7      | 35.5 | 36.1    | 52.5   | 42.3      | 48.3 | 49.1    |\n| Mixtral-Instruct       | 69.6   | 58.1      | 63.4 | 64.7    | 43.3   | 38.4      | 40.2 | 40.8    | 58.1   | 50.1      | 53.9 | 54.3    |\n| Qwen-72b               | 76.1   | 67.7      | 71.8 | 71.9    | 65.9   | 58.3      | 60.9 | 61.3    | 64.7   | 56.3      | 60.2 | 61.5    |\n\n\n\n**R W4/5 Comparisons with MCTS-based Methods:**\n\n1. GPU Hours Comparison: we provide a comparison of GPU usage for both training and annotation costs:\n\n| Dataset | GPU Hours (Annotation Cost) |                  | GPU Hours (Training) |                  |\n| ------- | --------------------------- | ---------------- | -------------------- | ---------------- |\n|         | Process (MCTS)              | Process (AutoCV) | Process (MCTS)       | Process (AutoCV) |\n| GSM8K   | 64                          | 3                | 2.5                  | 5                |\n| MATH    | 480                         | 6                | 2.5                  | 5                |\n\nAs the number of steps required to solve a question increases (e.g., MATH compared to GSM8K), the time required for MCTS annotation increases **quadratically**. Considering training and annotation costs, the total GPU hours for AutoCV are approximately **1/8** of those for MCTS on GSM8K and approximately **1/40** for MATH. We will update Table 9 to reflect this and provide a more balanced view of the efficiency of AutoCV compared to MCTS-based methods.\n\n\n\n\n\n2. Flexibility and Further Performance Enhancement: MCTS-based methods **require ground truth** on the correctness of final answers. In contrast, AutoCV generates process annotations by detecting relative confidence variations without needing ground truth annotations. This flexibility allows AutoCV to utilize redundant unlabeled questions, improving the model's performance. The results are provided below:\n\n| **Response Generator** | Pass@5 | Self-Cons. | OSV (GSM8K) | MCTS(GSM8K) | OSV+PSV (GSM8K) | OSV+PSV (GSM8K+WizardLM) |\n| ---------------------- | ------ | ---------- | ----------- | ----------- | --------------- | ------------------------ |\n| Mistral-Instruct       | 69.90  | 50.03      | 61.18       | 60.82       | 61.41           | 63.11                    |\n| Mixtral-Instruct       | 82.30  | 69.06      | 74.91       | 75.10       | 76.04           | 78.15                    |\n| Qwen                   | 91.13  | 81.27      | 84.91       | 84.         | 85.15           | 86.77                    |\n\nWhen MCTS and OSV-only training are limited to GSM8K labeled data, our AutoCV extends to apply 7K unlabeled math problems following the Evol-Instruct method from WizardLM [1]. Including unlabeled data further enhances the model's capabilities, demonstrating the value of the AutoCV approach.\n\n\n\n**R Q1 Verifier Performance with Different Generators** \n\nWe applied the Llama2-13B [2] model and followed the settings in our paper, testing the MATH test set. The results are presented below:\n\n| **Response Generator** | Pass@5 | Self-Cons. | OSV+PSV (Mistral-7B) | OSV+PSV (LLaMA-13B) |\n| ---------------------- | ------ | ---------- | -------------------- | ------------------- |\n| Mistral-Instruct       | 7.70   | 1.64       | 5.30                 | 6.13                |\n| Mixtral-Instruct       | 22.80  | 10.66      | 16.92                | 20.19               |\n| Qwen                   | 56.10  | 40.10      | 39.36                | 43.13               |\n\nFrom the table, it is evident that increasing the model size to 13B improves the performance of AutoCV across all settings. This demonstrates that our method works effectively even with strong response generators like Qwen, not just weaker ones. It shows that AutoCV performs better when applied to stronger and larger LLMs.\n\n\n\n[1] [WizardLM: Empowering Large Language Models to Follow Complex Instructions. ICLR 2024](https://arxiv.org/abs/2304.12244)\n\n[2] [Llama 2: Open Foundation and Fine-Tuned Chat Models. Arxiv 2023](https://arxiv.org/abs/2307.09288)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The submission has several strenghs for supporting its acceptance. First, the proposed AutoCV approach combines the strengths of output supervision and process supervision to automatically annotate reasoning steps, enhancing the performance in drawing correct outcomes, even though in some test cases the absolute improvement is small. Second, different from previous work on process reward models that only focuses on the math domain, AutoCV also shows effectiveness on commonsense reasoning and natural language inference. Finally, the  presentation of the submission is fine. Notable issues on the presentation have been addressed in rebuttal.
Decision: Accept (poster)