AutoPSV: Automated Process-Supervised Verifier

Authors: jianqiao  lu
zhiyang  dou
hongru  wang
zeyu  cao
jianbo  dai
yingjia  wan
yunlong  feng
zhijiang  guo
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 09:34:48

Abstract: In this work, we propose a novel method named \textbf{Auto}mated \textbf{P}rocess-\textbf{S}upervised \textbf{V}erifier (\textbf{\textsc{AutoPSV}}) to enhance the reasoning capabilities of large language models (LLMs) by automatically annotating the reasoning steps.
\textsc{AutoPSV} begins by training a verification model on the correctness of final answers, enabling it to generate automatic process annotations. 
This verification model assigns a confidence score to each reasoning step, indicating the probability of arriving at the correct final answer from that point onward.
We detect relative changes in the verification's confidence scores across reasoning steps to automatically annotate the reasoning process, enabling error detection even in scenarios where ground truth answers are unavailable. 
This alleviates the need for numerous manual annotations or the high computational costs associated with model-induced annotation approaches.
We experimentally validate that the step-level confidence changes learned by the verification model trained on the final answer correctness can effectively identify errors in the reasoning steps.
We demonstrate that the verification model, when trained on process annotations generated by \textsc{AutoPSV}, exhibits improved performance in selecting correct answers from multiple LLM-generated outputs.
Notably, we achieve substantial improvements across five datasets in mathematics and commonsense reasoning.  The source code of \textsc{AutoPSV} is available at \url{https://github.com/rookie-joe/AutoPSV}.

Main paper: 
 introduction: 
 Large language models (LLMs) have shown impressive performance on various reasoning tasks [1] [2] [3] [4] . Prior efforts primarily focus on specific prompting techniques, such as few-shot prompting with intermediate steps and augmented demonstrations [5] [6] [7] [8] . While these methods have shown promise, their effectiveness is often task-specific, and designing prompts can be labor-intensive, leading to inconsistent results [9, 10] . Another approach to improve reasoning in LLMs is through instruction tuning or knowledge distillation [11] [12] [13] [14] . These methods typically involve fine-tuning LLMs and require a large set of examples annotated with chain-of-thoughts (CoT; [5] ). However, these approaches can be resource-intensive and may not always produce reliable results.To address these challenges, verification techniques have emerged as a promising solution [15, 16] . Verification models are trained to evaluate and potentially correct the reasoning process generated by LLMs. This approach aims to mitigate the risk of relying solely on the top-1 result, which may not always be reliable [17, 18] . By reranking candidate responses, verification models can ensure higher accuracy and consistency in LLM outputs, and provide valuable feedback for improving LLMs [19, 20] further.Verification models generally fall into two training paradigms: outcome supervision and process supervision. In outcome supervision, the training annotations rely on the correctness of the final answer [21, 22] , while in process supervision, the annotations are based on evaluations of each reasoning step [23, 19] . However, process supervision is demanding in terms of annotations. Typically, it relies on either expensive and highly skilled human evaluators [23, 16] or model-induced process annotations [18, 17] to estimate the future correctness of the current reasoning step using Monte Carlo tree search [24, 25] . In contrast, outcome supervision only requires annotations for the output, making it more economical in terms of annotation effort but less effective. That being said when answers involve multiple reasoning paths, all aforementioned model-induced methods require numerous samples to ensure accurate estimations.In this paper, we introduce Automated Process-Supervised Verfier (AUTOPSV), a novel approach that synergistically combines the strengths of both process supervision and output supervision. Our method begins by training an outcome-supervised verification model using outcome supervision annotations. This model then assigns confidence scores to each intermediate reasoning step, estimating their likelihood of contributing to a correct final answer. A distinguishing feature of AUTOPSV is its ability to automatically generate process annotations through relative step-level confidence change analysis, significantly reducing annotation effort while maintaining supervision quality without requiring ground truth answers. These automatically generated process annotations subsequently serve as training data for developing an enhanced verification model that leverages both process-level and outcome-level supervision signals. The complete framework of AUTOPSV is illustrated in Figure 1 . We conduct extensive experiments across five datasets, including mathematical reasoning benchmarks and commonsense reasoning tasks. The results demonstrate that our method effectively improves the reasoning capability of the model with our highly efficient labeling scheme for process supervision. Our contribution is summarized as follows:• We introduce AUTOPSV to automate the labeling of process data to enhance LLMs' reasoning capabilities. By combining the strengths of output and process supervision, AUTOPSV effectively identifies variations in model confidence to annotate the correctness of intermediate reasoning steps, enabling efficient automatic labeling for process supervision. • Comprehensive experiments demonstrate that AUTOPSV significantly improves the performance and scalability of verification models in mathematical and commonsense reasoning tasks. This approach greatly reduces the need for manual intervention and extensive computational resources, making it a valuable tool for enhancing LLM capabilities. • AUTOPSV's versatility is evident in its applicability to both labeled and unlabeled dataset settings after completing the training process. This flexibility and generalizability highlight the method's potential for widespread adoption in various LLM applications.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The main claims made in the "abstract" and "introduction" sections accurately reflect the paper's contributions and scope.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related works: 
 Improving Reasoning Abilities of LLMs To enhance the reasoning capabilities of LLMs, prior research primarily focuses on specific prompting techniques [26] . Existing efforts include few-shot prompting with intermediate steps augmented demonstrations [5] [6] [7] 27] or zero-shot prompting with specific instructions [28, 29] . Although these methods have shown promising results, their effectiveness is often constrained by their task-specific nature and the labour-intensive process of designing prompts, leading to inconsistent outcomes across different tasks [9, 10] . Another strategy to facilitate reasoning involves instruction tuning or knowledge distillation, which elicits reasoning paths from LLMs without explicit prompting [11] [12] [13] 30] . These approaches typically involve resourceintensive fine-tuning over LLMs and require a large set of examples annotated with chain-of-thoughts (CoT). Unlike methods that directly modify parameters or prompts, AUTOPSV focuses on training an additional verification model to select the desired output from the original model's output. This approach is further discussed in the context of process supervision in the following paragraph.From Outcome to Process Supervision Recent efforts have focused on enhancing the reasoning capabilities of LLMs through the use of verifiers to select the best answer from multiple candidates. There are two main types of verifiers: the Outcome-Supervised Verifier (OSV) and the Process-Supervised Verifier (PSV). The OSV is supervised with a signal based on the final answer [21, 22] , while the PSV is with detailed feedback which requires evaluating individual reasoning steps [15,Outcome-Supervised Verifier Process-Supervised Verifier Problem:Anna spent 1/4 of her money, and now she has $24 left. How much did she have originally?Suppose Anna originally had $x. She spent 1/4 of her money, which is $x/4. This leaves her with $x -$x/4 = $3x/4. Since $3x/4 = $24, we can solve for x:Since $3x/4 = $24, we can solve for x: 3x/4 = 24, x = 48.The answer is $48.Step 1Step 2Step 3Step 1Step 2Step 19, 16, 23] . Despite the time-consuming annotation cost, the PSV offers several advantages that make it preferable to the OSV. It can provide fine-grained feedback by pinpointing the location of errors, which is valuable for reinforcement learning and automatic correction [16, 20] . To alleviate the extensive human annotation, recent approaches [17, 18] propose a machine annotation framework using Monte Carlo Tree Search [24, 25] . This process demands a lot of computing resources, potentially imposing a limitation on the usage. AUTOPSV is more efficient, as it utilizes an outcomesupervised verification model to assign confidence to each reasoning step and calculate relative step-level confidence changes, eliminating the need for additional sampling or manual labeling.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We provide the limitations in the "Limitations" section attached in the Appendix.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
autopsv: 
 In this section, we introduce the main problem that this paper focuses on in Section 3.1. We then discuss the motivation behind why we believe it is necessary to train a verification model in Section 3.2. Finally, we describe how we accomplish the transition from outcome supervision to process supervision during the training of the verification model in Section 3.3.Objective Our research addresses the challenge of response selection from multiple candidates generated by a Large Language Model (LLM). Specifically, given an LLM acting as a response generator, we seek to develop an effective method for identifying the correct response among multiple generated solutions. Our primary goal is to maximize the probability of selecting an accurate solution from the available candidates.Notation We define q as the input question presented to the model, and S (1:t) i as the sequence of intermediate reasoning steps up to step t for the i-th solution to question q. The binary correctness label for the i-th solution is denoted as y i , and Q represents our training question dataset. This formalization enables us to systematically approach the response selection problem while maintaining mathematical rigor in our methodology.Response selection methods can be broadly divided into two categories: models specifically fine-tuned for the selection task and those that employ various prompting strategies.In our exploration, we initially investigate whether existing open-source LLMs could serve as effective selection agents to evaluate model outputs and choose the correct response without fine-tuning. We choose Mixtral-Instruct [31] as the response generator, and its response results are listed Table 1 . Our objective focuses on identifying the correct response from five candidate solutions.To establish robust conclusions, we evaluate selector models ranging from 7B to over 70B parameters, applying various prompting strategies. The results, tested on the GSM8K test set [21] , are presented in Table 2 . Notably, even models exceeding 70 billion parameters demonstrate suboptimal selection performance when relying solely on prompting without fine-tuning.Table 1 : Performance of Mixtral-Instruct on GSM8K. All results are reported in accuracy (%).Response Generator Model Size (Parameters) Pass@1 (%) Pass@5 (%) Self-Consistency (%) Mixtral-Instruct [31] 8 Based on these findings, our research focuses on training dedicated verification models and enhancing their response selection capabilities.AUTOPSV integrates outcome supervision with process supervision to create an effective training methodology. Below, we detail each component of our approach.Outcome-Supervision We begin by training an outcome-supervised verification (OSV) model, denoted as f θ (•), where θ represents the optimized parameters. The training utilizes Mean Squared Error (MSE) loss for each solution step:EQUATIONThe complete objective function across all training questions Q is:EQUATIONWhere n represents solutions per question and m i denotes steps in the i-th solution. The OSV output approximates the expected correctness probability, as formalized as follows:Theorem 1 For a model trained with outcome supervision, f θ , characterized by optimally tuned parameters θ, the assigned score for the sequence S (1:t) is an estimation of the likelihood of ultimately deriving a correct answer, denoted by â, based on the progression observed in S (1:t) and the pertinent question q. This is mathematically represented as:f θ (S (1:t) ; q) ≈ p(â|S (1:t) , q)The proof follows from optimizing the MSE loss in equation ( 2), with details in [22] .We compute the relative confidence change between steps:EQUATION∆ t conf represents the relative variation in the model's confidence score from step t to step t + 1. A negative value of ∆ t conf signifies a reduced confidence in achieving a correct answer after incorporating information from the (t + 1)-th step. We denote the process label for the t-th step as y t i and θ as the variation threshold.For process labeling, we employ the "first error location" strategy [15] with threshold θ:• If ∆ t conf > θ: y t i = 1 • Otherwise: y t i = 0 and ∀t ′ > t, y t ′ i = 0The process-supervision loss function is:EQUATION4 Preliminary FindingsIn this section, we present our findings aiming to validate two key aspects: In Section 4.1, we present a comprehensive analysis of the OSV model, i.e., to validate that the initially trained OSV model is effective and robust. In Section 4.2, we further introduce a self-designed benchmark for process errors and calculate ∆ t conf to detect these errors, i.e., to demonstrate the effectiveness and reliability of relative step-level confidence change in the proposed method. The validation of these two components serves as a foundation for automatic process labeling via AUTOPSV, as described in Section 3.3.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
experiment on outcome-supervised verifier performance: 
 In this section, we validate the effectiveness and scalability of the OSV model. Initially, we fine-tune a pretrained language model using ground truth data from the GSM8K dataset. Then, we use this fine-tuned model to generate multiple response samples for the GSM8K training prompts. We label these samples based on the correctness of their final answers. After this, we train an OSV model using the method described in Eq. (1).To evaluate the OSV, we measure its ability to select a sample with the correct final answer from samples generated by various LLMs, denoted as the Response Generator. Specifically, our task involves selecting the correct candidate from five responses. We assess the effectiveness of outcome supervision on two models: Phi2 (OSV (Phi)) [35] and Mistral-7B (OSV (Mistral)) [32] . To explore the scalability of this outcome-supervised verifier effect, we choose Response Generators of varying scales, ranging from 7B to 72B parameters, i.e., Mistral-7B-Instruct (Mistral-Instruct) [32] , Mixtral 8 × 7B (Mixtral-Instruct) [31] and Qwen-72B-Chat (Qwen) [34] . This allows us to check the OSV's generalized ranking capability across different LLM scales. The results demonstrate the effectiveness and scalability of the OSV model in selecting the correct response among multiple responses generated by different generators. Specifically, the OSV models, trained using either Mistral or Phi, consistently outperform the self-consistency (SC) baseline across all generator configurations. The results validate the effectiveness of the OSV model in enhancing model selection strategies, particularly when applied to larger and more accurate LLM generators.We further analyze the performance discrepancy between the two OSV models:Performance Analysis of Different OSVs The performance disparity among the verifiers can be attributed primarily to variations in model sizes and the quality of their training data. It's important to note that the OSV model is continuously trained from the GSM8K fine-tuned model with the addition of a value head. This means that the training data for each OSV model is generated from its corresponding fine-tuned base model. Specifically, the training data for OSV (Mistral) is generated from the fine-tuned Mistral model, while the training data for OSV (Phi) is generated from the fine-tuned Phi model. It is worth noting that while both models are trained on the same quantity of data per question (100 samples), the quality of this data differs due to the capabilities of their respective base models. The Mistral model, being larger and potentially more capable, generates higher quality training data for its OSV, which in turn leads to better performance. In our content, we select the OSV (Mistral) model as the OSV model among other experiment settings due to its superior performance, as demonstrated Table 3 .In this section, we verify the effectiveness and reliability of our method AUTOPSV. Specifically, We calculate ∆ t conf to identify inaccuracies in the process, as outlined in Eq. ( 3). In Section 4.2.1, we introduce the concept of math calculation error and establish a preliminary benchmark. In Section 4.2.2, we assess the performance of calculating ∆ t conf to detect calculation errors against our established benchmark.We outline a method for identifying instances of math calculation error, which we define as occurrences where the numerical values on either side of an equals sign within a mathematical expression do not align. This misalignment indicates a breakdown in logical reasoning, categorizing the instance as a calculation error in the context of mathematical problem-solving. This process establishes a benchmark for math calculation error detection with more details in Appendix E.To identify calculation errors in mathematical reasoning, we monitor the relative step-level confidence changes between the intermediate steps as defined in Eq. ( 3). if ∆ t conf ≤ θ, we view the step as "incorrect". We also provide a detection example in Figure 2 for better understanding.We introduce three metrics for a thorough evaluation of math calculation error detection: Precision (Prec.), which calculates the proportion of samples with correct final answers but exhibiting hallucinatory errors during the reasoning process; Recall, which determines the proportion of samples with math calculation errors that the OSV model successfully identifies through step-level confidence changes; F1-score, which gauges the verifier's overall efficacy. In Table 5 , we explore how different threshold (θ) values affect the precision, recall, and F1-score for math calculation error detection.The results in Table 5 demonstrate that our method using step-level confidence change effectively detects calculation errors across threshold values from -0.5 to -0.9. As the threshold becomes more negative (stricter for labeling errors), the precision increases, indicating higher precision in identifying true errors. However, the recall decreases, meaning fewer actual errors are caught. Importantly, the F1score, balancing precision and recall, remains relatively stable across thresholds. This demonstrates that our method strikes a good balance between detecting real errors and minimizing incorrect We note that setting θ = -0.5 in our detection methods helps maintain a balance between precision and recall, which can ensure a balanced distribution of labeled "incorrect" and "correct" responses.Validation and Foundation for AUTOPSV Our empirical validation of the OSV model encompasses two key aspects: its efficacy in response selection (Section 4.1) and its capability in detecting calculation errors (Section 4.2). These experimental results provide a robust foundation for automating process annotations using AUTOPSV. Furthermore, Theorem 1 establishes the theoretical framework for utilizing OSV to estimate the probability of reaching correct final answers from any given intermediate reasoning step. This convergence of theoretical guarantees and empirical evidence provides the methodological groundwork for applying AUTOPSV to generate process-supervised training data in our subsequent experiments. 
experiment: 
 In this section, we first introduce the experimental setup in a subsection, which includes the response generator LLMs and evaluation settings in Section 5.1. We then present the main result of our process supervision-enhanced verification model on both mathematical and commonsense reasoning benchmarks, as described in Section 5.2.Models: We selected three instruction-tuned LLMs of varying sizes, ranging from 7 billion to over 70 billion parameters, to serve as the response generator. Specifically, we used Mistral-Instruct-7B (Mistral-Instruct), Mixtral-8x7B-Instruct-v0.1 (Mixtral-Instruct), and Qwen-72B-Chat (Qwen).Datasets: Our evaluation encompasses five benchmarks across two domains: For mathematical reasoning, we include GSM8K [21] , containing math word problems requiring multi-step reasoning, and MATH [36] , composed of high school-level competition problems covering a range of math subjects. For commonsense reasoning, we use HellaSwag [37] , a dataset for physically situated commonsense reasoning, Winogrande [38] , fill-in-the-blank problems requiring commonsense pronoun resolution and ANLI [39] , a dataset for natural language understanding and reasoning.Evaluation: For evaluation, we follow the methodology outlined in [40] to ensure consistency across benchmarks. Our protocol involves: (i) Our object is to select the correct answer from five candidate responses.(ii) For OSV models, we use the final value assigned to the whole solution for evaluation, while for PSV models, we utilize the product of step-level scores as the aggregation function.(iii) To obtain more reliable pass@k results, we implement the estimation method described in [41] . This involves generating n samples per task (where n > k) and assessing the number of correct samples that pass unit tests. We then calculate the unbiased estimator for pass@k. For self-consistency (Self-Cons.) and verifier results, we randomly select k out of n samples and perform separate calculations. All results are reported with an accuracy of ±0.1 at a 95% confidence level.Additional details regarding generation hyperparameters and different aggregation functions are provided in Appendix F.2.To evaluate the efficacy and scalability of our proposed approach (detailed in Section 5.2), we conducted comprehensive experiments across mathematics and commonsense reasoning tasks using five diverse datasets.Our experimental framework employs an autonomous process-supervision data annotation method where we calculate ∆ t conf based on model confidence from OSV, using a threshold of θ = -0.5. This annotated data is then utilized to continuously fine-tune the OSV model, resulting in our enhanced OSV + PSV model.We evaluate three distinct approaches: (i) Self-Consistency (Self-Cons.): Our baseline approach (ii) Outcome-supervised verifier (OSV): Our initial verification model (iii) Process-supervised enhanced verifier (OSV + PSV): Our proposed enhancement. For each approach, we assess Pass@5 performance, which represents the upper limit of achievable performance on these benchmarks.Mathematics Reasoning: As shown in Table 6 , the process-supervised enhanced verifier demonstrates superior performance over the outcome-supervised verifier and Self-Consistency models for all evaluated response generators on GSM8K. For the MATH benchmark, the process-supervised enhanced verifier outperforms the other two approaches for Mistral-Instruct and Mixtral-Instruct, but it is slightly less effective than the Self-Consistency model when applied to Qwen-72b. Commonsense Reasoning: According to Table 7 , OSV + PSV again leads to the best results among the three methods for each response generator tested on HellaSwag. For Winogrande, Mistral-Instruct paired with OSV + PSV achieves the highest performance, whereas, for Mixtral-Instruct and Qwen-72b, the original OSV without process supervision has a marginal advantage. When looking at the results of the ANLI benchmark, OSV + PSV is the highest-performing method for Mistral-Instruct and Mixtral-Instruct. Despite this, for Qwen-72b, the OSV model alone falls slightly behind the integrated OSV + PSV.Table 7 : Results on commonsense reasoning benchmarks.Response Generator HellaSwag Winogrande ANLI Pass@5 Self-Cons. OSV OSV + PSV Pass@5 Self-Cons. OSV OSV + PSV Pass@5 Self-Cons. OSV OSV + PSV Conclusion: Our experimental results indicate that the process-supervised enhanced verifier (OSV + PSV) consistently outperforms or matches the baseline models across most mathematical and commonsense reasoning tasks. By leveraging automatic process annotations, our approach enhances the model's capacity to verify reasoning processes, resulting in improved accuracy and robustness across a wide range of benchmarks and response generators.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: We provide the training scripts in the supplementary material.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
analysis: 
 In Section 6.1, we compare our process annotation method, AUTOPSV with two other modelinduced annotation methods to showcase the effectiveness and efficiency of our proposed approach. In Section 6.2, we validate the data quality constructed via AUTOPSV as described in Section 5.2.Aside from the labeling method defined by Eq. ( 3) in AUTOPSV, another labeling strategy is the Monte Carlo Tree sampling estimation (MCTS), as described in [17, 18] . To better demonstrate the effect of our method, we make a comparison with this approach and conduct experiments on mathematical benchmarks across both labeled and unlabeled settings (i.e., whether the ground-truth answers to the questions are provided).Performance in Labeled Settings We first evaluate the performance of AutoPSV in labeled settings. We follow the experimental settings described in [17, 18] to ensure a fair comparison. More implementation details are provided in Appendix F.3. Table 8 presents a comparison of process labeling methods across different response generators on the GSM8K and MATH datasets. The experimental results shown in Table 8 suggest that our proposed method for process labeling, which relies on detecting changes in model confidence, performs competitively with the MCTS method from [18, 17] . In some cases, our method even outperforms the MCTS method, especially on the more challenging MATH benchmark.A key advantage of AutoPSV is its computational efficiency. As shown in Table 9 , our method requires significantly fewer tokens for process labeling compared to MCTS-based methods. This efficiency stems from AutoPSV's ability to generate process annotations without requiring multiple samples for each reasoning step, making it particularly suitable for large-scale applications or scenarios with limited computational resources.To further demonstrate the flexibility of AutoPSV, we evaluate its performance in unlabeled settings. We generated an additional dataset of 7,000 unlabeled math problems using the Evol-Instruct method from WizardLM [42] . These problems, created by LLMs without accompanying gold solutions, represent a challenging scenario for traditional supervision methods. We then conducted an experiment incorporating these unlabeled questions alongside the GSM8K dataset. The results demonstrate that the addition of unlabeled data leads to noticeable improvements across all response generators. For instance, the performance of Mistral-Instruct improves from 61.18 (OSV) to 63.11 (OSV+PSV with GSM8K+WizardLM). These results further underscore the value of the AutoPSV approach, particularly its ability to effectively utilize unlabeled data for enhanced performance.In summary, AutoPSV offers several key advantages over MCTS-based methods:Consistent Performance with Computational Efficiency: AutoPSV demonstrates robust, consistent improvements across different response generators and datasets. Its ability to efficiently generate process annotations-without the need for extensive sampling like MCTS-makes it particularly well-suited for large-scale applications and environments with limited computational resources.Leveraging Unlabeled Data for Enhanced Performance: Unlike MCTS, which relies on ground truth labels, AutoPSV can effectively utilize unlabeled data. This capability not only enhances model performance in real-world settings but also offers scalability in scenarios where labeled data is scarce.The flexibility to harness unlabeled data ensures that AutoPSV can drive significant improvements even in data-constrained situations.We apply the OSV to relabel the process-supervised training data as in Section 5.2. We then retrain a new model using this relabeled data. This experiment highlights the performance gap between outcome-supervised and process-supervised training. The experimental results in Table 11 reveal that retraining the model with process supervision from AUTOPSV still yields better performance than self-consistency across three different response generators. We also noticed a small performance gap between the PSV and OSV. It is worth noting that our PSV was trained using data from the OSV. The small performance gap between the PSV and OSV models demonstrates that the relabeled process-supervised training method successfully inherits information from the outcome-supervised model without requiring ground truth annotations. This ablation study further provides quality assurance for automatic process labeling via AUTOPSV. Moreover, OSV training is limited to labeled datasets, while AUTOPSV demonstrates superior performance by utilizing both labeled and unlabeled data as shown in Table 10 . This comparison further highlights the versatility and effectiveness of AUTOPSV in real-world scenarios where ground truth annotations may be scarce or unavailable.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The paper provides comprehensive details regarding the training and testing in the "Appendix" due to the page limit.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 In conclusion, we propose a novel method for automatic process labeling in LLMs by detecting relative changes in model confidence. Our experimental results demonstrate that AUTOPSV significantly enhances the precision and scalability of the verifier models in various reasoning tasks, ranging from mathematical to commonsense reasoning. AUTOPSV therefore has the potential to considerably enhance existing LLMs' performance while drastically reducing the need for intensive computation and manual intervention. For future work, we aim to utilize the automatically constructed PSV to supervise the generator using step-wise proximal policy optimization, to enhance the accuracy of the generator's output during greedy decoding without the need for subsequent reranking. This avenue of research could lead to even more advancements in the capabilities of LLMs and their application in reasoning tasks. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We mentioned in "Experiment" section that The results are presented with an accuracy of ±0.1, calculated at a 95% confidence level.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We discuss the societal impact of the work in the "Broader Impact" section attached in the Appendix. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper does not present any high-risk data or models that could be prone to misuse, as it is focused on enhancing the performance of Language Model Support Systems (LLMS) within the field of Natural Language Processing (NLP). Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Yes, all experiments conducted in the paper utilize open-source models. Proper references and citations are included in the paper. Guidelines:• The answer NA means that the paper does not use existing assets.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: AUTOPSV is a promising solution for enhancing the reasoning capabilities of LLMs. However, it is important to acknowledge several potential limitations of the method. Firstly, while AUTOPSV aims to reduce the need for manual intervention, there is still a risk of inaccurate annotations. The relative step-level confidence change used to produce process annotations is an estimation and may not always accurately represent the actual correctness of a reasoning step. This could compromise the quality of the annotations and, in turn, the effectiveness of the method. Secondly, the success of AUTOPSV is heavily dependent on the performance of the verifier. If the model is not accurate enough in its step-level scores, the quality of the process annotations generated by AUTOPSV could be compromised. Thirdly, AUTOPSV is specifically designed to improve the reasoning capabilities of LLMs. Therefore, its applicability may be limited to tasks that involve complex multi-step reasoning.It is unclear how well the method will scale or generalize to other tasks or domains that do not involve intensive reasoning. This is an important consideration for future research and development of the method.Positive Societal Impacts The proposed AUTOPSV has the potential to bring about several positive societal impacts. By enhancing the reasoning capabilities of LLMs, AUTOPSV can lead to more accurate and reliable information, which in turn can support better decision-making in various sectors, including healthcare, education, and finance. Moreover, AUTOPSV combines the strengths of output supervision and process supervision to automatically annotate reasoning steps, significantly reducing the time, effort, and cost associated with manual annotation. This makes the process of training LLMs more efficient and accessible. Additionally, the process supervision data generated by AUTOPSV can improve the performance and scalability of verification models, allowing for the development of more complex and sophisticated LLMs capable of handling a wider range of tasks and applications.Positive Societal Impacts However, AUTOPSV also presents several potential negative societal impacts. The automation of the annotation process could lead to job displacement for individuals currently employed in this role. There is also a risk that AUTOPSV and the enhanced LLMs could be misused, for instance, to spread misinformation or manipulate public opinion. The increased reliance on LLMs for decision-making could potentially result in a decrease in critical thinking and problem-solving skills among individuals. Furthermore, the use of LLMs in various sectors could lead to privacy and security issues, as these models often require large amounts of data for training.Learning From Feedback Improving LLMs through learning from feedback has become a prevalent strategy, notably through reinforcement learning from human feedback, which seeks to align LLMs with human values by refining their outputs based on feedback [43] [44] [45] . However, this method faces challenges such as high costs due to manual labor and a lack of real-time feedback capabilities [46, 47] . An alternative strategy involves using self-correcting LLMs, which rely on automated feedback to iteratively adapt and understand the consequences of their actions without heavy reliance on human intervention. This feedback can be derived from outside sources such as other models [48, 49] , tools [50, 51] , knowledge bases [52, 53] , evaluation metrics [54, 55] or generation logits [56] .External feedback leverages external perspectives to identify errors and verify factual accuracy, offering insights that may not be recognized by the LLM alone. Conversely, feedback can also be internally generated, where the LLM evaluates and refines its output iteratively until a desired quality is achieved [57] [58] [59] [60] . This self-improvement mechanism is particularly valuable in scenarios where external feedback is scarce or restricted [61, 62] . However, recent effort [63] suggests that LLMs struggle to independently identify and correct errors through self-generated prompts.Classification: For this method, the evaluator is presented with multiple answers for a given question and is required to choose the best answer among them. The selection is made based on the evaluator's judgment of which answer most accurately addresses the question or provides the most relevant information.Given the following question: '[question]', and these five answers:1.[answer]2.[answer]3.[answer]4.[answer]5.[answer] Which answer is the best? Please provide the number of the best answer. You should strictly follow the output format requirements and not output any other content. Example: Answer [number] is better. Let's Begin! Classification + COT: In Classification + COT, the evaluator must not only identify the best answer but also analyze and compare all provided answers before making their decision. This method requires a deeper examination of the content and context of each answer to determine its quality and relevance to the question.Given the following question: '[question]', and these five answers:1.[answer]2.[answer]3.[answer]4.[answer]5 Scoring: In the Scoring method, the evaluator assigns a numerical score to each answer based on its quality or relevance to the given question. The scores typically range from 1 to 10, with 10 representing the highest quality or most relevant answer.Given the following question: '[question]', please score these five answers on a scale from 1 to 10, where 10 is the best:1.[answer]2.[answer]3.[answer]4.[answer]5.[answer] Please provide a score for each answer. You should strictly follow the output format requirements and not output any other content. Example: Answer i: [score]. Let's Begin! Scoring + COT: Similar to Scoring, Scoring + COT also involves assigning numerical scores to each answer. However, in Scoring_cot, the evaluator is required to provide an analysis of each answer before assigning a score. This analysis informs the scoring process and ensures a more informed evaluation of each answer.Given the following question: '[question]', please score these five answers on a scale from 1 to 10, where 10 is the best:1.[answer]2.[answer]3.[answer]4.[answer]5Please analyze each answer, and then provide a score for each answer. You should strictly follow the output format requirements and not output any other content. Pairwise Comparison: In this method, the evaluator is presented with pairs of answers for a given question and is tasked with determining which answer is better in each pair. The evaluator compares the content or quality of each answer and selects the one they deem superior. The Pairwise Comparison method differs from other evaluation methods in that it evaluates two candidate answers at a time and chooses the winner to proceed to the next comparison with the next candidate answer. For a set of n candidates, this method conducts n-1 pairwise comparisons. To mitigate the potential order preference bias exhibited by LLMs, we adopt a method similar to [64] which shuffles the positions of two answers during prompting. This ensures a fair evaluation process by eliminating any bias toward the position of the answers.Given the following question: '[question]', compare each pair of answers and decide which one is better: Compare 1. [answer1] with 2.[answer2] For the comparison, indicate the better answer with its number. You should strictly follow the output format requirements and not output any other content. Example: Answer i is better. Let's Begin! By employing these different evaluation methods, we aim to comprehensively assess the quality and relevance of the answers generated by our models for various questions. Each method offers a unique perspective and contributes to a more thorough evaluation process.Methodology We utilize the LlaMA2-chat model (LlaMA) for mathematical reasoning step generation. Using regular expressions, we extract computational steps and evaluate expressions to the left of the "=" sign using Python's eval function to verify their correctness against right-hand side results. We term this process "Math Calculation Error Detection" and present the results in Table 12 .Table 12 : Performance of the LlaMA model on GSM8K training data, evaluated through few-shot cot prompting strategy.Pass@5 (%) Self-Consistency (%) Math Calculation Error Detection (%) LlaMA 0.4791 0.2881 0.1824 Data Processing Non-computational expressions (e.g., "x + 1 + 2 = 4") that cannot be evaluated due to unsolvability or incorrect formatting are excluded from the ground truth data. This ensures our ground truth accurately reflects only computational errors during reasoning. While DeepMind [15] employs human labelers for "trace errors" annotation, our automated approach using Python's eval function provides a scalable alternative.Error Detection Example To demonstrate our calculation of ∆ t conf for math calculation error identification, we present an example in Figure 2 . The red highlights indicate calculation errors in the reasoning process, detected through significant decreases in model confidence.Computing Infrastructure Our experiments were conducted using 8 NVIDIA A100 GPUs, each with 40GB of memory. All models underwent full-parameter fine-tuning using the AdamW optimizer.Verifier Training Configuration For both process-supervised and outcome-supervised methods, we maintain consistent training parameters as detailed in Table 13 . The training process spans 1 epoch with a batch size of 512 and a learning rate of 2 × 10 -6 , incorporating a 3% learning rate warmup period. For comprehensive details regarding OSV implementation, refer to [22] .NeurIPS Paper Checklist† Corresponding Author. 


Decision: The submission has several strenghs for supporting its acceptance. First, the proposed AutoCV approach combines the strengths of output supervision and process supervision to automatically annotate reasoning steps, enhancing the performance in drawing correct outcomes, even though in some test cases the absolute improvement is small. Second, different from previous work on process reward models that only focuses on the math domain, AutoCV also shows effectiveness on commonsense reasoning and natural language inference. Finally, the  presentation of the submission is fine. Notable issues on the presentation have been addressed in rebuttal.

Reviews: review: paper_summary: paper_summary: The authors propose AutoCV, a method for solving multi-step reasoning tasks with chain-of-thought prompting that involving automatically labeling each step of the multi-step process based on its likelihood of leading to the correct outcome, and combining an outcome supervised classifier (OSV) and process supervised classified (PSV) to come up with a more effective method. Experiments on 2 math and 3 reasoning datasets show that OSV + PSV outperforms self-consistency as well as just OSV across the board, albeit by a small margin in several cases.
strength_weakness: strength_weakness: Not having familiarity with the area, I wasn't able to judge the novelty of the work and the substance (i.e., whether there is enough new material to warrant publication). It seems to me that f_theta() being a good indicator of the probability of reaching the correct final outcome is something known from reference [22]. The new part here is looking at the delta of f_theta() between time steps t and t+1, as done in Eq (3).

In terms of naming, I found the use of the phrase "confidence variation" not intuitive for referring to the change in the confidence from t steps to t+1. To me, confidence variation suggests how the confidence changes as one varies some parameter or some sampling, not as one takes another step. In a similar vein, I found "AutoCV" to not be an informative name for the system -- it doesn't connect well with the task of multi-step reasoning or with chain-of-thought prompting. That said, this is a subjective choice and it's up to the authors to decide if they want to look for a more intuitive name.

As noted above, I found the discussion in section 4.2 somewhat confusing.

Lastly, as also noted above, while OSV + PSV generally shows consistent gains over the OSV, the gains are often very small. This somewhat lowers the value of the proposed technique, but consistent gains remain a net positive.
questions: questions: Please see weaknesses above.
limitations: limitations: Yes
paper_summary: paper_summary: This paper proposes a new method (AutoCV) that bridges the gap in popular techniques for enhancing reasoning capabilities of LLMs. Prior work have proposed *verification models* (models that evaluate generated reasoning steps and rerank candidate responses) as a promising solution. Here the community has focused on two training paradigms for verification models: (1) outcome supervision (train on correctness of final answer; cheap but less effective) and (2) process supervision (train on labels for each reasoning step; expensive but effective). AutoCV attempts to bridge these two by first training an outcome-supervised model and then use it to annotate confidence for unannotated intermediate steps. Then, by calculating the relative confidence variation between steps, AutoCV generates process supervision data that can be used for training.
strength_weakness: strength_weakness: My concerns are mainly around the presentation, clarity, and significance of some results:

**Presentation/Clarity**
1. Throughout the paper, there are several experiments using verifier at inference to rerank generations from a *response generator*. For instance, Section 4.1 and Table 3. However, there is no mention or analysis of the #samples being reranked?

2. Similarly, there is no mention (or analysis) of the aggregation function used for process-supervised models at inference time?

3. Section 4.1 and Table 4 attribute the performance disparity among OSVs (for Phi2 and Mistral) to training data quality. Particularly confusing, was why the training data was different at all for each model? From the description, it seems like the training data was generated from _a single_ (GSM8K fine-tuned) pretrained LLM. If not, how does using a stronger model to generate data affect performance?

**Significance of results.** Table 6-7 describe the performance of a PSV finetuned on self-generated confidence variations of an OSV. From that, it seems like there is *very minimal gains* from using the PS data generated to enhance (finetune) the OSV model. Furthermore, when the PS data is used to retrain a new PSV, the performance even drops a few points (Table 10).

1. While one can argue that there is "minimal loss" in information from the OSV to a PSV, it concerns me that the quality of process labels might not be good. The generated labels add little-to-no value to overall performance? i.e., why use AutoCV to train a PSV at all if OSV is equally effective AND can provide process-annotations scores?

2. With minimal gains, the only advantage of AutoCV is that it is more sample efficient than MCTS based approaches for PSV training. Are there contributions beyond efficiency?
questions: questions: Note: Added most questions in the Weaknesses section. Additional:

Could the authors comment on how this would work with Agent based reasoning traces? Particularly, the procedure to generate process supervision labels (described in Section 3.3) would not work because it makes the following assumption: `any step, including the final result, following a step containing an error is considered incorrect`. This does not hold true for agentic approaches that use environment feedback to correct their mistakes (think code generation w/ execution feedback).
limitations: limitations: Yes
paper_summary: paper_summary: This paper introduces AutoCV, a novel method to create process reward models (PRMs). The approach involves training an an outcome-supervised verification model based on (S_{(1:t)}, y) pairs where S_{(1:t)} is the 1st~t-th steps in the response and y is the final correctness label of the entire response. The outcome-supervised verification model then generates process annotations by detecting confidence variations across reasoning steps. AutoCV combines outcome and process supervision, reducing the need for manual annotations or computationally expensive methods. Experiments across five datasets in mathematics and commonsense reasoning show a significant advantage of the PRM for best-of-N sampling. The method outperforms self-consistency baselines and outcome-supervised verifiers while requiring fewer tokens for annotations than existing approaches.
strength_weakness: strength_weakness: Though the method is interesting, the experiments are not very convincing and the paper is not well presented. 
- I cannot get the point of section 4.2.1 since it's a bit confusing to me. Is there a reason why the method is specifically useful for "process calculation hallucination detection"? It seems to me that it's a general method to detect all faulty steps in a response, which is what a PRM is supposed to do. Also, I think the term "hallucination" may be overloaded, which originates from an open-ended generation. I suppose the setting in section 4.2.1 is simply "calculation error detection".
- There is no baseline in the main experiment (Table 6&7). I'd suggest moving section 6.1 to the main exp, and results beyond the math domain as a separate subsection in section 5.
- The setup part (sec 5.1) does not mention if the experiments are all in-domain. OOD evaluations are required to examine the generalizability.
- From Table 8, AutoCV performs very comparable with MCTS-based methods with differences of <1% accuracy. The performance gain is marginal.
- Table 9 may not be a fair comparison of efficiency between AutoCV and MCTS-based methods. Despite using fewer tokens, AutoCV requires training an ORM/OSV for annotation which could also be costly. Maybe the sum of GPU hours for training/inference would be fairer?
questions: questions: - It seems the verifiers are based on the Mistral series and in general its gain is less significant when the response generator is Qwen, especially on MATH where AutoCV best-of-5 underperforms self-consistency. Is it because the method lacks generalizability or does the method only work well when the response generator is weak?
limitations: limitations: See above

Comments: title: Re: Rebuttal by Authors
Impact of Annotated Data on Verifier Capability and Final Performance
Reviewer Response
comment: Thank you for the detailed and informative response!  I am happy to see your willingness to reconsider the system name and for your efforts in explaining section 4.2 better.  Also happy to hear that the small-looking gains are statistically significant; please make sure to include that in case it's not already mentioned.

I remain in favor of accepting this paper.
Thank you for your feedback. We have conducted additional experiments to investigate how the quantity of annotated data affects the verifier's capability and its impact on the final outcomes.

1. **Verifier Capability:** We trained the OSV on subsets of the annotated data, specifically using 25%, 50%, and 75% of the full dataset. We then evaluated its performance in detecting mathematical calculation errors, applying the same metrics defined in Section 4.2.2 with a threshold of $\theta = -0.5$. The results obtained using the full training dataset are provided below, as also presented in Table 5 of the original manuscript.

**OSV Performance in Detecting Mathematical Calculation Errors:**

|              | **25%** | **50%** | **75%** | Full |
| ------------ | ------- | ------- | ------- | ---- |
| **Accuracy** | 0.78    | 0.82    | 0.84    | 0.85 |
| **Recall**   | 0.81    | 0.86    | 0.88    | 0.90 |
| **F1-Score** | 0.79    | 0.83    | 0.87    | 0.88 |

**Experiment 1 Analysis:** When the training data exceeds 1/2 of the full dataset, the incremental gains become less pronounced, with the performance at 3/4 of the data closely approaching that of the full dataset. This suggests that our method remains effective even with a moderate amount of annotated data, making it adaptable to scenarios where the availability of labeled data is limited.

2. **Impact on Final Outcomes:** Additionally, we evaluated the final performance of our OSV + PSV combination on both the GSM8K and MATH test sets.

**Final Performance of OSV + PSV on the GSM8K Test Set:**

|                        | **25%** |           | **50%** |           | **75%** |           | **Full** |           |
| ---------------------- | ------- | --------- | ------- | --------- | ------- | --------- | -------- | --------- |
| **Response Generator** | OSV     | OSV + PSV | OSV     | OSV + PSV | OSV     | OSV + PSV | OSV      | OSV + PSV |
| Mistral-Instruct       | 58.12   | 60.13     | 59.66   | 60.72     | 60.45   | 61.10     | 61.18    | 61.41     |
| Mixtral-Instruct       | 71.14   | 73.75     | 72.53   | 74.39     | 73.93   | 75.52     | 74.91    | 76.04     |
| Qwen                   | 80.59   | 82.93     | 82.47   | 84.10     | 84.01   | 84.83     | 84.91    | 85.15     |

**Final Performance of OSV + PSV on the Math Test Set:**

|                        | **25%** |           | **50%** |           | **75%** |           | **Full** |           |
| ---------------------- | ------- | --------- | ------- | --------- | ------- | --------- | -------- | --------- |
| **Response Generator** | OSV     | OSV + PSV | OSV     | OSV + PSV | OSV     | OSV + PSV | OSV      | OSV + PSV |
| Mistral-Instruct       | 4.13    | 4.41      | 4.55    | 4.72      | 4.95    | 5.15      | 5.10     | 5.30      |
| Mixtral-Instruct       | 12.59   | 13.87     | 13.97   | 14.50     | 14.80   | 16.52     | 15.20    | 16.92     |
| Qwen                   | 35.20   | 36.91     | 37.10   | 38.54     | 38.13   | 39.01     | 38.94    | 39.36     |

**Experiment 2 Analysis:** A similar trend is observed in Experiment 2, where the impact of annotated training data becomes even less pronounced when combining OSV and PSV. As the verifier's capability improves with more training data, the incremental gains in the final performance of the OSV + PSV combination decrease after the training data surpasses 50% of the full dataset. For example, on the GSM8K test set, the difference in performance between using 75% and the full dataset with OSV alone is 0.73 (60.45 vs 61.18), whereas, with the OSV + PSV combination, this difference reduces to just 0.31 (61.10 vs 61.41). Similarly, this effect is consistently observed on the MATH test set. This indicates that our approach remains not only effective with a moderate amount of ground-truth-labeled data but also that **the combined OSV + PSV system is particularly robust, showing even smaller performance drops when the training data is reduced.** This further reinforces the adaptability and efficiency of our method in scenarios where labeled data for the OSV are limited.



**Overall Conclusion:**   These results confirm that while more annotated data enhances verifier performance, our approach remains robust and effective even with limited data. The combined OSV + PSV system shows minimal performance degradation with reduced training data and **can also utilize unlabeled questions to improve its performance**, making it well-suited for practical applications where labeled data is limited. 

We will incorporate this into the revised findings to provide a comprehensive evaluation of our method. We would love to provide any further details if you have more questions. Thank you again for your thoughtful review.
Thanks for your clarification.

Most of my concerns have been addressed, but I think AutoCV still needs ground truth for verifier training. Yes, the process annotation stage does not require so, but the number of annotated data for verifier training may have an effect on the final result. 

Anyway, I am pretty sold by the approach now and will raise my score. I would love to further increase the score if the authors can show the effects of verifier capability on the final process reward annotation.