Incentivizing LLMs to Self-Verify Their Answers
Authors: Fuxiang Zhang
Jiacheng Xu
Chaojie Wang
Ce Cui
Yang Liu
Bo An
Venue: NeurIPS 2025 poster ()
Keywords: large language model
math reasoning
reinforcement learning
Primary area: deep_learning
TL;DR: We introduce an RL framework that unify the training of answer generation and verification in a single model.

Abstract: Large Language Models (LLMs) have demonstrated remarkable progress in complex reasoning tasks through both post-training and test-time scaling laws. While prevalent test-time scaling approaches are often realized by using external reward models to guide the model generation process, we find that only marginal gains can be acquired when scaling a model post-trained on specific reasoning tasks. We identify that the limited improvement stems from distribution discrepancies between the specific post-trained generator and the general reward model. To address this, we propose a framework that incentivizes LLMs to self-verify their own answers. By unifying answer generation and verification within a single reinforcement learning (RL) process, we train models that can effectively assess the correctness of their own solutions. The trained model can further scale its performance at inference time by verifying its generations, without the need for external verifiers. We train our self-verification models based on Qwen2.5-Math-7B and DeepSeek-R1-Distill-Qwen-1.5B, demonstrating their capabilities across varying reasoning context lengths. Experiments on multiple mathematical reasoning benchmarks show that our models can not only improve post-training performance but also enable effective test-time scaling. Our code is available at https://github.com/mansicer/self-verification.

Incentivizing LLMs to Self- Verify Their Answers
Fuxiang Zhang
Jiacheng Xul
Chaojie Wang²
Ce Cui
Yang Liu?
Bo An 1.2+
Nanyang Technological University, Singapore
Skywork Al
Abstract
Large Language Models (LLMs) have demonstrated remarkable progress com- plex reasoning tasks through both post-training and test-time scaling laws. While prevalent test-time scaling approaches are often realized by using external reward models to guide the model generation process, we find that only marginal gains can be acquired when scaling a model post-trained on specific reasoning tasks. We identify that the limited improvement stems from distribution discrepancies between the specific post-trained generator and the general reward model. To address this, we propose framework that incentivizes LLMs to self-verify their own answers. By unifying answer generation and verification within a single reinforcement learning (RL) process, train models that can effectively assess the correctness of their own solutions. The trained model can further scale its performance inference time by verifying its generations, without the need for external verifiers. We train our self-verification models based on Qwen2.5-Malte 7B and DeepSeek-RI -i-issil-----isssss---eeeee 1...--..BB demonstrating their capabilities across varying reasoning context lengths. Experiments on multiple mathematical rea- soning benchmarks show that our models can not only improve post-training performance but also enable effective test-time scaling. Our code is available at https://github. httph//...ooommmmineeeee/eeeee/e verification
MATH500 Score (avg@1) 1)
AIME24 Score (avg@10) DeepSeeek-R---islll--Weeeer
Model: Swen2.5-Math-2.
90.0
Selsortiication (Oun)
Sellveeriicaiion (Oun)
Selicanaddeney
Sell-corsiterny
60.0
Seat-cfi
Bear Search
55,0
DVTS
sobtnng Verfication lmmooling Only
Prob em-an Werlication
85.0
Prob em-uo
50.0
+14.4
13.4
62.0
35.0
Number of Sump mc
Number Sample mx
Figure 1: Average performance of post-trained models (dotted lines) and test-time scaling methods (solid lines) on the MATH500 and AIME24 benchmarks. Our self-veriffcation framework not only enhances post-training performance with RL on both problem-solving and verification, but also enables effective test-time scaling with increased generation numbers by verifying its own solutions.
Introduction
Large Language Models (LLMs) have demonstrated remarkable capabilities in wide range of natural language tasks [T,, particularly excelling in complex reasoning challenges such as mathematics
"Corresponding author. Email: boan@ntu. "han@ntu.ddu.st sg
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Problem-Solee Solviin
Solution Verification
Problem
Question #1
Question #2
the of 555,, 55, and 5,
aum of 555,,
555,
base 6.
Sample
Sample
SS, and S, in base 6.
55, and Sy in base B.
Self-
At The snswer is 110%,
A:.. The answer 715g-
Verification Model
Please the
Please verify the
stop by stop.
by
Responses
To 555, 55, and 5, in base well, we will follow these steps:
Verification
Verification
veriifyy the solution, let's go through the steps againe
ve the colution, let's through the steps again:
ORPO
Thus the
Answer
correct
correct
Verification reward
(Yes/Nof? Yes
MeshNoaj? No
correctness
reward
Test-time Answer Aggregation
Figure 2: The framework of our self-verification framework. The model is trained to solve mathemat- ical reasoning problems and verify generated solutions simultaneously.
and logic puzzles 4 [1. For these verifiable tasks with ground-truth answers, researchers have proposed scaling laws in both post-training and test-time reasoning that substantially enhance LLM performance [6]. On the one hand, reinforcement learning (RL) has been widely adopted align LLMs by rewarding predictions that match gold answers (7]. On the other hand, researchers leverage additional computational resources at inference time improve accuracy through ensemble approaches 8 9
and dedicated reward models (RMs) as verifiers [10H12]
Although both post-training and test-time scaling laws have significantly advanced the performance of LLMs, we observe that their combination yields limited synergistic benefits. Post-trained models often specifically tuned on training data, resulting in narrow generation distribution, which is often unrecognized by external verifiers trained on regular data. This distribution shift may lead to incorrect solution verifications and thus hinders test-time scaling performance [13] As illustrated in Figure I we find that commonly used test-time scaling methods such as best-of-N and beam search with external RMs offer minimal improvements compared to the simple self-consistency method through majority voting. It remains challenging to synergize post-training and test-time scaling within an effective framework.
In this work, we propose to leverage self-veriifcaatio to bridge the gap between post-training and test-time scaling. As illustrated in Figure 2 the core idea of self- silf-verificttion is straightforward: train the model not only to generate answers but also to verify the correctness of its solutions. Inspired by previous generative verifier works [64 we treat the pretrained LLM verifier, and then adopt RL algorithms to incentivize the L.M for self-verification, using reward signals derived from ground-truth answer correctness. We further design online policy-aligned buffer and dynamic verification rewards to stabilize the joint training of answer generation and verification. The online buffer ensures that the input distribution of the verifier remains aligned with the latest model outputs, while the dynamic reward function utilizes multiple rollouts from the GRPO algorithm [17 7 to automatically adjust reward signals, optimizing performance on challenging verification tasks. At inference time, we leverage the post-trained model for both answer generation and verification and adopt a weighted answer aggregation based on verification scores. Thanks to the unified model, our test-time scaling approach can be easily deployed within existing LLM inference engines, without relying on external RMs. Our experiments on mathematical reasoning benchmarks demonstrate that incorporating self aef-verifiiation into post-training not only improves model performance but also enables effective test-time scaling with increasing generation budgets.
2 Preliminaries
2.1 GRPO for Math Reasoning Tasks
Reinforcement Learning (RL) has proven to be an effective approach for post-training a pretrained
LLM with feedback from humans [1, Al proxies [88 19, ground-truth verification [2] 17 To model the LLM generation process as Markov Decision Process (MDP), we denote the language model as policy To(y 2). where 0 is the model parameters, is the input query and

---

(11, 32.............................. yr } is the generated output sequence. The policy xe can be decomposed into sequence of conditional probability distributions over tokens To 2) IT= P(y+ ,,;;,;; given the previous tokens.
To train policy with RL, we also need to define reward function to provide training signals. For our math reasoning tasks, we follow the previous DeepSeek-RI (7] work to adopt the correctness of the generated answer the reward, evaluated by rule-based verifier. Suppose the ground-truth answer to problem z is a". The correctness reward re can defined as re = [v) = a"), where is a rule-based extractor that extracts the answer from solution text and I is the indicator function with 0-1 binary output.
We use Group Relative Policy Optimization (GRPO) 17 i to train the policy x GRPO is variant of Proximal Policy Optimization (PPO) [2 22 that uses group-based policy sampling to compute the advantage as the training signal, providing higher efficiency and stability without introducing additional value functions. For given input æ; GRPO generates group of answers y!,..... yG and optimizes the policy according to the estimated advantage derived from all responses in this group:
(0) = BBuEDAAudE mo (-1=)
1
[o) A, clip
To(Ui (x) Te (V. (z) 1- c,1+)) A. A1) -DD
min
G
TT (v1(z))
Am
where D is the dataset of math questions, Tol is the old policy parameters for sampling, and / are two syper-parameters, and (-I() is the KL-divergence. The advantage A, of the generation ys is defined as
-)
A T: mean (re...
(2)
std
where r; is the reward on response which can be the correctness reward re for standard GRPO.
2.2 Generative Verifiers
LLM-based verifiers are widely used for LLM post-training and scaling LLM performance at inference time. Traditional verifiers typically trained as discriminative classifiers to the generated solution with additional network heads. Recently, generative verifiers [6] propose to train the verifier through token prediction, in the same way as classical text generation. Unlike discriminative verifiers, generative verifiers can synthesize verification rationales for improved explainability.
A typical generative verifier will prompt the model to verify the solution y for given problem X. The model is required to answer Yes or No after a template I like Is the answer correct? (Yes/No)) We also enable the model to generate intermediate rationales to justify its verification before outputting the final answer. This chain-of-though (CoT) generation process has proven helpful for the final verification result.
At inference time, to better measure the quality of each solution, we use the likelihood of the Yes token as the solution score. Let yo be the next token after the given template I, we have
s(x, v) P(y. Yes 2, 1).
In the original generative verifier work [14, the model is trained through supervised fine-tuning (SFT) with well-constructed data. Our work considers adopting RI. to train such a verifier model within the training of math reasoning.
Self- Verificatin Framework
Since conventional test-time scaling methods often fail to generalize effectively for post-trained models, we introduce velf-verjiication framework that enables both efficient multi-task reinforcement learning (RL) and robust test-time scaling. In Section 3.1 we detail how answer generation and
verification can be unified within a single RL process. Subsequently. in Section 3.2 we demonstrate how the trained model can leverage verification capabilities to further enhance performance at inference time.
3

---

Algorithm GRPO with Self-Verification
1: Input: Dataset D, initial policy иe. online buffer size To, group size G, total training steps T
2: Initialize: Policy-aligned buffer B 4-
3: for ( 1, do
4: Sample data batch from the joint dataset D U B
5: for input prompt in the batch do
6: Generate group of responses (y. from me
7: if the input x is from D then
8:
Math reasoning problem Compute correctness rewards for each generation y1,.... ........9 JG
9:
B 4 {(x, ((x,y))... yi)..... (z.
11:
else
12:
Verification problem
13:
Compute verification reward A. for each generation y1..... UG by Equation 5
14:
end if
15:
end for
16:
Compute the total reward r = re + f. for each sample
17:
Update the model Te according to Equation (I)
18:
Remove data samples from B that were collected before t To steps
19:
end for
3.1 RL with Verfieeriiinttion
We adopt the GRPO algorithm [17.. as described in Section 2.1 to train large language models (LLMs) on math reasoning tasks. This approach utilizes a correctness reward r' re( y) provided by rule-based verifier. To extend the model's 's capabilities of verification, we need to introduce an additional reward function. Since we have access the ground-trute correctness for each generated solution y, we can directly compare the model's predicted verification outcome with the ground truth:
[1,
if [yy, Yes) y),
(e v)
0, otherwise.
Here, the token yo is selected from Yes or No, representing the model's judgment on the verification task. Thus, To serves direct training signal for the verifier. Notably, the data for the verification task depends not only on the original math problem but also on the model's generated response, so use online data buffer to store real-time model responses during training.
Policy-Aligned Buffer. We leverage solutions generated during math reasoning training as data train the verifier. However, selecting appropriate verification data poses challenge. Retaining all historically generated solutions would force the verifier to learn from potentially meaningless or unrefined solutions produced during early training stages. To address this issue, emphasize that our verifier should primarily focus on solutions representative of the current model's capabilities. Therefore, we implement policy-aligned buffer B that stores only the most recent solutions, mitigating the risks associated with policy data. Specifically, we maintain only solutions generated within the last 73 training steps, ensuring the verification dataset is aligned with the evolving policy.
Dynamic Verification Reward. When training the verifier through RL, we find that a simple binary reward as defined in Equation (4) proves ineffective. As the model improves during training, we find that the correct solutions will soon dominate the verification dataset, creating imbalanced data input. We want our verifier focus on difficult verification problems during training. Hence, we introduce dynamic verification reward that identifies challenging verification cases and dynamically adjusts rewards according to their difficulty. Leveraging the GRPO algorithm, we can obtain group of generated responses ......, yc for each problem Denoting Ge and G; as the number of correct and incorrect generations in the group, we define the dynamic verification reward f. as:
20,
if Yes and re v) if yo No and re y) 0, otherwise.
(x, y)
20
(5)
0,

---

This reward design provides higher incentives when the verifier successfully identifies correct solution difficult problem or an incorrect solution for an easy problem. We maintain the expected reward for correct verification at 1, ensuring it remains comparable in scale to the problem-solving reward Tc-
As detailed in Algorithm [ our algorithm integrates self-verification into GRPO training by maintain- ing an online buffer of recent model generations. For each training step, we combine the original dataset and the buffer to sample problems. When processing dataset problems, we generate multiple solutions, compute their correctness rewards, and add them to the buffer. For problems from the buffer, we compute dynamic verification rewards based on the difficulty of verification. The final reward combines both correctness and verification components. This approach ensures the model learns to generate correct solutions while simultancously developing verification capabilities.
3.2 Self- seff-eeiffiatiin for Test-time Scaling
At inference time, our post-trained model possesses strong built-in verifier, we can scale the model performance by allowing the model to verify its own solutions. Given a generation budget of N. we first sample N candidate responses V1..... UN from the model. For each response, compute verification score s(z,3) y:) as defined in Equation (3). We then extract the set of unique answers A {E(y)),.. ..... {(yN) }}. We further aggregate the verification scores for each candidate answer to determine the final prediction à:
argmax Σ I[(()) al (1 ass.,
(6)
where a is hyperparameter that controls the scale of verification scores. When a = 0, this reduces to standard majority voting; as a + oo, the answer with the highest verification score dominates. By choosing moderate value for we can achieve balance between consensus and confidence, leading to more robust and reliable predictions. key advantage of this self-verification approach is its efficiency: it requires only single LLM deployment at inference time, eliminating the need for external verifier models. This makes our framework highly practical and easily compatible with modern LLM inference engines such as vI.LM 23 and SGL (24]
4 Related Work
RL for Reasoning. The OpenAl ol model series 6 has spurred a surge of research into the reasoning capabilities of large language models (LLMss). Early self-improvement methods 27
enable models to iteratively generate and refine their own reasoning paths by selecting high-quality solutions. With the emergence of DeepSeek-RI
77. RL has become increasingly prominent
means to directly incentivize solution correctness. Recent works including SimpleRL-Zoo (28), DeepScaleR [29]. Open RI [30], and Light-RI 31 demonstrate substantial improvements on reasoning tasks by applying RL to post-train pretrained models. Building on these advances, our work extends RL-based approaches jointly train the model both for problem-solving and verification, enabling simultaneous improvement in these two tasks.
Test-time Scaling. Numerous techniques have been developed enhance LLM performance at inference time by increasing computational resources and promoting output diversity. Self-consistency decoding, for example, samples multiple solutions and aggregates them via majority voting S while some works also consider scaling the response length for more exhaustive reasoning paths [32 Other approaches often leverage external reward models (RMs) to guide generation, including beam search 33 34], Monte-Carlo tree search [35) multi-turn correction [38] 42). For general pretrained models, process-based supervision- which rewards each correct reasoning step- has been shown to further improve performance [1] 12 43 33 However, our findings indicate that these process-based methods may be less effective for specifically post-trained models due severe distributional shifts.
Verifier Training. Robust verification mechanisms are crucial for reliable reasoning in LLMs. Early approaches typically train discriminative verifiers [4] by adding an auxiliary classification head to the model, with parameters learned from pairwise feedback [ [8 or ground-truth labels Inspired by the a.LdMas-a-juuge paradigm 47 used in the evaluation, recent work has explored generative verification, where the model produces verification results through text generation
45 46
5

---

Table Average greedy-decoding scores of different models on math reasoning benchmarks after post-training. The best scores from each model series are highlighted in bold. For AIME24, AIME25, and AMC23, we report the average scores over samples for each problem.
AIME24 (avg@ 10)
AIME25
AMC23
Olympiad Bench
Model
MATH500
(avg 10)
(avg@) 10)
Model Series: Qwen2.5-Math--7
Self-Verificatioon--eeee (Ours)
83.60
20.00
16.67
63.75
34.81
(Problem-solvingg verification)
Qwen2.5-Math-
62.00
14.67
5.00
45.25
17.63
(Base model)
GRPO-Qwen-7B (Problem-solving Only)
81.40
19.67
13.67
65.50
32.89
SimpleRL-Qwen-Mathh-B 28
80.80
23.33
10.00
63.75
32.15
Model Seriex: PrrepSeek-RI-Disiiie Qwen-1.5B
Self- -ereirfficatioo-R--1..E. (Ours)
87.00
43.00
31.33
77.50
44.30
(Problem-ssolving verification)
RI-Distill-Qwen-l.5t
80.00
24.33
25.00
64.25
32.89
(Base model)
GRPO-R1-1.58
87.00
37.67
26.67
72.50
40.74
(Problem-solvingg only)
DeepScaleR-115B-Preiiie ((2))
83.00
37.00
31.00
77.25
43.56
These generative verifiers can articulate chain-of-thought rationales 48 when assessing solutions, thereby making the verification process more transparent and interpretable. Previous works usually train generative verifiers via supervised fine-tuning (SFT) on datasets containing high-quality verification responses, whereas a concurrent work [49] also considers using SFT to tune verifier model within varying RI. training algorithms. To our knowledge, our work is the first to propose unified RL framework that unifies problem-solving and verification.
14
5 Experiments
In this section, we aim to examine the effectiveness and characteristics of our self-verification framework. We compare our methods with standard RL and test-time baselines answer the following questions: (1) How does imposing self-verifioation on the RL process compare to standard RL for problem-solving? (2) Can the learned model verify its own solutions better than external verifiers? (3) How does self-verification help with test-time scaling? (4) Is the self-verification process efficient enough for post- post-training and test-time scaling?
Models and Training. In our experiments, we primarily use two pretrained models for RL training with self velf-verification, which are Qwen2.5-Math-77 [50] and DeepSeck- DoepSeck-RI-Distilll --Distill-Qeen-..55 [7]. The DWwnh..5-Math-7B model is a 7B-parameter model pretrained in massive math data with relatively short generation length, while PeepSeck-RI-- -iaee--R---Dssiiil-e----------- the powerful DeepSeek-P RI model with the ability of generating long CoT responses. We use the popular verl framework [51] as the code base for RL training, where the maximal context length for Qwen2.5-Mor 5-t2--5hhhh-777 7B 4k and for SeepSeek-RI--iisee Sweilll-weee-..11 1.5B is 16k. We name the trained models as Self-Verifie Sirfiiaation-ween- and Self- VeriiiicctionnRII-1... respectively. We provide our training details in Appendix
is a distilled version of
Data and Benchmarks. We target our velf-veriiication framework on mathematical reasoning tasks. Considering previous popular implementations of GRPO on math reasoning [28] 22), we use the level 3-5 data from the math training dataset [ to train the Qwen2.5-Math-
model and a combined
math reasoning dataset sorted by DeepScaleR [29 to train the DeepSeek-RP 1-Distill-Qwen- Saaill--wen-115500 1.5B model. For the evaluation benchmarks, we adopt the MATH500 dataset [10), AIME 2024 and 2025
problems, AMC 2023 problems, and OlympiadBench [52). which commonly used benchmarks

---

Table The performance of different models on verifying MATH500 solutions generated by the Self- Perification-owerr 7B model. We highlight the best scores from the open source models in bold.
Category
Method
Accuracy
F1 Score
Self- Verification -erercation--enn--e- w Qwen- (Ours)
87.20
92.83
Open-source Models (-7B)
Qwen2.5-Math (Base model)
73.20
84.93
Llama-3.1-8 1amna-..-8B-Instuure
67.00
78.20
GPT-4o
85,20
91.57
Proprictary Models
Claude-3.7-Soma
90,20
94,46
DeepSeek-v3
89.00
93.73
Table 3: The performance of different models on verifying AIME24 solutions generated by the
Self- Verification-RI-I model. We highlight the best scores from the open- nource models in bold.
Category
Method
Accuracy
Score
Self- Perification-R--1.3 (Ours)
56.67
67.72
Open-source Models (1.5B or -7B)
Doistill-Qwenn 1.5B (Base model)
38.00
49.46
RI-Distill-Qwen-77
46.00
59.50
-
55.67
45.71
GPT-4o
59.33
65.54
Proprietary Models
64.33
71.16
DeepSeek-v3
57.67
66.67
for evaluating math reasoning models [50]. We enable context length of 4k for methods based on Swven22.5-Math-1 and context length of 16k for methods based on Qwen-1.5B. Unless stated otherwise, we use the accuracy of the generated answers as the score shown the experimental results, which automatically verified by the math-verify library from Hugging F As the evaluation data is relatively scarce in AIME24, AIME25, and AMC23, repeat each problem 10 times and use the average accuracy as the result.
Baselines. Our baselines include standard RL methods and test-time scaling baselines. For the RL algorithms, we compare the standard GRPO version without self-verification training and name them GRPO-Qwen-P and GRPO-RI-1.5B. respectively. To compare the validity of our RL process with prior works, we also include SimpleRL-Qwen 1aLL--ee--Math-77 Math-7B model (2 and DeepScaleR- --.ieeR------PP-- --rrvvew model 29 as baselines, which are also tuned based Qwen2.5-Mathe and DeepSeek-RI- For the test-time scaling baselines, we include the following methods:
Self-consistnncy g uses multiple CoT solutions to improve the accuracy of the final answer through majority voting.
Best-of-N adopts the sample with the highest score from N generated responses.
Process-based test-time scaling methods adopt process-basee reward model (PRM) [10 45) further supervise the generation in each step with various strategies like beam
search 43 33 and Diverse Verifier Tree Search (DVTS) [3
We can split the baselines into two categories. The self-consistency method is simple ensemble method without using other models, while other baselines including best-of-N, beam search, and DVTS require external reward models to guide generations. For the choice of models, we follow Beeching et al. [53] to use the RLHFlow Llama3
DeM-Deesseek Data PRM with 8B pa- best PRM revealed their work with similar size to our Qwen-based
rameters [55 46), which
7B verifier model.

---

Table 4: Average test-time scaling scores of different methods on various math reasoning benchmarks. All the test-time scaling methods have budget of 16 samples for each problem. The best scores from each model series are highlighted in bold. For AIME24, AIME25, and AMC23, we report the average scores over 10 samples for each problem.
AIME24
AIME25 (avg@) 10)
AMC23 (avg 10)
Olympiad Bench
Method@
MATH500
(avg @ 10)
Model Series: Qwen2.5-Math-7h
Self- Verification (Ours)
87.20
26.67
19.00
73.25
39.70
Self Seff-Consistency
86.00
23.67
21.67
71.25
39.11
Best-of-N
85,00
23.33
16.67
64.25
37.92
Beam Search
82.80
21.00
15.00
67.25
35.71
DVTS
82.80
21.67
20.33
67.50
35.85
Model Series: DeepSeek-PR RI RI-Distill Saeill-Qeen--...
1.5B
Self-Verification (Ours)
93.60
57.67
37.67
92.00
50.96
Self-Consistency
91.00
52.00
36.67
87.50
47.56
Best-of-N
86,00
46.33
33.67
83.75
43,41
Beam Search
88.40
42.67
33.00
85.25
44.59
DVTS
90.80
45.33
32.33
82.50
45.18
5.1 Post-training Model Performance
Performance Problem-Solving We begin by examining the impact of incorporating self- verification during post-training the greedy-decoding performance of the model, without employing any additional test-time techniques. To this end, we evaluate our self celf-verification models across several math reasoning benchmarks, as summarized in Table T Both Solf-Verification Serfiiaatioo-wwenn---n Qwen-7B and Self- Verification-R11 -1-1--5 consistently achieve strong results, outperforming their respective base models by significant margin due to the reinforcement learning process. Notably, integrating self-verification into RL yields even higher scores than the standard GRPO models, suggesting that self-verification intrinsically enhances the model's problem-solving abilities. We speculate that this synergy arises because verification task encourages the problem's 's logical structure, which in turn enhances its problem-solvine capabilities, Although no new input is introduced, verification training elicits knowledge and reasoning patterns that generalize back the generation task.
model to develop a deeper understanding of
Performance on Verifying Solutions We further evaluate the effectiveness of our self-verification models in verifying their own generated solutions. Specifically, prompt various LLMs to verify the solution for given problem, extract their binary verification answer (Yes or No), and compare against the ground-truth correctness. As presented in Table Sablleju and Table conduct this generative verification procedure with multiple LLMs,, including open-source models of comparable size to well as proprietary models accessed via APIs. On the MATH500 and AIME24 benchmarks, our self-verification models demonstrate substantial improvements in both accuracy and Fl score over similarly sized open-source baselines. Notably, our models also achieve performance on par with leading commercial systems such as GPT-4o [2], Siaude-3.7-Somee
(3], and DeepSeek-v3 [54]. Our model can even surpass GPT-4o in verifying their own responses despite having significantly fewer parameters. We also observe that verifying AIME24 solutions poses greater challenge for all models, likely due to the increased complexity of the problems.
5.2 Self-verification on Test-time Scaling
As stated in Section 3.2, our trained self-verification model can scale its performance by verifying its generated answers. We use Self- Verification Qwen-7B and Self- Prrfiication-R--1.53 to generate solutions with given sample budget and further adopt different test-time scaling methods to evaluate the performance of the final answer. In Figure I we show the scaling performance of different methods when sampling from Self- Seeff-eriiiiaiion-Oe -waatinnnQQee-BBBB on MATH500 and 1.5B on AIME24. The results show that test-time scaling with self-verification yields the best performance among all methods. Maybe surprisingly, find that the simple self-consistency

---

Problem Sobivig-Soiiin
Verification
700
8463,6
Token Usage
$.00
R1115 Taken Usage
500
6,00
Time Coettel
400
300
4000
300116
200
2000
M
100
0.94
0.66
Seff/eeniiiationn R3-1.58
Self Self-verifcationn
SefFcondsitenyy
Best-afN
Seun Search
DITTS
Figure 3: Token usage comparison between Figure 4: Average time cost different test-time
problem-solving and verification tasks.
scaling methods per problem from MATH500.
Table Ablation of the aggregation coefficient a on MATH500 with 16-sample budget. Accuracy is reported for different a settings when selecting answers using verification probabilities.
Self- Veriificaationn-wwe--BB 16
Self- VerificattionRI-11555
a
Accuracy
a
Accuracy
0.3
86.60
0.03
92.60
1.0
87.20
0.10
93.60
3.0
87.10
0.30
92.40
5.0
87.10
1.00
91.00
method also presents competitive performance by majority voting. contrast, methods based on external verifiers like best-of-N, beam search, and DVTS perform worse than self-consistency. This observation proves our hypothesis that external verifiers trained on specific data do not generalize to verifying our post-trained models. Process-basee methods like beam search and DVTS adopt step-wise generation, which may further amplify the distribution shift during the generation process. In Table show the detailed results more math reasoning benchmarks with generation budget of 16, which is a common number of generations balancing between the quality and the efficiency. We see that our self- seff-verification models can consistently outperform all other methods most tasks.
Analysis on the Efficiency of Self-Verifie Verification
In the post-training stage, our framework is comparable in training cost other RL methods. As the verification process is expected to less difficult than problem-solving, we find that the average tokens the model spends on verification is actually lower for both short-context Qwen 7B model and long-context RI distilled model. As shown in Figure 3 the average token usage in the verification task 24% of the problem-solving task for Weriification--een.......
and 35% for Seff-errffiation-R11 1.5B. The lower token usage makes our test-time scaling method more efficient since the verification process takes much less time than the problem-solving process. Figure illustrates the average time cost per problem on the MATH500 benchmark for various test-time scaling methods, evaluated on fixed hardware platform. Unlike approaches that rely on external Reward Models (RMs), our unified model architecture requires serving only single LLM, which reduces overhead and improves efficiency. Our self-verification method is faster than best-of-N sampling and demonstrates significant efficiency gains over process- process-based methods like beam search and DVTS. Although its time cost slightly higher than simple majority voting, this represents favorable trade-off for improved performance. We report the implementation details of our inference-time scaling in Appendix [
Ablation Studies We investigate how the mechanisms and hyperparameters proposed in our methodology affect the post-training and test-time performance. First, we examine the test-time aggregation coefficient a, which balances majority voting with verifier confidence. We show the MATH500 performance our two self verificatin models in Table The Veriiiicaaion-OOWee 7B model peaks at 1.0 while not showing significant performance degradation across different

---

settings. In contrast, Self- Verification-RI-1.55 is more sensitive to this hyperparameter, preferring smaller a 0.1, likely due to its small model size that limits the verification capability, We also report the ablation results on our two post-training mechanisms, including the policy-aligned buffer and the dynamic verification reward, Appendix F
Conclusion and Limitations
In this paper, introduced novel verfffeciiinntion framework for enhancing the mathematical reasoning of LLMs. Our approach integrates verification capabilities directly into the problem- solving model through specialized RL process, enabling models to evaluate their own solutions in generative verification process. Through extensive experiments across multiple mathematical benchmarks, we demonstrated that our self-verification models not only achieve superior performance in problem-solving tasks but also excel at verifying solution correctness. Our self-verification framework addresses the distribution shift issue that arises during test-time scaling with external verifiers, achieving better test-time scaling performance compared to other baselines.
Our work has several limitations. First, the velf-verificttion framework is primarily tailored for mathematical reasoning tasks, and its direct upplicability to other domains such as code generation and agentic tasks, remains to be tested. Adapting self- verification to these areas may require additional task-specific designs, integration with external tools, or new verification strategies. Second, our test- time scaling by self- velf-verification focuses on aggregating answers from multiple generations but does not consider the scale of response length. A multi-turn response generation between problem-solving and verification may further improve inference- sicrence-time performance, which is promising direction to extend our framework. We discuss the broader impact of our work in Appendix
Acknowledgments and Disclosure of Funding
Bo An is supported by the National Research Foundation Singapore and DSO National Laboratories under the Al Singapore Programme (AISGAward No: ASS4--GC-2023-00--111
References
[1] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022.
[2] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, al. GPT-4o system card. CoRR. abs/2410.21276, 2024.
[3] Anthropic. Claude 3.7 Sonnet and Claude Code. 3 2025. Accessed: 2025-05-14,
[4] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Advances in Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
[5] Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-RL: Unleashing LLM reasoning with rule-based reinforcement learning. CoRR, abs/2502.14478. 2025.
[6] OpenAI. Learning to reason with LLMs, 2024. URL nttps://openai.ccmm Accessed: 2025-05-05.
-
[7] DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin
Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu,
Z. F. Wu, Zhibin Gou, Zhihong Shao. Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan
10

---

Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin. Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingehang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang. Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge. Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, and S. S. Li. DeepSeek-RI Incentivizing reasoning capability in L.LS via reinforcement learning. CoRR, abs/2501. 12948, 2025.
[8] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in Lan- guage Models. In International Conference on Learning Representations, 2023.
[9] Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye. More agents is all you need. CoRR, abs/2402.05120. 2024.
[10] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike. John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In International Conference on Learning Representutions, 2024.
[1I] Jonathan Uesato, Nate Kushman, Ramana Kumar, H. Francis Song, Noah Y. Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback. CoRR, 0os/2211.142777 2022.
[12] Seungone Kim, Ian Wu, Jinu Lee, Xiang Yue, Seongyun Lee, Mingyeong Moon, Kiril Gash- teovski, Carolin Lawrence, Julia Hockenmaier, Graham Neubig, and Sean Welleck. Scaling evaluation-time compute with reasoning models as process evaluators. CoRR, abs/2503.19877, 2025.
[13] Benedikt Stroebl, Sayash Kapoor, and Arvind Narayanan. Inference scaling fLaws: The limits of LLM resampling with imperfect verifiers. CoRR, abs/2411.115011 2024.
[14] Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agar- wal. Generative Verifiers: Reward modeling next-token prediction. CoRR, abs/2408.1522 2024.
[15) Dakota Mahan, Duy Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative Reward Models. CoRR, abs/2410. 12832, 2024.
[16] Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling. CoRR, abs/2504.02495.. 2025.
[17] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. CoRR, 000/2400000300 2024.
[18] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosiute, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemí Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton. Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield- Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: Harmlessness from AI feedback. CoRR,
Ben
12. 0112222.00077 2022.
11

---

[19] Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. RLAIF vs. RIHEE Scaling reinforcement learning from human feedback with Al feedback. In International Conference on Machine Learning, 2024.
[20] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. WizardMath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. CoRR, abs/2308.00 009583, 2023.
[21] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 707 707.04347, 2017.
[22] John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. Yoshua Bengio and Yann LeCun, editors, International Conference Learning Representations, 2016.
[23] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Symposium on Operating Systems Principles, pages 611-626, 2023.
[24] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, lon Stoica, Joseph Gonzalez, Clark W. Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems, 2024.
[25] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. STaR: Bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems, 2022.
[26] Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron C. Courville, Alessandro Sordoni, and Rishabh Agarwal. V-STaR: Training verifiers for self-taught reasoners. CoRR, abs/2402.064577, 2024.
[27] Jing-Chinn Cheng Pang, Pengyuan Wang, Kaiyuan Li, Xiong-Hui Hui Chen, Jiacheng Xu, Zongzhang Zhang, and Yang Yu. Language model self- telf-improvement by reinforcement learning contempla- tion. In International Conference on Learning Representations, 2024.
[28] Weihao Zeng. Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma. and Junxian He. SimpleRL-Zoo: Investigating and taming zero reinforcement learning for open base models in the wild. CoRR, abs/2503. 18892, 2025.
[29] Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai. Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. DeepScaleR: Surpassing ol-Preview with a 1.5b model by scaling RL.
racio---rro-bbb b75 notion site/
SeepStalee--Srpassing---e 11631900:4460006eecaa00000011333
wi th- a-1----1-5- -B------M----------------------- Scwlanaaaannnnnnaeeee -R-
2025. Notion Blog.
[30] Hugging Face. Open RI: A fully open reproduction of DeepSeek-RL. January 2025. URL https://gg Ceigghtuu.... com http://hgginngcc///p.eeee
[31] Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, Haosheng Zou, Yongchao Deng, Shousheng Jia, and Xiangzheng Zhang Light-R1: Curriculum SFT, DPO and RL for long COT from scratch and beyond. CoRR, 104600 2025.
[32] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel J. Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. CoRR, abs/2501.19393, 2025.
[33] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling L.LM test-time compute optimally can be more effective than scaling model parameters. CoRR, http//2208.033111 14, 2024.
12

---

[34] Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min- Yen Kan, Junxian He, and Michael Xie. Self-evaluation guided beam search for reasoning. In Advances in Neural Information Processing Systems, pages 41618-41650, 2023.
[35] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of Thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems, 2023.
[36] Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. CoRR, abs/2305. 14992. 2023.
[37] Xidong Feng. Ziyu Wan, Muning Wen, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero- like tree-search can guide large language model decoding and training. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023.
[38] Xiaoyu Tian, Sitong Zhao, Haotian Wang, Shuaiting Chen, Yunjie Ji, Yiping Peng, Han Zhao, and Xiangang Li. Think twice: Enhancing LLM reasoning by scaling multi-round test-time thinking. CoRR, abs/2503. 19855, 2025.
[39] Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. Small language models need strong verifiers to self-correct reasoning. In Findings of the Association for Computational Linguistics, pages 15637-15653, 2024.
[40] Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D. Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M. Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal M. P. Behbahani, and Aleksandra Faust. Training language models to Self-Correct via reinforcement learning. In International Conference on Learning Reprexentations, 2025.
[41] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self- Refine: Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems, 2023.
[42] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Re- flexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, 2023.
[43] Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. CoRR, abs/2408 03311,, 2024.
[44] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jenyy Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/21 10. 14168, 2021.
[45] Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-Shepherd: Verify and reinforce LLMs step-by-step without human annotations. In Annual Meeting of the Association for Computational Linguistics, pages 9426 9116-9939,222 2024.
[46] Wei Xiong, Hanning Zhang, Nan Jiang, and Tong Zhang. An implementation of generative PRM. htttps//gitthu cww/RLHHooo http//LLHOOOORLFF-RRORR Rewe---eeaadddddddeeeee 2024.
[47] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems, 2023.
[48] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe Le, and Denny Zhou. Chain-of-Thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022.
13

---

[49] Kusha Sareen, Morgane M Moss, Alessandro Sordoni, Rishabh Agarwal, and Arian Hosseini. Putting the value back in RL: Better test-time scaling by unifying LLM reasoners with verifiers. CoRR, 00s/2505.00844, 2025.
[50] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-Math technical report: Toward mathematical expert model via self-improvement. CoRR, abs/2409. 12122, 2024.
[51] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang. Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A flexible and efficient RLHF framework. In European Conference on Computer Systems, pages 1 1297, 2025.
[52] Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Lun- wnn Ku, Andre Martins, and Vivek Srikumar, editors, Annual Meeting of the Association for Computational Linguistics, pages 3828-3850, 2024.
[53] Edward Beeching, Lewis Tunstall, and Sasha Rush. Scaling test-time compute with open models, 2025. URL Intpp::/huuge: https////ggggiggaceeee. co/ hoo//ppaces//uggggggaccee// glogpost-scalln test- time- compute
[54]
DeepSeck-AI, Aixin Liu, Bei Feng. Bing Xue, Bingxuan Wang. Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang. Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang. Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang. Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma. Shiyu Wang. Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, Wang. Tao Yun, Tian Pei, Tianyu Sun, W. L.. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang. Wenjun Gao. Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang. Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang. Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, 7. F. Wu, 7. 7. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. ReepSeek-V3 technical report. CoRR, 194427 2025.
[55] Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan. Chaojie Wang, Peng Cheng, Xi- aoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li. Liang Zeng, Tianwen Wei, Cheng Cheng. Bo An, Yang Liu, and Yahui Zhou. Skywork open reasoner series. Skywork-Open
https://cappiccoou--nnhooogeeee nww.onotiioioooo.
2025.
Notion Blog.
(56) Zitian Gao, Yitong Shu, Jiacheng Li, Shizhe Liu, Zixuan Han, Wang Zhou, Lingpeng Zhang, and Yang Liu. One-shot entropy minimization. CoRR, Abs/2505.20282. 2025.
14

---

A Broader Impact
Our self-verification framework for LLMs has several potential societal impacts, both positive and negative, that warrant discussion.
Positive Impacts The primary beneficial impact of our work is the improvement of LLM reliability in reasoning tasks. By enabling models to verify their own outputs, we can contribute to reducing hallucinations and factual errors in Al systems, which is crucial for applications in education, scientific research, and decision support. The self-verification approach also improves computational efficiency compared to methods requiring externl verifiers, potentially reducing energy consumption and computational costs associated with deploying reasoning systems at scale. Additionally, our unified model approach demonstrates that complementary capabilities (reasoning and verification) can be trained together, which may inspire similar approaches in other domains requiring accuracy and trustworthiness.
Risks and Limitations Despite the improvements in verification capabilities, approach does not eliminate the risk of incorrect outputs or the model confidently asserting wrong answers. There is potential concern that users might place excessive trust in self-verified outputs, believing them to be more reliable than they actually are, particularly in high-stakes domains like healthcare or financial analysis. The verification capability is also currently limited mathematical reasoning tasks, and the generalizability to other domains remains uncertain. Furthermore, the increased capabilities could potentially be misused in generating deceptive content that appears more legitimate due to self-verification signals.
Safeguards and Mitigations To address these concerns, we recommend several safeguards: (1) Clear communication to users about the limitations of self-verification, including transparency about error rates; (2) Complementary use of human verification for critical applications; (3) Continued research into detecting when models operating outside their reliable domains; and (4) Development of benchmark tests to evaluate verification capabilities across diverse problem types and difficulty levels. We have also published our methodology transparently enable further research into both the capabilities and limitations of self-verification approaches.
The overall aim of our work is contribute to the development of more reliable and trustworthy AI systems, with oef--erification representing one component in the broader ecosystem of techniques needed for responsible Al deployment.
B Technical Details on Post-training
In this section, we provide comprehensive details about our post-training process with self velffieeificttion. We use the popular verl framework which is the open-source version of Sheng et al. [51], as our RL training framework. For the training resources of our models, we use two nodes of 8 NVIDIA GPUs with 80GB memory each. The detailed training configurations and hyperparameters are listed in T
Model and Data Configuration For Self- Verification- Qarriion---e----ee 7B, we use Swve2.55MMath- as our base model and train on the MATH training data [4] with level to 5 curated by SimpleRL-Zoo 28 The model operates with a maximum prompt length of 2048 tokens and a maximum response length of 2048 tokens. For Self- Verificcaiion-PP R1-1.55 we use DeepSeek-Ri -aieeekk-R--iissse 1-Distill-11.51 as the base model and train on the comprehensive dataset from DeepScaleR 29). which is a collection from multiple math reasoning data sources. While keeping the same prompt length limit, extend the maximum response length to 14336 tokens to accommodate the model's capability for generating longer chain-of-thought reasoning. Different from most math RL implementations, we use longer prompt length of 2048 as we expect more input tokens for the verification task. For the Self- Verification- -itffcaationRR11..55 R1-1.5B model, we remove the thinking content from the original response, wrapped by the <think>... </think> tags, to reduce the input prompt length, which also follows the implementation of DeepSeck-PP R1 7 when building multi-turn tulli-tuunnne chat conversation.
15

---

Table 6: GRPO hyperparameters on the post-training stage with self verificaiin
Hyperparameter
Self-Verffication--wnn-- Seff-Verification-Rl--.
Model and Data
Base Model
Qwen2.5-Math-7B
Training Data
MATH-level3toS 28
DeepScaleR [29]
Max Prompt Length
2048
2048
Max Response Length
2048
14336
RL Hyperparameters
Train Batch Size
128
128
PPO Mini-batch Size
64
64
Learning Rate
le-6
le-6
KL Loss Coefficient
0,001
0,001
Entropy Coefficient
0.001
0.005
Adaptive Entropy
False
True
Target Entropy
N/A
0.2
Entropy Coeff Delta
N/A
0.0001
Max/Min Entropy Coeff
N/A
0.005/0
Rollout Settings
Engine
vLLM 23
vLLM 23
GPU Memory Utilization
0.8
0.8
GRPO Group Size
8
8
Temperature
0.6
0.6
Training Schedule
Total Training Steps
1000
2000
Rejection Sampling
/
Online Data Buffer
Data Buffer Size
5000
40000
Save Frequency Tb
60
60
RL Settings We adopt the GRPO algorithm for both models with the same batch size: a training batch size of 128 and PPO mini-batch size of 64. Other training-related configurations are similar to the original implementation in verl. The learning rate set to 1e-6 for both models. For the KL loss coefficient, use 0.001 to maintain balance between exploration and policy improvement. The entropy coefficient is set differently: 0.001 for Qwen- 7B and 0.005 for R1-1.5B. Notably, we follow He et al. (55) to enable adaptive entropy for R1-1.5B with target entropy of 0.2, allowing the entropy coefficient to adjust between 0 and 0.005 per step, as we find that a fixed entropy coefficient for -Distilll ----ssii-----isl---------------- 1.5B model results in an extremely unstable training process. We leverage the vLLM engine (23] for efficient inference during training, setting the GPU memory utilization 0.8. For both models. we use a group size of for GRPO and temperature of 0.6 for sampling. The training runs for 1500 steps for Qwen- 7B and 2000 steps for R1-1.5B, with rejection sampling enabled filter invalid generations.
the DeepSeek-RR
Additional Rewards for Qwen-7B Model When using the Qwen2.5-Math-7B model for RL. we find that the generated output of this base model is unstable, with useless and unverified code text, potentially due its special pretraining data. Surprisingly, we find that the unexpected code snippets in the generated output do not significantly affect the RL performance in standard RL process. However, this meaningless code generation can be harmful for verification tasks. As result, we add an additional code generation penalty of -0.5 for all generations containing code output. Additionally, we also add a short response penalty -0.5 for verification task when the model only outputs the final verification result without COT. These rewards are specific to the Qwen 7B model. We believe that these auxiliary rewards have minor effect on our main methodology since the model quickly convergess to these rewards in a few RL training steps.
16

---

Table Hyperparameters for test-time scaling with self-verification and baselines
Parameter Category
Self- Velf-eerfficaioon-vee--B
Self- Wew.Veiicaaaion-R-----
Generation Settings
Number of Samples
16 (unless specified)
16 (unless specified)
Temperature
0.6
0.6
Top-p Max Tokens
1.0
1.0
2048
14336
Self-verification Configuration
Verifier Max Tokens
2048
14336
Verifier Temperature
0.0
0.0
Verification Scale Coefficient
1.0
0.1
Process-based Methods (Beam Search and DVTS) Configuration
4,
4, or
Beam Width
when number of samples two linebreaks
when number of samples 2 two linebreaks
Step Delimiter
Max Search Steps
27.0
se (Oun)
22.0
Sdiverffcct
26.0
Seltcorelciency
Best-ff
25.0
Beam Search
19.0
DVTS
18.0
23.0
15.0
14.0
21.0
21.0
Numberaf Samples SomberafSomstes
Numberat Samples
(a) AIME24 Score (avg @ 10)
(b) AIME25 Score (avg @ 10)
on (Our))
73.2
Sefiverficat on
30.7
Sett-crriutency
Seli-coraisenny
29.1
71.2
Beai-afN
Beamn Search
DVTS
37.0
36.0
64.2
35.0
36.0
16
36
Samples
(c) AMC23 Score @avg@) 10)
(d) OlympiadBench Score (avg@) 1)
Figure 5: Test-time scaling performance of Self- Verification Qweenn--eeeeiinee 7B on math reasoning benchmarks including AIME24, AIME25, AMC23, and OlympiadBench.
Online Buffer Management The online buffer is crucial for our self-verification framework. For Qwen-7B, maintain buffer size of 5000 samples, while for R1-1.BB, we expand it to 40000 samples to accommodate the larger dataset and longer responses. Both models update their buffers every 60 training steps (Tb) ensure the verification data stays aligned with the current policy.
C Technical Details on Test-time Scaling
In this section, we provide the details of our test-time scaling process. We implement an efficient inference-time generation framework on top of Beeching al. [53] substitute the inference engine to SGLang [24], which provides highly controllable frontend for guided generation. We use the
17

---

:90
Scloericction (Dus)
sofccoriieency
23.5
38.0
Sdiverficaton (Dur))
Sdfcorrizency
Beal-at-
370
Beam Search
Bearn Search
91.0
DVTS
36.0
DVTS
25.0
10.
337
32.5
66.0
56.2
Kumber of Samples
Number Samples
(a) MATH500 Score (avg @ I)
(b) AIME25 Score (avg 10)
92.0
52.0
510
41.0
90.0
60.0
$4.0
47.0
52.0
45.0
430
Numberaf Samsess
Number
(c) AMC23 Score (avg@) 10)
(d) OlympiadBench Score (avg@) 1)
Figure 6: Test-time scaling performance Self- CenfiicatiooRR11.-555 on math reasoning benchmarks including MATH500, AIME25, AMC23, and OlympiadBench.
same generation settings for all methods, whose hyperparameters are listed in Table 7 RLHFlow 46 PRM for all baselines as their reward models. According to the practice of Beeching et al. [53], we use the last step score of the PRM as the reward for the current step or response. On the hyperparameter a, which balances the answer selection between model verification score and the majority answer, we choose a relatively large value for Self- Perification-Qwent -B model and smaller value for Self- Verification-PP R1-1.5B. This decided according to their verification accuracy during the training time, since we find the 7B model often has better verification performance. In contrast, using smaller a value for the 1.5B model helps alleviate its verification error. For inference-time rollouts, we maintain the same temperature of 0.6 as that in the training time.
We choose the
Computational Resources We typically use NVIDIA GPUs with 80GB memory each for all test-time scaling experiments. We first deploy the SGLang server all GPUs with data parallel strategy. For baselines requiring simultancous reward model deployment, we lower the GPU memory utilization accordingly enable additional memory for the reward model. The inference of reward models are directly through the Hugging Face Transformers library since it is non-trivial to deploy the PRM forward pass on our inference engine. In contrast, the verification process of our self-verification model can be fully realized by the LLM generation process. As result, we only need to serve one model simply through the deployed SGLang server, where we show the efficiency our framework at test time in Figure
D Additional Results on Test-time Scaling
Tigure[5land Figure we show the test-time scaling performance our Self-Verification--weeee - and Self- Verification-R1-1155 models on different math reasoning benchmarks, which corresponds the results shown the main paper. We find that our self-verification test-time method can achieve better performance than our compared baselines in most tasks.
18

---

Veriication Acouracy Different 7a
Verficalionn Accunicy for Different Reward Types
Ta-80 (Our chokee)
Owenn 7B Dynamic Reward
Tu+20
0.96
Qwen 76 0-1 Reward
0.90
T,-
0.80
R1 1.58 Dynamic Reward
R1 1.58 0-1 Reward
Verfoaloon Accurscy
Reward Megh
0.65
0.80
0.80
0.70
0.75
0.68
0.70
100
200
300
400
500
900
700
800
0.58
200
400
600
800 1000
Staps
Stops
Figure 7: choices of the policy-aligned buffer update fiee quency T3 from the Qwen2.5-Math-7B base model. The values are smoothed with a window size 5.
Verification accuracy different
Figure 8: Verification accuracy difference be- tween the dynamic reward design and a simple 0-1 reward for RL with Qwen 7B and RI 1.5B models. The values are smoothed with window size of
E Analysis on Distribution Discrepancy
To substantiate our claim that a distribution discrepancy exists between post-trained generators and general reward models (RMs), evaluate the calibration of external RM outputs from different generators. well-aligned RM should assign higher scores to correct answers regardless of the gener- ator model. We measure the Pearson correlation coefficient between the RM scores and ground-truth correctness on the MATH500 dataset. We use the RLHFlow model as the external RM. As shown in Table X the correlation drops significantly for responses generated by our post-trained model compared to standard instruction-tuned models, indicating that the external RM is misaligned with our model's output distribution. This misalignment limits the effectiveness of test-time scaling methods that rely on such RMs.
1--0--..--PPRM--- 1------M----ee-e PRM-Deepseek -Dta
Table 8: Pearson correlation coefficient between external RM scores and ground-truth correctness on MATH500.
Generator Model
Pearson Correlation Coefficient
Llama-3. 3.1-8B-Instruet Qwen2.5-Mathe Mat.555--a------ntttte
44.9
42.3
Self-Verificttion--wee--
37.5
F
Ablation Studies
We investigate how our framework's 's core design choices influence verification quality. First, examine the policy-aligned buffer, which keeps the verifier on-policy by refreshing stored trajectories every T3 steps. As shown in Figure 7 the verification accuracy curves remain tightly clustered when sweeping T3 around our default value. While larger buffers amortize variance, they provide little extra signal during the online training process and thus show lower accuracy at later training stages. In contrast, Te = 20 shows less stability during the training process. We therefore retain Tß 60 as robust trade-off.
Next, we analyze the dynamic verification reward, which reweights samples by their inferred difficulty. Figure plotss the difference in verification accuracy between using the dynamic reward and simple binary reward. The figure shows that for the 7B model, the dynamic reward yields a significant and sustained positive accuracy difference, indicating clear performance improvement. For the 1 1.5B model, the gain less pronounced throughout training. This suggests that the smaller model's 's limited capacity may prevent it from fully leveraging the nuanced signal from the difficulty-awee reward.
19

---

The dynamic reward efficient in our implementation since the difficulty statistics are derived from GRPO rollouts already generated during training.
G
Comparison with Entropy-Minimizatton Baseline
Recent work has shown that entropy minimization (EM) be a strong baseline for post-training L.MMs. We compare our Self- Veriffiiccatinnn--- Sifiicattiio-Qee-75bb model with the one-shot entropy minimization model (EM 1-shot) from Gao et al. [56]. which is also post-trained from the @wen2.5-Math-- base model. As shown in Table 9 our model outperforms the EM baseline across all benchmarks, demonstrating the effectiveness of our joint problem-solving and verification framework.
Table 9: Comparison with the entropy- miniimzaion (EM) baseline on greedy-decoding scores.
AIME24
AIME25
AMC23
Olympiad Bench
Model
MATH500
(avg@ 10)
(avg@ 10)
(avg@ 10)
Self -eniocatttiin--en----e
83.6
20.00 19.33
16.67 8.67
63.75 54.50
34.81
EM 1-shot
68.8
27.26
H Prompting on Generative Verification
Our generative verification process follows the previous work 14 turn chat template generate the verification output. A template is shown in the box below. We first append the problem and solution for the given data sample, and then add the verification prompt in fixed format.
Additionally, we adopt multi-
Listing 1: Generative Verification Prompting Template
User: [Math problem]
Assistant:
[Solution]
User:
Please verify the solution step by step. the end of the solution verification, when you give your final grade, write it in the form "Is the answer correct (Yes/No)? x", where X is either Yes No.
Assistant:
With this template, the LLM is prone answer the verification question with 'Yes' or 'No' the end. When performing generation with inference engine, we can simply set the stopping string at this template and thus we can acquire the probability of the 'Yes' token according Equation 3
I Examples of Problem-Solving and Verification Outputs
In this section, provide generation examples problem-solving and verification tasks from our Self- Verification Serification-Qwen--8B (Listing 2 and Listing 3) and Self- Verification-RI models (Listing 4 and Listing 5)
Listing 2: Seaf-errfiicaiio--wvee--- Problem-Solving Example
User: Suppose that have $68 different books, $28 of which are math books. In hox many ways
my be next to each other?
Assistant:
To determine the nunber of that the 2 math booke are not next to each other, xe can use the principle of complementary counting. This involves calculating the total number of possible arrangenents of the books and then subtracting the number of arrangements whero
stack 6 different books on shelf auch
20

---

Step 1: Calculate the total number of arrangements of the 6 books.**
The total number of vays to arrange different books is given by the factorial of 6:
61 \tines 5 Atines 4 Atimes 3 \tinea 2 \tines 720
s*ap 2: Calculate the nunber of arranganents whare the 2 math booka are naxt to other.**
treat the nath books as single unit, se can think then as forming "auper book." This reduces the problen to arranging 5 "books" (the 'super book" and the other individusl books). The number of vays to arrange these "books" is:
M
5! \tinea 4 \tines 3 \times 2 \tinea 1 120
Vithin the "super book, the nath books can be arranged in different ways (either math book folloxed by math book 2, nath book followed by math book 1). Therefore, the total number of arrangements where the 2 math books each other is:
AI
51 \tinea 2 120 \tines 2 240
.tepp 3: Subtract the number of arrangements where the math booke are next to each other fron the total number of arrangements.
number arrangenenta
booka
each other
A[
\tines
Thus, the number of vays stack the books shelf such that the math books not next to each other is (480)))
Listing Self-Veriffcation---ee---- Verification Example
$u3, $b3 the value of $1 (blcdoo d)8?
$a8
Bc$ tines what
Assistant: To find the value of \(Arracaa Aedot <){b Aedor d)), dn start by expressing MaV, MBV, and Mol) terms of M(d).
Given:
Ma 3b1)
2. Abb - 2c1)
4d1)
First, substitute Me 4d1) into Mlb 2e\);
A[ 2(4d) 8d.M
8d1) \a 3bl): Ma 3(8d) - 24d.A]
subetitute
Now have ((a 24d1), Alb sdl), and Me 4d)). Ne substitute these expressions into (a ledot cb Aedot d))):
Mracfaa cedot eHb Aedot d) (24ac((24d) Aedot (84) Vedot d} \frac (96d 2) (8d
-2 (9)) 12.
V
Therefore, the value is \(bboxed(12)).
verify solution step by step. verification vhen you give your final grade, vrite it in the form "Is the ansxer correct (Yes/No)? where either No.
the solution
Assistant: To verify the solution, let re-evaluate expression step by atep.
Given:
\a 3bl) 2. \lb - 2c1)
4d1)
First, express \(b) in terms of MaV): \lb 2c 2(4d) Bd.
exprees M(a))
A(dV:
21

---

Ma 36 3(8d) 24d.
Now, substitute \(a 24d1), M(b sa)), and Ale 4d)) expresaion V(Arracaa \edot c){b ledot dA):
Afracfa ledot eH Aedot d} \Trac((24d) Aedot ((8))(88) Vedot d} Afrac (96d (2) (9r 12.
is indeed (127))
the anaver correct (Yea/Bo)7 Yea
Listing 4: Self-Verriication-RR-L.. Problem-solving Example
User: Suppose that I have $68 different books, $28 of uhich are math books. In hox many ways can stack ny 868 booke on a ahelf if do not want the math booke to be next to each other?
Assictant:
thinking part ways arrange any restrictiona. This is given by (
loag.)
calculate
M
61 720
nead subtract to each other. To find this, ve treat the tvo math books as single unit er "block." This reduces the problen to arranging 5 "books" (the "super book" and the other individual books). The number of vays arrange these "books" is:
arrangements whare tuo
51 \tines 4 Itines 3 \immes 2 \times 1 120
Vithin the "super book," the tvo math booke
arranged in different ways ( either math book 1 folloved by math book 2, or math book folloved by math book 1). Therefore, the total number of arrangenents where the 2 math books
to each other is:
51 \tines 2 120 \times 2 240
Subtracting the number bad arrangements from the total number of arrangements gives the nunber of valid arrangements vhere the math books each other:
M
720 240 480 V
the number of ways stack baoks shelf auch that math booka not next to each other is (480)).
Listing 5: Self- Verification-RI-- Verification Example
mutually exclusive event C. It is known that (P) 0.24, (B) - 0.68, and SP(C) 0. Find the probability that at least one of A, B, or accurs, denoted as $P(A+B+C)S.
events
independeat
Aesistant: Given that events and are independent, and both are nutually ezclusive B,
C, need find probability that occurs, denoted as (( P(A Acup B Aeup C) V..
Ne knov the folloving probabilities: P(A) 0.2
C P(B) 0.6 V)
-. ((C) 0.14 V)
Since and B are independent, the probability of their intersection is: P(A) \tines 0.2 \times 0.12
\eap
Siace and are mutually exclusive with c, the probabilitiee of and overlapping and B and overlapping, both AL P(A \cap C) 0
A[ Aeap
22

---

Uning the excluuinnn principle for three events, se have:
AL P(A Acup Yeup c) P(A) P(B) P(c) P(A \eap B) PCA Neap c) P(B \cap
P(A \eap B cepp C) VJ
Substituting the known values:
A P(A (eup B Neup C) 0.2 0.6 0.14
0.12
A PCA Acup B ceup 0) 0.2 0.6 0.14 0.12
AI P(A \eup B \eup c) 0.94 0.12 \I
A PEA Acup Neup
Thus, the probability that at least one of h, or occura 1a VO 82
User: Please verify the solution step by step At the end of the solution verification, when you your final grade, write in the form "Is the anszer correct (Yea/No)? x", where is either Yea or No.
Assiatant:
<think>
Oxay. I need verify if ny step by step.
0.82 is Lat think through this
Firat, I recall that eventa A and B are independent. That neans the probability of both and happening together is the product their individusl probabilities So, P(A B) should be P(A) P(B) 0.2 0.6 0.12. That right.
with That at the sane tine, and B and can't happen at the sane time. So, P(A n C) - Okay, that sense.
mutually
happen
Now, I used the inclusion exclurion principle for three evente to find P(A 0)
The formuls is:
PAUUBU P(A) P(B)
P(A C(A P(B P(Ann)))
Plugging
P(A U U C) 0.2 0.6 0.14
0.12 0
Wait a minute, why are P(A n c) and P(B n a) zero? Bacause the events are mutually yes, they shouldn't overlap.
exclusive with three events A, B, and C night overlap? That is, is P(A n B Q) not zero?
there
that
Sam, the problem
and B are nutually exclusive with C, shich I think means that A cannot occur with C, and cannet occur with C. But and B can happen separately from
perhape
P(B) P(C) C) + P(A n B n C). But since P(A A C) is zero, and P(B I C) is zero, and P(A A zero,
probability
B) 0.12,
Wait, maybe I'm confusing things.
If A and B are independent, then P(A n B) P(A) P(B) 0.12. Since and B are autually exclueive with about P(A n B n C)? Since if 1 An is zero, and B n C is zero, then A n B a sould also zero bacause
means
P
subset
So, P(A B o.
Therefore, the formuls simplifies to:
P(A U B C) P(A) P(B) P(C) P B) P(A n C) P(B n P(A n B
Plugging the numbers:
0.14 0.12
0.94 0.12
0.82
So, the calculation seens correct. Therefore, the ansver of 0.82 is accurate.
</think>
Is the correct (Yes/Bo)? Yee
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes. The claims made in the abstract and introduction are supported by the results our paper in the experiments section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes. The limitations of the work are discussed in Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

This work does not provide additional theoretical results. The theoretical correctess of this work follows the previous work on GRPO for LLM post-training.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The technical details of our framework are disclosed in Appendix B and Ap- pendix C We also provide the code and data for reproducing the results in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
25

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The data and code provided in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.co. ccippi.cc guwdes/ddeeeee CooeeSumeeieeeeeeeeeee ssi onyolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The technical details on data, hyperparameters, and other information of our framework are disclosed Appendix B and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to the extensive compute resources required train our models through RL, we did not report eтror bars from repeated experiments associated with our method and experiments. An analysis of experimental stability is reported Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The computational resources required for our experiments are stated in Ap- pendix and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., - if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts of our work in Appendix A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
27

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We discuss the safeguards of work in Appendix
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We credited the creators of the assets used in our paper and checked the licenses of the assets as stated Appendix B and Appendix C
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
28

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code is contained in the supplementary material of this submission, which is well documented within the readme file.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
29

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The methods in this paper do not use large language models (LLMs) in any non-standard way. LLMs were used for assistance with writing and editing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
30


Reviews:
[{"review_id": "sqs53mLnRs", "reviewer": "Reviewer_j7pR", "summary": "The authors propose a unified reinforcement learning framework that trains large language models to both generate and verify their own solutions within a single policy. They introduce a dynamic verification reward and a policy-aligned buffer to address distribution shifts and data imbalance during joint optimization. The approach is tested on mathematical reasoning benchmarks using Qwen2.5-Math-7B and DeepSeek-R1-Distill-Qwen-1.5B, their self-verification models improve post-training performance and enable effective test-time scaling without external verifiers, achieving good results.", "questions": "- How well does self-verification generalize to non-mathematical domains such as code generation or open-ended commonsense reasoning?\n\n- How sensitive are results to hyperparameters of the dynamic verification reward and buffer size, and have the authors performed ablations on these components?\n\n- Recent discussions have revealed that even entropy-minimization alone can be a strong baseline. Can the authors provide comparison against those?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strength\n1. Strong performance advantages.\n2. Clear motivation. \n\n## Weakness\n1. The paper’s primary impetus—that a “distribution discrepancy” between a post-trained generator and a general reward model constrains test-time scaling—is asserted but never substantiated with distributional distance metrics or error‐mode analyses\n2.  Although joint RL training is said to eliminate external verifiers, there is no measurement of inference latency, sampling overhead, or energy consumption to demonstrate a net practical benefit.\n3. Complexity - deploying a unified RL framework requires managing complex components—policy and value networks, replay buffers, scheduler tuning, distributed rollout servers—versus the far simpler pipelines of prompt-based self-consistency or supervised fine-tuning.", "was_revised": true, "final_justification": "Thanks for the author's response. I will maintain my positive score of 4.", "rebuttal": "> **Weakness 1** *The paper’s primary impetus—that a ‘distribution discrepancy’ between a post-trained generator and a general reward model constrains test-time scaling—is asserted but never substantiated with distributional distance metrics or error-mode analyses*.\n\nWe not only propose the issue of distribution discrepancy but also show its influence through the test-time performance. In Figure 1, Starting from a post-trained base model, *all* test-time scaling strategies that rely on an off-the-shelf reward model (RM-guided beam search, best-of-*N* with RM reranking, DVTS, etc.) fail to beat simple majority voting. The results diverges from previous results, e.g., [the results from DVTS](https://huggingface.co/spaces/HuggingFaceH4/blogpost-scaling-test-time-compute#the_best_of_all_worlds:_compute-optimal_scaling), implying that the external RM is mis-aligned with the new generator’s output distribution.\n\nDirectly measuring distributional distance in the high-dimensional space of LLM outputs is non-trivial, so we further conduct experiments to evaluate *calibration*: the Pearson correlation between RM scores and ground-truth correctness. A well-aligned RM should assign higher scores to correct answers regardless of the generator. As shown in the following table, we find that the correlation coefficient drops significantly on responses generated by our post-trained model compared to open-source Llama and Qwen models. \n\n| Generator model                 | Llama-3.1-8B-Instruct | Qwen2.5-Math-7B-Instruct | Self-Verificaiton-Qwen-7B |\n| ------------------------------- | --------------------- | ------------------------ | ------------------------- |\n| Pearson correlation coefficient | 44.9                  | 42.3                     | 37.5                      |\n\nThese two pieces of evidence substantiate our claim: without re-alignment, a general reward model generalizes poorly to a newly post-trained generator, limiting the effectiveness of test-time scaling. \n\n\n\n> **Weakness 2** *Although joint RL training is said to eliminate external verifiers, there is no measurement of inference latency, sampling overhead, or energy consumption to demonstrate a net practical benefit*.\n\nOur joint RL procedure is deliberately matched to the standard GRPO baseline, so its resource profile is comparable.\n\n1. Inference latency: Training uses the same batch size and number of gradient steps as GRPO. At inference time, our policy produces *fewer* tokens per problem because verification prompts are shorter than full solutions (see Figure 3). \n2. Sampling overhead: the number of samples required in each training step is the same as standard GRPO. For the test-time scaling, Figure 4 shows that our method is faster than other baselines requiring reward models.\n3. Energy consumption: Our RL training and test-time strategies use the same hardware settings as our RL baselines. Because token count and external-model calls are lower, our method inherits the same upper bound on energy use and in practice consumes *less* GPU time.\n\n\n\n> **Weakness 3** *Deploying a unified RL framework requires managing complex components—policy and value networks, replay buffers, scheduler tuning, distributed rollout servers—versus the far simpler pipelines of prompt-based self-consistency or supervised fine-tuning.*\n\nOur method only has marginal complexity over GRPO, as our method only extends the data buffer to include verification problems. The table below compares **greedy decoding** from our single model to **16-sample self-consistency** on the Qwen2.5-Math-7B base model—arguably the simplest prompt-only baseline. \n\n| Model                                           | MATH500   | AIME24    | AIME25    | AMC23     |\n| ----------------------------------------------- | --------- | --------- | --------- | --------- |\n| Self-Verification-Qwen-7B (**greedy decoding**) | **83.60** | 20.00     | **16.67** | **63.75** |\n| Qwen2.5-Math-7B (self-consistency@16)           | 79.00     | **24.00** | 15.67     | 62.00     |\n\nDespite using *zero* extra samples at test-time, our method surpasses the prompt-based approach on three of four benchmarks and narrows the gap on the fourth, validating that the modest training complexity translates into material accuracy gains. On the other hand, our dataset provides pass/fail verdicts but not gold-standard rationales; hence pure SFT cannot learn to both solve problems and judge its own answers. RL with a scalar correctness signal is therefore more appropriate and effective.\n\n\n\n> **Question 1** *How well does self-verification generalize to non-mathematical domains such as code generation or open-ended commonsense reasoning?*\n\nThis is an interesting point. Our framework is task-agnostic in the sense that it only presumes access to the *correctness signal* for each sampled answer. In mathematics that signal is provided by a given math verifier. In other domains, we may need more efforts on designing the verification processes. For example, the verification process for code tasks can be generate reliable unit tests to examine the code, where a recent work provides related idea [1]. Besides, we can also design more complex verification mehtods using tools, which make our methodology able to extend more scenarios. \n\n\n\n> **Question 2** *How sensitive are results to hyperparameters of the dynamic verification reward and buffer size, and have the authors performed ablations on these components?*\n\nYes, we ablate the effects of our hyperparameters, including $T_b$, which decides the buffer size, and the use of dynamic verification reward, in Section D from the appendix. We find that the choice of $T_b$ will not result in significant performance difference, while the use of dynamic verification rewards can lead to better performance than simple 0-1 rewards, especially for the 7B-size model. \n\nAccording to the feedback from Reviewer nBSh, we also conduct experiments on the effect of the test-time hyperparameter $\\alpha$. The results are shown in the following table. In general, different $\\alpha$ will result in performance difference in a small range. The chocie of $\\alpha$ will be more sensitive to the small 1.5B-size model, since its model size limits its verification performance. As a result, we adopt a smaller value of $\\alpha$ (0.1) for the 1.5B model and a larger $\\alpha$ (1.0) for the 7B model as stated in Appendix C.\n\n| $\\alpha$                               | 0.3   | 1.0       | 3.0   | 5.0   |\n| -------------------------------------- | ----- | --------- | ----- | ----- |\n| Self-Verification-Qwen-7B@16 (MATH500) | 86.60 | **87.20** | 87.10 | 87.10 |\n\n| $\\alpha$                               | 0.03  | 0.1       | 0.3   | 1.0   |\n| -------------------------------------- | ----- | --------- | ----- | ----- |\n| Self-Verification-R1-1.5B@16 (MATH500) | 92.6  | **93.6**  | 92.4  | 91.0  |\n\nIn sum, both training and test-time hyperparameters are ablated, and the results show that self-verification remains stable across a broad range of settings.\n\n\n\n> **Question 3** *Recent discussions have revealed that even entropy-minimization alone can be a strong baseline. Can the authors provide comparison against those?*\n\nYes. We compare our 7B model with the one-shot entropy minimization model (EM 1-shot) [2]. We note that they are generally comparable since they are post-trained from the same base model, despite different training configurations. The results are shown in the following table. We can see our trained model still outperforms the entropy-minimization model. \n\n| Model                     | MATH500  | AIME24    | AIME25    | AMC23     | Olympiad Bench |\n| ------------------------- | -------- | --------- | --------- | --------- | -------------- |\n| Self-Verification-Qwen-7B | **83.6** | **20.00** | **16.67** | **63.75** | **34.81**      |\n| EM 1-shot                 | 68.8     | 19.33     | 8.67      | 54.50     | 27.26          |\n\n[1] Wang, Yinjie, et al. \"Co-evolving llm coder and unit tester via reinforcement learning.\" *arXiv preprint arXiv:2506.03136*(2025).\n\n[2] Gao, Zitian, et al. \"One-shot Entropy Minimization.\" *arXiv preprint arXiv:2505.20282* (2025)."}, {"review_id": "TkD4G8Glsi", "reviewer": "Reviewer_nBSh", "summary": "Inspired along the line of the generative verifier works[14-16], the paper applies GRPO algorithm to align the model to answer and verify the question at the same time. During the inference time scaling, the same model is used to answer and question and verify the generated answer. A weighted aggregation is used to determine the final answer.", "questions": "* In table 1, is the verification used in avg@10, if yes, how?\n* In table 1, GRPO-Qwen-7B seems duplicate, what's the GRPO-R1-1.5B performance?\n* What's the verifier model for best-of-N baselines? How good is the trained model's ability to serve as verifier other than the answer generated by itself? (e.g we can verify this by using the trained model as another model's verifier)", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The clarity of the paper's writing and presentation is good.\n\nWeakness:\n\nMethod-wise:\n\n1. The novelty of the proposed method, both technically and method-wise, is quite limited. The method builds on the success of previous generative-verifier works [14–16]. The major difference lies in: (1) instead of using SFT, the model uses an RL method; and (2) following naturally from point 1, the method relies on RL–online sampled data instead of the pre-constructed dataset used in previous generative-verifier works.\n2. One proposed benefit of the method is that no external verifier is needed for test-time scaling. However, the same amount—or even more—compute is required, since the model must be prompted to verify the generated answer. Thus, compute-wise, the proposed method offers no advantage beyond saving a copy of the model weights in storage. Moreover, when scaling to larger models, this approach is debatably limited by the need for both generation and verification to rely on the same large model, thereby losing the flexibility of using a large model for generation and a smaller model for more efficient verification.\n\nExperiment-wise:\n\n3. The performance improvement seems to be marginal compared to the GRPO baselines shown in Table 1.\n4. Despite being closely related to previous Gen-Verifier works, there seems to be no comparison against previous work [14], It's expected to see a comparison on verifier's accuracy, as well as performance difference of using different verifiers for test-time-scaling\n5. Ablation studies against some of the important method design are missing, e.g. Policy-Aligned Buffer and $\\alpha$ from Test-time Scaling", "was_revised": true, "final_justification": "My major concern has been resolved. See my discussion with the author.", "rebuttal": "> **Weakness 1** *The technical novelty is limited; prior works [14–16] already train a generative verifier. Your differences are merely (1) using RL instead of SFT and (2) sampling data online*\n\nThe novelty of our work does NOT come from **learning another generative verifier** but showing the **integration of problem-solving and self-verification**. The works [14-16] you list **only consider learning verifier models** but not problem-solving ones. The purpose of training a generative verifier in our work is to **realize better test-time scaling** on the model itself, different from these works. Besides, replacing SFT with RL is *not* a cosmetic swap. We design a RL training scheme to utilize online samples and discuss specific challenges in our methodology, which are technically distinguished from previous works.\n\n> **Weakness 2** *The 'no-external-verifier' claim confers little benefit: the same—or even greater—compute is needed because the model still has to verify its own answer. When scaling up, tying generation and verification to a single large model removes the flexibility of pairing a large solver with a smaller, cheaper verifier.*\n\nWe would like to answer this comment from the following points:\n\n1. **“More compute is required in self-verification”**. External verifiers also need extra compute. In Figure 4, we show that our method is indeed **more efficient** than other test-time methods using other reward models. The extra compute is always required unless we do not rely on any form of verifiers, e.g., using simple voting. In this case, the cost of ours is typically less than other baselines.\n2. “**The method offers no advantage beyond saving a copy of the model weights in storage”**. From the infrastructure perspective, our method not only saves the memory but also **reduces the complexity of deployment**. For example, a method with external verifiers needs to deploy both the generator and the verifier with careful workloads balance. Otherwise, the **imbalanced hardware utilization** between these two models will result in **inefficiency**. However, our method only needs **one simple deployment** since the problem-solving and verification are through the same generation interface.\n3. “**The method loses flexibility due to relying on the same large model**”. We acknowledge that using a single checkpoint removes the option to combine different solvers and verifiers. We believe this is an acceptable trade-off for our efficient unified structure. We also would like to argue the need of choosing a smaller model for verification, as recent works find that verification quality is also important. For instance, [1] shows that using the large Gemini 2.5 Pro model for both problem-solving and verification is capable of winning IMO gold, highlighting the significance of the verifier. We will describe our limit on this flexibility as the trade-off in the next version of our paper.\n\n[1] Huang, Yichen, and Lin F. Yang. \"Gemini 2.5 Pro Capable of Winning Gold at IMO 2025.\" *arXiv preprint arXiv:2507.15855*(2025).\n\n> **Weakness 3** *The performance improvement seems to be marginal compared to the GRPO baselines shown in Table 1.*\n\nWe clarify that the results in Table 1 are comparisons of **greedy-decoding scores** for all models, **without introducing any verification process** to select answers. Because solving and verifying draw on overlapping reasoning skills but are trained with different losses, a modest transfer gain is expected and, we argue, already noteworthy. Besides, the full benefit of our approach appears once the verifier is *used* at test time, as shown in Figure 1 and Table 4. We will revise related descriptions to make the results clearer.\n\n> **Weakness 4** *Despite being closely related to previous generative verifier works, there seems to be no comparison against previous work [14], It's expected to see a comparison on verifier's accuracy, as well as performance difference of using different verifiers for test-time-scaling*.\n\nWe treat the generative verifier [14] **as a methodology** in our work **rather than a baseline**. The core idea of [14]—train and ask the LLM to produce the judgment—is the *prompting mechanism* we adopt for **every model** in Tables 2 and 3. In other words, all systems are evaluated under the *identical* generative-verifier interface of [14]. In our work, the principal question is not which verifier alone is better, but which **complete system** (the post-trained model + test-time methods) yields the best math scores. In this domain, our self-verification method outperforms all other baselines. We would like to elaborate on this idea better in our next version.\n\n> **Weakness 5** *Ablation studies against some of the important method design are missing, e.g. Policy-Aligned Buffer and $\\alpha$ from Test-time Scaling*.\n\nDue to the page limit, we put our ablation studies to **Appendix D**. We find that the choice of $T_b$ that decides the buffer size will not result in significant performance difference, while the use of dynamic verification rewards can lead to **better performance than simple 0-1 rewards**. We also supplement experiments to show the effect of the test-time hyperparameter $\\alpha$ in the following table. In general, different $\\alpha$ will result in performance difference in a small range. The chocie of $\\alpha$ will be more sensitive to the small 1.5B-size model, since its model size limits its verification performance. As a result, we adopt a smaller value of $\\alpha$ (0.1) for the 1.5B model and a larger $\\alpha$ (1.0) for the 7B model as stated in Appendix C.\n\n| $\\alpha$ | 0.3   | 1.0       | 3.0   | 5.0   |\n| -------------------------------------- | ----- | --------- | ----- | ----- |\n| Self-Verification-Qwen-7B@16 (MATH500) | 86.60 | **87.20** | 87.10 | 87.10 |\n\n| $\\alpha$ | 0.03 | 0.1      | 0.3  | 1.0  |\n| -------------------------------------- | ---- | -------- | ---- | ---- |\n| Self-Verification-R1-1.5B@16 (MATH500) | 92.6 | **93.6** | 92.4 | 91.0 |\n\nIn general, we ablate both training and test-time hyperparameters, and the results show that self-verification remains stable across a broad range of hyperparameter choices. We will add a short pointer to these appendix sections in our main paper.\n\n> **Question 1** *In Table 1, is the verification used in avg@10, if yes, how?*\n\nAs described in Weakness 3, the benchmark results are all greedy-decoding scores of different models without using verification. avg@10 means that we repeat the questions for 10 times to ensure the stability of results for some benchmarks with few questions. We will update the descriptions in the next version.\n\n> **Question 2** In Table 1, GRPO-Qwen-7B seems duplicate, what’s the GRPO-R1-1.5B performance?\n\nWe apologize for the duplicate data, where we forgot to fill in the results of GRPO-R1-1.5B. Here we supplement the results of GRPO-R1-1.5B in the following table:\n\n| Model | MATH500   | AIME24 (avg @ 10) | AIME25 (avg @ 10) | AMC23 (avg @ 10) | Olympiad Bench |\n| ----------------------------------- | --------- | ----------------- | ----------------- | ---------------- | -------------- |\n| Self-Verification-R1-1.5B (Ours)    | **87.00** | **43.00**         | **31.33**         | **77.50**        | **44.30**      |\n| R1-Distill-Qwen-1.5B (Base model)   | 80.00     | 24.33             | 25.00             | 64.25            | 32.89          |\n| GRPO-R1-1.5B (Problem-solving only) | **87.00** | 37.67             | 26.67             | 72.50            | 40.74          |\n| DeepScaleR-1.5B-Preview             | 83.00     | 37.00             | 31.00             | 77.25            | 43.56          |\n\nWe can find that our self-verification model also outperforms the standard GRPO-R1-1.5B RL baseline on most benchmarks, indicating that training with verification problems can synergize problem-solving tasks, with similar findings in the 7B model. We will revise this table to include the results in the next version.\n\n> **Question 3** *What's the verifier model for best-of-N baselines? How good is the trained model's ability to serve as verifier other than the answer generated by itself? (e.g we can verify this by using the trained model as another model's verifier)*\n\nWe use the `RLHFlow/Llama3.1-8B-PRM-Deepseek-Data` model as the general reward model for the best-of-N baseline, following the settings from [the previous work](https://huggingface.co/spaces/HuggingFaceH4/blogpost-scaling-test-time-compute). Different from our generative verifier, it is a discriminative model that outputs the reward score directly through an additional network head. \n\nWe also conduct a new experiment to perform our test-time scaling on the outputs of the `Qwen2.5-Math-7B-Instruct` model. We choose three different models as the verifier: the `RLHFlow/Llama3.1-8B-PRM-Deepseek-Data` reward model, the `Qwen2.5-Math-7B-Instruct` model itself as a generative verifier, and our `Self-Verification-Qwen-7B` post-trained model. The test-time scaling results with 16 samples are shown in the following table. As a generative model, we can expect that our model can better generalize to different distributions than numerical models, since the model can generate text-based rationales to help make judgments. Meanwhile, our verifier model also outperforms the instruct model itself, indicating the advantages of verification training through RL.\n\n| Verifier Model | MATH500   | AIME24    | AIME25    | AMC23     | Olympiad Bench |\n| ------------------------------------------------------ | --------- | --------- | --------- | --------- | -------------- |\n| RLHFlow/Llama3.1-8B-PRM-Deepseek-Data (Discriminative) | 85.20     | 15.67     | 20.33     | 71.75     | 36.89 |\n| Qwen2.5-Math-7B-Instruct (Generative) | 85.40     | 15.67     | 9.67      | 63.00     | 34.67 |\n| Self-Verification-Qwen-7B (Generative) | **88.80** | **17.33** | **20.00** | **73.00** | **38.67** |"}, {"review_id": "3K0k4sYT80", "reviewer": "Reviewer_HSim", "summary": "This paper introduces a novel \"self-verification\" framework to address the distributional shift between specialized language models and external verifiers, a key problem that limits performance scaling at test-time. The authors propose unifying answer generation and verification within a single RL process, training the model to simultaneously solve complex reasoning tasks and assess the correctness of its own solutions. By leveraging a policy-aligned buffer to train on its own recent outputs and a dynamic reward function that incentivizes learning on difficult cases, the model effectively becomes its own highly capable and efficient verifier. Experiments on mathematical reasoning benchmarks demonstrate that this method not only improves the model's intrinsic problem-solving ability but also enables superior test-time scaling by self-evaluating multiple candidate responses, outperforming traditional methods like self-consistency and those relying on external reward models.", "questions": "1、Regarding Table 1, the comparison seems potentially imbalanced. If I understand correctly, the \"Self-Verification (Ours)\" method involves generating multiple responses which are then verified, whereas the baseline methods only generate a single response. Could you clarify if this is an apples-to-apples comparison, or am I misinterpreting the experimental setup for this table?\n\n2、The performance improvement shown in Table 1 appears to be somewhat marginal. Specifically, when comparing the \"Self-Verification\" method to the \"GRPO (Problem-solving Only)\" baseline, the gain seems modest. Could you elaborate on the significance of this improvement?\n\n\n3、Regarding the training process, my understanding is that for each problem-solving sample, G responses are generated, which then create G verification samples. If G=8, this seems to imply a significant increase in computational overhead for training the verification task. Could you comment on the cost-benefit analysis here, as the performance improvement from this additional training seems modest?\n\n\n4、I acknowledge that the dynamic reward function for self-verification is quite sophisticated. However, looking at Figure 6 in the appendix, the performance of the simple 0-1 reward function doesn't seem substantially worse, especially for the smaller R1-1.5B model?\n\n5、I'm interested in the claim that problem-solving and self-verification are mutually beneficial. From one perspective, the verification task could be seen as a simple re-framing of the original task's output, potentially not adding substantial new information. Could you provide a more detailed or intuitive explanation for how training a model to verify its own solutions leads to a genuine improvement in its core problem-solving abilities?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The proposed unified framework for problem-solving and self-verification is quite intriguing, as it appears, at least on the surface, to lead to more strong reasoning capabilities. Furthermore, the authors' point regarding verifier distribution shift is compelling, and I concur with this perspective.\n\nHowever, a conceptual challenge arises from this design. While using a single model for both problem-solving and self-verification elegantly addresses the issue of distribution shift, it seems that, theoretically, the responses generated by the model itself do not introduce new, external information. This raises the crucial question of why this unified training approach should enhance the model's fundamental reasoning abilities. The paper seems to lack a more thorough or intuitive explanation for this synergistic effect.", "was_revised": true, "final_justification": "Thanks for the author's reply, which has resolved some of my confusion. I will keep my score, and it is already a positive score.", "rebuttal": "For the weakness stated in the review, please refer to the response to **Question 5**.\n\n> **Question 1** *Regarding Table 1, the comparison seems potentially imbalanced. If I understand correctly, the \"Self-Verification (Ours)\" method involves generating multiple responses which are then verified, whereas the baseline methods only generate a single response. Could you clarify if this is an apples-to-apples comparison, or am I misinterpreting the experimental setup for this table?*\n\nThe results in Table 1 are comparisons of **greedy-decoding scores** for all models, **without generating and selecting from multiple responses**. This is an evidence that introducing verification tasks indeed helps problem-solving by enhancing the potential reasoning capabilities (refer to Question 5 for more in-depth discussion), as our self-verification model has higher greedy scores than standard GRPO. To further utilize the verification abilities, our main results on test-time scaling are shown in Figure 1 and Table 4. We will emphasise this distinction more clearly in the revised version.\n\n> **Question 2** *The performance improvement shown in Table 1 appears to be somewhat marginal. Specifically, when comparing the \"Self-Verification\" method to the \"GRPO (Problem-solving Only)\" baseline, the gain seems modest. Could you elaborate on the significance of this improvement?*\n\nAs stated in our response to Question 1, the improvement is based on **greedy-decoding** scores. Compared to the GRPO baseline, our method only has extra verification training in the RL process. This process has no direct correlation with the problem-solving performance, while we still prove that it can improve greedy-decoding scores due to better reasoning abilities.  \n\n> **Question 3** *Regarding the training process, my understanding is that for each problem-solving sample, G responses are generated, which then create G verification samples. If G=8, this seems to imply a significant increase in computational overhead for training the verification task. Could you comment on the cost-benefit analysis here, as the performance improvement from this additional training seems modest?*\n\nWe keep the training cost of our self-verification model **similar** to the GRPO baseline by keeping the **same training batch size and training steps**. As shown in our listed algorithm, we sample data batches from a mixed dataset with both problem-solving and verification problems. Therefore, we use the same total amount of data as the baseline GRPO algorithm. Our policy-aligned buffer design maintains a fixed verification data size, without selecting all samples generated during training. \n\n> **Question 4** *I acknowledge that the dynamic reward function for self-verification is quite sophisticated. However, looking at Figure 6 in the appendix, the performance of the simple 0-1 reward function doesn't seem substantially worse, especially for the smaller R1-1.5B model?*\n\nYour finding is correct. We speculate that the **model size** is the bottleneck of limited verification performance for the 1.5B-size model, while adopting the dynamic reward cannot improve the performance with a large margin. On the other hand, we can see that the dynamic reward function leads to significantly better performance improvement for our 7B model. Besides, the dynamic reward is not so complex to implement. We only need to record the difficulty of the original question as additional data, which can be naturally acquired through the sampling process of problem-solving tasks. Compared to other training operations like sampling and weight updates, its cost can be very small.\n\n> **Question 5** *I'm interested in the claim that problem-solving and self-verification are mutually beneficial. From one perspective, the verification task could be seen as a simple re-framing of the original task's output, potentially not adding substantial new information. Could you provide a more detailed or intuitive explanation for how training a model to verify its own solutions leads to a genuine improvement in its core problem-solving abilities?*\n\nThe verification task can enhance the model's **reasoning capability** on the given problem, which **further helps solve the problem in subsequent training epochs**. This means that the knowledge required by the verification task can finally generalize to problem-solving. We may find similar insights in models like OpenAI Codex [1], where they find that pretraining the model on code data acquiring stronger logical abstractions that then transfer to natural-language reasoning tasks. A recent work [2] also finds that training meta-abilities on some tasks can help improve general reasoning capabilities on many benchmarks. For our task, although no new input is introduced, the verification training can also elicit responses that are also useful for problem-solving, resulting in the improvement in Table 1, indicating the effectiveness of our self-verification training. \n\n[1] Chen, Mark, et al. \"Evaluating large language models trained on code.\" *arXiv preprint arXiv:2107.03374* (2021).\n\n[2] Hu, Zhiyuan, et al. \"Beyond'Aha!': Toward Systematic Meta-Abilities Alignment in Large Reasoning Models.\" *arXiv preprint arXiv:2505.10554* (2025)."}, {"review_id": "Q90DtYmUS4", "reviewer": "Reviewer_Wty9", "summary": "This paper presents a unified RL framework that trains LLM by synergizing answer generation and verification via Group Relative Policy Optimization (GRPO), eliminating the need for external reward models at inference time. The authors integrate a generative-verifier reward into GRPO, using a policy‐aligned buffer and a dynamic verification reward to focus the model on challenging cases. This method is then evaluated on mathematical reasoning benchmarks (MATH500, AIME24/25, AMC23, OlympiadBench), showing that “Self-Verification” improves post-training performance over standard GRPO and enables more effective test-time scaling compared to self-consistency, best-of-N, beam search, etc.", "questions": "1. Verification without Ground Truth: During test-time scaling (Section 3.2), how does the model reliably assess correctness in the absence of explicit ground-truth labels? What confidence calibration mechanisms prevent propagation of false positives/negatives?", "limitations": "Yes, the authors have explained the limitations, including the scope and potential constraints of their approach.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The single‐model approach bridges post-training and test-time scaling, avoiding distributional mismatch with external verifiers.\n\n2. Evaluation: On multiple benchmarks, Self-Verification models outperform base models and prior RL baselines (e.g., SimpleRL, DeepScaleR) in both problem-solving and verification tasks. In terms of efficiency, verification consumes significantly fewer tokens (24% in problem-solving) and lower inference overhead compared to external-verifier methods. Ablation studies \n\nWeaknesses:\n\n1. (Data Contamination): The paper does not discuss whether evaluation sets (e.g.\\ AIME24, MATH500) might overlap with the training data or pretraining corpora, especially for the self-verification verifier (Tables 2–3). Without contamination analysis, reported verification accuracies may be overly optimistic.\n\n2. The authors focus exclusively on LLM-based generative verification (which may contain hallucinations and black-box mechanisms) and do not compare to symbolic or formal engines (e.g.\\ SAT/SMT solvers) that could provide deterministic verification paths for many math sub‐tasks.", "was_revised": true, "final_justification": "The authors have provided a detailed response to clarify my confusion about formal verification (W2, W3) and explained my concerns about the data contamination issue in the experiment. Based on their response, I will keep my rating and increase confidence.", "rebuttal": "> **Weakness 1** *The paper does not discuss whether evaluation sets (e.g. AIME24, MATH500) might overlap with the training data or pretraining corpora, especially for the self-verification verifier (Tables 2–3). Without contamination analysis, reported verification accuracies may be overly optimistic.*\n\nAll our training data are inherited exactly from prior works (please refer to SimpleRL [27] and DeepScaleR [28] in our paper). The dataset used in SimpleRL is the *training* portion of the original MATH dataset and well removes the test data. The dataset of DeepScaleR contains AIME and AMC problems, all prior to our test benchmarks according to [their description](https://huggingface.co/datasets/agentica-org/DeepScaleR-Preview-Dataset). Therefore, all the training data used in our method has **no overlap** with the benchmarks. Our verifier training is based on the same query data as the problem-solving data, which does not overlap with our evaluation benchmarks. \n\nRegarding possible leakage from the underlying *pre-training* corpora of Qwen and DeepSeek, those corpora are **proprietary and unavailable for direct inspection**. However, every baseline we compare against is built on the **same pretrained models**, so any residual contamination, if existing, affects *all* methods equally and therefore cannot explain the performance gap we observe.\n\n> **Weakness 2** *The authors focus exclusively on LLM-based generative verification (which may contain hallucinations and black-box mechanisms) and do not compare to symbolic or formal engines (e.g. SAT/SMT solvers) that could provide deterministic verification paths for many math sub‐tasks.*\n\nOur benchmarks involve **full natural-language contest problems**. We note that it is infeasible to use SAT/SMT solvers for our benchmarks, since they are only for problems that fit their formal theories -- pure propositional logic for SAT and a few decidable first-order fragments for SMT. Encoding these challenging math questions into these formal forms is labor-intensive and leaves many sub-goals outside our scope.  Here we list the differences between our method and SAT/SMT solvers in the problem categories and encoding methods. \n\n| **Dimension**    | **SAT Solvers** | **SMT Solvers** | **Self-Verification** |\n| ---------------- | -------------------------------------------- | --------------------------------------------------------- | ---------------------------------------------------------- |\n| Problem category | Boolean satisfiability | Algebra/logic within supported theory | Open math problems  |\n| Problem Encoding | Manual CNF translation | Manual, theory-specific constraints | Natural-language prompt |\n\nOur LLM pipeline operates directly on the raw natural-language statement, generates a complete derivation, and self-verifies it probabilistically, covering the full breadth of contest-level mathematics while requiring zero manual encoding.\n\n> **Question 1** *Verification without Ground Truth: During test-time scaling (Section 3.2), how does the model reliably assess correctness in the absence of explicit ground-truth labels? What confidence calibration mechanisms prevent propagation of false positives/negatives?*\n\nAs shown in Equation (6), we adopt a weighted score to aggregate different generated responses and select the answer with the highest score. This **ensemble** has two safeguards:\n\n1. The verifier is trained jointly with the solver and receives a reward that penalises wrong judgments, so its Yes/No probabilities are already aligned with empirical correctness.\n2. Even if an individual judgment is noisy, combining *N* independent solutions dilutes single-case errors. In practice, we see a superior math performance with aggregating 16 samples in Table 4.\n\nTherefore, the weighted aggregation can mitigate both false positives and negatives without requiring external ground-truth labels."}]

Author rebuttal:

Meta-review: This paper proposes to add self-verification reward in addition to correctness reward during RL, to incentivize LLM's reasoning and verification capabilities simultaneously. The improvements on both generation and verification were demonstrated with the separate experiments. Initially, the reviewers raised some concerns, but the authors successfully resolved them during the discussion. Consequently, all the reviewers reached a unanimous agreement on the acceptance of this paper, and hence AC also recommend to accept this paper.
Decision: Accept (poster)