Process Reward Models That Think
Authors: Muhammad Khalifa
Rishabh Agarwal
Lajanugen Logeswaran
Jaekyeom Kim
Hao Peng
Moontae Lee
Honglak Lee
Lu Wang
Venue: Accepted by TMLR ()

Abstract: Step-by-step verifiers—also known as process reward models (PRMs)—are a key ingredient for test-time scaling, but training them requires expensive step-level supervision. This work aims to build data-efficient PRMs as verbalized step-wise reward models that verify every step in the solution by generating a verification chain-of-thought (CoT). We propose ThinkPRM, a long CoT verifier fine-tuned on orders of magnitude fewer process labels than those required by discriminative PRMs. Our approach capitalizes on the inherent reasoning abilities of long CoT models, and outperforms LLM-as-a-Judge and discriminative verifiers—using only 1% of the process labels in PRM800K—across several challenging benchmarks. Specifically, ThinkPRM beats the baselines on ProcessBench, MATH-500, and AIME ’24 under best-of-N selection and reward-guided search. In an out-of-domain evaluation over subsets of GPQA-Diamond and LiveCodeBench, our PRM surpasses discriminative verifiers trained with the full PRM800K by 8% and 4.5%, respectively. Lastly, under the same token budget, ThinkPRM scales up verification compute more effectively compared to LLM-as-a-Judge, outperforming it by 7.2% on a subset of ProcessBench. This work highlights the value of generative, long CoT PRMs that can scale test-time compute for verification while requiring minimal supervision for training.

Published in Transactions on Machine Learning Research (03/2026)
Process Reward Models That Think
Muhammad Khalifa
Khaljaam@miich.ccm edu
University of Michigan
Rishabh Agarwal Mila
Lajanugen Logeswaran LG Al Research
Jaekyeom Kim
LG Al Reseorch
Hao Peng
University of Nlinois Trrbnna-Champen
Moontae Lee
LG Al Reseorch
Honglak Leet
University of Michigan LG Al Research
Lu Wang! University of Michigan
Equal supervision.
Reviewed on OpenReview: https: // openreview. net/ forum? id= FPVC6OWM
Abstract
Slep-by-step verifiers also known as process reward models (PRMs) are a key ingredient for test-time scaling, but training them requires expensive step-level supervision. This work aims to build data-efficient PRMs as verbalized step-wise reward models that verify every step in the solution by generating verification chain-of-thoughh (CoT). We propose THINKPRM, long CoT verifier fine-tuned orders of magnitude fewer process labels than those required by discriminative PRMs. Our approach capitalizes the inherent reasoning abilities of long CoT models, and oulperforms LLM-as-a-Judge and discrininative verifiers- Sestlerr-usiig usinn only 1% of the process labels in PRMS00K across several challenging benchmarks. Specifically, THINKPRM beats the basclines on ProcessBench, MATH-500, and AIME '24 under best- of-N selection and reward-guided search. an out-of-domain evaluation over subsets of GPQA-Diamond and LiveCodeBench,, our PRM surpasses discriminative verifiers trained with the full PRMS00K by 8% and 4.5%, respectively. Lastly, under the same token budget, THINPRMM scales up verification compute more effectively compared to LLM-as-a-Jndge, oulperforming it by 7.2% generative, long CoTT PRMs that can scale test-time compute for verification while requiring minimal supervision for training."
subset. of ProcessBench. This work highlights the value of
OQur code, data, and models released https://github.ce httpp:///gtthhh...omm////////hhs

---

Published in Transactions on Machine Learning Research (03/2026)
1
Introduction
ThinkPRM
DisCPRM
LLM-as-a-judge
Training data efficiency: ProcessBench
Verifier-guided search: MATH-500
90
70
reasoning accuracy
verificttion F1
85
65
8K process labels
80
-700K process labels
75
70
50
103
10
10
2
2
2
2
2
Training samples
Number of beams
Figure 1: Left: Verifier Fi-score on ProcessBench (Zheng et al., 2024). THINKPRM-14B, trained on 8K process labels or 1K synthetic examples, outperforms discriminative PRMs trained on about 100x more data. Right: Verifier-guided search accuracy on MATH-500 with 1anua-..--B-Insttuute as generator. THIKPPM-1.5B, trained using the same 8K labels, oulperforms LLM-as-a-judge and discriminative verifiers in -wward-giided search on MATH-500. The LLM-as-an as-juude in both figures uses the same base model THINKPRM.
Reasoning with large language models (LLMs) can substantially benefit from utilizing more test-time compute (Jaech et al., 2024; Guo et al., 2025; Akyürek et al., 2024). This typically depends high-quality process reward model (PRM)-also known as a process verifier that scores (partial) solutions for selecting promising paths for search or ranking (Cobbe et al., 2021; Li et al., 2023; Wu et al., 2024; Brown et al., 2024). PRMs have typically assumed the form of discriminative classifiers, trained to discern correct from incorrect reasoning (Uesato et al., 2022; Zhang et al., 2025). However, training discriminative PRMs requires access process labels, i.e., step-level annotations, which either require extensive human annotation (Lightman et al., 2023; Zheng et al., 2024), gold step-by-step solutions (Khalifa et al., 2023), compute-intensive rollouts (Luo et al., 2024; Chen et al., 2024a). For instance, training reasonably performing math PRMs requires hundreds of thousands of step-level annotations (Lightman et al., 2023; Wang et al., 2023b).
Generative verification either via LLM-as-a-judge (Wang et al., 2023a; Liu et al., 2023b; Zheng et al., 2023) or GenRM (Zhang et al., 2024a) treats verification as a generation problem of a rationale followed by a decision. However, L.M-as-a-judge known to perform poorly compared to specialized reward models (Lambert et al., 2024; Zhang et al., 2024b; Chen et al., 2024c), as general-purpose LLMs frequently fail to recognize reasoning errors (Huang et al., 2023; Zhang et al., 2024a; Ye et al., 2024). Moreover, GenRM is limited to outcome verification via short chain-of-thoughts (CoTs), fundamentally limiting its ability for test-time scaling.
This paper builds on the insight that generative step-by-step verification can greatly benefit from scaling up the verifier's inference compute-sseccffeally specifically, by enabling it to think through a COT. Specifically, we repurpose open-weight. large reasoning models (LRMss) as the foundation for generative PRMs through lightweight training. This training uses synthctic data (Kim et al., 2023; Zhu al., 2023; Wang et al., 2024), utilizing as few as 8K step labels, and yields THINKPRM outperforms discriminative PRMs trained on two orders of magnitude more data across variety of test-time scaling scenarios.
PRM that not only surpasses LLM-as-a-judge, but also
We obtain THINKPMM by training four reasoning models, namely R1-Distill-Qwen (1..ee(11.B,700 (155,,B,,14)) (Guo et al., 2025), and BooQ-32B-Previee (Team, 2024), and extensively evaluate it both as a standalone verifier on ProcessBench (Zheng et al., 2024), and combined with a generator under Best-of-N and verifier-guided beam search. THINKPRM-14B outperforms a discriminative PRM based on the same base model in terms of accuracy while using far fewer supervision signals as in Figure/ 1 left. In addition, THINKPRM.
2

---

Published in Transactions on Machine Learning Research (03/2026)
demonstrates strong performance MATH-500 (Hendrycks et al., 2021) under guided beam search, shown in Figure 1 right. Lastly, as shown in Figure 2, THINKPRM can effectively utilize more verification compute than LLM-as-a-judge, by forcing it to think for more tokens. All these results are obtained while training only on 8K stop labels.
Our work highlights the promise of long CoT PRMs that verify reasoning with reasoning, effectively scaling both generator and verifier compute. Our main findings are as follows: THINKPRM outperforms strong PRM baselines in best-of-N and guided- search sctups on two math reasoning benchmarks: MATH-500
Scaling verifier compute: ProcessBench
88
1-sccre (%)
and AIME 2024, and surpasses LLM-as-a-judge baselines under the same base model by thinking longer curing verification (64). Moreover, THINKPRM generalizes under two types of domain shift. First, outperforms basclines on out-of-domain lasks such as scientifie reasoning and code generation. Second, despite being trained only short solutions, it generalizes to long-form reasoning without explicit step delimiters ($5.3). Third, THINKPRM outperforms colfisonnst (Wang et al., 2022) when using the same compute budget, especially under high sampling regimes ($5.4). Finally, fine-grained filtering of synthetic data based on step supervision is crucial for training high-quality PRMs ($5.7).
82
80
78
76
74
8K
16K
24K
32K
Thinking up to (#tokens)
ThinkPRM
DiscPRM
LLM-as-a-judge
Figure 2: THINKPRM enables scaling veri- fication compute with more CoT tokens.
2 Background and Related Work
Discriminative PRMs. Discriminative PRMs are trained as classifiers that directly predict numerical correctness scores for each solution step, and typically rely on extensive step-level annotations (Uesato et al., 2022; Lightman et al., 2023; Zhang et al., 2025). Given a solution prefix, discriminative PRMs encode the solution text and employ a classification head to produce step-level scores, usually optimized with binary cross- entropy. An overall correctness score for a solution is obtained by aggregating these step-level scores (Beeching et al.). PRMs are effective and straightforward but they do not utilize the aanuuagemodelling head of the base language model, making training expensive and labor-intensive (Yuan et al., 2024). Additionally, they offer limited interpretability and utilize fiaed compute, restricting their dynamic scalability at test-lime (Zhang et al., 2024a; Mahan et al., 2024). Thus, there is need for data-efficient PRMs that can scale with more test-time compute.
Generative Verification. Generative verification (Zheng et al., 2023; Zhu et al., 2023; Zhang et al., 2024a) frames verification as language-genene generation task, producing step-level decisions as tokens (e.g., "correct" or "incorrect"), typically accompanied by a chain-of-thought (CoT). One can train generative verifiers using the standard language modeling objective on verification rationales rather than on binary labels. This approach leverages the strengths of LLMs in text generation, making gencrative verifiers inherently interpretable and scalable (Zhang al., 2024a; Mahan et al., 2024; Wang al., 2023a; Ankner et al., 2024). However, prior work generative verifiers has relied mainly on short verification CoT (e.g., few hundred tokens) (Zhang et al., 2024a), which highly limits their scalability. Thus, there is need for verifiers that can "think" longer through verification, utilizing test-time compute more effectively. While auddee has been employed for step-level verification (Zheng et al., 2024), it tends to be sensitive to prompt phrasing, and prone to invalid outputs, such as infinite looping or excessive overthinking (Bavaresco et al., 2024)-issucs issues we further confirm this work. Prior results with reasoning models like QwQ-32B-Previis (Team, 2024) show promise, but their practical utility in test-time scaling remains limited without additional training (Zheng al., 2024).
Test-Time Scaling with PRMs. Test-time scaling techniques, such as Best-of-N selection (Charniak & Johnson, 2005) and troc-basod scarch (Yao et al., 2023; Chen et al., 2024c; Wan al., 2024), leverage additional inferenco-time compute improve reasoning performance. Central to these approaches is the quality of the verifier used to score and select. solutions. A major advantage of generative PRMs is that they
3

---

Published in Transactions on Machine Learning Research (03/2026)
Sample verification chains
<chinic
2. Compare against process labels
Step accurstely...
(bored(ccrrect)
Step onita. (bored(tncersset)
Srep Step
Cumner tnoassect
X Discard!
X
Solution
Step (bosed(srcazeett)
Problem
</thinks
?
Scep
Reasoning Model
3. Keep good chains
Step calesl atee. Themefone (coeecen
Step
(bosndcorssett)
Step Cornect
Step
Curnect
(bosed(nncorrett)
Finetuning data
</thanio
Figure 3: Collecting verification chains for finetuning. First, we prompt a reasoning model, in our case QwQ-32B-Preview to critique a given solution to a problem. Then, we sample multiple verification chains, which we judge against gold process labels from PRM800K, only keeping chains that match the gold process labels.
uniquely support simultaneous scaling of both generator and verifier compute (Zhang et al., 2024a; Kalra & Tang, 2025). In particular, our work shows that generative PRMs trained based on long CoT models (Jaech et al., 2024; Guo al., 2025) enable both parallel and sequential scaling of verifier compute.
3 ThinkPRM
Our goal verbalized PRM that, given a problem-solution pair, verifies every step in the solution via an extended chain-of-thought (CoT) such as the one shown in Figure 31 in Appendix H. This section introduces issues with LLM-as-a-jude- verification and proposes data collection process (shown in Figure 3) to curate high-quality synthetic verification Co'Is for training such PRM. The rest of the paper addresses the following rescarch questions:
RQI: How well do LRMs perform under T.I.Mas-a-judde for process-level verification? $3.1
RQ2: Can lightweight finetuning on synthetic verification Colls improve the reliability and effectiveness of these models as process verifiers? $3.2
RQ3:
How does finetuned verbalized PRM (THINKPRM) compare to discriminative PRMs and LLM-as-a-Juude baselines under different test-time scaling scenarios? $4
3.1 LLM-as-a-judge PRMs are suboptimal
This section highlights limitations observe when using off- off-the-shelf reasoning models as process verifiers, suggesting the need for finetuning. For evaluation, we use ProcessBench (Zheng et al., 2024), which includes problem-solution pairs with problems sourced from existing math benchmarks, along with ground-truth correctness labels. We report the binary F1-score by instructing models to verify full solutions and judge whether there exists a mistake. We use the two most challenging subsets of ProcessBench: OlympiadBench (He et al., 2024) and OmniMath (Gao et al., 2024), cach comprisod of 1K problem-prefix pairs. For LLM-as- a-judge, we use the same prompt template as in Zheng et al. (2024), shown in Figure 29, which we found to work best overall. Table 3 shows LLM-as-a-judge F1 scores and sample output by QwQ-32B-Previen is displayed in Figure 28 in Appendix G. We provide details on how we parse and extract binary labels from model outputs in $E.1.
We observe different issues with LLM-as-a-judge verification. First, the verification quality is highly sensitive the instruction wording: a slight change in the instruction can affect the Fi-score by up to 3-4 points. A substantial number of the generated chains include invalid judgments, i.c., chains without an extractable overall label clear in Figure 5. Such invalid judgements are caused by the following. In some cases, the
4

---

Published in Transactions on Machine Learning Research (03/2026)
final decision was in the wrong format than instructed e.g., the model tries to solve the problem rather than verify the given solution behavior likely stemming from the model training. Additionally, we noted multiple instances of overthinking (Chen et al., 2024b; Cuadron et al., 2025), which prevents the model from terminating within the token budget, and infinite alternative lechniques to verify the solutions.
where the model gets stuck trying
Figure 4 (left) shows a histogram of verification CoT lengths generated by R1-Qwen- 14B in the LLM-as-a- judge setting. Accurate CoTs tend to be shorter, typically under 3K tokens, while inaccurate CoTs are more evenly distributed and spike sharply around 7K-8K tokens, highlighting the prevalence of overthinking and looping in long chains. We show examples of these behaviors in Appendix B. In the next section, we mostly fix these issues via lightweight finetuning over synthetic verification Co'Is
3.2 Finetuning synthetic data boosts LLM-as-a-judge verification
Inspired by recent work on reducing overthinking in long CoT models by training (Yu et al., 2024; Kang el al., 2024), we aim to improve LLM-as-a-judge performance via fineluning on high-quality verification data. Collecting real data would be expensive, so we rely on filtered synthetic data (Zelikman et al., 2022; Singh et al., 2023; Dong et al., 2023; Zhang et al., 2024b; Wang et al., 2024) also known as rejection sampling finetuning. To keep our approach simple, we refrain from more expensive training techniques, such as reinforcement learning or preference-based learning.
Synthetic data collection. As training data, we sample synthetic verification CoTs from QwQ-32B- Preview, prompting it to verify each step in solution prefix, using the instruction shown in Figure 13. The problems and corresponding step-by-step solutions come from the PRM800K dataset (Lightman et al., 2023), which provides both model-generan generated solutions and hurin-veriffed step-level labels.
The sampling process continues until we obtain 1K verification CoTs which correspond to 8K step labels in total. For data filtering, we use the following criteria: (i) the CoT must follow the expected format (i.., include an extractable decision label for each step inside \boxed(] as shown in Figure 12, and (ii) the generated step judgements match the gold step labels from PRMS00K, and (iit) the CoT length is within a maximum budget to avoid the excessive overthinking behavior we observed in Figure 4 (left). The filtering process ensures our training data is of sufficient quality. Note that process- process-based filtering is crucial for the performance of the resulting PRM as we show in $5.7. We ensure that our 1K synthetic verification traces do not overlap with any problems used in the evaluation datasets (c.g., MATH-500, AIME '24 GPQA, LiveCodeBench), verified via string match filtering as we detail in $A.3. Data collection is illustrated in Figure 3, data statistics are in $A.1 and a training example is in Figure 12.
Notably, our training relies only on supervision in the form of gold step-level annotations, not on gold verification rationales or CoTss-oal gold step-level annotations, one can obtain silver labels via Monte Carlo rollouts (Wang et al., 2023b; Chen et al., 2024a). While we train only on math data, the resulting PRM remains robust under other domains such as science QA and code generation as we show in $4.2. We then proceed to train our models on the 1K collected chains. Our training is very lightweight; finetuning 1--Q-3BB-Previce takes only 4.5 hours on a single A100 80GB GPU. Refer to $C.1 for training details.
this pipeline supervision-efficient and low-overhead. In the absence of
Finetuning on synthetic verification CoTs substantially improves the verifter. THINKPMM trains the 1K chains and is evaluated on ProcessBench and compared to LLM-as-a-judge under the same base model. Figure 6 shows verifier accuracy of different models before and after our finetuning. We note a substantial boost in F1 across all models, with the 1.5B model gaining most improvement by over 70 F1 points, and the 14B model performing best. Looking at the ratio of invalid judgements in Figure 5, we also note a significant reduction in invalid labels with all models, except for QwQ, where it slightly increases. Lastly, the reduction in overthinking and infinite looping behavior discussed in the last section is evident, as in Figure& (right), where THINKPRM generations maintain a reasonable length (1K-5K) tokens while being substant more accurate.
5

---

Published in Transactions on Machine Learning Research (03/2026)
LLM-as-8 aaMM-as-ajuuuee j-juge
ThinkPRM
300
300
250
250
Frequency
200
Frequency
200
Overthinking. repetition,
150
150
infinite looping, etc.
100
100
50
50
0
3
4
7
8
1
2
5
7 8
Length (K tokens)
Length (K tokens)
Accurate Verification
Inaccurate Verification
Figure 4: Verifier performance ProcessBench in light of CoT lengths. On the left, LLM-as-a-judde produces excessively long chains including repetition, infinite looping, and overthinking, leading to worse verifier performance since the oulput never lerminates. Training on collected synthetic data substantially reduces these issues as shown in the THIKPRM plot on the right.
Verifier performance on ProcessBench
CoTs without valid label on ProcessBench
Percentage total (%)
100
53.2
50
80
73.2
70.3
73.7
F1-score
40
60
53.0
45.2
40
random
16.0
11.5
20
10
1.9
0
OwQ-328-previl 32B-preview
R1-Owen-148
R1-Qwen-78
R1-Qwen-1.5B
QwQ-32B 100-32--preee -review
NI-Qwen-14B
R1-Qwen-7B
R1-Qwen-1.55 1.58
ThinkPRM LLM-as-a-judge
ThinkPRM LLM-as-a-judge
Figure 5: LLM-as-a-judge suffers from significant ratio of verification CoTs that do not terminate with a parsable label, i.e, \boxed(yes} or (no), Our finetuning process that yields THINKPRM, substantially mitigates this issue. Both verifiers are based on RI-Distill-Qwen--1a
Figure 6: Verification accuracy on 2K question- solution pairs from two most challenging subsets of ProcessBench: OlympiadBench and OmniMath. THINKPRM obtained by finetuning the correspond- ing model over only 1K verification chains performs better.
4 Test-time Scaling Experiments
This section aims to answer RQ3 introduced in $3 by comparing THINKPMM to baselines under different scaling sccnarios. We study how THINKPRM performs under different generation budgets (i) best-of-N selection (Wu et al., 2024; Brown et al., 2020) and (ii) guided beam search (Snell et al., 2024; Beeching et al.). We also explore how THINKPRM performs when verifier compute scaled either in parallel by aggregating decisions over multiple verification CoTs or sequentially through longer CoTs by forcing the model to double check or self-correct its verification.
4.1 Experimental Setup
In the remainder of the paper, we will mainly use our finetuned verifiers based on R1-Distil--wen--I and R1-Distill- R1-Distill-Qven-114 as these provide the best tradeoff between size and performance. We will refer these and THINKPRM-14B, respectively.

---

200-problem subset from the v5 release of Published in Transactions on Machine Learning Research (03/2026)
Best-of-N: AIME 24
Best-of-N: MATH-500
32.5
Generator: 1een2.5-32--nnstruu
Generator: Owen2.5-144.
80
Accuracy (%6
30.0
Accuracy (%)
27.5
70
25.0
22.5
20.0
50
2
2
2
2
2
2
2
2
25
2
Number of solutions
Number of solutions
ThinkPRM-144
DiscPRM 14B
LLM--a-a-judge
Majority
Figure 7: Best-of-N on AIME '24 and MATH-500. Compared LLM-as-a-judge, DiscPRM, and (unweighted) majority vote, THINKPRM-14B exhibits best accuracy scaling curve.
Baselines. We compare THINKPRM to DiscPRM, which uses the same base model as THINKPM,, finetuned with binary cross-entropy the entire PRMS00K dataset, totaling 712K process labels, which is two orders of magnitude larger than our training data. Details on finetuning DiscPRMs are in $C.2. We also compare to unweighted majority voting, which merely selects the most frequent answer across the samples (Wang et al., 2022), and to LLM-as-aJJudge using the same base model as THINKPRM, prompted as in $3.1.
Tasks and Models. We show results on two math reasoning tasks, namely 100 problems from MATH-500 (Hendrycks et al., 2021) covering all difficulty levels (see $E.5 for more details), and American Invitational Mathematics Examination (AIME) problems for 2024. Since THINKPRM was finetuned only on math data, we study the out-of-domain generalization on two tasks: scientific reasoning and code generation. For scientific reasoning, we use the physics subset of GPQA-Diamond (Rein al., 2024), consisting of 86 PhD-level multiple choice questions. For code generation, LiveCodeBench (Jain et al., 2024).
Over MATH-500, we report results with THINKPRM-1.5B and THINKPRM-14B using two different generators: Qwen-2.52 3-14BBInnstrue and Llama-3.2 3...-33B-nnstreee We use Qwen-2.5-144 24B for best-of-N evaluations, and Llama-3.2 3.2-3B for beam search, since scarch more compute-intensive. Using different generators helps ensure that our conclusions are not tied to particular model family or size. For the more challenging benchmarks AIME '24 and GPQA-Physics we use stronger generator, Swven-2.5-32B-Inssttrr generation, we use Owen-2.55-Cder-77 (Hui et al., 2024). In best-of-N and beam-search settings, the verifier must output numerical correctness score; describe how extract these scores from THINKPRM and L.M-as-a-Judde in 1F.1, and we provide additional implementation and hyperparameter details for best-of-N and beam search in Appendix E.
For code
Scaling verifier compute. Compared to DiscPRMs, generative reward models enable an extra dimension of scaling to squeeze more performance: scaling the verifier compute. Specifically, THIKPRMM allows for two types of scaling. First, we use parallel scaling (Mahan et al., 2024; Brown et al., 2024). by sampling K independent CoTs and averaging their scores. We will refer to this scaling using "@K" throughout the rest of the paper. Second, and more specific to long reasoning models, we use sequential scaling e.g., by enabling the model double-check its initial verification (Xiong et al., 2025; Kumar et al., 2024; Ye al., 2024). Inspired by Muennighoff al. (2025), we use trigger phrase such "Let's verify again" elicit colf-correction of earlier verification. See $E.4 for more details.

---

Published in Transactions on Machine Learning Research (03/2026)
Guided beam search: MATH-500
Filtering based on Process vs. Outcome
72.5
Generator: Liama 3.2 Instruct
68
Generator: Llama 30mma-3.--B-nntttee 2-3B-Instruct
70.0
66
Accuracy (%6
Accuracy (%)
67.5
64
65.0
62
62.5
60.0
60
57.5
58
55.0
2
2
2
2
24
56
Number beams
20
2
2
2
2
Number of beams
ThinkPRM-1.58
DiscPRM-1.58 MathShepherd-78
ThinkPRM-1.BB@4
Process-bass (ours)
LLM-as-a-judge
RLH#Flow-8B-Deepseek
Outcome-based (GenRM)
Figure Comparison off thelshelf PRMs trained much more step labels than THINKPRM. @K represents parallel scaling by averaging scores over K CoTs.
Figure 9: Ablating the data filtering mechanism,
where process-based filtering yields better PRMs. LLM-assa-judge is shown with number of beams = 16.
4.2 Results
ThinkPRM outperforms DiscPRM and LLM-as-a-Judge. Under best-of-N selection with MATH-500 shown in Figure 7 (right), THINPRMM leads to higher or comparable reasoning accuracy to DiscPRM under all sampling budgets. The trend holds on the more challenging AIME '24 shown in Figure left. Additionally, Figure (right) shows beam search results on MATH-500, with THINKPRM 1.5B surpassing DiscPRM and LLM-as-aJJudge.
ThinkPRM surpasses off-tho-shelf PRMs. We compare THINKPRM-15B to two strong off-tho-shelf PRMs, namely RLHFFlow- Deeeeeeeww (Xiong et al., 2024) and MATH-Shepherd PRMM (Wang et al., 2023b). These PRMs are trained on even more data than PRM800K and are larger than 1.5B. We show results under verifier-guided search on MATH-500 in Figure 8, with THINKPRM-1.5P 5-..5 scaling curve surpassing all baselines and outperforming RLHFFlow- www--Depseek-PPee PRM, the best off-the-shelf PRM among the ones we tested, by more than 7% across all beam sizes.
ThinkPRM excels on out-of-domain tasks. As for OOD performance on GPQA-physics (Figure 10 left), THINKPRM scales better than FiscPRM-whiee drops substantially at N=32 outperrononi it by 8%. On LiveCodeBench (Figure 10 right), THINKPRM also outperforms DiscPRM by 4.5%. On LiveCodeBench, (Zhang et al., 2025)-a discriminative PRM trained on substantial amount of process labels obtained from LLM-as-a-judge data and Monte Carlo rollouts struggles when applied out-of-domain. Our results shed light on the fragility discriminative PRMs under domain shifts in contrast with generative PRMs. Refer Figure 23 in $F.4 for results on MATH-500, GPQA-physics and LiveCodeBench with bootstrapped confidence intervals.
Swen2.5-7B-Mathh--
Scaling ThinkPRM compute boosts performance. Under verifier- search (shown in Figure 8), parallel scaling with THIKKPMM-15B8 boosts the accuracy by more than 5% points, and yields the best accuracy on MATH-500. In addition, parallel scaling with THINKPRM-14B04 and THINKPRM-14B0G boosts best-of-N performance on MATH-500 as shown in Figure 19 in $F.1. Now we move lo sequent ial scaling of verifier compute by forcing THIKPRRM to recheck its own verification. Since this can be compute-intensive. intensive, we only run this on 200 problems from OmniMath subset of ProcessBench, and observe how verification F1 improves as we force the model think for longer as shown in Figure 2. THINKPRM exhibits better scaling behavior compared to I.MM-as--juude,, which drops after 16K tokens, and outperforms DiscPRM-14B
8

---

Published in Transactions on Machine Learning Research (03/2026)
Best-of-N: GPQA-physics (OOD)
Best-of-N: LiveCodeBench (OOD)
70
Generator: Qwen-2.5-322 1et.2-5322--nnnnuuee
65.0
Generator: Pwen2.5-Coder-78
Accuracy (%)
Pass rate (%)
65
62.5
60
60.0
55
57.5
55.0
2
2
2
2
2
2
2
2
2
2
2
2
Number of solutions
Number of solutions
ThinkPRM-14B
DiscPRM-14B
Qwen2.5-7B-Math-PP5
Majority
Figure 10: Bost-of-N on two out-of-domain tasks: science QA (GPQA-Physics) and code generation (Live- CodeBench).. Although THINPPMM was only finetuned math, it exhibits superior OOD performance than the baselines, especially at larger sampling budgets, where the baselines fall short. Discriminative PRMs struggle despite being trained on orders of magnitude more process labels.
by 15 Fl points. In summary, THINKPRM is consistently betler than LLM-as-a-judge under parallel and sequential scaling.
Parallel scaling vs. sequential scaling. Is it preferable to scale verifier compute in parallel or sequentially? We investigate this by comparing the two modes of scaling under the same token budget. Figure 20 in $F.1 shows performance of best-of-N with Qwen-2.5-14B under parallcl and sequential scaling with K 2, under both parallel scaling and sequential scaling. Overall, the performance of both methods is fairly close, but observe slight advantage to parallel scaling under certain budgets.
5
Analysis and Discussion
5.1 Training data efficiency
A major strength of THINKPRM is training data efficiency compared to discriminative versions. Here, we study the training scaling behavior of THINKPRM-14B by training it over 500 and 1K examples in total collected using the pipeline in $3.2, which roughly corresponds to 4K and 8K process labels from PRM800K in total. We compare that to DiscPRM-14B trained with 1K, 10K, 50K and 98K examples, where 98K corresponds training on the full PRMS00K train set that includes 712K step labels. Figure 1 (Left) contrasts the training data scaling behavior of THNKPRM-14B with that of DiscPRM-14B, where performance scales substantially better with two orders of magnitude fewer process labels. This primarily stems from THINKPRMF utilization of text generation and reasoning abilities of the underlying models.
While we train THINKPRM using only 1K data points, we investigate whether it will benefit from training on more data. Using the pipeline, we collect and filter additional verification CoTs and obtain a total of 65K chains. We then finetune R1-Distilll-Qeen-1.1t and R1-Distill-Qwen-I 14B on these for single epoch while keeping all other training hyperparameters fixed. We then compare the resulting models the 1K-trained version of THINKPRM under best-of-N selection MATH-500. Figures 21 and 22 in $F.2 show performance boost from training on the 65K examples compared to only 1K. This suggests that THINKPRM can utilize more training data when available.
5.2 Effect of Verification CoT Length on PRM Quality
We study whether the length of verification chains of thought affects the quality of the resulting generative verifier. Specifically, we compare ThinkPRM trained the full, long synthetic CoTs with variant trained

---

Published in Transactions on Machine Learning Research (03/2026)
short, compressed versions of the same 1K CoTTs To obtain the short CoIs, instruct gpt-4o-mint to rewrite each original CoT into concise version that preserves only the essential reasoning. We then train RI-Qwen-1.5B and RI-Qwen-144B on these short Co'Is and evaluate verification Fl on ProcessBench. Table reports the comparison.
Long CoT (THINKPRM)
Short CoT
Model
OlympiadBench OmniMath
OlympiadBeuch
OmniMath
R1-Qwen-1.5B
87.3
75.7
64.8 55.3
66.7
R1- R1-Qwen-1415
87.3
85.7
60.8
Table 1: Verification F1 when training R1 models on long versus short CoTs.
The substantial performance drop when training on short CoTss emphasizes how THINKPRM benefits from extended reasoning. Since verification is a complex task, throwing more reasoning effort at it via thinking improves performance. These results support the value of using long verification CoTs for training.
5.3 Reasoning traces without clear step boundaries
So far, have used TuNKPRM to verify short CoTs with clear steps delimiters. Here, we investigate whether THINKPRM still verify long CoTs that involve extended reasoning, backtracking, and self-correctinn. As generator, we use Qwen3-1.77 (Yang et al., 2025) with thinking mode. Although THIKPRMM was only trained on short solutions from PRM800K, we find that it can still verify long CoTT and outperforms the baselines as shown in Figure 11 left. Inspecting THINKPMM's outputs, we found that it extracts and verifies individual steps embedded in the long CoT-an an example is in Figure 32.
5.4 Compute-matched comparison across verifiers
A natural question is how THINKPRM compares to alternative verification strategies under fired test-time compute budget. We therefore perform a compute-matched analysis where vary the number of sampled solutions and plot accuracy as a function of estimated FLOPs for different verifiers.
Figure (right) shows results on MATH-500, comparing THINKPRM (ThinkPRM-14B) to DiscPRM-14B, LLM-as-a-Judge, and majority vote. At low compute budgets, majority voting and DiscPRM are competitive with or slightly ahead of THINKPRM. However, the FLOP budget increases, THINKPMM steadily improves and ultimately achieves the highest accuracy among all methods, while LLM-as-a-Judge lags behind despite consuming substantially more compute. This indicates that the additional verifier compute used by THINKPRM is more elfective than simply sampling more candidate solutions.
We report an analogous compute-matclnn matched analysis on GPQA-Physics in Figure 25 (Appendix F.7), where we observe the same qualitative trend: THINKPRM makes better use additional verifier compute compared to the other methods.
5.5 ThinkPRM with Monte Carlo step labels
To train THIKPPM,, have relied on manual gold step labels from PRMS00K. Since automatic labels c.g., via Monte Carlo rollouts Luo et al. (2024) are cheaper as they only require knowledge of correct final answer, we validate whether we can train THINKPRM using automatic labels. We train THINKPRM-L5B using 1K synthetic chains based on labels from Math-shepherd dataset (Wang et al., 2023b). Performance on BrochssBench is shown in Table 5, where training THINKPRM with automatic labels yields very comparable performance to training with manual labels, showing that our training pipeline is agnostic step-labeling strategy.
10

---

Published in Transactions on Machine Learning Research (03/2026)
Best-of-N: MATH-500
Compute-matcher analysis: MATH-500
Generator: Qwen3-1.78-thinking
85
88
Accuracy (%)
75
86
Accuracy
70-
84
ThinkPRM-1.58
ThinkPRM-1.5BG4
ThinkPRM-144
60
DiscPRM-14B
82
Majority
DiscPRM-1.58
55
LLM-as-a-judge
Majority voting
2
2
2
24
1x
1
1x
Number of samples (N)
Estimated FLOPs (log scale)
Figure 11: Left: Best-of-N with Qwen3-1.7B the full MATH-500 test set, showing how THINKPRM generalizes well to verifying long reasoning traces. Right: Compute-matched comparison under best-of-N with THINKPMM, majority vote, DiscPRM and LLM-as-a-judge.
5.6 ThinkPRM helps with difficult reasoning problems
THINKPRM's reasoning ability should enable it to tackle verification of hard problems. To check if this is the case, we analyze performance of THINKPRM vs. DiscPRM in light of problem difficulty over MATH-500 and GPQA-physics (how we estimate difficulty for GPQA-Physics explained in $F.6), shown in Figure 24. The generators here are Qwen-2.5-14B for MATH-500 and Qwen-2.5-32B-Inshhh
for GPQA-Physics. Primarily, THINKPMM improves reasoning on the difficult problems (levels 3, 4, 5 in MATH-500 and 2, 3, 4 in GPQA-Physics) substantially more than DiscPRM.
5.7 Filtering based on process vs. outcome labels
In $3.2, we describe our process-based filtering strategy, which selects verification CoTs based on agrecment between generated step-level decisions and gold process labels. To validate its effectiveness, we compare it to outcome-based filtering, in GenRM (Zheng et al., 2024), which retains chains solely based on final answer correctness keeping CoT if its final answer is correct and the final step is \boxed(correct), answer is incorrect and the final step is \boxed(incorrectt thereby ignoring intermediate step labels. We obtain 65K and 128K CoTs using process- and outcome-based filtering, respectively. Figure 9 shows that finetuning L--.Diilll-Qen--lee on process-filtered data yields significantly better verification performance, despite using fewer examples, which reflects the importance of our process-based filtering in training strong PRMs.
if the
5.8 Limitations of Generative PRMs
While generative PRMs are more powerful and data-efficient than their discriminative counterparts, they also introduce some limitations that we view as promising directions for future work. In particular, they incur additional compute overhead from generating verification chains compared to scalar-valued discriminative PRMs. We quantify this effect in $F.8, where we provide a FTOP and latency-based comparison of the inference officiency of THINKPRM, DiscPRM, and LLM-as-a-Judge. Sccond, overconfidence in LLMs (Liu et al., 2023a; Stechly et al., 2023; Zhou et al., 2024) and, for PRMs, tends to push scores toward the extremes (near 0 or 1), especially when they are derived from the probabilities of discrete decision tokens (e.g., "yes"/"nno"). In practice, we find that generative verifiers such as THINKRMM and LLM-as-a-Judge are more miscalibrated than DisCPRM. $F.9 shows reliability plots on MATH and GPQA-Physics where DiscPRM stays closest to the diagonal while generative verifiers are clearly overconfident.
known issuc
Future work should explore more reliable techniques extract calibrated scores from generative reward models. Another limitation due to the autoregressive nature of LLMs, leading them prematurely commit to earlier judgment.. For example, observe a phenomenon term step label interference, where
11

---

Published in Transactions on Machine Learning Research (03/2026)
verification errors for earlier steps impact verification of later steps. For example, if the PRM judges particular step incorrect, it becomes more likely to label subsequent steps as incorrect even if they are not.
6 Conclusion
We introduced THINKPRM, generative process reward model trained with minimal synthetic supervision for scalable step-by-step verification. With just 8K process labels, THINKPRM significantly outperforms LLM-as-a-judge and even surpasses discriminative PRMs trained on orders of magnit more data. These results highlight the benefits generative PRMs for interpretability, scalability, and data efficiency.
Broader Impact
This work proposes a more data-officient way to train process reward models for step-by-step verification of reasoning, which we expect to be primarily useful as a contribution for scaling test-time compute, search, and alignment of foundation models. However, stronger generative verifiers also carry the risk of false confidence: they can produce fluent, convincing critiques or chains of thought that are nonetheless incorrect, and our calibration analysis shows that THINKPRM and LLM-as-a-Judge are often overconfident, especially out of domain. For these reasons, view THINKPRM as a component to be used with human oversight rather than standalone authority, and we do not recommend direct deployment in high-stakes domains (e.g., medical or safety-critical decision making) without additional safeguards and careful evaluation of failure modes.
Acknowledgments
This research is supported by LG AI Research and partially by the NVIDIA Academic Grant Program. We also thank Yunxiang Zhang, Xin Liu, Sean Welleck, and Lewis Tunstall for very useful discussions.
References
Ekin Akyürek, Mehul Damani, Linlu Qiu, Han Guo, Yoon Kim, and Jacob Andreas. The surprising offectiveness test-time training for abstract reasoning. ar preprint orXiv:2411.07279 2024.
Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D Chang, and Prithviraj Ammanabrolu. Critique- out-loud reward models. arXiv preprint arXiv:2408. 1 1791, 2024.
Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, et al. Lims instead of human judges? large scale empirical study across 20 nlp evaluation tasks. ar preprint orXiv:2406.18. 2024.
Edward Beeching, Lewis Tunstall, and Sasha Rush. Scaling test-time compute with open models. URL https://huggingface http:/ggnn.ac..ocopccsseeeeeeeee
seatts---stttttttttttttteeeee compute.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V Le, Christopher Ré, and Azalia Mirhoscini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint
27Xiv:240..21787, 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Eugene Charniak and Mark Johnson. Coarso-to-fine n-best parsing and maxent discriminative reranking. In Proceedings of the 43rd Annual Meeting of the Association for Compulational Linguistics (ACL '05), pp- 173-180, 2005.
Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Alphamath almost zero: process supervision without process. Xix preprint ar arXiv:2405.0955 03553, 2024a.
12

---

Published in Transactions on Machine Learning Research (03/2026)
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of ol-like llms. Xrviv preprint arXiv:2412.21187, 2024b.
Ziru Chen, Michael White, Raymond Mooney, Ali Payani, Yu Su, and Huan Sun. When is tree search useful for 1lm planning? it depends on the discriminator. arviv preprint or arXiv:2402. 10890. 2024c.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiihhrr Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXi::211011468,, 2021.
Alejandro Cuadron, Dacheng Li, Wenjie Ma Xingyao Wang, Yichuan Wang, Siyuan Zhuang, Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, et al. The danger of overthinking: Examining the reasoning-action dilemma in agentic tasks. arXiv preprint arXi::2022.8833: 2025.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. Xrv preprint arXiv:234.00666, 2023.
Bofei Gao, Feifan Song, Zhe Yang, Zelan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arxiv preprint ftt:2410.079555 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepsook-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arx preprint arXiv:2501. 12948, 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14 14008, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the math dataset. arXiv preprint wrXii:2103.0387/ 2021.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. ICLR, 12)):3 2022.
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. arXie preprint arXiv:2310.0112 2023.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 12186, 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint X2v:2412211672 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXie preprint arXiv:2403.0790 079.0794. 2024.
Nimit. Kalra and Leonard Tang. Verdict: A library for scaling judge-time compute. arXiv preprint 17X1v:2502.18018, 2025.
Yu Kang, Xianghui Sun, Liangyu Chen, and Wei Zou. C3ot: Generating shorter chain-of- thought without compromising effectiveness. arXiv preprint Xr 11664, 2024.
13

---

Published in Transactions on Machine Learning Research (03/2026)
Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, and Lu Wang. GRACE: discrininatto-guidded chain-offthought reasoning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, Decem- ber 6-10, 2023, pp. 15299-15328. Association for Computational Linguistics, 2023. doi: 10.18653/V1/222 FINDINGS- EMNLP. 1022. URL https:///oi... http:///doo..rgg1//// http://.rr/.....1//////////01
18653/v1/2023 http:///2222...2nneeeeeee penddingg-ennl.... 1022.
Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, al. Prometheus: Inducing fino-grained evaluation capability language models. In The Twelfth International Conference on Learning Represenlations, 2023.
Aviral Kumar, Vincent. Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Sharig Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning. Xiv preprint arXiv:2409.128 2024.
Nathan Lambert, Valent ina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403. 13787, 2024.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making language models better reasoners with step-aware verifier. In Proceedings of the 61st Annual Meeting of the Association for Computational Lánguistics (Volume. 1: Long Papers), pp. 5315-5333, 2023.
Hunter Lightman, Vincet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lec, Jan Leike, John Schulman, Ily Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20057 2023.
Xin Liu, Muhammad Khalifa, and Lu Wang. Litcab: Lightweight language model calibration over short-and long-form responses. arviv preprint or Xrxiw:2310.100 19208, 2023a.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303. 16600, 2023b.
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint orXiv:2406. 06592, 2, 2024.
Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. arXiv preprint ar Xix:222 10. 12332, 2024.
Niklas Muennighofff Zitong Yang. Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Fmnanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXie preprint arXiv:2501.19993, 2025.
David Rein, Betty Li Hou, Asa. Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&ta benchmark. In First Conference on Language Modeling, 2024.
Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jachoon Lee, Kelvin Xu, et al. Beyond human data: Scaling self-training for problem-solving with language models. arxiv preprint urXim:2312.0655 2023.
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling 11m test-time compute optimally can be more effective than scaling model paramoters. arXiv preprint Xrxi::2408.00 033114 2024.
Kaya Stechly, Matthew Marquez, and Subbarao Kambhampati. Gpt-4 doesn't know it's wrong: An analysis of iterative prompting for reasoning problems. Xrx preprint ar arXiv:2310. 210:22310.12397, 2023.
14

---

Published in Transactions on Machine Learning Research (03/2026)
Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Scan Welleck, and Chuang Gan. Easy- to-hard generalization: Scalable alignment beyond human supervision. arv preprint or arXiv:2403.094 2024.
Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown, November 2024. URL https:
//quenlm... geggeelm.githu...... wwtpoo/////g-aa//// 3wb-preeeew.......
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa. Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and basoom---sedd foedback. arXiv preprinl Xr::222 1. 1275, 2022.
Ziyu Wan, Xidong Feng, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-lin like tree-search can guide large language model decoding and training. In forty-first International Conference on Machine Learning, 2024.
Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfong Qu, and Jie Zhou. Is chatgpt a good nlg evaluator? a preliminary study. aiviv preprint or arXiv:2303.04 040448 2023a.
Pciyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui. Math-shepherd: A label-free step-by-step verifier for Ilms in mathematical reasoning. arXiv preprint arXiv:2312.0880 2023b.
Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi- Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. Self-tanght evaluators. arXiv preprint 2rx:::2408.022 2024.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint Xixx:2203.11172 2022.
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. arXie preprinl 200.2408.0072, 2024.
Wei Xiong, Hanning Zhang, Nan Jiang, and Tong Zhang. An implementation of generative prm. https: 2024.
Bwwoo///RLLReeRee // //ggthub.ccom wwt..//////////III.IIIEI
Wei Xiong, Hanning Zhang, Chenlu Ye, Lichang Chen, Nan Jiang, and Tong Zhang. Self-rewarding correction for mathematical reasoning. arXiv preprint arXiv:2502. 19613, 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Cheuxu Lv, el al. Qwen3 technical report. orXiv preprint arXiv:2505.0933 2025.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809-11822, 2023.
Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of language models: Part 2.2, how to learn from mistakes on grade-school math problems. arXiv preprint arXiv:2488 16293, 2024.
Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. Distilling system into system Xixiv preprint arXiv:2407.00023, 2024.
Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. arXie preprint orXiv:2412.01991 2024.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 15:15476-15488, 2022.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling next-token prediction. arXie preprint arXiw:208.. 15240, 2024a.
15

---

Published in Transactions on Machine Learning Research (03/2026)
Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Jackyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. Small language models need strong verifiers to self-correct reasoning. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 15637-15653. Association for Computational Linguistics, 2024b. doi: https://doi. org/ 10. 18653/v1/2024. findings-acl.99 .244.
URL
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprinl trXi:2251.00033 003000,,, 2025.
Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Processbench: Identifying process errors in mathematical reasoning, arXiv preprint 24 1244122.055, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhnang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Newral Information Processing Systems, 36:46505-46623, 2023.
Kaitlyn Zhou, Jena D Hwang, Xiang Ren, and Maarten Sap. Relying on the unreliable: The impact of language models' reluctance to express uncertainty. arXiv preprint 0r1:02401.006730 2024.
Lianghui Zhu, Xinggang Wang, and Xinlong Wang. Judgelm: Fine-tuned large language models are scalable judges. or Xiu preprint ar 2ivi 17631, 2023.
16

---

Published in Transactions on Machine Learning Research (03/2026)
Synthetic training sample
Problem:
Find the area in square feet of square with a perimeter of 32ft.
Solution:
Step 1: A square has four equal sides, so the perimeter is four times the length of one side.
Step 2: I can write an equation to relate the perimeter and the side length: ds = 32, where a is the side length in feet.
Step 3: To solve for s, I can divide both sides by 4: s 32/4 = 8.
Step 4: Now I know the side length of the square is & feet.
Step 5: The area of square is the side length squared, so the arca is 8 8 64.
Step 6: The area is in square foet, so the units are ft?.
Step 7: The answer is 64 ft
<think>
Alright, let's verify step step the solution provided for finding the of square with a perimeter of feet.
Step 1: "A square has four equal sides, so the perimeter is four times the length of one side.' This statement is accurate. square indeed has four sides of equal length, and the perimeter is the of all sides. Therefore, perimeter (P) equals 4 times the side length (s), The step is \\ boxxe(correett)
Critique:
4a.
Step 2: "I write equation to relate the perimeter and the side length: 4s 32, where is the side length in feet." Critique: Given that the perimeter 32 feet, setting up the equation 4s 32 is correct. It properly represents the relationship between the perimeter and the side length of the square.
The step is (boreccc
Step 3: "To solve for can divide both sides by 4: 32/4 8.'
Critique: This step correctly solves for by dividing both sides of the equation by 4. The calculation 32 divided by equals accurate.
The step \) mooxedfcorrece
Step "Now know the side length of the square is feet.' Critique: Based on previous step, feet, which correctly stated here.
The step (/boxed (correct)
Step "The area of square the side length squared, so the area s² 8 64." Critique: The formula for the
square is indeed s squared. Calculating 8 squared equals correct.
The step is \boxed (correct)
Step "The area square feet, the units ft?."
Critique: Since the side length is in feet, the will in square feet (ft)). This is the correct unit for The step bbbooedfcorree correct)
Step 7: "The answer 64 ft*
Critique: This is the final answer, correctly stating the area 64 square feet.
The step is [booeefcorrect
</think
Is the solution correct? Yes
Figure 12: Training example from our sampled 1K synthetic chains.
A
Training data
A.1 Sampling
Figure: 3 shows the prompt used with QwQ-32R-Preview to sample verification chains for training. We use the problem-prefix pairs from the PRMS00K train split (Lightman et al., 2023), which based on MATH (Hendrycks et al., 2021). We sample 4 verification chains for each prefix in the dataset with temperature = 0.1 until we have 1K verification CoTs that (1) follow the desired format i.e., the model generated an extractable label for every step in the prefix and (2) the verbalized step labels match the gold labels in the PRMS00K dataset annotation, (3) at most 4096 tokens long, and are (4) roughly balanced with respect to final answer correctness.
Roughly 20% of the initial samples satisfied criteria 1 and 2 in our case, which means that about 5K samples in total are needed to obtain 1K filtered samples. Table 2 shows the statisties of the final dataset. We also show example from the 1K training CoTs in Figure 12.
17

---

Published in Transactions on Machine Learning Research (03/2026)
A.2 Preprocessing
From the sampled verification CoTs, extract boxed predictions (e.g. \boxed(correct) (boxed(incorrect)) for each reasoning step. Then we filter the data as follows:
CoTs with missing malformed verification decisions discarded.
CoTI where verification decisions don't match ground truth labels are filtered out. Precisely, we require all step-by-step verification decisions to match the ground truth labels.
Then we clean the data as follows:
We standardize boxed notation formats.
Content after the last verification decision removed.
Special tokens (e.g., <think> and </think>) are added to designate verification reasoning.
A.3 Contamination Control
To avoid test data leakage, we ensured that none of the 1K verification traces used for training THINKPRM overlap with the test problems in MATH-500, AIME '24, ProcessBench (OlympiadBench; OmniMath)), GPQA-Physics, LiveCodeBench. We applied exact-match filtering between PRM800K problem texts and each evaluation set. We also verified that ATME '24 problems are not present in PRM800K, which only includes pre-2023 math problems.
Correctness Distribution
# correct
486 (48.6%)
# incorrect
514 (51.4%)
Step label distribution
# correct
7474 (92.3%)
# Incorrect
625 (7.7%)
Prefix statistics
# unique questions
average steps per prefix
min # steps
max # steps
53
Verification Chain Statistics
Average verification chain length (tokens) 1037.0
Min verification chain length (tokens) Max verification chain length (tokens) 3669
Table 2: Statistics of the 1K finetuning dataset.
18

---

Published in Transactions on Machine Learning Research (03/2026)
Data generation prompt
You are given a math problem and a propused multiplo-step solution (with a step on each line):
[Math Problem] (problem)
[Solution]
(solution)
Review and critique the proposed solution stepe and determine whether cach stop correct. If the solution is incomplete, only critique the steps that provided. Your output must be the following format:
Let's verify step step:
Step 1: <critique>... step (boxcc<corrccc//neeom
Step 2: <critiquc>... step (boooc/corroc///ccorre
Step n: <critique>... The step (coxee(corrett// (onneedcoorett/nooreeee
Once you find an incorrect step, you should stop since you don't need to analyze the remaining steps.
Figure 13: Prompt used to generate synthetic verification chains as described in $3.2.
B
aaM-as-a-Judge Limitations
We show examples of failures from QwQ-32B-Preview and LLM-as-judge verification in Figures 15 and 17.
when used for
C Training details
C.1 Training ThinkPRM
We train all THINKPRM variants on the filtered set of 1K verification chains.
LoRA finetuning. For large-base models, use LoRA (Hu et al., 2022) with rank 32 and 16. We finetune Qwen-2.5-14P Instrrcct and R1-DeepSeek- 1eB 14B for 3 epochs, and QwQQ-3B--Peeveew for 5 epochs. We use an effective batch size of 16 and constant learning rate of 4 x 10-4 without warmup. Training takes roughly 1.5 hours for the 14B models and 4.5 hours for QwQ-32B-Previiz on a single A100 80GB GPU. practice, we found D particularly difficult to train with LoRA: even after finetuning, still produces a relatively high fraction invalid judgments. We suspect that full finetuning would alleviate these issues but would require computation than we had available.
Full finetuning for smaller models. For the R1-Distill- Qwen models ({1.5B, 7B)), use full-parameter finetuning rather than LoRAA. The 1.5B model is trained for 3 epochs with effective batch size of 32 and a constant learning rate of x 10 (no decay or warmup). Both the 1.5B and 7B variants are trained using data parallelism on four RTX A6000 48GB GPUs. On the 1K chains, training the 1.5B model takes about 30 minutes and the 7B model about 2 hours.
C.2 Training Discriminative Verifiers
We train R1-Qwen-14B for 1 epoch over the entire PRMS00K dataset using two A100 80GB GPUs with batch size of and a learning rate of 6 x 10-5, We use constant learning rale scheduler with no warmup. Following prior work (Wang et al., 2023b; Zhang et al., 2025), we train the model using binary cross-entropy maximize the probability of the tokens "4" and u_* for correct and incorrect steps, respectively. The R1-Qwen-1.55 1.5B model is trained with the same infrastructure with batch size of 64 and learning rate of 10-4 with warmup ratio of 10%.
19

---

Published in Transactions on Machine Learning Research (03/2026)
QwQ-33BB- Pree-er stuck
loop
Okay, so I need to analyze each step of the proposed solution to the math problem. The problem is: If f(x) br 2 what is the value of f(-2) + [(-1) + f(0?? The proposed solution is given, and have to check if each step is correct.
First, let me read through the problem and the solution carefully.
The function is f(x) --2 They the of f(-2) + f(-1) f(0).
Looking the solution steps:
Step They plug -2
and calculate
2) 2
Step 1:
They mention that plugging x = -2 into the denominator gives zero, which would be division by zero.
Step 2:
They generalize that we should avoid plugging in values that make the denominator zero.
Step 3: Similarly, plugging = -1 into the denominator also gives zero.
Step 4:
Plugging 0 into the denominator also results in division by
no! The original function given f(z)=
zumerator
Nol The problem says: f(z)
Eumerasar
Jenomitnator
No! The problem says: f(x) Tepominstor
No! The problem says: If f(x) demomitnttor
No! The problem says: If f(z)
Jenominntto
No! The problem says: If f(x)
cumerniar
No! The problem says: f(x)
Eumerator
No! problem says: f(z)
gumeratog
No! The problem says: f(x)
Eumeratar.
Jenominattrr
No! The problem says: f(x)
Figure 14: Degeneration behavior from QwQ-32B--Previee with LLM-as-judge verification. Problem and solution are omitted for brevity.
20

---

Published in Transactions on Machine Learning Research (03/2026)
Deepseck-Hi Rtl---t--iiiiiia Distilll Qwen- 14B stuck in loop
Okay, so need to review and critique the solution step by step for the given math problem. The problem is to find the exact value the continued fraction expression:
1 1/(1 2/(1 + 1/(1 2/(1 + +..))))
The solution provided is split into steps, enclosed in tag. need go through each step, analyse it, and
thore errors if the logic holds, After that, I'll determine if the solution correct by outputting "yes" or "no" in a box. expression
Let me start by understanding the problem. It's continued fraction that alternates between 1 and in the numerators. The
x 11+ 1/(1 2/(1 + 1/(1 + 2/(1 + +..]]]]
2, etc. The solution starts by denoting expression and then tries to find relationship by substituting the inner expression back into
the pattern repests every two steps:
then
Wait, original expression
+ 1/(1 2/(1 + 1/(1 + 2/(1 ...))). So, part after the first
2/(1 + 1/(1
2/(1 ...))) But that part isn't because starts with + 1/(..). So, maybe the substitution isn't directly x. Let think.
Wait, perhaps the inner part is similar but not exactly x. Let me denote the iuner part as y. So, y = 1 2/(1 1/(1 + 2/(1
...))) But then, the expression for would involve another similar structure. Hmm, mayhe the solution oversimplifying here.
Wait, actually, when you multiply both sides by (1 2/x), the left side is x511 + 2/x) x
The right So, the correct equation after multiplying both sides by 2/x) is:
[1 + 1/(1 21x)]* 2/x) = 1*(1 2/x) [1/(1 21x) 2/x) = (1 2/x) +1= 2/x.
x + = + 2/x.
But in step they have:
+ 2= 1
1/(1 2/x) + 2/x + 4/x²
Which scems
incorrect. They serm have distributed multiplication incorrectly.
Wait, let me see. If you have x+(1 + 2/x)
1/(1 2/x) + 2/x) [2/x*(1 + 2/x).
But that doesn't ' make sense because when you multiply both sides by (1 + 2/x), you should only have:
Left side: x*11 + 2/x)
Right side: [ 1/(1 (1x)]"(1 2/x)
Which simplifies
2/x.
But in the solution, step 6 says:
x*(1 2/x)
1/(1 2/x) 2/x*(1 2/x)
Which seems be adding extra term, 2/x*(1 2/x), which is not correct. So, step is incorrect.
Therefore, the solution has an error step Instead correctly multiplying both sides, they incorrectly expanded the right- hand side, leading
incorrect equation.
Bocause of this mistake, the subsequent steps are likely incorrect as well. So, the entire solution is flawed from stop onwards.
Wait, let me compute the continued fraction numerically to check. depth.
me compute the continued fraction up Let me compute it step by step: Start with the innermost part and work outwards.
it seems converge approximately 2,414, which Wait, but according to earlier solving, x should be sqr((2) s11414, but the numerical computation suggesta it converges to approximstely 2,414, which + sqrt(2). So, which one la correct? Wait, perhaps I made mistake in setting the equation. Wait, let's is re-examine the original expression:
sqrt(2) #2.4142.
But when tried to solve it earlier, set 1 + 1/(1 + 2/xx,, which led to x s2rt(2). But the mumerical computation suggests it converges to 1 + eqrt(2). So, perhaps my initial subatitution was incorrect. Wait, perhaps I need to define differently. Let try define the continued fraction properly.
Wait, let's solve the system:
From
1/y and
+ 2/x.
Substitute y = 2/x into x + 1/y:
1/(1 2/x)
Multiply both sides by (1 2/x):
Alternatively, perhaps I need define differently. Let try define the expression starting with 1/(..): = + 1/(1 2/(1 + 1/(1 2/(1 + +..]))] brings
But y
the original substitution.
Which is the equation before, leading x2rt(2). But the nunerical computation suggests it's approaching 1 sqrt(2). So, perhaps my initial substitution incorrert. Alternatively, perhaps the continued fraction structured such that after the first 1/(..), the denominator another continued fraction similar to but shifted.
Figure 15: Degeneration behavior from Problem and solution are omitted for brevity.
with LLM-as-judge verification.
21

---

Published in Transactions on Machine Learning Research (03/2026)
Problem and
ProcessBecch
Problem: a, positive
> a. squares lengths a, each other, such
larger square has lower left corner at (0 and the smaller square has lower left corner at [a, 0). Draw the line passing through (0, a) and (o + b, 0). region the two squares lying above the line has area 2013, If (a, b) the unique pair maximizing
a 6, compute f.
Solution:
We begin by finding the equation of the line through (0, and (a + 0) The slope ls:
Using point-slope form from (0, a):
8 =
Area above the line in the larger square: At x the height is:
a²
ab
9
+
+ b
So the trapesoid:
a+ ab
8 20
Arcaj
a
2(a b)
Area above the line in the smaller square: This triangle with height ab and base
ad²
Areag
b
a+
2(a +
Total Area:
a²
2h ab?
a(a? 2ab +
a(a + h)2
a(a b)
Total Area; + Areaz
2(a b)
2(a
2(a b)
We are given:
b) = 2013 = a(a + b) 4026
Express b
1026
Apply the constraints
b to:
1028
2013
2013
b ta A020 ta a² 2684 /2644
So /2013 < a < /2684. To maximize a + b 4026 we minimize a, so let
/2013. Then /2013, satisfying the
Figure 16: An example of problem and solution from OmniMath subset of ProcessBencc
22

---

Published in Transactions on Machine Learning Research (03/2026)
Incorrect verification
Okay, so I need to review and critique the given solution stop by step, Let me go through each step carefully. by identifying key points properties. mentions that J the excenter opposite A, which is
Step The solution cotrect. A₁ is where the excircle touches BC, and B,, Ct are where it touches the extensious of AC and AB. Then, it says B1 la perpendicular to AB and intersects Hmm, this seems okay, but wonder if they's ve correctly identified all the properties. Maybe they should have mentioned about the excenter's s properties, like the fact that it's the intersection knowledge.
D.
the foot of the perpendicular from C1 to DJ.
external angle bisectors. But maybe that's 's assumed
Step 1: They note that since A; is perpendicular to AB, D is the projection of A1 onto AB. So, D lies on AB and D is perpendicular AB.
Wait, is that necesarily true? If A B, is perpeudicular to AB, and they intersect at D, then D is the foot of the perpendicular from Ai AB. That seems correct.
Step 2: They examine the excircle and the triangle, mentioning that the excircle langent to BC at A1, and the tangents from A₁ to B1 and Cj form right angles with the radii Then, DJ is line through the excenter and the intersection point of the perpendiculars from A₁ and to AB.
Hmm, I'm not sure about this part. The line DJ is through the excenter J and point D, which the foot from A₁ But does involve perpendiculars from An and B,? think the solution might couflating different properties equidistant from all sides, but DJ is just a line from D to J, not necessarily involving perpendiculars from both A₁ and Bj- Maybe this is misstatement.
The excenter is
Step 3: They state that since is the foot of the perpendicular from C1 to DJ, angle C₁ ED is 90 degrees. Then, they argue that angles BEA; and AEB; are equal due to symmetry and properties of the cyclic quadrilateral.
Wait, why is symmetrically placed? The solution says it's because the properties of the excircle and right angles, but I'm not entirely convinced. Just because E is a foot of perpendicular doesu't automatically make it symmetric with respect to reason, cyclic quadrilateral:
similar triangles, but this step feels a bit havd-wavy. Step 4: They conclude both angles are 45 drgrees herause of symmetry and right angles, forming a cyclic quadrilateral. But not sure lead that?
B,. Maybe there's
eyelic quadrilatera) part clearly established. Also, 45 degroes seems specific does the problem necessarily
Wait, maybe should draw a diagram to visualize. If A₁ B3 is perpendicular to AB, and E the foot from C2 to DJ, which is connected to the excenter. Hmm, maybe the angles indeed degrees because of the right angles and the properties the excentral triangle, but I'm not entirely certain.
some correct steps might making assumptions without sufficient. justification. For example, in Step 3, the symmetry argument isn't fully explained. Also, in Step 2, the deseription of DJ might not be accurate. The conclusion of degrees might be correct, the reasoning feels rushed. Final Answer After considering the steps, think the solution might have gaps justifiration but arrives at the correct. answer. So, I'll say yes, but with some reservations. yea
Overall Thoughts:
solution seems
Figure 17: Incorrect verification by
for the problem-solution pair in Figure 16.
23

---

Published in Transactions on Machine Learning Research (03/2026)
D
Results on ProcessBench before and after finetuning
Table 3 shows the performance numbers of LLM-as-a-Judge and THINKPRM on ProcessBench.
LLM-as-a-Judge
THINKPRM
Model
OlympiadBench
OmniMath
OlympiadBench
OmniMath
Random baseline
39.1
32.7
39.1
32.7
R1-Qwen-1.5E
5.0 (61.4
5.4 (66.1 %6
76.3 (1.4 9
75.7
R1-Qwen-7B
44.8
(18.2
45.7 (20.9
73.4 (1.1 %0
74.0
R1-Qwen-14B 14B
72.8 (18.3
67.8 (18.6 %6)
87.3
85.7
(2.3
QwQ-32B-previcw
50.6
55.5 (10.0 98)
73.1 (16.1
73.2 (7.9 95)
Table 3: Average Fi-score on OlympiadBench and OmniMath subsets of ProcessBench (Zheng et al., 2024) comparing LLM-as-a-Judge to THIKPRMM finetuned on 1K examples. Random baseline for OlympiadBench is 39.1% and for OmniMath 32.7%. Percentage of bad outputs (repetitions, invalid label formatting, overthinking, etc.) are shown in red. LLM-as-a-judge with reasoning models suffer from issues that limit. their utility generative verifiers.
E Evaluation details
E.1 Extracting verification labels/scores
Extracting binary labels for ProcessBench. For evaluation on ProcessBench ($3.1), we follow Zheng et al. (2024) and extract binary yes/no labels from the verifier outputs using a simple regex-based parser. Concretely, we first strip LaTeX \text(...] wrappers (e.g., \text<(yes} + yes), then search for the final \boxed(.... span and take its contents as the model's decision. We lowercase and trim this string and map common variants such as yes/cors ht/////rre/ttttt/ to the positive label and no/incorrect/fall label. If no boxed span is found the extracted string does not match any of these patterns, we treat the output invalid and count the corresponding step incorrect in all ProcessBench metrics.
the negative
Extracting numerical scores. Following prior work (Snell et al., 2024; Beeching et al.), we aggregate scores from DiscPRM by using the score of the last step. For THNNPRMM and LLM-as-a-judge, we first prompl the model to generate the verification chain up a maximum of 8192 tokens where the model generates a label for each step inside \boxed(), then we force decode the string "Is the solution correct?" P("yes") and use P("yes") P("ooo as the solution or prefix score. That is, we do not predict a score per step then aggregate as in DiscPRM, but a score per the whole solution, which we found to work best and simplify the need for aggregation over step-level scores.
E.2 Best-of-N selection
We sample solutions using a temperature of T 0.8 for Llama-3.2-3B: Instrunttt and T' 0.4 for Qwen-2.5-141 We instruct all models to think step by step and put the final answer in \boxed All our Best-of-N experiments use weighted majority voting, which scores final answers based on the sum of the verifier scores of their solutions (Uesato et al, 2022; Wu et al., 2024; Sun et al., 2024), except for our experiments on AIME 24, where we use the verifier score directly to rank the solution, as we found this to perform better for all verifiers.
E.3 Verifier Verifier-guided beam search
Under verifier-guided beam search, we sample candidate next steps and score them with the process verifier, then selects the top-K out these to further expand and so on. Our implementation based on (Snell et al., 2024; Beoching al.), which maintains boams total, and samples M candidate next steps per beam. We set M for all experiments and run search for a maximum of 20 steps per beam. To sample next steps, T 0.6 and double newlines the step delimiter.
24

---

Published in Transactions on Machine Learning Research (03/2026)
E.4 Sequential scaling of verifier compute
We achieve budget forcing (Muennighoff et al., 2025) by triggering the model to think again for R rounds, where each round uses a unique trigger phrase that incites the model to revisit or double-check its earlier verification. Each round allows the model to think for up to 8192 tokens and we truncate the output if it exceeds that maximum length and then force decode the label. We use different trigger phrases for each round since we found that using the same phrase causes the model repeat what it did in the last round.
We do maximum of R 4 thinking rounds, and use the phrases "Tet me double check', "Let's verify again', and "Did I miss something?", for rounds 2, 3, and 4 respectively. We do not investigate deeply into optimizing the trigger phrase, but we note that performance can be sensitive to the choice of trigger phrase used. We use the same phrases for both THINKPRM and L.MM-as-a-judge to ensure fair comparison.
E.5 MATH-500 test examples
As running all 500 examples from MATH-500 will require lot of compute, all experiments 100 randomly sampled subsets from MATH-500 (Hendrycks et al., 2021). We pick the 100 problems such that they cover different difficulty levels, shown in Figure 18.
Difficulty levels in the MATH-500 split we use
Parallel scaling of verifier compute: MATH-500
25
85
80
Accuracy Ecuracy [%6
75
20
70
65
Count
15
60
55
10
50
2
2
2
2
24
2
Number of solutions
ThinkPRM- -1BB
DiscPRM-144
ThinkPRM-14B@4
Majority
5
ThinkPRM-1444 -48@@
Problem Level
Figure 18: Histogram of difficulty levels in our 100-
Figure 19: Scaling of verifier compute by parallel sampling of multiple verification CoTs and aver- aging their scores. Parallel scaling (THINKPRM- 14B@4 and TuNKPRMM14BBS performance curve compared to scoring based on single CoT (THINKPRM-14B))
problem subset from MATH-500.
further boosts
F More results and analysis
F.1 Additional results scaling verifier compute
Figure 19 shows results of THINKPRM-14B of parallel scaling verifier compute by sampling K = 4 and K = 8 CoTs with temperature ure = 0.6 and aggregating their scores. Parallel scaling indeed lifts up the accuracy curve THINKPMM-14B compared to standard K = 1 with greedy decoding. However, performance plateaus rather quickly and 8 remains comparable to K 4, while slightly better smaller sampling budgets. Figure 20 compares parallel to sequential scaling under the same token budget. While there no clear winner, parallel scaling seems perform slightly better at best-of-8.
25

---

Published in Transactions on Machine Learning Research (03/2026)
Parallel vs. Sequential Scaling: MATH-500
Parallel vs. Sequential Scaling: MATH-500
80
80
Accuracy (%)
Accuracy
70
65
65
60
60
55
55
50
2
2
2
2
Number of solutions
Number of solutions
+ ThinkPRM-144B
ThinkFRM-14B (2 thinking rounds)
ThinkPRM-114B ThinkFRM4-44B@4
ThinkPRM-14B (4 thinking rounds)
TrinKRRM-14BB2
Figure 20: Parallel sequential scaling of THINKPRM compute under the same generation budget with Qwen-2.5-14B generator. Parallel scaling (model@K) is done by independently sampling K verification CoTs and aggregating their scores. Sequential scaling is done by prompting the model K times to revise its own verification for K thinking rounds. Both setups generate up until 8192 tokens per generation. We do not observe a clear winner although parallel scaling seems slightly better especially larger sampling budgets.
Scaling training data: MATH-500
Scaling training data: MATH-500
70
Generator: Uama 3.2-3B-instruct
85
Generator: Qwen2.5-12
80
Accuracy (%)
Accuracy (%)
75
60
70
65
55
60
50
55
50
45
2
2
2
2
2
2
2
2
2
2
Number of solutions
Number solutions
ThinkPRM-148 (1K)
DiscPRM-14B
ThinkPRM-1.5BBE (1K)
DiscPAM-1.58 RLHFFlow BB-Deepeeek
ThinkPRM-1.5B@@ (65K)
ThinkPRM-14B (65K)
Majority
Best-of-N results with THIKPRM-- 1.5B comparing the version trained on 1K examples (used throughout the paper) and version trained on 65K examples. THINKPRM benefits from train- ing on more synthetic data the performance can further improve with more training.
Figure 21:
Figure 22: Best-of-N results with THIKKRM-144B
comparing the version trained on 1K examples (used Uhroughout the paper) and version trained on 65K examples. THINKPRM benefits from train- ing on more synthetic data the performance can further improve with more training.
F.2 Scaling training data of ThinkPRM
Here, show results when training THINPPRM-14B and THINPPM--155 using synthetic data from all PRM800K. The goal is to show that THINKPRM can still benefit from training on more synthetic data. Here, we train both RR--Disill-Qween-1.BB and RI- Ri-Distill-Qwen-111 on a total of 65K verification CoTs we obtained by sampling and filtering as explained in $3.2. Figures and 22 show best-of-N performance with THINKPRM-1.5B and THINKPRM-14B respectively when trained on 65K and compares examples. Interestingly, THINKPRM benefits from additional training, and can further improve the accuracy curve compared to the 1K-trained version MATH-500. We note, however, that while training on more
training on 1K
26

---

Published in Transactions on Machine Learning Research (03/2026)
Best-aff of-N: MATH-5D0
Best-of-N: GPQA-physics (OOD)
Best-of-N: LiveCodeBench (OOD)
Cererratoo:: 2-5-1111-1111
80
Accuracy 180
90
70
Accuracy
70
Pass rate
65
60
50
2"
2
2²
24
23
Number of salutions
Number of solutions
Number solutions
ThinkPRM-14B DiscPRM-14B
LLM-as-a-Judge
ThiniPRM-148
DiscPRM-14B
Majortty
Figure 23: Best-of-N performance on MATH-500, GPQA-Physics, and LiveCodeBench with 95% confidence intervals (bootstrap = 100). Despite wider intervals due to limited test set size, ThinkPRM-144 consistently outperforms DiscPRM-14B and LLM-as-a-Judgo, espocially at larger sampling budgets.
math data boosts performance on MATH-500, we observe some performance drop on out-of-domain tasks due to the distribution shift.
F.3 Results on the Full MATH-500 Dataset
In addition to the 100-problem subset used in several main-text analyses, we report best-of-N results on the full MATH-500 test set. All results use Qwen-2.5-14B as the gencrator, and we evaluate a range of verifiers under identical decoding and sampling budgets.
Table 4 shows that ThinkPRM-14B consistently outperforms all baselines across best-of-8, best-of- 16, and bfst-of-32 settings. Notably, ThinkPRM-14B improves over strong off-tho-shelf PRMs such as Math- Shepherd and Qwen-225-Maath-PPRMM despile being trained with substantially fewer step-level labels. These results confirm that the gains observed on smaller subsets extend to the full MATH-500 benchmark and are not an artifact of limited evaluation size.
Verifier
N-8
N=16
N=32
Majority
58.6
69.0
73.2
DiscPRM-14B
71.4
73.2
75.0
Math Bathh-Shepherd7B
68.6
74.0
76.0
Quen-225-Math--RREE.
72.6
76.8
78.8
ThinkPRM-14B
73.6
77.2
80.2
Table 4: Best-of-N accuracy (%) on the full MATH-500 dataset using Qwen-2.5-14B as the generator. ThinkPRM-14B achieves the strongest, performance across all sampling budgets.
F.4 Bootstrapped Confidence Intervals for MATH-500, GPQA, and LiveCodeBench
To assess statistical robustness on small-scale evaluations, we report bootstrapped 95% confidence intervals for the best-of-N accuracy curves three benchmarks: MATH-500 (100 problems). GPQA- Physics (86 problems), and LiveCodeBench (200 problems). These results are visualized in Figure 23, from left to right.
Confidence intervals are computed via bootstrap resampling (n = 100) over test problems, with shaded regions indicating the 2.5th 97.5th percentile range. While evaluation sets are relatively small, the observed performance trends remain consistent across tasks. We focus here on best-of-N- scaling setup-but but note that. trends align with beam search results reported in the main text.
primary test- time
27

---

Published in Transactions on Machine Learning Research (03/2026)
F.5 Results with automatic labels
Table shows performance when filtering training data based manual labels (PRM800K) automatic labels (Math-Shepherd) (Wang et al., 2023b). THINKPRM still performs well even with automatic labels, and comparably to manual labels.
Model
OlympiadBench OmniMath
ThinkPRM-1.5B (PRM800K)
76.3
75.7
ThinkPRM-1.BB (Math-shepherd)
75.8
76.5
Table 5: Comparison of ThinkPRM-1.5B trained on PRMS00K vs Math-shepherd step labels.
F.6
Verifier performance terms of problem difficulty
Best-of- 32: Math-500
Best-of-32: GPQA-Physics
100
100
Accuracy (%)
Accuracy (9))
60
60
40
20
20
0
2
3
4
2
4
Problems binned by diffficutyy
Problems binned by difficulty
ThinkPRM-14B
DiscPRM-148
Figure 24: THINKPRM helps with challenging reasoning problems compared to DiscPRM. The generator model here Qwen-2.5-14B for MATH-500 and Qwen-2.5-322--Inst Instruct GPQA.
Each problem in MATH is annotated with one of 5 difficulty levels, so we just use that information. For GPQA-Physics problems, we first compute the pass 1 rate of Qvee22.532B-Instruv Instrucc for every problem by sampling 32 solutions for each problem. Then, bin the pass@1 rates into four different bins and use the bin assignment for each problem the difficulty level. As discussed in $5.6, Figure 24 show how THNKPRM helps more with verifying hard problems compared to DiscPRM. OlympiadBench
Prompt
OmniMath
Single Yes/No
50.6 (7.9 96)
55.5 (3.2 %)
QwQ-32B
Rad-step-index
65.4 (18.3 %)
69.0 (16.8 %)
Decislon-for-acch-se eacch--ste
40.1
36.6
Single Yes/No Bad step index
72.8 (13.3 %))
67.8 (18.8 %0)
R1- f1l-Qwen11a 14B
77.7
76.6 (17.9
caatiioon-oo--ecc-sttee
73.3 (14.6 %)
71.4 (14.7 96)
Table 6: Fi-score on OlynpiadBench and OmniMath subsets from ProcessBench with different prompts.
Ratio of invalid judgments is shown in red.
F.7 Compute-matcher comparison on GPQA-Physics
For completeness, we report a compute-matched comparison on the out-of-domain GPQA-Physics benchmark, analogous to the analysis on MATH-500 in $5.4. We vary the number of sampled solutions and the verifier
28

---

Published in Transactions on Machine Learning Research (03/2026)
calls and plot accuracy function of the estimated FLOPs budget for different verification strategies: THINKPRM (ThinkPRM-14B) DiscPRM-14B, LLM-as-a-Judge (14B), and majority vote.
Figure 25 shows the resulting trade-off curves. At low compute budgets, DiscPRM achieves the highest accuracy, reflecting its ability to cheaply score many candidates. As the FLOP budget increases, however, THINKPRM consistently improves and ultimately attains the best accuracy among all methods, while majority voting saturates and LLM-as-a-Judge lags behind despite consuming the same or more compute. These results mirror the MATH-500 trends and further support our claim that THINKPRM makes more effective use of additional verifier compute than both discriminative PRMs and self-consistency out-of-domain tasks.
Compute-matched analysis: GPQA-Physics
Generator: (wen2.5-328-il 5-328-Instruct
Accuracy (%))
60
ThinkPRM-14B
Majority voting DiscPRM
55
LLM-as-a-judge
10tt 50 1x10 2 10th 10
Estimated FLOPs (log scale)
Figure 25: Compute-matoie analysis on GPQA-Physics. We plot accuracy as a function of estimated FLOPs (log scale) for different verifiers. While DiscPRM is competitive at low compute, ThinkPRM-144 achieves the best accuracy at higher FLOP budgets, indicating more effective use of verifier compute.
F.8 Inference- Time Cost Comparison of Verifiers
Verifier
Avg. Time
Estimated FLOPs
Notes Autoregrossive generation Autoregressive generation One forward pass (1 token)
LLM-as-a-Judde (14B)
87.6s
1.02 x 10
ThinkPRM-14B
29.3s
3.48 x 1013
DiscPRM-14B
0.46s
2.80 10
Table 7: Inference cost comparison of verifiers 128 OLYMPIADBENCH problem-solution pairs. We report average wall-clock time and estimated FLOOPs per input. FLOPs computed using FLOOPs T P with p= 1.4 x 10a0 (14B parameters) and T'' the number of generated tokens; for DiscPRM we approximate T = 1 (one forward pass per input).
To compare the inference efficiency of different verifiers, we measure the end-to-end latency and estimated compute (in FLOPs) of ThinkPRM--14B, DiscPRM-11BB, and LLM-as-a-Judge (14B) on 128 problems from OLYMPIADBENCIL All models run using batch size of 4 single NVIDIA A40 GPU.
We follow prior work in estimating FLOPs using the approximation FLOPs 2xTxP, where T is the number of generated tokens and P = 1.4 x 10th is the number of parameters for 14B models. This approximation is widely used in LLM scaling analyses and provides a fair estimate of autoregressive inference cost.
For ThinkPRM and LLMeas-a-Judge, T corresponds to the number tokens generated during the verification Thouuhtt (up to 8192 tokens). For DiscPRM, which produces scalar judgments via a single forward pass, we approximate compute as equivalent to generating a single token per input. This reflects its non-autoregressive nature and its extremely low inference cost.
29

---

Published in Transactions on Machine Learning Research (03/2026)
As shown in Table 7, ThinkPRM-14B is significantly more compute-cffciiet than LLM-as-a-Judge, reducing both FLOPs and wall-clock time by ~ 3x. DiscPRM-14B remains the most efficient by far as it requires only single forward pass input. However, it underperforms generative PRMs and, unlike THINPRMM, does not enable scaling verification compute.
F.9 Calibration Analysis
Because our best-of-N and search procedures rely on verifier scores as confidence signals, we examine how well these scores are calibrated. We plot reliability diagrams for each verifier obtained by binning predictions according to their reported confidence and plotting the empirical fraction of correct predictions each bin. We also report the expected calibration error (ECE) in every panel.
Across both MATH-500 and GPQA-Physics (Figure 26), clearly overconfident, while ThinkPRM-14B is somewhat better calibrated but still puts a lot of mass very high-confidence predictions. DiscPRM-14B is the closest to the diagonal on both datasets and has the lowest ECE. This matches our main takeaway: generative verifiers, including ThinkPRM, are useful primarily as ranking signals rather than as well-calibrated probability estimates, and calibration tends to degrade further under OOD shift.
consistent pattern. L.M-as-a-Judge is
LLM-as-a-judge (MATH)
ThinkPRM-14 (MATH)
DiscPRM (MATH)
1.0
Perfect callbratian
1.0
Perfect calibration on ECE=0.168
1.0
Perfect calibration
% Correct
% Correct
0.8
% Correct
0.8
0.6
0.6
0.4
0.4
0.2
0.0 0.1 0.2
0.3
0.4
0.5
06
0.7
08
0.9
1.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9 1.0
Confidence
Confidence
Confidence
LLM-as-a-judge (GPQA-Physics)
ThinkPRM- 14b (GPQA-Physics)
DiscPRM (GPQA-Phyyscs)
calibration ECE=0.311
calibration ECE=0.336
ECE=0.142
Correct
% Correct
Correct
0.6
0.4
0.0
0.0 0.1
0.2
0.7 0.8 Confidence
0.4 0.5
0.9
1.0
0.0 0.1 0.3 0.4
0.6 0.7 0.8
0.9
1.0
0.0
0.1
0.2
0.3
0.4 0.5 0.6 0.7 0.8 0.9 1.0
Confidence
Confidence
Figure 26: Reliability diagram for LLM-as-a-Judge, ThinkPRM-14B, and DiscPRM-14B on MATH-500 (top) and GPQA-Physics (bottom). Bars show empirical accuracy in confidence bins; the dashed line indicates perfect calibration. The legend in each panel reports the expected calibration error (ECE). Generative PRMs (LLM-as-a-Jure a-duge and THINKPMM) exhibit worse ECE compared to discriminative PRMs, mostly due to overconfidence.
30

---

Published in Transactions on Machine Learning Research (03/2026)
Example problen-solution pair from ProcessBench
Problem:
In quadrilateral ABCD, m BBBB + mZD = 270", The circumcirele of AABD intersects CD at point E, distinct from D. Given that BC 1, CE and DE T, compute the diameter the cireumcirele of AABD.
Solution:
Step To solve this problem, we will use several properties of geometry, particularly related circles and quadrilaterals, Let's break down the steps: First, we understand the given information. We have quadrilateral ABCD where LB LD = 270". The circumcircle of AABD intersects CD at point E, distinct from D. Additionally, BC = 4, CE 5, and DE 7.
Step 1: Next, apply properties of cyclie quadrilaterale and circles. Since E lies on the circumcirele of AABD, E la also part of the circumcircle of DABCD (assuming ABCD is cyclic, which will be justified later). In a eyclic quadrilateral, opposite angles to 180". Given 2B LD 270". suggests not sum to more than 360 in a quadrilateral. However, we consider the context, let's assume a correction was intended for the sake of explanation: Correct interpretation: If adjust for logical consistency, perhaps the intention angles at B or D is 90" (since this would fit the property of cyclic quadrilaterals better, aligning with the geometry principles in- volved). For this explanstion, let's proceed with an sasumption that either ZB or LD is 90", aiming
initial understanding
misinterpretation, since angles can-
imply that of the
coherent solution path.
We then use the Power a Point Theorem. The Power of Point Theorem states that for a point P outside a circle, the product of the lengths of the segments from to the circle equals the product of the lengths of the other segments when considering tangents or secants from that point. However, in this case, E Is not outside but on the circumference of the circle, and we apply differently: Given BC and apply the property of interscttngg chords in a circle, which states that the product of the leugths of the segments of one chord is equal the product of the lengths of the segments the other chord. Thus, in this scenario, we consider BC and CD as intersecting chords, where CD= = BC CE
Step 2:
5, DE
and knowing E part the cireumcircle,
can
BC CD CE DE
Substituting the known values:
4-9 9 = 5 DE
36 m 5 DE
Thus,
36 7.2.
Step 3:
This discrepancy suggests reevalustion of the given values or misunderstanding their application within the context provided. However, for the purpose of demonstrating the method:
Step 4:
Finally, we find the diameter of the circumcircle of AABD. The key to finding the diameter involves rec- ngmizing that the circumcirele's radius diametor crucial for calculating distancrs invalving points A, B, and D. However, without specific details about how E relates these points geometrically beyond being the circle, ad- ditional information such as the length of another segment or angle measures, directly caleulating the diameter becomes complex.
Step 5: The question's 's forus on and its relation to BC, CE, and DE might suggest lnoking at the propertics of chords in circle applying trigonometric relationships based the angles given initially. However, due the discrepancy noted and the complexitt introduced by the misinterpretation of the angle condition, direct calculation for the diameter of the circumcirele AABD isn't straightforward with the information provided.
Step 6: Given the complexities and discrepancies in the original problem statement, particularly with the angles summing to 270" in quadrilateral, the solution path becomes less clear. The explanation provided aimed to creatively interpret and apply geometric principles under the constraints of the given information. However, the direct calculation for the diameter of the circumcircle of AABD requires additional, precise information about the positions and relationships of the points A, C, D, and E within the quadrilateral and its circumcircle.
An example problem and solution from the OlympiadBench (He et al., 2024) in ProcessBench (Zheng et al., 2024).
Figure 27:
G More details on LLM-as-judge verification
The instruction used for LLM-as-judge verification in $3.1 is shown in Figure 29, which we refer to as Single-label. Table 6 shows performance of QwQQ-3BB-Previee and Deven-Deepseek-oo R1-14B with two more instructions. Bad-step-index refers to instructing the model to find the index of the first incorrect step and then if step found generate a single yes/no label regarding the solution correctness. Decision-for-each step instructs the model to critique and produce boxed label for each step. This instruction is shown in
31

---

Published in Transactions on Machine Learning Research (03/2026)
Example from S0-Q--2B-PPeeiie
Let's verify step step:
Step The initial step seems correctly identify the given information the problem. It mentions the quadrilateral ABCD with angles at and summing 270", and the circuncircle of &ABD intersccting CD E, distinct from The lengths BC 4, CE 5, and DE are also noted. This step is accurate summarizing the problem statement.
This step attempts to apply propertics of cyclic quadrilaterals but seems to get confused about the sum of angles in quadrilateral ABCD. It is mentioned that in cyclic quadrilateral, oppositc angles sum to 180%, , but the given sum of angles LB and 2D is 270", which indeed seems inconsistent. However, the step then suggests assuming that one of the angles is 90" for the sake of explanation, which is an adjustment to praceed with the solution. This part is a bit problematic because it alters the given information, but perhaps there was a misinterpretation in the problem sta statement.
Step 1:
Step 2: Here, the Power of Point Theorem is applied to points related to the circle. However, there is confusion in calculating the length of CD. It states that CD = BC + CE 4 == 9, but since E is point on CD distinct from D, and DE is given as 7, this suggests that CD CE DE = 7 12, not 9. This is error in calculation. Furthermore, the application of the Power of Point. Theorem seems misapplied berause E is on the circle, not outside it, the theorem should be applied differently, possibly considering intersecting chords.
Step 3: Recognizing the discrepancy the previous step, the solution acknowledges the need for reevaluation, which is good practice. However, without correcting the earlier mistake caleulating CD, subsequent steps may be built on incorrect foundations.
This step aims to find the diameter the circumcircle of AABD, which the main goal the problem. It notes that the diameter is related to the circumradius, but without specific details about the triangle's sides or angles, it's challenging to compute directly. The solution hints at the need for more information, which is accurate.
Step 4:
Step 5: The insight here is that the relationship between E and the other points might provide the necessary in information through chord properties or trigonometric relationships. However, due to the earlier discrepancies, this step does not lead to concrete calculation.
Step The conclusion reflects on the complexities and discrepancies in the problem and the solution approach. It acknowledges that the direct calculation for the diameter is not straightforward with the given information, which is fair assessment.
Overall, while the solution attempt shows understanding of relevant geometric principles, there significant in calculation and application of theorems, particularly Step 2. These propagate through the subsequent steps, leading an inconclusive solution. Therefore, the solution is not correct.
(no)
Figure 28: Example verification chain generated by BorQ-32BB-Previee in response to the problem-solution pair in Figure 27.
32

---

Published in Transactions on Machine Learning Research (03/2026)
LLM-as-a-judge prompt
The following is math problem and a solution (split into steps, enclosed with tags and indexed from 0):
[Math Problem] (problem)
[Solution] (prefix)
Your task review and critique the solution step by step. Your output must be in the following format:
Step <critique>>
Step <critique>
Step n: <critique>>
After your analysis, return yes/no depending on whether the solution is correct.
Please put your final answer (iee, yes/no) in \boxed{}
Figure 29: Prompt used for LLM-as-a-judge verification in $3.1
Figure 13. We observe high variance verification performance and invalid judgment ratio depending on the prompt.
We also show example from ProcessBench in Figure 27 and the generated verification chain from QwQ 32B-Preview in Figure 28,
H Examples from ThinkPRM
Figure 31 shows example of verification CoT from THINKPRM-14B problem and solution from the OmniMath subset in ProcessBench, shown in Figure 30.
33

---

Published in Transactions on Machine Learning Research (03/2026)
Example verification thain-of-thoughh
Problem: solid reclangular prism has dimensions
2. by by cube is out corner creating the new solid shown. What surface area the new solid?
Solution:
Step 1: determine the surface area of the new solid after cutting a 1 by 1 by 1 cube from one corner of by by rectangular prism, we will follow these steps: First,, calculate the surface area the original rectangular prism. The original rectangular prism has dimensions by by 2. The surface area Aariginal of a rectangular prism given by:
Aoriginal = 2(1w + + wh)
where 1 4, in = 2, and h Substituting these values in, we get:
Aoriginal = 2(4 2 4- 2 2) 2-
Step 2: Second, determine the effect of cutting out the by 1 by cube. When we cut out 1 by by cube from one corner, we remove three faces the cube (one face for each dimension) but add three new faces of the same size. Each face of the cube is 1 by 1, so the area each face is square unit.
Step 3: Third, calculate the change in surface area. Removing the three faces of the cube reduces the surface area by x 1 = 3 square units. Adding the three new faces increases the surface area by x = 3 square units. Therefore, the net change in surface area is:
- = 0
Step 4: Finally, calculate the total surface area of the new solid. Since the net change in surface area is zero, the surface area of the new solid remains the same as the original surface area:
Anew
Aoriginal 40
Thus, the surface area of the new solid is 40
Figure 30: Problem and solution from the OmniMath subsct ProcessBench. The verification CoT output by THINKPRM is shown in Figure
34

---

Published in Transactions on Machine Learning Research (03/2026)
Example verification chain-of-thought from THINKPRM-IdR
Verification: <think>
Alright, I've got this math problem about finding the surface area a solid after cutting out a small cube from a larger rectangular prism. The original prism has dimensions by by 2, and a by by cube is removed from one corner. The provided solution has four steps, and need to review each one to see it's correct. Let's start with Step 1.
Step 1: The solution calculates the surface area of the original rectangular prism. The formuls used is Aoriginal = 2(w + th + wh), which is correct for a rectangular prism. Plugging in the dimensions = 4, w = 2, and h = 2, the calculation goes like this:
Aoriginal
2(8 + 4) = 20
This seems right, So, Step 1 is \ \boxed [correct]. Step Here, the solution talks about the effect of cutting out the by 1 by 1 cube from one corner. It says that removing the cube takes away three faces hut adds three new faces. Each face is by each has area of 1 square unit.
Wait minute, this accurate? When you does that affect the surface area?
cube from corner, you indeed removing material, but how
Let visualize this. Imagine rectangular prism and cutting small cube from corner. The cube has three outside
faces exposed
prism. When you remove cube, those three faces are no longer part the surface area. However, the removal also creates three new internal faces where the cube was taken out, which are now part of the surface area.
So, the solution says that removing three faces reduces the surface area by 3 square units and adding three faces increases it by square units, leading to no net change. That seerss logical at finst glance.
But is this actually correct? Let me think deeper.
When you remove the cube, you are taking away three ixl faces from the original prism, which indeed reduces the surface area by 3 square units. However, the new faces created are also 1x1, and there are three of them, so that adds 3 square units back. So, the total surface area remains the same.
But I'm not entirely sure. Maybe should consider if the new faces are actually exposed or if they are somehow internal. Wait, no. When you cut out the cube from the corner, the three new faces are now part of the exterior of the new solid, so they should be included in the surface area.
Therefore, the net change is indeed zero, and the surface area remains 40.
Hmm, but I recall that sometimes when you make cut, the surface area change in unexpected ways. Maybe should double-check this.
Let consider simpler example. Suppose I have cube and cut smaller cube from one corner. How cloes the surface change?
Original cube: surface After cutting out a smaller cube from one corner, remove three faces of the smaller cube and expose three new faces. So, the change in surface area -3s + 3x2 0, similar to what the solution says.
is 6s", where the side length.
Alternatively, perhaps can think about the new solid's dimensions.
But I think the initial approach is correct. Therefore, Step 2 is bboxed (correct).
Step 3: This step ea calculates the net. change in surface area as zero, which follows from Step 2.
Since the net change is zero, the surface area remains the same as the original.
So, Step 3 is \ cooreeett
Step 4: Concludes that the surface area of the new solid is 40, the same the original. Therefore, Step 4 is bbxxee (correct).
Wait, but recall that sometimes when you make cut, especially certain ways, the surface area change. But in this case, seems hold up.
35

---

Published in Transactions on Machine Learning Research (03/2026)
Alternatively, maybe should consider the cube cut. from a corner where only two faces are on the exterior. Wait, no. In rectangular prism, cach corner where three faces mect, so cutting out a cube from corner would remove three faces and expose three new ones. But perhaps I should consider if the cube is cut from a corner where one of the original faces is larger than Ixl. Wait, the original prism is 4x2x2, and the cube being removed is 1x1x1, so the faces being removed are indeed 1xl. So, the calculation soems accurate.
Alternatively, maybe I can think about it in terms of the total surface area. Alternatively, perhaps can think about the surface area of the new solid directly.
Let try that.
The solid has the original surface minus the three 1xl faces that removed, plus the three new 1xl faces that
exposed.
So, 40 3= Alternatively, perhaps can think about the surface the of all exterior faces. But I think the initial approach Therefore, the solution is correct.
But the instruction says to review and critique each step, and determine the whole solution is correct. So, this case, all steps seem correct, leading to conclusion that the surface area remains 40. </think>
Figure 31: Example output by THNKPRM-14B, where it verifies and labels every step the provided prefix via long verification CoT. The problem and solution are in Figure 30
36

---

Published in Transactions on Machine Learning Research (03/2026)
Verification long CoT solution
Problem:
Convert the point (0,3) in roctangular coordinates to polar coordinates. Enter your answer in the form (r, 0), where > and 0 2 0 < 2n.
Solution:
Okay, so need to convert the rectangular coordinates (0,3) to polar coordinates. Let me recall how to do this. From what I remember, polar coordinates given (r, 0), where is the distance from the origin to the point, and 0 is the angle made with the positive z-axis. The formulas to convert between rectangular (x., and polar (r, coordinates
cos 0,
sin
To find r,
V x² y²
To find 0, we use:
arclan y
But I need to he careful with the quadrant since arclan may be undefined or ambiguous when = 0.
Given 0 and = compute:
V 02 + v
So 3. For 0 since x = O, the expression arctan(3//3 (1//2)) undefined due to division by zero. However, because the point lies on the positive y-axis, = 3. This is the standard angle for points directly above the origin.
Therefore, the polar coordinates are \boxed[(3, (frac((pi)(22))
Wait, let me check there's another way to get this. Since the point is (0, 3), which is on the positive y-axis, the angle is indeed 2" Could 4 No. is negative g-axis, which the opposite direction. Since the point on the positive y-axis, f 4-
Let me recall again that when converting from rectangular to polar, 0 calculated as arctan(y/x)), but when = 0, is 3n depending on the sign of y. Here > 0, so the correct angle is
either or 2
ThinkPRM-115BB
<think>
Step 1: The student correctly recalls the formulas for converting from rectangular to polar coordinates:
cos O,
sin
and
22 v3 0 arctan
This step (oorect) Step 2: The student computes:
0 32 =
This is (ooreett)
Step The student identifies that ar/tan(3/0) undefined but correctly infers that the point lies the positive y-axis, hence
3. This reasoning also (ooree(c)
</think
Figure 32: Example of verification of long CoT solution from Owen3-1.7B-thinkkee using THINKPRM-L.BB. THINKPRM extracts and verifies individual steps embedded in the long CoT.
37


Reviews:
[{"review_id": "O1wGqlTJMX", "reviewer": "Reviewer_2C82", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Paper summary**\n\n- The paper discusses that there are three ways to build process reward models (step-by-step verifiers):\n    - discriminative PRMs (expensive)\n    - generative PRMs (short CoT only)\n    - LLM-as-a-judge (prone to prompt format and invalid outputs)\n- They develop a generative PRM that can “think” for longer (ThinkPRM). They show the advantage of this model over LLM as a judge and discriminative PRMs.\n- They train and evaluate the model on math reasoning tasks:\n    - training: 1K samples of the PRM800K dataset.\n    - evaluation: MATH500 and AIME 2024 datasets.\n\n**Strengths**\n\n- Their training method is much more efficient compared to discriminative PRMs. they use only 1% of the data.\n- They perform extensive experiments and ablations to highlight the strength of their model and the validity of their design choices:\n    - ThinkPRM outperforms LLM-as-a-judge and discriminative PRMs.\n    - ThinkPRM outperforms other generative off-the-shelf PRMs.\n    - ThinkPRM generalizes to out-of-domain data (physics-QA and code).\n    - Parallel (averaging the scores of multiple samples) and sequential scaling (forcing the model to think for longer) of ThinkPRM’s compute improves its performance, with a slight preference of parallel scaling.\n    - They can train their model with automatic labels (obtained via Monte Carlo rollouts) instead of gold human labels of step verifications.\n    - ThinkPRM generalises to extended reasoning solutions that involve backtracking and self correction, even though it was not trained on such problems. It also extends to more “difficult” math and physics problems.\n\n**Weaknesses**\n\n- In section 3., have you tried to investigate the reason for the increase in invalid labels for QwQ?\n- In 5.2 you show that long CoT reasoning is better than short CoT. I am thinking that this is definitely task and step dependent. I assume that some tasks and steps of the solution require less thinking that others. Have you explored training thinkPRM with a mixture of long and short CoTs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The main claims in the paper are supported by exhaustive experiments.", "audience_yn": "Yes", "audience_explanation": "People working on developing process reward models will find this work very relevant.", "requested_changes": "### \n\n- **Critical**\n    - Please respond to my questions in the weaknesses section above.\n    - Section 4.1, Tasks and Models, you mention: “We show results on three math reasoning tasks” while it is two tasks.\n\n- **recommended to strengthen the work**\n    - The paper is difficult to follow, as it requires frequent back-and-forth between the text and the figures/tables. I recommend placing explanatory text closer to the figures it refers to and keeping the main figures in the main body rather than relegating most of them to the appendices.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zP0Txu2UyJ", "reviewer": "Reviewer_qWhF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces ThinkPRM, a generative process reward model that verifies step-by-step reasoning by producing its own verification chain-of-thought and step-level judgments. It trains this verifier using a small, high-quality dataset of synthetic verification traces filtered by agreement with a limited number of gold step labels, and shows improvements both as a standalone verifier and as a verifier for best-of-N / search, with promising cross-domain transfer.\n\nStrengths that stood out include (i) a clear and practically attractive data-efficiency story (orders-of-magnitude fewer process labels than discriminative baselines), (ii) a concrete demonstration that longer verification “thinking” can be more compute-scalable than judge prompting while also reducing invalid/looping behavior after finetuning, and (iii) broad evaluation across ranking/search settings and domain shifts.   \n\nMain weaknesses/risks are that the proposed pipeline still relies on access to step-level labels (even if only ~8K) for filtering and quality control, the contribution may depend substantially on the teacher model used to generate candidate verification chains (raising questions about portability to weaker teachers or non-math domains), and several reported evaluations appear to use relatively small subsets (e.g., 100 problems from MATH-500; 86 GPQA-physics; 200 LiveCodeBench subset), which makes it harder to judge statistical robustness without confidence intervals or repeated runs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper’s central claims are backed by a coherent experimental narrative: it first diagnoses failure modes of LLM-as-a-judge verification (prompt sensitivity, invalid/unparsable labels, overthinking/looping with very long CoTs), then shows that finetuning on filtered synthetic verification CoTs reduces these issues and improves ProcessBench F1 across multiple base reasoning models.  \n\nThe downstream claims about test-time scaling are also supported by multi-setting comparisons: best-of-N on MATH-500/AIME’24, verifier-guided beam search on MATH-500, compute scaling via parallel aggregation (@K) and sequential “verify again” prompting, and out-of-domain tests on GPQA-physics and LiveCodeBench, where ThinkPRM is reported to surpass discriminative verifiers trained on substantially more process labels.  \n\nFinally, the analysis section strengthens credibility by probing mechanisms (long-vs-short verification CoTs) and by ablations that justify the process-based filtering choice over outcome-based filtering, plus a discussion of limitations (calibration extremes from token-prob scoring; step label interference).  \n\nThat said, a few parts would benefit from additional clarification (e.g., contamination/leakage controls, statistical uncertainty, and how sensitive results are to the particular teacher used for chain generation). These do not undermine the main evidence, but they matter for how broadly one should trust the reported gains.", "audience_yn": "Yes", "audience_explanation": "Test-time scaling for reasoning is an active direction, and process verifiers are a central bottleneck due to supervision cost. This work directly targets that bottleneck by proposing a verifier training recipe that uses *very little* process supervision while enabling both parallel and sequential scaling of verification compute, i.e., an angle that is likely broadly interesting to readers working on reasoning, reward modeling, and inference-time search/reranking.", "requested_changes": "1. Clarify data leakage / contamination controls. Since training uses PRM800K-derived problems/solutions and evaluation includes MATH-500/AIME/ProcessBench-style problems, the paper should explicitly state what overlap checks were performed (problem text overlap, source dataset overlap, AIME-year overlap, etc.) and how strict they are. Even a small amount of leakage could inflate best-of-N/search results.\n\n2. Add uncertainty estimates (or repeated runs) for key downstream metrics, especially where evaluation sets are relatively small (e.g., 100 MATH-500 problems, 86 GPQA-physics, 200 LiveCodeBench subset). Reporting confidence intervals or bootstrap estimates for the main scaling curves would materially strengthen the empirical claims. \n\n3. Since verification CoTs are sampled from QwQ-32B-Preview and filtered, it would be valuable to show how performance changes if the teacher is weaker/different, or if chains are generated by the same base model (self-generation) vs. a stronger model.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nLMZSOOk1B", "reviewer": "Reviewer_nG2L", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes ThinkPRM, a generative Process Reward Model (PRM) that verifies step-by-step reasoning solutions by generating a verification Chain-of-Thought (CoT) and extracting a reward signal from that structured verification. The authors introduce a data-efficient training pipeline where a strong reasoning model (QwQ-32B-Preview) generates synthetic verification traces, which are then filtered using ground-truth process labels from PRM800K. The resulting filtered traces are used to fine-tune smaller reasoning models (e.g., R1-Distill-Qwen 1.5B/14B) into reliable verbalized verifiers.\n\nKey contributions are: \n\n1. Data efficiency: ThinkPRM can outperform discriminative PRMs (DiscPRM) trained on the full PRM800K supervision (712K process labels) while using only around 1% of the process supervision . \n2. Strong in-domain performance: ThinkPRM outperforms LLM-as-a-judge and DiscPRM on ProcessBench, MATH-500, and AIME ’24 under best-of-N selection and reward-guided / verifier-guided search.\n3. Out-of-domain generalization: ThinkPRM improves over baselines on LiveCodeBench (code) and GPQA-Diamond (physics subset), suggesting robustness beyond math.\n4. Test-time scaling: ThinkPRM supports both parallel scaling (averaging scores from @K independent verification CoTs) and sequential scaling (self-reverification via prompts like “Let’s verify again”). Experiments indicate verification improves with more verifier compute, and finetuning reduces invalid/looping behaviors commonly observed in LLM-as-a-judge.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper’s main claims are supported by evidence that is generally clear, comparative, and aligned with how PRMs are used in practice (selection, search, and test-time scaling). Specifically:\n\n1. Data efficiency is directly tested. ThinkPRM is trained on ~1K filtered synthetic verification traces (≈8K process labels) and compared against discriminative PRMs trained with full PRM800K supervision (≈712K labels), demonstrating strong performance despite using ~1% of the step-level supervision.\n\n2. In-domain performance is evaluated in realistic regimes. On MATH-500 and AIME ’24, the paper reports improvements under best-of-N selection and reward-guided / verifier-guided search, with comparisons to LLM-as-a-judge, DiscPRM, and strong open-source PRMs/verifiers (e.g., MathShepherd and other public PRM baselines).\n\n3. Key design choices are supported by ablations. The paper contrasts process-based filtering vs outcome-based filtering and analyzes the effect of verification CoT length, showing that compressed/short CoTs perform substantially worse than long verification CoTs, supporting the necessity of the proposed components.\n\n4. Out-of-domain generalization is demonstrated. Results on GPQA-Physics and LiveCodeBench (Figure 10) show ThinkPRM outperforming discriminative verifiers under domain shift, supporting the claim that the approach generalizes beyond the training domain.", "audience_yn": "Yes", "audience_explanation": "The topic of \"System 2\" scaling (test-time compute) and Process Reward Models (PRMs) is currently of immense interest to the machine learning community, particularly given the recent focus on reasoning models.\n1. Practical Value: The high cost of collecting human process supervision (step-by-step labels) is a major bottleneck for practitioners. This paper offers a viable recipe for training high-quality verifiers with minimal labeled data (1% of standard datasets), which is highly relevant to researchers working on data-efficient alignment.\n\n2. Methodological Shift: The empirical validation that generative verifiers (producing CoT critiques) generalize better than discriminative reward models (scalars) addresses a key open question in the field regarding the optimal architecture for reward modeling.\n\n3. Broader Applications: Researchers working on reasoning (math/code) and search algorithms (Best-of-N, Beam Search) will find the scaling laws of verification compute presented here directly applicable to their work.", "requested_changes": "I request the following adjustments to strengthen the manuscript, clarify limitations, and improve reproducibility. I label items as Critical (required for acceptance) vs Strengthening (not strictly required, but would materially improve the work).\n \n ### 1. Inference latency and compute normalization (Critical)\nThe paper convincingly shows that long-CoT verification can reduce invalid terminations and overthinking relative to LLM-as-a-judge (Figure 4; Figure 5), but the main text does not fully quantify the inference-time cost of generative verification relative to discriminative scoring. Figure 4 indicates judge-style verifications can run into multi-kilotoken regimes (spiking around 7K–8K tokens for inaccurate chains), and the proposed approach explicitly generates a full verification CoT per candidate, which may dominate cost in best-of-N and search settings. \nThe paper includes a compute-matched analysis (Figure 11), but it is framed against majority/self-consistency rather than comprehensively against DiscPRM/LLM-as-judge across all key settings. \n\n*Requests*: \n- Report end-to-end inference cost per problem for ThinkPRM vs DiscPRM vs LLM-as-judge in the main experimental regimes (ProcessBench verification, best-of-N, guided beam search): at minimum total verifier tokens and wall-clock latency on a specified GPU; ideally also an “estimated FLOPs” proxy consistent with the paper’s Figure 11 methodology\n- Provide compute-matched comparisons including DiscPRM and LLM-as-judge (not only majority/self-consistency), under a fixed inference budget: e.g., allow DiscPRM to score more candidates / deeper beams so total compute matches one ThinkPRM verification pass, then compare final accuracy/F1.\n- Clarify evaluation handling of non-terminating / invalid outputs (noted in §3.1 and Figure 5) when doing compute matching, since truncation/timeout policies can materially change effective compute and measured performance. \n \n\n ### 2.  Statistical significance and small evaluation subsets (Strengthening; especially for OOD claims)\nSeveral headline results use limited-size subsets: GPQA-Physics is 86 questions (§4.1), sequential scaling experiments are run on 200 OmniMath problems (main text), and MATH experiments are run on a 100-problem subset (§E.5). \nGiven these sample sizes, improvements like 4.5%–8% in Figure 10 could plausibly be within sampling variance without uncertainty reporting. \n\n*Requests*: \n- Report confidence intervals / standard errors (bootstrap over problems is fine) for key accuracy/F1 curves, particularly OOD results (Figure 10) and AIME ’24 best-of-N curves (Figure 7). \n- For sampling-based regimes (best-of-N, @K parallel scaling), report variability across multiple random seeds or bootstrapped resampling.\n\n\n ### 3) Data contamination / overlap checks (Strengthening; could become Critical if unaddressed)\nThe method constructs synthetic verification chains from PRM800K problems/solutions and filters them based on agreement with PRM800K gold step labels (§3.2; Figure 3). Given evaluations on standard benchmarks (e.g., MATH/AIME/GPQA subsets referenced throughout §4.1), contamination and overlap are plausible, especially when using strong pretrained reasoning models as generators/verifiers.  \n\n*Requests*: \n- Add an explicit contamination/overlap analysis describing what checks were performed (deduplication, string similarity, provenance filtering) between PRM800K-derived training material, generated verification traces, and evaluation sets (ProcessBench subsets, MATH/AIME splits used, GPQA-Physics, LiveCodeBench). \n \n- If rigorous checks are not feasible, clearly state what is and is not ruled out, and how that affects interpretation of OOD/generalization claims.\n\n ### 4) Clarify dependence on “gold” process labels and supervision framing (Strengthening)\nThe synthetic-data filtering explicitly relies on matching generated step judgments to PRM800K gold step labels (§3.2; Figure 3).This is a valid and pragmatic design, but the paper would benefit from clearer phrasing in the main method section that this is supervision-efficient rather than unsupervised, and from a clearer statement of what supervision is essential vs optional.\n\n*Requests*: \n- In §3.2, explicitly state the required supervision (gold step labels from PRM800K used for process-based filtering) and characterize ThinkPRM as supervision-efficient. \n- If the paper supports a “silver labels / automatic labels” variant elsewhere, summarize its performance and label quality sensitivity in the main text (not only appendices), so readers understand how dependent the approach is on PRM800K-style step supervision.\n\n ### 5. Scoring extraction, parsing robustness, and invalid-output policy (Critical for reproducibility)\nA central claim is that finetuning reduces invalid judgments and improves termination/format compliance (Figure 5), and the pipeline depends on extractable boxed decisions and yes/no outputs (§3.2).  Because formatting and parsing choices can materially change results, the paper should make the evaluation pipeline fully explicit.\n\n*Requests*: \n- Specify exactly how scalar scores are extracted from generated outputs (tokenization choices for yes/no or correct/incorrect, any normalization, how multiple steps are aggregated into a solution-level score).\n- Specify policies for invalid/unparsable outputs (missing boxes, truncated generations, non-terminations): how often they occur per method and how they are scored in final metrics (counted incorrect vs excluded). \n- Make these details easily discoverable in the main paper (with appendix pointers as needed).\n\n ### 6) Calibration analysis for generative PRM scores (Strengthening)\nThe paper itself flags overconfidence as a limitation of generative PRMs (§5.8), noting score clustering near extremes. \nGiven that best-of-N selection and search heavily rely on score calibration, quantifying and (even minimally) mitigating miscalibration would strengthen claims.\n\n*Requests*: \n- Add a small calibration study (e.g., reliability curves / ECE) on at least one in-domain and one OOD setting. \n- Include a lightweight remedy baseline (temperature scaling or isotonic regression on a held-out set) and report whether it changes selection/search performance materially.\n\n\n ### 7. Sequential scaling details and sensitivity (Strengthening)\nSequential scaling is an interesting contribution and is implemented via multi-round trigger phrases (§E.4), and it is discussed as compute-intensive with experiments on 200 OmniMath problems. Because triggering can be brittle, the paper would benefit from a bit more sensitivity reporting.\n\n*Requests*: \n- Provide a short sensitivity analysis over trigger phrases or number of rounds (or at least report whether results are stable across a small set of alternative triggers). \n- Clarify the token budgets used in sequential scaling and how truncation is handled.\n\n ### 8) Presentation and copy-editing (Strengthening)\nThere are multiple typos/grammar issues that interrupt readability, e.g., “syntehtic” in Figure 4 caption, and awkward phrasing in Appendix C.1 training details.  \n\n*Requests*: \n- Thorough proofreading and minor layout polish (especially in captions/appendices) to improve clarity and reduce ambiguity.", "broader_impact_concerns": "No major concerns. The paper focuses on improving the accuracy and reliability of mathematical reasoning in LLMs. However, the authors should briefly acknowledge the risk of \"false confidence\"where a generative verifier produces a convincing-sounding but incorrect critique potentially misleading human users relying on the system for explanations.", "additional_comments": "The paper is a bit hard to follow in second half, and the presentation could be improved by fixing the disjointed figure placement and typos mentioned above. The distinction between parallel scaling (averaging over K samples) and sequential scaling (self-correction) is a valuable contribution to the literature on test-time compute. The finding that fine-tuning is necessary to stabilize \"LLM-as-a-Judge\" behavior (preventing loops) is a practical insight for the community.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)