VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
Authors: Thomas Zeng
Shuibai Zhang
Shutong Wu
Christian Classen
Daewon Chae
Ethan Ewer
Minjae Lee
Heeju Kim
Wonjun Kang
Jackson Kunde
Ying Fan
Jungtaek Kim
Hyung Il Koo
Kannan Ramchandran
Dimitris Papailiopoulos
Kangwook Lee
Venue: ICML 2025 oral (2025)
Keywords: Proccess Reward Model
Primary area: deep_learning->large_language_models
TL;DR: Math PRMs show limited generalizability beyond math. The fix: further train on a synthetically generated multi-domain CoT dataset.

Abstract: Process Reward Models (PRMs) have proven effective at enhancing mathematical reasoning for Large Language Models (LLMs) by leveraging increased inference-time computation. However, they are predominantly trained on mathematical data and their generalizability to non-mathematical domains has not been rigorously studied. In response, this work first shows that current PRMs have poor performance in other domains. To address this limitation, we introduce ***VersaPRM***, a multi-domain PRM trained on synthetic reasoning data generated using our novel data generation and annotation method. VersaPRM achieves consistent performance gains across diverse domains. For instance, in the MMLU-Pro category of Law, VersaPRM via weighted majority voting, achieves a 7.9% performance gain over the majority voting baseline–surpassing Qwen2.5-Math-PRM's gain of 1.3%. We further contribute to the community by open-sourcing all data, code and models for VersaPRM.

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
Thomas Zeng' Shuibai Zhang' Shutong Wu' Christian Classen' Daewon Chae? Ethan Ewer' Minjae Lee
Heeju Kim 3
Wonjun Kang²
Jackson Kunde' Ying Fan Jungtaek Kim Hyung II Koo
Kannan Ramchandran" Dimitris Papailiopoulos Kangwook Lee'
Abstract
to rank multiple generated solutions and select the best an- swer (Lightman et al., 2024). Reward models evaluate the quality of solutions, helping guide L.LMs toward better out- puts. In particular, Outcome Reward Models (ORMs) are used to provide supervision based solely on the correctness of the final outcome. However. ORMs fail address er- rors in intermediate steps, limiting their effectiveness for complex, multi-step reasoning tasks (Luo et al., 2024; Light- man et al., 2024; Sun et al., 2024). Because ORMs suffer from this limitation, Process Reward Models (PRMs) have been proposed to offer f fine-nga step-by-step feedback feedbback on the correctness of each reasoning step (Lightman et al., 2024; Uesato al., 2022). PRMs have proven highly ef- fective during inference, improving the reranking of gen- erated solutions and guiding LLMs through search-based algorithms (Wan et al., 2024; Wang al., 2024a).
Process Reward Models (PRMs) have proven effective at enhancing mathematical reasoning for Large Language Models (LLMs) by leverag- ing increased inference- computation. How- ever, they predominantly trained math- ematical data and their generalizability to non- mathematical domains has not been rigorously studied. In response, this work first shows that current PRMs have poor performance in other domains. To address this limitation, we intro- duce VersaPRM. multi-domain PRM trained synthetic reasoning data generated using novel data generation and annotation method. Ver- saPRM achieves consistent performance gains across diverse domains. For instance, in the MMLU-Pro category of Law, VersaPRM via weighted majority voting, achieves a 7.9% per- formance gain over the majority voting baseline- surpassing Owen2.5-Math-PRM. gain of 1.3%. We further contribute the community by open- sourcing all data, code and models for VersaPRM.
Several studies have shown that PRMs trained on exten- sive process supervision significantly outperform ORMs in mathematical reasoning tasks, with notable improvements reported on datasets such as MATH500 and GSM800K (Luo et al., 2024; Lightman al., 2024; Uesato et al., 2022). While substantial investigation has been made in creat- ing training data (Lightman et al., 2024; Wang et al., 2024b), training PRMs (Xiong et al., 2024), and evalu- ation (Zheng et al., 2024) with respect to mathematical reasoning, the application of PRMs to non-mathematical domains-such as Biology, Chemistry, and Law-remains underexplored. To investigate the capability of math PRMs in non-mathematical domains, we test open-source math PRMs such as Math-Shepherd (Wang et al., 2024b) and Qwen-2.5-Matth Mathh-PPMM (Zheng et al., 2024). Not surprisingly, these PRMs demonstrate poor performance, indicating their limited domain generalizability. They exhibit only marginal improvements over the baseline Law, Philosophy, and Biology illustrated in Figure 1.
1. Introduction
Large Language Models (LLMs) have demonstrated signifi- cant potential in tackling complex reasoning tasks. Specif- ically, they can employ step-by-step Chain-of-Thoupee (CoT) approach to generate more accurate and reliable so- lutions (Wei et al., 2022b; Kojima et al., 2022; Yao et al., 2023; Madaan et al., 2023). Moreover, by using additional test-time computation, the reasoning performance of LLMs can be further enhanced (Snell et al., 2024; Yao et al., 2024).
An important and addete- test-time computation method is using external verifiers, such reward models
To address this limitation, propose fine-tuning PRMs synthetically generated multi- CoT dataset, to significantly enhance reasoning capabilities beyond math- ematics. We call this resulting multi-domain PRM Ver- saPRM, short versatile PRM. Notably, by sampling ques- tions from the MMLU-Pro dataset (Wang et al., 2024c), we generate CoTs to produce step-by-step reasoning using an
University of Wisconsin-Madison "Korea University 2 Furiossal Universityy of Califomia, Berkeley. Correspondence to: Kangwook Lee <kangwook. www.:/........i.................
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

VersaPRMr Multi-Domain Process Reward Model via Synthetic Reasoning Data
(Current)
Tost-Timee compute with Moth PRM only offective for Math
(Asouracy wi PRN) -Acourrcyy wio PRM)
Labebet DaT
Buse LLM
Muth PRM
Teat-Time Compute Agurtia
Step 20113
3721
Labeled
Dataset
Pine-tue
(Ours) VersaPRM
LUN
Mulh
Low
(louueecy oi PRN) -Aoourecy wu PRM)
The 13th
Caf Generatin Lubb-Labwling
lat bases are DNAA
Phllccephty
Stop Adkiii V
VirsAPRM
Test-Tims Compute
Algeriiins
Who
What DNA?
Stop Jinai
Looking
enciter ona.
Multi-Domain Questions
Multi-Domaln
Test Time compute with VersaPRM bay
CoT Dataset
Figure 1. Existing open-source PRMs trained on math datasets achieve strong math performance and can outperform a majority voting baseline when used via weighted majority voting. However, these PRMs fail to generalize to other domains (e.g.. Law, Philosophy. and Biology), performing no better than baseline. We propose multi-domain PRM, VersaPRM, obtained by further fine-tuning a math PRM on synthetically generated multi-domain dataset. The resulting PRM effectively generalizes beyond math, improving test-time reasoning across multiple domains.
LLM-based generator, i.e., Llama-3. 1-81a-3.1-8BBInsttree (Dubey et al., 2024), and then auto-label them using an LLM-based (Dubey et al., 2024).
2. Related Work
Mhain-of-Thoughe and Process Reward Models. PRMs have proven more effective than ORMs in enhancing LLM reasoning via COT, particularly for mathematical tasks (Luo et al., 2024; Lightman et al., 2024; Sun et al., 2024). Un- like ORMs, which focus on final outcomes, PRMs provide step-by-step feedback. improving error detection in inter- mediate steps and multi-step tasks (Luo et al., 2024; Light- man al., 2024; Uesato et al., 2022; Wang et al., 2024b). Techniques like OmegaPRM (Luo et al., 2024) and Math- Shepherd (Wang et al., 2024b) reduce reliance on costly human annotations, while RLHflow (Xiong et al., 2024), OpenR (Wang et al., 2024a) and ProcessBench (Zheng et al., 2024) advance PRM evaluation and training. However, the expertise of existing PRMs is mainly limited mathemati- cal reasoning. This mathematical focus aligns with findings from Sprague et al. (2025), who show that CoT prompting yields its largest gains on mathematical problems, with only modest improvements in other domains. contrast, our work demonstrates that when CoT prompting is paired with VersaPRM test time, consistent improvements can still be achieved across several non-mathematical domains.
labeler, i.e., I.lama-3 VersaPRM, which is trained on the resulting synthetic multi- domain reasoning data, shows strong performance across diverse domains. We validate its superior performance using various empirical analyses on VersaPRM against existing open-source PRMs.
Our contributions are summarized as follows:
1. We identify the limited domain generalizability of open- source math PRMs in Section 4.
2. We propose novel data generation and annotation pipeline across multiple domains in Section 5.
3. We introduce a large-scale, high-quality multi-domain process supervision dataset, dubbed MMLU-Pro-CoT- Truin (Labeled).
4. We train a well gell-generalized PRM that outperforms exist- ing baselines, demonstrating strong generalization across diverse domains in Section 6.
Test-Time Inference Algorithms. Test-time inference algorithms enhance LLM reasoning by adding compu- tation during inference, offering a distinct axis of seal- ing compared to the conventional trade-off between train- ing larger models versus smaller, specialized architectures
5. We open-source the implementation of VersaPRM with training details, our multi-domain reasoning data, and its model checkpoint; available at ht https:/ ht/////tttl com/Uke Mta/uuuMMada son- Laa//aabb/eereeeeee
2

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
(Hoffmann al., 2022). AlphaCode (Li al., 2022) shows how test-time computing boosts competitive pro- gramming performance, while Snell et al. (2024) argue that scaling inference-ting compute, rather than model param- eters, yields better results by adapting compute allocation to prompt difficulty. Test-time inference includes verifier free methods- Tree of Thought reasoning (Yao al., 2024), self velffieiiiiation (Weng et al., 2023) and stepwise self- evaluation (Xie et al., 2024)-a0 well methods such as verifier reranking (Cobbe et al., 2021; Wan et al., 2024), tool feedback (Gou et al., 2024), reward-grn reward-guided reasoning (Yang et al., 2024b), and multi-agent debate (Du et al., 2024). Our work uses PRMs for reranking solutions and guiding reasoning within the external verifier paradigm.
S = (81, $2,...; ;.k.....))))))) 8k) as a sequence of k reasoning steps, where 8, is the i-th step in the Co'T' for each [k:] A PRM can then be formally characterized
function that maps each CoT S to an associated k-dimensional vector of rewards: (RM(S) E [0, 1] The i-th coordinate of the output score vector, denoted as PRM(S); represents the PRM score for the correctness of the reasoning step si.
iiternal-verffier
3.1. Score Aggregation Methods
Using a PRM, we can obtain scores for each reasoning step. To then scalarize the reward score vector of the whole CoT, we consider the following three aggregation methods.
Min-Aggregaas Mii-Agggregtiin. We use the minimum PRM step score in a CoT as the aggregated score:
Synthetic Data Generation. Obtaining fine-grained step- wise labeling of CoTs via expert annotation costly and time-consuming. Automated annotation methods, such as rollout-based approaches, reduce human effort but quire numerous model inference, which is computationally expensive. Recent effort to mitigate these limitations in- clude multi-rollout generation per reasoning step (Wang et al., 2024b;d) and efficiency improvements via binary search (Luo et al., 2024). Despite these advancements, the reliance on multiple model calls remains a bottleneck. Li al. (2024) introduce novel arithmetical-puzzee and show that fine-tuning large-scat scale synthetic examples yield pre- cise multi-step math reasoning capabilities- though its ap- plicability outside mathematics remains unclear.
Aggr min (S) min PRM(S);
iE|K]
We utilize the PRM score of the last step in a CoT as the aggregated score:
cast-Aggregation.
Aggrial (S) PRM(S))-
AverageAggreegation We employ the average PRM step score of the CoT as the aggregated score:
Ager avg (S)
1
Σ PRM(S);-
(E(A)]
These aggregated scores are particularly useful for solution reranking and are employed in the test-time inference al- gorithms described below (Wang et al., 2024b; Sun et al., 2024; Lightman al., 2024).
This challenge of efficiently generating supervision data mirrors the goals of knowledge distillation, where insights from larger "teacher" model are transferred to a smaller "student" model, often by using the teacher to generate la- beled training data (Hinton et al., 2015; Gu et al., 2024). Indeed, recent studies by Gilardi et al. (2023) and Fonseca & Cohen (2024) demonstrate the potential of LLMs as data labelers. The effectiveness of leveraging a larger model as a direct evaluator or labeler in such a process relies on its inherent zero-shot evaluation capabilities (Wei et al., 2022a). This use of L.LM-geeeatee feedback is conceptually similar to methods like Constitutional AI, where larger models are used to guide smaller ones (Bai et al., 2022). Inspired by these approaches and the prompting techniques of Zheng et al. (2024), our work employs LLMs as labelers for au- tomated process reward annotation, enabling cost-effective synthetic data generation across multiple domains.
3.2. Inference-Tinee Methods
In this section, we introduce three reranking- meth- ods-Majority Voting, Weighted Majority Voting, and Best-of-N allon with two search-based methods-Bii Search and Monte Carlo Tree Search.
Let ug denote the final answer in a CoT S, which in prac- tice can be extracted using a suitable parser. Further let SN = {S1, S2.... Sx denote a set of N Co''s sampled i.i.d. from a generator over a particular question.
Majority Voting (MV). MV (Wang et al., 2023) is ro- bust baseline inference-time method that does not require a PRM. Specifically, we first sample N candidate solutions to a problem from generator. The final answer is then deter- mined by selceting the solution that appears most frequently among these N candidates:
3. Process Reward Models
Similar to the work of Lightman et al. (2024), we define process rewards to represent the correctness of each step, and whether it is logical and follows from previous steps.
MV(Sx)
arg max ag:SESy [E[N]
Σ
1la (as,):
To formally define PRM, we begin by specifying a CoT
3

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
Weighted Majority Voting (WMV). This method, as used by Uesato al. (2022), is similar to MV. We sample N candidate solutions. However, we weight the frequencies of CoTss with identical answers by the aggregation scores. The final answer is the one with highest sum of weights: WMV(Sx) = arg max Sg:SESN
Table I. Results of two open-source math PRMs on different do- mains in MMLU-Pro -oTT--lll (Unlabeled) when using WMV with aingrgggeat on 16 CoTs generated per question using Llama-3. 1-8B--ntuett In parenthesis we report absolute differ- ence between WMV and MV (WMV-MV), While WMV us- ing math PRMs exhibits greater improvement in Math and Math- adjacent domains, there is no significant improvement on MV in other domains.
Σ NE[M]
Las (as;) Aggr(S:).
Best-of-N (BoN). This method also samples N candidate solutions. It then reranks them using the aggregation score from a PRM. The answer of the solution with highest score chosen as final answer:
Category
MV
Math-Shepherd
Qwen-2.5-Math- PRM
All
57.15
57.66 (+0.51)
58,17 (+1.02)
All except math Math
56.61
57.01 (+0.40)
57.32 (+0.71)
62.40
64.13 (+1.73)
67.20 (+4.80)
(SN = arg max Aggr(S).
ag:SESN
Chemistry Physics
58.67
60.13 (+1.46)
60.67 (+2.00)
58.53
61.87 (+3.34)
61.47 (+2.94)
Biology
75.38
75.38 (+0.00)
75.69 (+0.31)
Beam Search. This method (Snell et al., 2024) is initial- ized with fixed number of beams N and width M. The process starts by sampling N initial predictions for the first reasoning step. These are ranked via the PRM's step score, N retaining the top M candidates. For each retained candidate, M proposals for the next step are sampled, yielding N new candidates. This iterates until all beams reach solutions or maximum iteration limit. The final prediction is selected based on the highest aggregated PRM score across steps. See Algorithm I for details.
Psychology
61,60
61.47 (-0.13)
62.27 (+0.67)
Law
35.93
37.24 (+1.31)
36,28 (+0.35)
History
49.20
49.87 (+0.67)
49.40 (+0.20)
Philosophy
44.83
44.70 (-0.13)
45.17 (+0.34)
4.2.
Evaluation Dataset
For our multi-domain evaluation dataset, we curate ques- tions sampled from the MMLU-Pro dataset (Wang et al., 2024c). MMLU-Pro is designed to benchmark the reason- ing abilities of LLMs and consists of college-level multi- ple choice questions in the following 14 domains: Math, Physics, Chemistry, Law, Engineering, Other, Economics, Health, Psychology, Business, Biology, Philosophy, Com- puter Science, and History.
Monte Carlo Tree Search (MCTS). MCTS is a search algorithm used during test-time inference (Hao al., 2023; Wan et al., 2024) that iteratively builds a search tree to find the CoT with the highest aggregated PRM score. A detailed description is presented in Appendix C and the pseudo-code is provided in Algorithm 2.
To craft our evaluation dataset, we randomly sample 150 questions from each domain. Due to duplicate questions, we discard 41 questions-23 from Biology, 10 from Health, 5 from Law, and 1 each from Business, Economics, and Philosophy. For each remaining question, we generate 128 candidate solutions using Llama-3. 1-3.1118---ssteee (Dubey et al., 2024) for MV, WMV, and BoN test-time inference algorithms. Prompt details and generation parameters are provided in Appendix A.2. We refer to this multi-domain evaluation dataset as MMLU-Pro-CoT--B
4. Limitations of Process Reward Models Trained on Math Domain Data
We introduce various math PRMs used for comparison in Section 4.1, present our multi-domain evaluation dataset in Section 4.2, and provide a detailed analysis of the evalua- tion results in Section 4.3.
(Unlabeled).
4.1. Open-Source Math PRMs
4.3. Multi-Domain Performance of Math PRMs
For evaluation, we conduct experiments on a diverse set of models. Our analysis includes four open-source math PRMs: Math-PSA, (Wang et al., 2024a), Sheh-Sh (Wang et al., 2024b), LLHHLLo-Deepseene (Xiong et al., 2024), and 2.wen-2.5-Math-PRA (Zheng et al., 2024).
We conduct comprehensive analyses on a diverse set of models. For clarity, we report results for two representative models here, with additional evaluations available in Ap- pendix E. The first model, Math-Shephen (Wang et al., 2024b), is trained on synthetically generated math data la- beled via a rollout-based method. The second model, Qwen- 2.5-Math-PRR (Zheng et al., 2024), is a best- persforernn open-source PRM, trained on the high-qualin expert labeled PRM800K math dataset (Lightman et al., 2024).
In addition to the open-source models, two math PRMs based on open-source models are specifically trained in this work. They are denoted as LlamaPRM800K and Qwen- PRM800K. More details are given in Appendix D.1
4

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
The PRMs are applied using WMV with min-aggregation. While math PRMs show significant improvements in math- ematical reasoning domains, their effectiveness in broader, non-mathematical areas remains limited. Notably, in the and Math-Shepherd achieve relative gains of +4.80 and +1.73, respectively, outperforming the MV baseline. Similar improvements are observed in Math-adjaces disciplines: Chemistry (+2.00 for Siven-2.5-Math Shepherd), underscoring their utility in tasks requiring math- ematical reasoning.
unparsable answers.
5.2. Auto-Labeling
Math category, Qwen-2.5-Mathh-
To annotate our synthetic CoT data, we adopt an approach inspired by the critic models in the work of Zheng al. (2024). Specifically, we utilize Llama-3.1-70B Inntt-utttucttt as strong LLM to evaluate each CoT using step-by-step reason- ing, locating the earliest erroneous step, if any. To enhance accuracy and consistency, we identified two key additional components.
and Physics (+3.34 for Math-
First, we incorporaee explicit step evaluation definitions, in- spired by Lightman et al. (2024), into the system prompt. Steps are categorized GOOD, OK, or BAD: BAD for incorrect, unverifiable, or irrelevant steps; GOOD for cor- rect, verifiable, and well-aligned steps; OK for intermediate cases. Second, we also provide the ground-truth reference answer for the question whose CoT is being graded in the prompt. The full prompt is detailed in Appendix A.2.
Finding 1: Math PRMs struggle to generalize to broader domains.
However, the benefits diminish sharply in non-mathematical areas. For example, in Philosophy and History, we see gains of only +0.34 and +0.20% respectively for the most performant PRM Qwen-2.5 2.5- -.-.Math-PRMM.
To convert the auto-labeling output to stepwise labels, we apply the following rule: if no steps are detected as incorrect, all steps in the CoT are labeled as 1. If a step is detected as incorrect, all preceding steps are labeled as 1, the incorrect step is labeled as 1, and all subsequent steps are discarded.
The "All except math" aggregate further underscores this disparity, with PRMs achieving a maximum gain of +0.71 2.een-.5-Math-PP compared with the majority voting baseline.
These results highlight critical limitation: math PRMs trained exclusively on mathematical data lack the versa- tility to generalize beyond mathematical reasoning tasks. While they excel in contexts aligned with their training- quantitative reasoning- capacity to evaluate reasoning quality in broader domains remains insufficient.
In total, we sample 5,750 questions from MML.UPro. Among the 84,098 generated CoTs that passed filtering, 36,935 were labeled as having no incorrect steps and 47,163 were labeled as having at least one (see Table 2). The CoTs generation and labeling were done using AWS Bedrock batch inference at a total cost under $100 (USD). This dataset, denoted as MMLU-Pro- Train (Labeled), is the first open-source multi-domain reasoning dataset with step-wise labels.
5. Automatic Generation of Multi-Domain Reasoning Data with Labels
In order to obtain step-wise reasoning data for non-Math domains, we devise a pipeline, as outlined in Figure 2, to generate synthetic reasoning CoTs from existing question- answering data. These CoTis are then given step-wise labels based on reasoning correctness, We detail the synthetic data generation process in Sections 5.1 and 5.2, including methods to create and annotate reasoning steps. We also provide additional analysis on the quality of the generation pipeline in Section 5.3.
To assess the quality of our auto-labeled data, conduct manual evaluation on random sample of 64 questions from the dataset. For each question, we randomly select one CoT classified as entirely correct and two CoT' flagged as containing an incorrect step. We then manually validate whether the auto-labeled judgments align with our own assessments.
For the Co'ss labeled correct by the auto-labeler, we observed an agreement rate of 80% (95% CI: 0.69-0.89) with our manual evaluations. For CoTs labeled as incorrect, the agreement rate was 71 % (95% CI: 0.63-0.79).
5.1. Chain-of-Thought Generation
For the generation of CoTs, we prompt Llama-3.1-8B Instruct to produce step-by-step reasoning for each in- put question. For training, we source questions from the MML.-Prr dataset (Wang et al., 2024c), by randomly sam- pling up to 500 questions per domain, ensuring that it is disjoint to the subset used for evaluation. We then generate 16 CoTs for each sampled question. Post-generation, we fil- ter out CoT's exceeding the 2,048-toker limit or containing
Based on these results, we estimate that approximately 75% of the CoTs in the entire dataset are correctly labeled. This level of accuracy is comparable to that of manually-labeled CoT datasets, such as PRM800K (Lightman et al., 2024), which is estimated to achieve around 80% accuracy.
'See the following Github issue for discussion: ht https:/ g thub. , con/ wwwp///ppannnooo//oo// http://rrn8000/sssss////22/ 1ssucs/12.
5

---

VersaPRM: Process Reward Model via Synthetic Reasoning Data
Instructions the first slep...
Labeled
Step 1: Napoleon in
Step 1: Napolson 1821.
Instructions Think step by stop..
Step 2: World War began in 1914.
Step 2: World War 1 began in 1814.
Answer: No.
Answer: No.
3.1 BB
70B
Question Did Nepoleon fight World War 1?
Napoleon died in 1821. France World War 1812. Answer: Yes.
Unlabeled CoT
Step Napoleon
Labeled
Step 2: France joined World War in 1812.
CoT Generation Stage
Auto-Labeling Stage
Figure 2. A diagram of the synthetic data generation pipeline. In the CoT Generation Stage, each question is used to generate 16 CoT solutions. Then, in the Auto-Labeling Stage, each CoT evaluated to create step-wise labels. a CoT step is labeled as BAD, all subsequent steps will be discarded.
5.3. Auto-Labeling Prompt Analysis
However, incorporating augmentations from either method during PRM training did not lead to significant performance improvements. As result, we defer the specific details and experiments on counterfactual augmentation and rewrite augmentation to Appendices B.1 and B.2, respectively.
To further understand the factors influencing auto-labeling performance, we conduct an evaluation of the auto-labeling using simplified prompt. Specifically, we remove the system prompt defining the types of reasoning steps and exclude the reference ground-truth answer from the prompt. When re-evaluating the auto-labeling quality on random subset of 30 questions from the original sampled 64, we observe a drastic drop in performance, with the agreement rate for CoT's labeled correct by the original auto-labeler decreasing by over 70%, to 7%, while the agreement rate for CoT's labeled incorrect decreased from 71 1% to 62%. Additionally, in Appendix B. 4. we provide an end-to-end experiment that evaluates a PRM trained on data labeled using this ablated prompt. The findings mirror our manual evaluation the PRM trained with the ablated prompt used via BoN achieves less than half of the performance uplift over MV that the full model attains.
6. Multi-Domain Process Reward Model
We present the implementation and evaluation of VersaPRM, structured follows. First, Section 6.1 covers the vari- ous training configurations used. We then evaluate Ver- saPRM via BoN and WMV in Section 6.2, showing im- proved domain generalization compared to math PRMs. In Section 6.3, we additionally discuss results using Beam Search and MCTS. Lastly, we examine VersaPRM's abil- ity to scale test-time compute for larger models such as Deepseck-RI (Guo et al., 2025) in Section 6.4.
6.1. Training of Our Multi-Domain PRM
These results highlight the importance of providing both well-defined prompt with step label definitions and access to the ground-truth answer in achieving high auto-labeling ac- curacy. The ground answer provides essential context on CoT final correctness and enhances the model's ability evaluate reasoning steps effectively.
To train VersaPRM, we employ a classification head atop LLM, optimizing with cross-entropy loss applied to special classification token appended at the end of each CoT step in AMLU-Pro-CoT-TIE (Labeled). Detailed specifics and hyperparameters are provided in Appendix D.2.
We explore several training configurations, including: 1) LoRA (Hu al., 2022) vs. full me-tunnng for efficient train- ing, base LLM vs. math PRM for initializing the PRM, and 3) a Qwen-based PRM vs. Llama-based PRM for train- ing. Comprehensive experimental results for these studies are presented in the next section. Based on those findings, our final, our final multi-domain PRM, named VersaPRM, Appendix D
5.4. Synthetic Data Augmentations
To further augment the dataset, we experiment with two synthetic augmentation methods. The first, which term counterfactual augmentation, involves generating additional examples of incorrect reasoning by prompting an LLM to modify steps in correct CoTs, thereby introducing targeted errors. The second method is a straightforward rewriting approach, where an LLM rewrites a step to preserve its original meaning but alter its style.
is initialized from our LlamaPRM800K for its finill--finetteedd using LoRA on our multi-domain training dataset.
6

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
6.2. Math PRM vs. VersaPRM Reranking Based Inference- Time Methods
Domain diversity of CoI's in train- ing dataset plays an integral role in generalization of PRMs multiple domains.
Finding 3:
We first report results of the reranking methods WMV and BoN on ods, we adopt Min-aggregation, as it outperforms Average and Last in aggregating PRM step scores; see Appendix E.3 for comparison. We also include MV as a baseline.
(Unlabeled). For both meth-
As shown Figure 4, VersaPRM (Math subset) obtains modest lift over SaamaaPR8OK-eevideeee that learn- ing the in-domain question format partially helps. Far more striking is that VersaPRM (random subset) obtains markedly higher WMV accuracy across both math and non-math tasks These findings suggest two key insights. First, our PRM is not simply learning the question format but is acquir- ing knowledge on how to label reasoning across diverse domains. This why training on diverse data enables bet- ter overall performance than training on same sized data in only domain. Second, VersaPRM (random subset) also demonstrates slightly better performance in the math domain, indicating that training on a diverse dataset may facilitate positive transfer from other domains to math.
Comparison with Math Open-Source PRMs. We evalu- ate our multi-domale PRM, VersaPRM, against open-source math PRMs by partitioning UMLU-Pro-CoT--Evall (Unla- beled) into three groups: 1) Math, 2) Math-adjacent, i.e., Chemistry, Computer Science, Engineering, Physics, and 3) non- Hom-Math-adae adjacent domains, As shown in Figure 3, our model consistently outperforms baselines in both WMV and BoN across all domain groups.
Fine-tuning with synthetic multi- domain data enhances the PRM's 's performance in non-math domains.
Finding 2:
Ablation Experiments using Hold-out Domain. In or- der to verify the domain-general reasoning capabilities of VersaPRMM we conduct additional ablation experiments us- ing hold-out domain approach. Specifically, we excluded one domain category (law or biology) from the VersaPRM training set and evaluate the model's performance on the held-out domain. As illustrated in Figure 5, the performance of VersaPRM trained with one domain held out remains comparable to the fully trained model across both evaluated domains. Additionally, they both surpass the performance of the math PRMs. These results underscore that the gener- alization ability of VersaPRM is not merely due to broader coverage in the training data, but rather represents genuine Nomain-general reasoning capabilities. Additional results on other domains are presented in Appendix E.7.
For WMV, we can see the relative performance difference in- crease with domain distance from core mathematics. While performance of math PRMs converges to the majority voting baseline non-mathematical domains, VersaPRM main- tains robust performance in these other domains.
In BoN the superiority of VersaPRM is even more pro- nounced. Unlike math PRMs, which fail to surpass the baseline of MV in Math-adjacent and non-Math-adjacent domains, our model consistently surpasses it across all do- main groups.
See Appendix E.5 for more fine-grained details where we plot WMV and BoN for every domain of MMLU-Pro-Cofs Eval (Unlabeled). The results consistent with Figure 3, and VersaPRM outperforms math PRMs in all domains.
Experiments Using Other Training Configurations. While our final version of VersaPRM is trained from LlamaPRMM80K on our synthetic data using LoRA, we also test the following training configurations on our multi- domain dataset:
Ablation Experiments Using VersaPRM Trained on Math Only Subset vs. Random Subset. We further con- duct an ablation study to evaluate the impact of training data diversity on the performance of our LlamaPRMM80K Math PRM. Specifically, we train one PRM using only the math subset of our multi-domain training data and another using a random subset of the same size. We refer to these two models as VersaPRM (Math subset) and VersaPRM (ran- dom subset), respectively. This experiment tests that the improved performance of our multi-domain PRM is due to the domain-diversite of the CoT data and not merely from learning the in-distribution question and CoT formats of MMLU-Pro questions. If the latter is the case, both PRMs should perform similarly, given that they are exposed to the same amount of questions and CoT examples with the in -n-ditrriuttin format.
We initialize training from -lama-3.1-8B---a and use LoRA fine-tuning with our multi-domain dataset.
VersaPRM (Llama Base):
VersaPRM (Qwen): We initialize training from Qwen- PRM800K PRM, and utilize LoRA fine-tuning with our multi-domain dataset.
VersaPRM (full-tuned): We initialize training from LlamaPRM800K PRM, and do full fine-tuning with our multi-domain dataset.
The results are presented in Figure 6. Comparing Ver-
saPRM (Qwen) and VersaPRM (Llama), we observe that
7

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
Math
Math-Adjacent Domains
Non-Math-Adjacee Domains
Inference Accuracy (%
Inference Accuracy (%)
Inference Accuracy Inference Accuracy (%)
70
60
60
60
50
16
64
128
16
64
128
2
16
32
64
128
of generated CoT (log scale)
of generated solitions (log scale)
generated solutions (log scale)
Inforence Accurney (%6
70-
Inference Accurcc
50
60
60-
50
55
50-
40
50
16
128
16
64
128
16
128
Number of generated Col solutions flog scale)
Number of generated Col solutions (log scalle)
Number of generated CoT solutions scale)
Majurity Voting
Weighted Majerity Voting
BestoffN
Math-PSA
Qwen-2.5-Moth-Pee
VersaPRM (Ouns)
Matt-Shhpberd
RLHFFoo-DDepsock
Figure 3. Comparison of WMV (top) and BoN (bottom) using VersaPRM against open source math PRMs on MMLU-PPN EMLU-Pro-CoT-Ewa. -oa--val (Unlabeled). use min-aggregatton and the CoTs are generated using Llama-3. I-St-Instruct. VersaPRM has consistently better performance than math PRMs, and the differences become larger in domains not adjacent to Math.
Non-Math Dommins
Binlogy
(steveee Aucuoo 199
Namber of generted Oul aduuiion oalg)
Number genorated Cal solution scalc)
Number Coll selusione
Narbaz of gecerned Cal vcnl)
Waghe Hoeag
Maves Wilk
Ver PRM Leark
supetty Volka
Wegginal Suiurty
YouFFRM
Figure Comparison of WMV using LlamaPRM800K, Ver-
Figure 5. Hold-out domain ablation experiment results. WMV performance comparison of VersaPRM trained with each of the indicated domains held out versus the fully trained VersaPRM.
saPRM (Math subset) and VersaPRM (random subset). Ver- saPRM (random subset) achieves better performance than Ver-
saPRM (Math subset) in Math and non-Math.
domain-specific training.
the QwenPRM800K VersaPRM performs worse. This high- lights the importance of base model choices. Although Qwen-2.5-Math-7B. the base model for QwenPRM800K, specialized in mathematical reasoning, its limitations in general-domain knowledge hinder its ability to fully lever- age multi donliin training data.
We note that VersaPRM (full-tuned) has worse performance than VersaPRM. This may be due to suboptimal hyperpa- rameters leading to overfitting during full fine-tuning.
6.3. Math PRM vs. Multi-Domain PRM on Search Based Inference- Methods
Exposure to mathematical data be- forehand can enhance a PRMs' ability to effectively leverage multi-domain CoT fine-tuning.
Finding 4:
We evaluate the performance of math PRMs (using LlamaPRMS00K) and VersaPRM with beam search and MCTS on MMLU-Pro-CoT-Evall (Unlabeled), The results over questions in all domains, presented in Figure 7, show that MCTS outperforms beam search and that they both do better than the MV baseline. Regardless of the search algorithm, consistent with our WMN and BoN results, Ver- saPRM gives boosted performance over the math PRM. also interesting to note that the performance of these search
Next, comparing VersaPRM (Llama Base) with VersaPRM,, we find that the latter achieves superior performance in Math while maintaining comparable performance in non-Math domains. This suggests that prior exposure to mathematical data enhances the model's s ability benefit from further
8

---

Versa Multi-Domain Process Reward Model via Synthetic Reasoning Data
Inference Accuracy (%)
Verng
Checad Mta--31 (0ho
73-
69
bioepee Avause (20
68
67
Number generated CoT solutions
Nanbar of generted Cul audaicen volg)
Namber genorated CeT solation sckl))
(a) Law Domain
Mipes Wike
Unwitttt (the))
Vew Poo- (1lim hevd
Wagged Yoeas
Voraet (iveoo
VersuFRA L1Bi-aac
Inference Accurccy
78
Figure Comparison of MVW using VersaPRM against other multi-domain PRMs trained using different configurations. Ver- saPRM has better WMV performance than all other models in both Math and non oatMatth domains.
13 14 15
Number of generated CoT solutions
Mijucity Varing
Inferenee Accuracy
70
Toan Seaihh VeuuRRA (thnd)
MCSS with Nommttat
(b) Philosophy Domain
Boon with Hecses
MCTS beea
60
Figure 8. Comparison of WMV using VersaPRM against Qwen- 2.5-Math-PRM, and Majority Voting on DeepSeek-RI generated CoTs for Law and Philosophy subsets. VersaPRM consistently outperforms the math PRM and MV baseline in both subsets.
50
8
16
32
128
Number of generated CoT solutions (log scule)
inference.
Figure 7. Comparison of VersaPRM and LlamaPRM800K with beam search and MCTS. The x-axis compares MCTS and Beam Search in terms of computational cost for an equivalent num- her of generated CoT solutions. Overall in the diverse domains from MMI.U--Pr-CCoT-ER (Unlabeled), VersaPRM achieves bet- ter performance.
7. Discussion and Future Directions
We proposed VersaPRM trained using synthetic reasoning data address the limitations of existing math PRMs. By leveraging cost-efficient synthetic data generation pipeline, enabled production of high-quality step-wise reasoning data and demonstrate that PRMs can effectively scale rea- soning ability at inference time in diverse domains.
algorithms saturate much slower than WMN and BoN. De- tails by category results are presented in Appendix E.6.
6.4. Does PRM with Test-Time Compute help Reasoning Models?
Future Work. Several directions remain for advancing multi-domain PRMs. Can VersaPRM be effectively used reward model for reinforcement learning (RL. train- ing? Can VersaPRM also improve RL training in domains beyond mathematics? Could more sophisticated counter- factual augmentation enhance PRM effectiveness? From an alignment perspective, given that both the generator and labeler LLMs are Llama models, could VersaPRM be biased towards performing better with Llama-based generator mod- els? Furthermore, evaluating PRMs on more challenging, open-ended problems, and distinct task categories such as puzzles games, would better assess their generalization capabilities. Finally, a more thorough investigation of PRMs with larger models (e.g.. GPT-4 (Achiam et al., 2023) and DeepSeek- -R) could clarify their scalability and potential role in state-of -ate----thhht-- reasoning systems.
Having shown that VersaPRM effectively leverages inference-time compute enhance LLM performance, next examine whether this benefit extends to strong reason- ing models such as DeepSeek-RI (Guo et al., 2025). Given DeepSeek-R11 extensive reasoning-foeur focused training, one might hypothesize that reranking methods (e.g., WMV and BoN) offer minimal additional improvements.
To test this, evaluate VersaPRM using WMV on DeepSeek-RI for the Law and Philosophy subsets, sampling 16 CoT responses per question. As illustrated in Figure 8, VersaPRM achieves a modest yet clear improvement over both the math PRM and MV baselines. These preliminary results counter the hypothesis and indicate that even highly capable reasoning models can benefit from PRM-enhanced

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
Acknowledgements
Gou, Z., Shao, Z.,, Gong, Y., yelong shen, Yang. Y., Duan, N., and Chen, W. CRITIC: Large language models can self-correct with tool-interactive critiquing. In Proceed- ings of the International Conference on Learning Repre- sentations (ICI.R), 2024.
Kangwook Lee is supported by NSF Award DMS-2023239, NSF CAREER Award CCF-2339978, an Amazon Re- search Award, and a grant from FuriosaAI. In addition, Thomas Zeng acknowledges support from NSF Award DMS-2023239 and Daewon Chae is supported by the Hyundai Motor Chung Mong-Koo Koo Foundation.
Gu, Y., Dong, L., Wei, F., and Huang. M. Minillm: Knowl- edge distillation of large language models. In Proceedings of the International Conference on Learning Representa- tions (ICLR), 2024.
Impact Statement
Guo, D., Yang, D., Zhang, H., Song. J., Zhang, R., Xu, R., Zhu, Q., Ma. S., Wang, P., Bi, X., et al. DeepSeek-RI: Incentivizing reasoning capability in LLMs via reinforce- ment learning, arXiv preprint arXiv:2501.
Given the potential for LLMs be used in unethical ways, such as spreading misinformation or manipulating public opinion, VersaPRM could inadvertently contribute to such misuse. To mitigate these risks, it is essential to implement robust safeguards in training and inference.
2025.
Hao, S., Gu, Y., Ma. H., Hong, J., Wang, Z., Wang, D., and Hu, Z. Reasoning with language model is planning with world model. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2023.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L.,, Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the MATH dataset. In Advances in Neural Information Processing Systems (NeurIPS), 2021. Datasets and Benchmarks Track.
Amini, M.-R., Feofanov, V., Pauletto, L.,, Hadjadj. L., De- vijver, E., and Maximov, Y. Self-training: A survey. Neurocomputing, 616:128904, 2025.
Hinton, G., Vinyals, O., and Dean, Distilling the knowledge in neural network. arXiv preprint 1eXiv:5503002311 2015.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:22 2022.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford. E.. de Las Casas, D., Hendricks, L. A., Welbi, J., Clark, A., et al. Training compute- optimal large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint 2x 2021.
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In Proceedings of the International Conference on Learning Representa- tions (ICIR), 2022.
Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., and Mordatch, Improving factuality and reasoning in language models through multiagent debate, 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7B. arXiv preprint 1XXI:23110008255 2023.
Fonseca, M. and Cohen, S. Can large language models follow concept annotation guidelines? case study scientific and financial domains. In Findings of the Asso- ciation for Computational Linguistics: ACI. 2024, 2024.
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwa- sawa, Y. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Gilardi, F., Alizadeh, M., and Kubli, M. Chatgpt outper- forms crowd workers for text-annotation tasks. Pro- ceedings of the National Academy of Sciences, 120(30): e2305016120, 2023.
Li, H. Ma, Y., Zhang, Y., Ye, C., and Chen, J. Explor- ing mathematical extrapolation of large language models with synthetic data. In Findings of the Association for Computational Linguistics ACI., 2024.
10

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Fecles, T. Keeling, J., Gimeno, F., Dal Lago, A., et al. Competition-Ievel code generation with alpha- code. Science, (78 1097, 2022.
Wang, P., Li, L.., Shao, Z., Xu, R., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z. Math-Shepherd: Verify and reinforce LLMs step-by-step without human annotations. In Proceedings of the Annual Meeting of the Association for Computational Linguisties, 2024b.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verity step by step. In The Twelfth International Conference on Learning Representations, 2024.
Wang. X., Wei, J., Schuurmans, D., Le. Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency im- proves chain of thought reasoning in language models. In Proceedings of the International Conference Learning Representations (ICLR), 2023.
Luo, I.., Liu, Y., Liu, R., Phatale, S., Lara, H., Li, Y., Shu, L., Zhu, Y., Meng, L., Sun, J., et al. Improve mathemati- cal reasoning in language models by automated process supervision. arXiv preprint ar Xiv:2406.006592, 2024.
Wang. Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Aruiraj, A., He, X., Jiang, Z., Li, T., Ku, M., Wang, K., Zhuang, A., Fan, R., Yue, X., and Chen, W. MMLU-Prr: A more robust and challenging multi- task language understanding benchmark. In Advances in Neural Information Processing Systems (NeurIPS), 2024c.
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao. L.. Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang. Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., and Clark, P. Self-Refine: Iterative refinement with self felfbeeed In Advances in Neural Information Processing Systems (NeurlPS), 2023.
Wang, Z., Li, Y., Wu, Y., Luo, I.., Hou, L.,, Yu, H., and Shang. J. Multi-step problem solving through a verifier: An empirical analysis on model-induced process supervi- sion. In Findings of the Association for Computational Linguistics: EMNLP, 2024d.
Shen, Y. and Sanghavi, S. Learning with bad training data via iterative trimmed loss minimization. In Proceedings of the International Conference on Learning Representa- tions (ICLR), 2019.
Wei, J., Bosma, M., Zhao, V., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. In Proceedings of the International Conference on Learning Representations (ICLR), 2022a.
Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling IIm test- time compute optimally can be more effective than seal- ing model parameters. arXiv preprint 03X3344408.03314 2024.
Sprague, Z. R., Yin, F., Rodriguez, J. D., Jiang, D., Wad- hwa, M., Singhal, P., Zhao, X., Ye, X., Mahowald, K., and Durrett, G. To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning. In Pro- ceedings of the International Conference on Learning Representations (ICLR), 2025.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thouglle prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2022b. 2022b.
Weng, Y., Zhu, M., Xia, F., Li, B., He, S., Liu, S., Sun, B., Liu, K., and Zhao, J. Large language models better reasoners with self-verification. In Findings of the Association for Computational Linguisties: EMNLP, 2023.
Sun, Z., Yu, L., Shen, Y., Liu, W., Yang. Y., Welleck, S., and Gan, C. Easy-to-hard generalization: Scalable alignment beyond human supervision. arXiv preprint arXiv:2403.09472 2024.
Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving, G., and Higgins, I. Solv- ing math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.1422
Xi, Z., Yang, D., Huang. J., Tang. J., Li, G., Ding, Y., He, W., Hong, B., Do. S., Zhan, W., et al. Enhancing I1m reasoning via critique models with test-time and training- time supervision. arXiv preprint 2rXiv:2411.16577 2024.
2022.
Wan, Z., Feng. X., Wen, M., Wen, Y., Zhang, W., and Wang. J. Alphazero-like tree-search can guide large language model decoding and training, 2024.
Xie, Y., Kawaguchi, K., Zhao, Y., Zhao, X., Kan, M.- Y.. He, J., and Xie. M. Self-evaluation guided beam search for reasoning. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Wang. J., Fang, M., Wan, Z., Wen, M., Zhu, J., Liu, A., Gong, Z., Song, Y., Chen, L., Ni, L. M., et al. OpenR: An open source framework for advanced reasoning with large language models. arXiv preprint arXiv:2410.09671, 2024a.
Xiong, W., Zhang, H., Jiang, N., and Zhang, T. An im- plementation of generative PRM. https: ////thuh.. com/ RLHF ow /RLFF -Rewa Model 2024.

---

VersaPRM: Multi-Domain Process Reward Model via Synthctic Reasoning Data
Xu, W., Deutsch, D., Finkelstein, M., Juraska, J., Zhang. B., Liu, Z., Wang, W. Y., Li, L.,, and Freitag, M. L.MM- Refine: Pinpointing and refining large language models via fine-graine the-graiied actionable feedback. In Findings of the Association for Computational Linguistics ACI, 2024.
Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C.,, Liu, D., Tu, J., Zhou, J., Lin, J., et al. Qwen2.5-math techni- cal report: Toward mathematical expert model via self- improvement. arXiv preprint arXiv:2409. 12122, 2024a.
Yang, D., Zeng, L., Chen, K., and Zhang, Y. Reinfore- ing thinking through reasoning-enhanced reward models. arXiv preprint arXiv:2501.0145) 2024b.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, Narasimhan, K. R., and Cao, Y. ReAct: Synergizing reasoning and acting in language models. In Proceedings of the Interna- tional Conference Learning Representations (ICLR), 2023.
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Zheng, C., Zhang, Z., Zhang, B., Lin, R., Lu, K., Yu, B., Liu, D., Zhou, J., and Lin, J. Processbench: Identifying process errors in mathematical reasoning. arXiv preprint arXiv:2412.06559, 2024.
12

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
A. More Details Synthetic Data Generation Pipeline
A.1. Dataset Composition
The total composition of MMLU-Pro-CoT-Tran (Labeled) is as follows.
Table Composition MOMLLU-Pr--CO-TWAE (Labeled)
Total
Fully Correct
Incorrect
Number of CoTs
84,098
36,935 440,217
47,163
Number of Steps
487,380
47,163
A.2. Data Generation Pipeline Prompts
To generate Mhain-of-thoughe (CoT) reasoning for MMLU-Pro questions, utilize the prompt shown in Figure 12. To ensure the generated CoT adhere to the proper moomattwheee steps are separated by two newline characters and the final step follows the structure "the answer is (X)"-we include five few-shot examples. These examples are derived from the CoTss provided in the validation split of MMLU-Pro, with additional processing to ensure each step is delimited. The code for generating the complete prompt will be open-sourced alongside the rest our code and data.
During generation, we use temperature of 0.8 and set the maximum generation length to 2,048 tokens. During auto-labeling. temperature of 0, and the maximum generation length remains 2,048 tokens.
The following is multiple choice question and its ground truth answer. You are also given students solution (split into step, enclosed with tags and indexed from 0):
[Multiple Choice Question] {question}
[Ground Truth Answer] [answer]
[Student Solution]
<<step_0>>
Student solution step </step_0>
<step_1> Student solution step </step_11
...}
Figure User prompt template for auto- latollab
13

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
You an experienced evaluator specializing in assessing the quality of reasoning steps in problem- solving. Your task is to find the first BAD step in student's solution multiple choice question.
You will judge steps as GOOD, OK or BAD based on the following criteria:
1. GOOD Step
A step is classified as GOOD if it meets all of these criteria:
Correct: Everything stated is accurate and aligns with known principles or the given problem.
Verifiable: The step be verified using common knowledge, simple calculations, quick refer- ence (e.g., recalling basic theorem). If verifying requires extensive effort (e.g., detailed calculations obscure references), mark it BAD instead.
Approppiate: The step fits logically within the context of the preceding steps. If a prior mistake exists, a GOOD step can correct it.
Insightful: The step demonstrates reasonable problem-solving direction. Even if ultimately progress in the wrong direction, it acceptable as long as represents logical approach.
2. OK Step
A step is classified as OK if it is:
Correct and Verifiable: Contains errors and can be verified.
Unnecessary or Redundant: Adds little value, such as restating prior information or providing basic encouragement (e.g., "Good job!").
Partially Progressing: Makes some progress toward the solution lacks decisive or significant advancement.
3. BAD Step
A step is classified as BAD if it:
- Is Incorrect: Contains factual errors, misapplies concepts, derives an incorrect result, or contradicts
the ground truth answer.
Hard to Verify: Requires significant effort to confirm due to poor explanation.
Is Off Off-T: Includes irrelevant or nonsensical information.
Derails: Leads to dead ends, circular reasoning, unreasonable approaches.
#### Task Description
You will be provided with:
1. A Question
2. A Ground Truth Answer
3. Reference explanation of the answer
4. A Student's Step-by-Step Solution, where each step enclosed with tags and indexed from 0
You may use the ground truth answer and reference explanation in classifying the type of each step. A student's final answer is considered correct if it matches the ground truth answer or only differs due to differences in how the answer rounded. Once you identify a BAD step, return the index of the earliest BAD step. Otherwise, return the index of -1 (which denotes all steps are GOOD or OK). Please put your final answer (i.e., index) boxed.
Figure 10. System prompt for auto-labeling
14

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
B. Additional Training Experiments and Ablations
In this section we detail additional training and ablation experiments for VersaPRM. The final evaluation results of these methods are presented in Appendix B.6.
B.1. Counterfactual Augmentation
Error Type
Instructions
Choose a step and add a [Type 1] or [Type 2] error
Error Type
?
Question 1+1+1=?
Augmented CoT
Llama 3.1 70B
Error Step Step 2: 2+1=4
Step 1: 1+1=2 Step 2: 2+1=4
Correct CoT
Step 1+1=2 Step 2: 2+1=3
Counterfactual Augmentation Stage
Figure 11. Diagram of the counterfactual augmentation pipeline
After generating and labeling synthetic reasoning CoTs (as described in Section 5), we attempted to create additional incorrect steps by augmenting the correct reasoning steps. Our pipeline is depicted in Figure 11. We provide the full CoT to Llama-3. 1-70B-Instrucc instructing it to select and rewrite step where it would be appropriate to introduce an error. Additionally, we define list of possible fine-grained error types. To encourage the generation of a variety of different error types, we only include a random selection two these error types in each system prompt, forcing the LLM to choose one. The error types are:
Conflicting Steps: The reasoning step includes statements that contradict previous steps.
Non-sequitur: The reasoning step introduces information that is irrelevant to the question.
Factual: The reasoning step contains incorrect statements of factual information.
False Assumption: The reasoning step makes incorrect assumption about the question.
Contextual:
The reasoning step misinterprets information given within the question/context.
For the prompt format used in counterfactual augmentation, see Figure 13. In total, we generated 73,829 augmented incorrect steps.
The following are multiple choice questions (with answers). Think step by step and then finish your answer with "the answer (X)" where X is the correct letter choice.
Figure 12, Prompt to generate CoTs for MMLU Pro.
15

---

VersaPRM: Multi-Domann Process Reward Model via Synthetic Reasoning Data
You are highly knowledgeable philosopher with expertise across many domains, tasked with analyzing reasoning processes. Your goal is to identify how reasoning process could naturally deviate toward an incorrect conclusion through the introduction of subtle errors.
Here are a of potential error types, all which are equally valid:
[ERROR TYPE I]: [ERROR TYPE 1 DEFINITION] [ERROR TYPE 2J: [ERROR TYPE 2 DEFINITION]
Instructions:
You will provided with:
1. A question.
2. A complete chain of reasoning steps, where each step is numbered (e.g., Step X).
Your
1. Identify the major factual information, reasoning, and conclusions within reasoning steps.
3. Explain how to generate an incorrect step to replace one of the existing steps. This should include: Identifying step where the reasoning could naturally deviate.
Speculating what type of eиror would be most appropriatc introduce the chosen step.
Introduce an incorrect next step that aligns stylistically with previous steps. This incorrect step should: significantly harms the correctness.
Reflect deviation reasoning
Appear natural and believable in the context of the reasoning process.
5. Clearly explain how the incorrect step is eroo, highlighting the specific logical conceptual flaw.
Output Format:
SIEPSSUMMAYY:
[Summarize the reasoning within the steps in 1-2 sentences, identifying major information, logical steps, and conclusions.]
INCORRECT STEP GEN:
[Explain how the reasoning at specific step could deviate naturally into being incorrect. Clearly describe the type of error that could be introduced this step.]
ERROR_ TYPE:
[The name of the type of error chosen to be introduced.]
STEP NUM: [The number step was identified a place where the reasoning could naturally deviate. Only include the number here.]
INCORRECT.STEP
[Write the incorrect step the same tone and style as the other steps. Wrap the incorrect step inside curly braces (c.g. (incorrect stop))..
ERROR EXPLANATION:
[Explain how the incorrect step fits the definition of the selected type, identifying the specific flaw.]
Figure 13. System prompt counterfactual augmentation.
16

---

VersaPRM: Multi-Domain Process Reward Model via Synthctic Reasoning Data
B.2. Rewrite Augmentation
To further enrich the training dataset and enhance model robustness, we implemented rewrite augmentation. We randomly sampled one intermediate reasoning step (excluding the final answer step) from each CoT in the labeled MMLU-Pro-COS Train dataset and tasked Llama-3.1 1.mma-3.1-70B--. 11-70-IInnsttue with rewriting the step to preserve its logical content while varying its wording.
The prompt for rewrite augmentation explicitly required substantial rephrasing without introducing or omitting critical reasoning content (see Figure 14).
You will be given a reasoning step from a larger chain of thought. Your task is to rewrite this step using different phrasing, while keeping the underlying reasoning and deduction the same.
The rewritten step must:
Preserve the same logical content and conclusion
Match the tone and level of formality of the original
Use different wording and phrasing, not just minor edits or clause reordering Do not introduce new information or omit key reasoning. Format your output as:
<rewritten_step>>yor rewritten version here<//ewritten.ste> ### Original Step:
foriginal reasoning step)
Figure 14, Prompt to generate rewrite augmentations.
B.3. Self-Filtering
Motivated by prior works self-training (Amini al., 2025) and self-filtering (Shen & Sanghavi, 2019), applied self-filtrring procedure to the labeled IMLU-Pro-CoT-TTeee dataset. Specifically, we utilized VersaPRM to assign scores to each step within the dataset and filtered out entire CoTs containing any step whose predicted score deviated by more than 0.4 from its autolabeled score. This self elffffiltring process resulted in removing approximately 37% CoTs from the original dataset. We then train a PRM on the resulting filtered dataset.
B.4. A blation Study on Inclusion of Ground Truth in Auto-Labeling Prompt
As an additional ablation, we do an end-to-end experiment evaluating the impact of omitting the ground truth answer from the auto-labeling prompts. We performed an end-to-end experiment by auto-labeling the entire UMM.U-Pro-CoT-TEE dataset using modified prompt that explicitly excluded the correct answer. Subsequently, we trained variant of VersaPRM on this modified dataset.
The detailed ablated prompt used for autolabeling (excluding the ground truth answer) is provided in Figure 15.
B.5. Llama-as-a--Judge Baseline
Finally, considered additional baseline leveraging Llama-3.1 1.mmma-3.11700 3.B- Instructt directly process reward model (PRM). Here, we used Llama-3.1170BB -nstruu with the ablated auto-labeling prompt (see Figure 15) to identify the earliest incorrect step in each CoT. Scores of 1 were assigned to steps preceding the identified incorrect step, while scores of 0 were assigned to the incorrect step and all subsequent steps. Importantly, this approach did not utilize the ground truth answer at inference, reflecting a realistic scenario.
B.6. Experimental Results
We evaluate all of the methods presented in this section against VersaPRM using both BoN (Figure 16) and WMV (Figure 17). For each metric we apply min-aggregation across all categories of MMLU-Pro-Cort CoTT-val- (Unlabeled).
We find that the training enhancements counterfactull augmentation, rewrite augmentation, and self-filterd seff-filtrring- -rrovide no
17

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
significant benefit over VersaPRM. Under BoN, VersaPRM attains the best aggregate performance across categories, while under WMV the enhancements offer only marginal gains, and only large values of N.
Aggregated over all categories, VersaPRM also outperforms (i) directly using Llama 3.1-70B as a judge and (ii) the end-to-end variant of VersaPRM trained on data from the ablated prompt without ground-truth labels, for both BoN and WMV at every value of N. These results underscore that including the ground-truth answer in the labeling prompt is essential for the autolabeler to assign labels accurately.
18

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
You experienced evaluator specializing in assessing the quality of reasoning steps in problem- solving. Your task is to find the first BAD step in student's solution to multiple choice question.
You will judge steps as GOOD, OK or BAD based on the following criteria:
1. GOOD Step
A step is classified as GOOD if it meets all of these criteria:
Correct: Everything stated is accurate and aligns with known principles the given problem.
Verifiable: The step be verified using common knowledge, simple calculations, quick refer- ence (e.g., recalling basic theorem). If verifying requires extensive effort (e.g., detailed calculations or obscure references), mark it BAD instead.
Appropriate: The step fits logically within the context of the preceding steps. a prior mistake exists, GOOD step can correct
Insightful: The step demonstrates reasonable problem-solving direction. Even if ultimately progress in the wrong direction, it is acceptable long represents logical approach.
2. OK Step
A step is classified as OK if it is:
Correct and Verifiable: Contains no errors and can be verified.
Unnecessary or Redundant: Adds little value, such as restating prior information or providing basic encouragement (e.g., (Goodjob!")
Partially Progressing: Makes some progress toward the solution but lacks decisive or significant advancement.
3. BAD Step
A step is classified as BAD if it:
Incorrect: Contains factual errors, misapplies concepts, derives an incorrect result, or contradicts the ground truth answer.
Is Hard to Verify: Requires significant effort confirm due to poor explanation.
Is Off-Topic: Includes irrelevant or nonsensical information.
Derails: Leads to dead ends, circular reasoning, unreasonable approaches.
#### Task Description
You will be provided with:
I. A Question
2. Ground Truth Answer
Once you identify BAD step, return the index of the earliest BAD step. Otherwise, return the index of -1 (which denotes all steps are GOOD or OK). Please put your final answer (i.e., the index) in boxed.
Figure 15. Ablated system prompt used for auto-labeling where the ground truth not provided.
19

---

VersaPRM: Multi-Domail Process Reward Model via Synthetic Reasoning Data
All
Non-Math Domains
Chemistry
Compurer Science
60
Engineering
Inforess Acomey (5)
Infernce (5)
70
Inleerace Accuncc (N0)
60
Inference Accuncy (16
60
15
50-
55
30
50
50-
-0
40
40-
15
30-
128
16
generated scale)
generated scale)
generated scale)
Number generated
generated (log scalo)
(log scale)
Physics
Biology
Healthh
Psychology
70
Business
Inforenes Accoracy (86
(86)
Inference Accurazy (1)
Inference Accuracy (16)
Accuracy
83
&
65-
60
60-
75-
80
50
30-
20-
-5
40
128
Number generated
Namber of penemtted solutions
Number of penernted solutions
Number
penutated solutions
Number
genemted CoT solations (log scele)
(log soale)
(lag scale)
(log scale)
(lng scalc)
Economics
Law
History
Philosoply
Other
12.
Inforemes Arcincy (9g)
(%6
Infwracee Accurcy (N))
Informee Accuncy (%)
65
-50
40-
15
55
45
55
40
64 129
32 64 128 Number of penenttdd Col solutions (log scale)
2
18
16 32 64 128
16
32
64
15
128
Number of genarated CoT solutons
Nomber of genented CoT solutions
Number of ganorated CoT solutions
Number of penemtted CoT solutions (lug scele)
(log scald)
(Fg scals)
(log scale)
Mojority Wring BesteC
ViroRMM (Chen)) VassPRM (Gel-Sitered)
Vire piblandd peompt)
VeeaPRM (ccemorfactaaly sugmorid))
VencPPDA gevilte angiennte))
3.1-70B Jodge
Figure 16. Comparison of BoN using VersaPRM against the methods proposed Appendix B over all categorics of MMLU-Pro-CoTE (Unlabeled). We min-aggregation and the CoTs generated using Llama-3. 1.--a-3.-----ttrree
20

---

VersaPRM: Multi-Domail Process Reward Model via Synthetic Reasoning Data
All
Non-Math Domains
Chemistry
Compurer Science
60
Engineering
Inforess Acomey (5)
(Sg
70
Inleerace Accuncyy (Na)
60
Inference Accuncy (%)
60
15
50-
55
30
Infernce
50
50-
-0
40
40-
15
30-
128
16
generated (log scale)
generated scale)
generated scale)
Number generated
generated (log scalo)
(log scale)
Physies
85.
Biology
Healthh
Paychology
70
Business
Inforenes Accoracy (86
(86)
Inference Accurazy (16
Inference Accuracy (96)
Accuracy
80
83
&
65
60
60-
75-
80
50
30-
20-
-5
40
128
Number
generated solutions
Namber of penemtted solutions
Number of penernted solutions
Number
penstated solutions
Number
genemted CoT solations (log scele)
(log soald)
(lag scale)
(lgg scals)
(lng scalc)
Economics
Law
History
Philosoply
Other
12
Informmes Arcincy (Pg)
(%6
Infwence Accurcy (N6)
Informee Accuncy (%)
65
-50
40-
1.5
15
55
55
40
64 129
18 32 64 128 Number of penenttdd Col solutions (log scale)
2
18
16 32 64 128
1
16
32
54
5 15 128
Number of generated CoT solutons
Nomber of genented CoT solutions
Number of ganurated CoT solutions
Number of penemtted CoT solutions
(log scald)
(kg scals)
(log scale)
(lug scele)
Mojority Wring
ViroRM (Chen)) VasaPRM (Gel-Sitered)
Vine piblandd peompt)
VeeaPRM (ccumorfactaaly sugmarid)) 3.1-08 Jodge
Best NCN
VencPPDA gevilte angiennte))
Figure 17. Comparison of WMV using VersaPRM against the methods proposed Appendix over all categorics of MMLU-Pro-CoT. Eval (Unlabeled). We min-aggregatiin and the CoTs generated using Llama-3. 1----..-----nttulle
21

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
C. Additional Search Algorithm Details
Algorithm Beam Search with Process Reward Model
Require: Large Language Model LLM(-), Process Reward Model PRM(-), Prompt 80, Number of Beams N, Beam width
M, Maximum step length L
1: B + [8o]
2: + [0]
3: for = 1 to L
4: B - Expand len(B]
N
5: B + LLM.step(8))
6: Q + Aggr(B)
7:
best_idxs - Indexes of the highest scores in
8:
B B|best.idxs]
9:
Q[best_ice idxs]
10: if All sequences in B contain a terminal leaf node then
11:
break
12: end if
13:
end for
14: Return the sequence with the highest score from B
Algorithm a greedy search algorithm that uses a PRM select the best CoT during search. More details are given in Section 3.2.
22

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
Algorithm 2 Monte Carlo Tree Search with Process Reward Model
Require: Large Language Model -., Process Reward Model (RM()), Prompt 8p, Maximum step length L, Number of roll-outs K, Number generated child nodes d, Exploration weight w Initialize the value function Q + and visit counter N -
2: for O..... do
3: // Selection
4: 1. x-0
5: while is not a leaf node do
6:
(8.) < (s)) +1
81+1 f arg maXchildee((ss.) [(child(ss)) + w N(child )
N(a,)
7:
8: t ++t+1
9:
end while
10: I/ Expansion & Simulation (equivalent to the beam search with N = M d)
11: B + [st)
12:
while is not a terminal leaf node A t S do
13:
(st) < (s,)
< Expand d)
B LLM.step(B))
16:
for B do
17:
((s) Aggr(s) end for
18:
19:
8t+1 <- arg c (8))
20:
t t+1
21:
B+ [sr]
end while
23:
I1 Back Propagation
24: for t' +- ... do
25:
((s²) + ((s)(Q(s)).(ss.)
26: end for
27:
end for
28: Return sequence with the highest score among the terminal nodes
Algorithm a tree-basee based search algorithm that iteratively expands search tree find the CoT with the highest PRM score. MCTS iteratively builds search tree through the following steps:
1. Selection: Starting from the root node, the algorithm traverses the tree by selecting child nodes according to a selection policy.
2. Expansion and Simulation: Upon reaching non-terminal leaf node, the tree is expanded iteratively by generating fixed number of child nodes and then greedily selecting the child node with the highest value (which for us is determined by the PRM). This process continues until a terminal node is reached.
3. Backpropagation: The results from the simulation are propagated back through the tree, updating value estimates and visit counts for each node along the path.
These steps are repeated for fixed number of iterations until computational time limit is reached. To determine the final prediction, choose the terminal node with the highest value.
23

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
D. Additional PRM Training Details
D.I. Open-Source Math PRM Training Details
The open-source PRMs evaluated in this work utilize CoT training data derived from two mathematical datasets: MATH (Hendrycks et al., 2021) and GSM8K (Cobbe et al., 2021). The Math-Shepherd and OLHHlow/Deepsee--PRM datasets synthetically generated following the rollout method proposed Wang al. (2024b). Similarly, the MATH- APS dataset is produced using the synthetic generation technique introduced by Luo al. (2024). PRM800K, in contrast, consists of manually annotated labels and was specifically curated for the study by Lightman et al. (2024).
All PRMs are trained using the base LLMs of comparable model size and class, including Mistral-7B (Jiang et al., 2023), 1omm--.1-8B-Inssiie (Dubey et al., 2024), and Qwen-2.5-Mait 8B (Yang et al., 2024a).
Table Training details various Math PRMs
PRM
Base Model
Training Data
Training Method
Math-PSA
Qwen-2.5-Malle Mrath-h----s-trucc
PRMS00K, Math-Shepherd and MATH-APS
LoRA
Math- Shepherd
Mistral-7B
Math-Shepherd
Full fine-tuning
Qwen-2.5- M--M-MPaPM
Qwen-2.5 Swen-2.5-Math-
PRM800K
Full fine-tuning
RLHFLow Deepseck
Llama3. 1--.. 1-88--Instrect
RLHFlow/Deepseck FRM-Datt
Full fine-tuning
LlamaPRM880K QwenPRM8OOK
Llama3.1 1ee.1-8BB-nntiee
PRM800K PRM800K
Full fine-tuning
Qwen-2.5 1aat-22.--athh-25 Math-7B--Istruuu
Full fine-tuning
D.2. Details of PRM Training
For training, we extract logits from the tokens + and in the final layer of the LLM. The logit for + corresponds to a correct reasoning step, while the logit for represents an incorrect step. We use four newline characters \n as the classification token, which appended to the end of each reasoning step. We use standard cross-entropy loss and only compute it over our classification token.
For training math PRMs on the PRMS0OK dataset (QwenPRMS00K and LlamaPRMM800K) we employ batch size of 128 and perform full fine-tuning. For experiments mixed-domain datasets, we reduce the batch size to 32 due to smaller dataset size.
All training is conducted over single epoch. For full fine-tuning, we use a learning rate of 1.25 10 10-6, while for LoR LoRA-ban A-ased fine-tuning, we use a learning rate of 0 x 10-1
24

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
E. Additional PRM Comparisons
This appendix compiles a set additional results that fill out the main paper's findings. Concretely, it is organized as follows:
Table is a more detailed version of Table in the main text. It contains results over additional open source PRMs.
Figure 18 provides side-by-side comparison of the effects of using three different aggregation strategies with WMV and BoN
Figure shows WMV and BoN results when the generator swapped from L Llama-3. 1--00- Instruct
Figure 20 shows results obtained when the PRM is downsized from Llama-3. 3.1--B-Insttuc to a smaller model, -lama-3.1-3B-Imstuue
Figures 21 and 22 provide WMNV and BoN over every MMLU-Pro category for VersaPRM.
Figure 23 gives domain-leve, comparisons of Beam Search and Monte-Carlo Tree Search when powered by VersaPRM versus math-only PRM baseline.
Figure 24 is a more detailed version of Figure 3-nonniining experimental results for additional tested domains.
E.1. Evaluation Results for Math PRMs and VersaPRM Across all Categories
In this section, we list various other miscellaneous results
Table 4. Comparison among various math PRMs and VersaPRM on different domains in MMLU-Pro-CCS CoT-Evall (Unlabeled) when using WMV with aggregation N 16 CoTs generated per question using Llama3. 10mma3.1-BB... 1-8B-Instruch In parenthesis report the relative difference between WMV and the MV baseline (WMV-M) MVV.. While WMV using math PRMs exhibit greater improvement in math and aajh-adjceent domains, there significant improvement on other domains.
Category
MV (Baseline)
Math-PSA
Math-Shepherd
Qwen-2.5-Math-PRM RI.HFL.ww-Deepe
LlamaPR
VersaPRM
57.15
57.87(+0.72)
57.66(=0.51)
58.17(+1.02)
57.59(+0.44)
58.16(+1.01)
61.22(+4.07)
All except math
56.61
56.82(+0.21)
57.01(-0:40)
57.32(+0.71)
56.96(+0.35)
57.71(+1.10)
60.29(+3.68)
Math
62.40
64.20(+1.80)
64.13(+1.73)
67.20(+4.80)
64.07(+1.67)
65.40(+3.00)
68.87(+6.47)
Math-Adjacent
56.75
(3)9)(+1.23)
57.48(-0.73)
58.30(+1.55)
57.33(+0.58)
58.27(++1.22)
61.22(44.47)
Non-Math-Adaaeen
56.79(+0.10)
57.14(-0145)
57.09(+0.40)
57.022(0.33)
57.55(+0.86)
60.00(+3.31)
Chemistry
58.67
60.47(++.80)
60.13(+1.46)
60.67(+2.00)
59.13(40046)
60.47(+1.80
66.13(47.46)
Computer Science
55.80
56.93(+1.13)
56.077(0)27)
56.13(+0.33)
56.07(+00.7)
56.40X+0.60)
58.60((2.80)
Engineering
51.67
(0.67(-1.00)
(-.60)))
13.13(+1.46)
51.87(+0.20)
52.27(+0.60)
55.27(+3.60)
Physics
58.53
(1.87()3.34)
61.87(+3.34)
61.47(+2.94)
60.80(+2.27)
61.47(+2.94)
(4.87)+6.34)
Biology
75.38
75.23(-0.15)
75.38(-0.00)
75.69(+0.31)
7
76.38(+1.00
80.00(+4.62)
Hcalth
63.36
63.00(-0.36)
63.93(-0.57)
63.50(+0.14)
63.57
1..50
65.50(+2.14)
Psychology
61.60
61.47(-0.13)
61 4.7(-0.133 13)
62.27(+0.67)
61.47(-0.1))
(1.87()
64.53(
Business
61.34
61 (1.95)))))1)
62.21(-0.87)
63.02(+1.68)
62.21(+0.87)
62.62(+1.28)
64.50(+3.16)
Economics
62.00
62.67(40.67)
62.33(-0.33)
62.53(+0.53)
62.67(+0.67)
62.40(+0..0)
64.27(+2.27)
Law
35.93
35.72(-0.21)
37.24(11.31)
36.28(+0.35)
36.07(+0.14)
36.90(+0.97)
43.86(47.93)
History
49.20
49.00r 0.200
49.67(-0.67)
49.40(+0.20)
49.40(+0.20)
49.87(+0.67)
50.67(+1.47)
Philosophy
44.83
44.90(+0.07)
44.70(-0.13)
45.17(+0.34)
44.56(-0.27)
45.30(+0.47)
49.13(+4.30)
Other
55.53
55.80(+0.27)
55.47(-0.06)
56.07(+0.54)
55.87(+0.34)
57.07(+1.54)
59.00(+3.47)
25

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
E.2. WMV and BoN using different aggregation methods
64
64
62
62
Inference Accuracy (%)
Inference Accuracy
56
56
50
50
48
48
2
16
32
64
128
2
16
32
64
128
Number of generated CoT solutions (log scale)
Number of generated CoT solutions (log scale)
Majority Voring
Weighted Majority Voring
ofN
Min- Siirenggggtttionn
Last Aggregation
Average Aggregation
Figure 18. Comparison of WMV (left) and BoN (right) using VersaPRM with different reward aggregations on MMLU-Pro Pro-CoT--vval (Unlabeled). The CoT's are generated using Llama 3.1 8B Instruct. Overall, min-aggregation brings the largest inference performance boost.
26

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
E.3. Larger Generator Inference with PRM Rewarding
74
74
Inferenee Accuracy (%)
Inference Accuracy
70
70
68
68
66
16
32
64
8
16
32
64
Number of generated CoT solutions (log scale)
Number of generated CoT solutions (log scale)
Majority Voring
Weighted Majority Voring
BfN
VersaPRM (Ours)
LlamaPRMKOOK
Qwen 2.5 MRM PRM
Figure 19. Comparison of WMV (left) and BoN (right) using VersaPRM against math PRMs on MMLU-Pro-CoT-S -va-Eval (Unlabeled), We min-aggregation and the CoT's generated using Llama-3.1 1-----0--Intttuee Instrcct... Similar trends to using 8B model as the generator observed, indicating that our Multi- Domain PRM can generalize across generators with different capacities.
27

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
E.4. Inference with PRM of smaller model size
Inference Accuracy (%)
Inference Accurscy (%)
Math-Adjacemt
Inference Accurscy (%)
65
Maa--Maat--jjjeem
70
60
60
60
55
30
50-
50-
40
45
2
Number of generuted CoT solutions (log scale)
Number of generated CoT solutions (log scule)
Number of generated CoT solutions (log scule)
Inference Accuracy (%)
Accurncy (%)
forerccce Accurncy (%)
70
60
60
60
50
55
50
40
45
16
16
16
128
Number of generuted CoT solutions (log scale)
Number of generated CoT solutions (log scale)
Number of generated CoT solutions (log scule)
Majority Voting
Weighted Majority Voting
ofN
VersaPRM (Liama3B Basc)
VersaPRM (Llama Base)
Figure 20. Comparison of WMV (top) and BoN (bottom) using VersaPRM (Llama3B Base), PRM based Llama-3.2-30 Battaa-..2---Inttttee trained on our multi-domain dataset. We use min-aggregation and CoTs are generated using Llama-3.1-8B-Insttee with using VersaPRM (Llama Base), which applies the same training data and configurations but is based on Llama-3. 1at--3..-8B--ntttue VersaPRM (Llama3B Basc) brings less significant performance boost. However, the overall trends are similar, indicating the efficacy the inference pipeline using PRM.
Compared
28

---

VersaPRM: Process Reward Model via Synthetic Reasoning Data
E.5. Comparison of VersaPRM against Other Open-Source Math PRMs WMV and BoN by Category
All
Non-Math Domains
Chemistry
Computer Science
Engincering
70
Inference Accunay (NQ
Inforemee Accumuy (%)
Inference Accuncy
Inference Accuracy
:0-
55
55-
-55
Inforence
-00
40
30-
400
2
52 66 12s
16 32 64 124
16 52 64
16
52 64
8
16 32 128
Number generated Col solutions
Number of generated Coll solutions
Number of generated CoTl solutions
Number of generated Col solutions
Number of generated Col solutions
(log scale)
(log scale)
(Log suale)
(log scale)
(lug scall)
Plyyics
Biology
Health
Psychology
Business
20
s0
Iaferrnne Accunnry (%)
Inforrnce Accurayy (%)
65
fefermncc Accunccy (%)
66
Infercnce Accuncy
femonce Accumary
60
75
60
20
55
55
55
8
50-
50
128
Number generated Coll solutions (log scale)
Number of generated Call solutions
Namber of getterated CoT solutions
Number of generated Co solutions
Number of generated Cal aolutions
(log scall)
(log scale)
(log scale)
(lug scall))
Economics
Law
History
Philosophy
Other
70
15
Inferscce Avcuracy (%)
fevencc Accurcy (%)
60
Inforcice Accuncy
Accuracy
Inforesco Acconncy
50
-5
45
30
40
128
32
64
16
Number generated solutions
Number
genersted CaT
Number penerattd CoT solutions
of gggeentte
Number penerated solitions
(log scale)
[log scale)
(log scale)
(log scale)
(lug scale)
Majudiy Vicing
Weighted Majarit Valing
MaB-PSA
Muth Shephard
Qwen Moen-25-Mub-PPe MRM
OL:HFlow.Deepeck
VanuPRM (Ouna)
Figure 21. Comparison of WMV using VersaPRM against open-source PRMs on more other categories of MML.UPrr-Coo--vall (Unlabeled), We use min min-aggregation and CoTs are generated using Llama-3. 1.1ma-311-80 1--8B-Inttrct.
29

---

VersaPRM: Multi-Domail Process Reward Model via Synthetic Reasoning Data
Non-Math Domains
Chemistry
Computer Science
Engincering
Inference Acenney (%)
Inferance Accomey (%)
70
Infermnce Accumcy (%)
Infermme Ascunacy (%)
60
40
60
55
55
.55
30
40
2
52
12x solutions
64
2 Number
16 32 64
124 Coll solutions
2
8
16
32 64 128
16
52
64
128
64 128 (log
8
16 32
Number
Number of gonerated Coll solutions
Number of generated Coll solutions
Number gemarated Coll solutions
(log
scale)
seale)
(log
Physics
Biology
Health
Psychology
Business
Inforesce Accuncy (8)
20
40
Accmacy (N4
Inforucce Aveuracy (%K
Informnce Accuracy (%)
65
665
feforncc Acconecy
60
60
60
50
20
&
50-
64 12x
128 Number of generated Col solutions
16 32 64
16 32 64
16 52 64
16 8
12%
Number generased CoT solutions (log scale)
Number of getierated CoT solutions
Number of generated CoT solutions
Number of generated Col solutions
scale)
scall)
(log scale)
(log scalle)
Economics
Law
History
Philosophy
Other
55
45
Infoessic (SQ
50
Inferesze Accuracy (95)
Infrrmsee AateeerueAcccaantt Accurwcy (%)
Inforence Accaney
40
30
Anforeson
45
60
-5
10
40
55
128
64
Number ganerated Coll solutions
Number of gemerated Coll solutions
Number of gonerated CoT solutions
Number of generated CoT solutions
Number of genarated CoT solutions
(log scale)
[log scale)
(log scale)
(log stale)
(lug scall)
Majonty Vering
Shepherd
RLtt7fow-Deepoonk
Figure 22. Comparison of BoN using VersaPRM against open-source PRMs on more other categories of MMI.U-MPo (Unlabeled)) We use min-ag agnreaggggat and the CoTs are generated using Llama-3. 3.1-BBBBBBBB -------Instuu Instruct.
30

---

VersaPRM: Multi-Domaii Process Reward Model via Synthetic Reasoning Data
E.6. Comparison of VersaPRM against Other Open-Source Math PRMs MCTS and Beam Search by Category
Health
Physics
Math
Diology
Philosophy
75
70
(S)
Infermnce Accuray (S)
Infennce Accurecy (%)
70
K5
70
Inference Accuracy
80
80
Inforeroe
63
Inforescs
60
50-
-30
32 64 128
16 32 64 124
16 52 64
16 32 64
16
128
Number generated Col solutions
Number of generated Coll solutions [log scale)
Number of generated Co'l solutions
Number of generated Col solutions
Number of generated Col solutions
(log scale)
(log suale)
(log scale)
(lug scall)
Computer science
Other
Engineering
Econsomics
Hiskery
80
Accuraay(%)
Inference Accurasy
Inference Ascuracy (%)
70
50
70
60
Infrewse
Infrrncee
40
60
50
30
Number generated Coll solutions
Number of generated Call solutions
Namber of getterated CoT solutions
Number of generated Co solutions
Number of generated Cal aolutions
(log scale)
[log scalé)
(log scale)
(log scale)
(lug sealle)
Chemistry
Psycholugy
Law
Dusiness
Non-Math Demains
70
Infermnce Accuncy (%)
Inference Accumcy
Inference Accuncy
Accemnay
60
Ailerene-Avccumpy Avcunay
60
70
.00
60
50
Inferencee
.00
55
60
50-
30
30
128
32
16
Number generated solutions
Number genersted
Number penerated CoT solutions
of genernttd
Number penerated sohitions
(log scale)
[log scale)
(log scale)
(log scale)
(lug scale)
Maosity Voice
MCTS with VeMPRR
MCTS wath LiaNAPRMEOOKK
Bean Search with VersPPMM
Beun: Seaach with LiMiOOO
Figure 23. Comparison of VersaPRM and LaamPPRM00KK with beam search and MCTS. In more other categories from MML.U-Pro-CoT. Eval (Unlabeled), VersaPRM achieves better performance.
31

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
E.7. Comparison of VersaPRM against VersaPRM Trained with One Categories Held-out
Biology
Law
Computer Seience
Inferanc (60
Inforence Accuincy (10)
60
Infermnce Aucuracyy
23
70-
40
128
10
52
64
125
Number of genemite CoT solutions. (log scale)
Number of genemted CoT solutions (log scalo)
Number of genersted CoT solutions (log scale)
Philosophy
Psychologyy
$2.5
Inferenee (N)
fererccc Accurncy (%)
47.5
62-
Mijority Voring Weightel Majurity Vasing
VenaPRM
45.0
60-
40.0
Numbor of penortted CoT solutions (log scalo)
Number of genomtedd Col solutions (log scalo)
Figure 24, Hold-out domain ablation experiment results across all tested categories. WMV performance comparison of VersaPRM trained with each of the indicated domains held out versus the fully trained VersaPRM.
32

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
F. Manual Analysis of VersaPRM and Math PRM Mistakes
In this section, we manually analyze few handpicked examples of reward scores outputted by VersaPRM and a math PRM (specifically (wennPRM800K). For the following examples, steps bolded and colored in red denote steps manually inspected be incorrect.
F.I. Examples where the Math PRM Fails but VersaPRM Succeeds Question ID: 1141 mmlu law Category: Law
Question:
man and a woman were involved in a car accident that occurred when the man abruptly switched lanes front of the woman without signaling. Unable stop, the woman hit the man's car into median. Immediately after the accident, as the two drivers emerged from their vehicles, a pedestrian, obviously emotional and upset from having just observed an accident, ran up to the man and shouted, "You careless driver, you. Didn't you ever learn that you're 'r supposed to use a turn signal when you switch lanes?" The pedestrian is never identified and is not present trial. The pedestrian's 's statement is...
Options:
A. inadmissible, because the bystander was not involved in the accident; B. admissible, because it is relevant to the case; C. admissible under the excited utterance exception; D. admissible as non-hearsay; E. admissible, even though it is hearsay; F. admissible under the present sense impression exception; G. inadmissible, because it is an opinion, not fact; H. inadmissible, because the bystander cannot be identified and is not present at trial: inadmissible, because it is prejudicial the defendant; J. admissible under the spontaneous statement exception.
Answer: C
CoT 1:
1. The pedestrian's 's statement is spontaneous statement made while the person is under the stress of excitement caused by the event, and before they have time to reflect on the matter.
2. The statement is an out-of-court statement that is being offered for the truth of the matter asserted, which is that the man switched lanes without signaling.
3. Therefore, the statement is hearsay, but it is admissible under the excited utterance exception to the hearsay rule.
4. The answer is (J).
Math PRM rewards:
0.75, 0.82, 0.84, 0.86
VersaPRM rewards: 0.98, 0.95, 0.93, 0.17
Explanation:
The Math PRM does not check that the selected answer lines up with the prior reasoning steps.
CoT 2:
1. The statement is an outburst from the bystander immediately following the accident, and it describes what he just saw.
2. This meets the requirements of the present sense impression exception, which allows for the admission of statements describing or explaining an event or condition made while the declarant is perceiving the event or condition.
33

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
3. The statement is describing the man's actions in switching lanes without signaling, and it is being made immediately after the accident.
4. The answer is (F).
Math PRM rewards: 0.94, 0.92, 0.96, 0.87
VersaPRM rewards: 1.00, 0.43, 1.00, 0.09
Explanation:
The Math PRM fails to detect that the wrong hearsay exception was applied.
CoT 3:
1. The pedestrian's statement is out-of-court statement that is being offered for the truth of the matter asserted; Le., that the man was careless driver who failed to use turn signal.
2. The statement is admission by bystander, but it is still hearsay.
3. There is exception to the hearsay rule that allows statement to be admitted if it was made by bystander.
4. The answer is (E).
Math PRM rewards:
0.70, 0.90, 0.77, 0.65
VerSaPRM rewards: 0.97, 0.80, 0.42, 0.08
Explanation:
The Math PRM fails to recognize that there are exceptions to the hearsay rule such as the excited utterance exception. Thus, the PRM might have mistakenly judged the logical structure as correct without verifying if the statements were factually accurate in the legal context.
34

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
Question ID: 2518 _nshuupsyychology
Category: Psychology
Question:
Which of the following terms is the best synonym for test validity?
Options:
A. Dependability; B. Authenticity; C. Generality; D. Reliability; E. Stability; F. Truthfulness; G. Precision; H. Consistency; I. Replicability
Answer: F
CoT:
1. Validity refers to how accurately test measures what it is supposed to measure.
2. Reliability refers to how consistently a test measures what it is supposed to measure.
3. However, there is another term that is often confused with validity, which is reliability.
4. The answer is (D).
Math PRM rewards:
0.98, 0.99, 0.86, 0.80
VersaPRM rewards: 1.00, 1.00, 0.96, 0.02
Explanation: The Math PRM again does not check that the selected answer lines up with the prior reasoning steps.
35

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
Question ID: 5362 3362_mmlu_othhe
Category: Other
Question:
Which of the following is an example of devolution?
Options
A. Allowing states to decide how meet federally mandated greenhouse gas emissions targets; B. The federal government setting minimum wage that all states must meet; C. A state being prevented from implementing a fuel efficiency standard that is more stringent than that set by the federal government; D. A "race to the bottom" in which states compete with one another to have the fewest environmental regulations; E. A state being forced to comply with federal immigration policies; F. The federal government establishing nationwide educational standards; G. A state choosing to implement stricter gun control laws than those mandated by the federal government: A state that decriminalizes all drug use, contrary federal law; state that legalizes marijuana possession in conflict with federal law; The federal government overriding a state's decision legalize same-sex marriage
Answer: A
CoT:
1. Devolution refers to the transfer of power or authority from central government to a lower level of government, such as a state or local government.
2. Option C states that state is prevented from implementing a fuel efficiency standard that is more stringent than that set by the federal government. This is an example of devolution because the federal government is limiting the power of the state to act on its own, effectively "devolviigg" power back to the federal level.
3. The answer is (C).
0.98, 0.63, 0.89 Math PRM rewards:
VersaPRM rewards: 1.00, 0.10, 0.47
Explanation:
The Math PRM likely lacks the domain-specific understanding of political science and government structure. It does recognize that devolution inherently involves granting more autonomy rather than restricting it.
36

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
F.2. Examples where VersaPRM Fails
Question ID; 4410 mmlu chemistry
Category: Chemistry
Question:
Question: The solubility product constant of magnesium hydroxide is 8.9 x 10-12 12, calculate its solubility in (a) water and
(b) .05M NaOH.
Options:
A. 1.4 "om"mol/liten and 3.7 x 10 100-mollliter B. 1.5 x 10 /liter and 3.8 x 10 ooolmollleee C. 1.1
10 100 *mol and 3.0 x 10-9 moolm/irrrr D. 1.2 x mmol /liter and 3.5 10 m000mmll/lise E. 1.8 10 4mol "mol/liter and 3.2 x 10-9mol /liter; F. 2.0 x 10 /liter and 2.5 x 10-9mol/liters G. 1.6 x 10-4 /liter and 4.0 x mol//itr,,
10 10-2 /liter; H. 2.2 x 10 1001000000 "mol/litr and 3.9 x 10 10-9mol/liter I. 9.0 x 10 and 4.5 x
J.
1.3 x 10 1000mmoo/liee and 3.6 x 10 10 moolmmllte
Answer:
CoT
1. Let's calculate the solubility of magnesium hydroxide in water and in 0.05M NaOH.
2. The solubility product constant sKsp) is given by:
3. Kep [Mg=+][OH=]?
4. For magnesium hydroxide, the dissolution reaction is:
5. Mg(OHH)(s) = Mgq+(aq) + 20H)(ag).
6. Let be the solubility of magnesium hydroxide in moles per liter (M).
7. Then, [Mg x and [OH]]= = 2x.
8. Substituting these values into the Ksp expression, get:
9. 8.9 x 10 12 = x(2x)²
8.9 10 4x.
x² 2.225 10-12
1.17 14 1.2 10-4M:
10. Therefore, the solubility of magnesium hydroxide in water is approximately 1.2 x 10-4M.
11. When 0.05M NaOH is added, the concentration of OH- ions increases, and the equilibrium shifts to the left. The new concentration of OH ions is 0.05 + 2x.
12. However, the concentration M 2 ions is still x, because the NaOH does not react with Mg2+.
13. Substituting these values into the Ksp expression, we get:
14. 8.9 x 10-12 .0.005-- 2n:
15. This equation is difficult to solve algebraically, but we can make an approximation. The concentration of OH ions is much larger than the concentration of M ions, so we can assume that 0.05 + 2r R 0.05.
16. Substituting this value into the Kap expression, get:
17. 8.9 x 10-12 12 x x00.05) x R 3.56 x 10-9M M
18. Therefore, the solubility of magnesium hydroxide in 0.05M NaOH is approximately 3.6 x 10 M.
19. The answer (J).
37

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
Math PRM rewards: 0.99, 0.98, 0.98, 0.97, 0.99, 0.99, 0.97, 0.96, 0.87, 0.98, 0.82, 0.84, 0.92, 0.95, 0.83, 0.91, 0.68, 0.87, 0.91
VersaPRM rewards: 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 0.88, 1.00, 0.82, 0.93, 1.00, 0.80, 0.28, 0.99, 0.72, 0.97, 0.43
Explanation:
Neither the math PRM or VersaPRM label the calculations in step 9 as incorrect. VersaPRM marks step 15 as incorrect.
CoT 2:
1. The solubility product constant is given by ssp [Mg²+][OH-).
2. The solubility of magnesium hydroxide s.
3. We know that spp = Kar since s moles magnesium hydroxide will produce s moles Mg and 2s moles of OH and we know that K = 10-14
4. We can now calculate the solubility of magnesium hydroxide:
5. V Kip K_
8.9x10 12 10-14
V.9 10a 9.43 10 10-2M.
6. In the presence of .05M NaOH, have excess of hydroxide ions, assume that the concentration of OH is approximately equal to .05M.
7. We can now calculate the solubility of magnesium hydroxide:
8. Ksp 2
(M0g5)
9. [Mg K.p 8.9x10-12
11.8 10
.05
10. The answer (J).
Math PRM rewards:
0.97, 0.99, 0.41, 0.94, 0.40, 0.87, 0.95, 0.98, 0.89, 0.63
VersaPRM rewards: 1.00, 1.00, 0.53, 1.00, 0.69, 0.96, 1.00, 0.95, 0.83, 0.12
Explanation:
While the math PRM is able to detect the errors in steps 3 and 5, neither PRM recognizes the incorrect formula in step 1, which leads to the incorrect result in step 9. This suggests that both PRMs have gaps in their knowledge about this domain.
38

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
Question ID: 8925 _mmmu_mathhhhhhh
Category: Math
Question:
In Canada 2014, the average wholesale price of soybeans was $0.24 per pound. 2015, the average wholesale price of soybeans was $0.16 per pound. If retailer purchased 20,000 pounds of soybeans in 2014 and in 2015, what was the percent change in the retailer's expenses from 2014 2015?
Question:
Options:
A. -25%; B. 33%; C. -33%; D. -8%; E. 16%; F. 8%; G. -20%; H. -50%; I. 25%; J. 0%
Answer: C
CoT:
1. The retailer purchased 20,000 pounds soybeans in 2014 and 2015.
2. The cost of 20,000 pounds of soybeans in 2014 was 0.24 20, 000 4800.
3. The cost of 20,000 pounds of soybeans in 2015 was 0.16 20,
000 200000
3200.
4. The percent change in the retailer's expenses from 2014 to 2015 is 4800 4800-3200 3200
1600
=33%.
4800
5. The answer is (B).
Math PRM rewards:
0.99, 0.98, 0.99, 0.18, 0.20
VersaPRM rewards: 1.00, 1.00, 1.00, 0.68, 0.78
Explanation: VersaPRM does not recognize the error in the order of subtraction in step 4, which causes the result to have an incorrect sign.
39

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
Question ID: 112_mmlu_bbsinees
Category: Business
Question:
Question: The following entries appeared in the ledgers of Kane's Record Store: Beginning Inventory, Purchases, Ending Inventory $16,000, $58,000, $14,000. Find the store's rate of inventory turnover.
Options:
A. 4 times; B. 6 times; C. 2 times; D. times; E. 4.5 times; F. 8 times; 3.5 times; H. 7 times; 1. times; J. 2.5 times
Answer: A
CoT:
1. "To find the rate of inventory turnover, we need to use the following formula:
2. Inventory Turnover Cost of Goods Sold Average Inventory.
3. The cost of goods sold is the difference between purchases and ending inventory:
4. Cost of Goods Sold = Purchases Ending Inventory
$$58,000 -14,000-$14,000 = $$440000.
5. The average inventory is the average of beginning and ending inventory:
6. Average Inventory (Beginning Inventory Ending Inventory) / ($16,000 $14,000) $15,000.
7. Now, we can calculate the inventory turnover:
8. Inventory Turnover = Cost of Goods Sold / Average Inventory = $44,000 / $15,000
2.93 (or approximately times).
9. The answer is (D).
Math PRM rewards:
0.97, 0.98, 0.56, 0.55, 0.97, 0.93, 0.99, 0.21, 0.23
VersaPRM rewards:
1.00, 0.99, 0.96, 0.93, 1.00, 1.00, 1.00, 0.99, 0.95
Explanation:
VersaPRM does recognize the improper method of calculating the cost of goods sold in step 4, which causes accept incorrect results.
40

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
G. Other Experiments
G.I. Evaluation on Open-Ended Law Questions
We run an additional (preliminary) experiment testing the ability Versa-PRM to work on open-ended problems. Specifically, we first take the 145 law questions from AMLLU-Pro-CoT--vve and filter out the questions that contain the wording "which of the following [X] is correct", that have total of 116 questions remaining.
For each question, similar to before, we generate 16 responses to each question using -ettaa-..--B--nstttee However, differently, we remove all the answer choices from the question in the prompt. Thus the question effectively becomes open ended. The specific prompt is given in Figure 26.
We next use BoN to rerank these CoTss using both VersaPRM and a math PRM (Owen-PRM8000) as the reward model. (Figure 27). The final results are seen
We grade the final selected CoTs for correctness using in Figure 25. We see that with VersaPRM, accuracy increases with larger values of N N-albbit saturates around N 6. On the other hand the math PRM fails entirely and in fact sees negative change in accuracy with increasing values of N. This thus provides preliminary evidence that VersaPRM can also work in open-ended non-multiple choice settings.
56
34
52
Accuracy (96
50
48
46
44
VinsPRM (Oun)
42
Quen P MMMIIKK
4
10
11
12
13
14
16
Number of sampled CoT solutions
Figure 25. Performance VersaPRM vs. a math PRM (wwen-PRMM00K] on 116 open-ended law questions from MMLU-Pro-CoT-ES (i.e., by removing the answer choices from the original multiple choice questions). using BoN (min-aggregatiin). VersaPRM improves with larger N, while the math PRM fails to generalize.
Given the following question, respond with the best possible answer.
(open-ended question]
Your response should end with "Final Response: [your answer]", where [response to the question] should he replaced with your actual response. Each reasoning step should be separated by two newline characters.
Let's think step by step.
Figure 26. Prompt used to generate the CoTs used for BoN evaluation. The generator model is Llama3. 1-8------nstree Crucially, we do not provide the answer choices for the question when inserting it into the prompt.
G.2. Evaluation of Iterative Refinement Using PRM
In addition to the test-time scaling methods presented in the main body of the paper, another approach to solution generation is iterative refinement. In this method, the initial response generated by the LLM is critiqued by a critic model, whose feedback is then used by the LLM iteratively refine its answer (Madaan al., 2023; Xu al., 2024; Xi al., 2024).

---

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
Given the following open-ended question, reference answer, and student response, evaluate the student's response for correctness.
Question: (question)
Reference Correct Response: (reference answer)
Student Response: {parsed answer]
Provide step-by-step analysis of the student's response using the reference answer as a guide. Your response should end with [GOOD] if you believe the students response be correct and valid with respect the question and [BAD] otherwise.
Let's think step by step.
Figure Prompt used by grader LLM Allamma33-70B-Instrrre evaluate correctness the CoTs selected BoN
We evaluate iterative refinement with one iteration, specifically employing the PRM as the critic model. Initially, we score all CoT reasoning in the MMLU-Pro-CooT-Bvall dataset using VersaPRM and 2.em-2.5-Math-PPe We then prompt Llama-3.1 10mma-3.-8B--Instree to generate revised responses, following the template in Figure 28. After discarding responses that either timed failed parsing, our refined dataset comprises 220,640 CoT responses covering 1,754 questions from MMML-PPr-CCTT-Ewal
We compare the average @ass@1 rate-ccomee computed by calculating the proportion of CoT's with correct final answers per question, then averaging across all questions-for the original CoT dataset, refinement using VersaPRM, and refinement using Qwen-2.5-Math-PPM. These methods yield Pass@1 rates of 51 %,, 45%, and 44%, respectively.
These results indicate that VersaPRM slightly outperforms Qwen-2.5 -.tt---55--a------ further confirming the results in the main paper that has superior multi-domain generalization. However, notably, both refinement methods perform worse than the baseline without refinement. This outcome implies that the current refinement prompt in this experiment may not be potentilll overly biasing the LLM by the provided scores and suggests that additional prompt engineering is necessary for better performance. We leave more detailed study of iterative refinement with PRMs to future work.
Given the following question and answer choices: {question)
Below is previously generated chain-of -ffoouuhtthh reasoning for this question. Each step in the reasoning has an associated score (between and 1), provided by a reward model predicting the correctness that step:
{cot and step scores)
Re-examine and reconsider each step carefully. you rethink the reasoning process, explicitly use the provided scores as indicators to identify potentially correct or incorrect reasoning steps. Clarify your reasoning and adjust it accordingly based these scores.
Clearly separate each reasoning step by two newline characters.
After thoroughly reconsidering the reasoning process, select the best answer choice.
Conclude your response clearly with: "The answer is (X).", where X is the letter corresponding to your chosen answer.
Let's think step by step.
Figure 28, Prompt used to generate refined response given feedback
42

---

VersaPRM: Multi-Domail Process Reward Model via Synthetic Reasoning Data
Step O: To determine the effect of the purchase on GDP, we need to consider the components of GDP. Score 0.9765625
Step 1: GDP (Gross Domestic Product) is calculated as the sum of Consumption (C), Investment (I), Government Spending (G), and Net Exports (X).
Score 1: 0,9453125
Step 2: In this case, XYZ Corporation is buying the Matisse painting for its board room. This is an example of Investment (I), which is the purchase of capital goods or assets that are expected to increase future production. Score 0.6953125
Step 3: Since the purchase is investment, it will increase the Investment (I) component of GDP.
Score 3: 0.92578125
Step 4: The answer is (J).
Score 0.93359375
Figure 29, Example of CoT and corresponding step that would go in the {cot and step scores) part of the prompt in Figure 28
43


Reviews:
[{"review_id": "RfWSEAtklz", "reviewer": "Reviewer_gKUV", "summary": "This paper describes an automated pipeline for annotating chain-of-thought rationales with stepwise correctness labels. The authors generate a set of these annotated CoTs for MMLU-Pro, then train a model to predict these correctness labels. The result is VersaPRM, a process reward model for reasoning beyond math domains. The authors evaluate VersaPRM as a reranker against several math-specific PRMs using weighted majority voting, as well as plain majority voting without a reward model; they additionally evaluate VersaPRM as a scoring function for MCTS and beam search. Results on held-out MMLU-Pro questions show that weighted majority voting with math-specific PRMs fails to improve on simple majority voting in non-math-adjacent domains, whereas WMV with VersaPRM provides clear improvements in all considered domains. The authors also demonstrate that VersaPRM works well as a scoring function for search, providing better accuracy scaling with inference budget than majority voting or search with a math-specific PRM.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful feedback on the paper, recognizing that (i) its claims are well supported and (ii) this work is the first attempt to broaden PRMs beyond the math domain. We will incorporate your suggested revisions into our final paper.\n\n> The comparisons the authors carry out in their main experiments are valid. However, to truly demonstrate that VersaPRM is “domain-general” I would have preferred to see some evaluation beyond just MMLU-Pro, especially since the training traces are gathered from it.\n\nWhile we do not currently have results for VersaPRM on datasets beyond MMLU-Pro, we have conducted additional hold-one-out evaluations, which we will include in the final paper. Specifically, we exclude one domain category (law, biology, or CS) from training and assess whether the resulting PRM generalizes to the held-out domain.\n\nResults are available [here](https://drive.google.com/drive/folders/1n1bMkNSQ9u923b9P_IVW1nl3ENmSJCec?usp=sharing). As shown, the WMV generalization performance of VersaPRM *trained with one domain held out* is comparable to that of the full model. This suggests that its generalization ability is not merely due to broader coverage of the training data, but rather indicates genuine domain-general reasoning capabilities.\n\n> I am not aware of any essential references that the authors missed (with the exception of work currently in submission). \"To CoT or Not to CoT\" (Sprague et al. '24) could help contextualize the authors' finding that math PRMs are ineffective outside math domains - and the authors' demonstration that WMV with VersaPRM outperforms MV on non-math domains is extra surprising (positively) in light of the conclusion from that work that CoT primarily supports math performance and not other domains.\n\nWe appreciate this reference and will include it in Section 2. The following sentence that will be added:\n\n*“According to the work of Sprague et al. (2024), most of the reported advantages using CoT stem from math or math-related tasks.”*\n\n> I'm glad to see folks working to broaden the applicability of CoT and test-time search beyond math, these are encouraging results. The MCTS performance under VersaPRM is especially cool - looks like performance doesn't saturate with increasing budget nearly as quickly as it does with the other setups.\n\nThank you for this positive feedback. We will emphasize it in the final version.\n\n> On line 306 there is a missing space between \"PRM\" and \"via\".\n\nThanks for catching this–we will fix it.\n\n**Final Note:** We appreciate the reviewer’s insights and suggestions and are encouraged by the positive assessment of our work on broadening test-time scaling methods to domains beyond math.", "claims_and_evidence": "The authors' claims about their method are basically well-supported - see \"Experimental Designs or Analyses\" for a nitpick.", "methods_and_evaluation_criteria": "MMLU-Pro makes sense as a benchmark to evaluate this approach.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The comparisons the authors carry out in their main experiments are valid. However, to truly demonstrate that VersaPRM is \"domain-general\" I would have preferred to see some evaluation beyond just MMLU-Pro, especially since the training traces are gathered from it.", "relation_to_broader_scientific_literature": "There have been a handful of previous process reward modeling efforts in the literature, but as the authors note, these have mostly focused on the math domain, as this is where most step-by-step reasoning with LMs has been developed and evaluated due to easy answer verification. This is the first work I have seen attempt to train a domain-general reasoning PRM, but I may have missed one.", "essential_references_not_discussed": "I am not aware of any essential references that the authors missed (with the exception of work currently in submission). \"To CoT or Not to CoT\" (Sprague et al. '24) could help contextualize the authors' finding that math PRMs are ineffective outside math domains - and the authors' demonstration that WMV with VersaPRM outperforms MV on non-math domains is extra surprising (positively) in light of the conclusion from that work that CoT primarily supports math performance and not other domains.", "other_strengths_and_weaknesses": "I'm glad to see folks working to broaden the applicability of CoT and test-time search beyond math, these are encouraging results.\nThe MCTS performance under VersaPRM is especially cool - looks like performance doesn't saturate with increasing budget nearly as quickly as it does with the other setups.", "other_comments_or_suggestions": "On line 306 there is a missing space between \"PRM\" and \"via\".", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LADbKZBVzl", "reviewer": "Reviewer_baxV", "summary": "Process Reward Models (PRMs) have been effective in improving mathematical reasoning for Large Language Models (LLMs) by utilizing increased inference-time computation. However, their generalizability to non-mathematical domains remains unproven. This work demonstrates that current PRMs perform poorly in non-mathematical domains. To address this, VersaPRM is introduced—a multi-domain PRM trained on synthetic reasoning data generated through a novel data generation and annotation method. VersaPRM achieves consistent performance gains across diverse domains. For example, in the MMLU-Pro Law category, VersaPRM improves performance by 7.9% using weighted majority voting, significantly outperforming Qwen2.5-Math-PRM's 1.3% gain. Additionally, all data, code, and models for VersaPRM are open-sourced for community use.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback, and for acknowledging that (i) our method fills a gap in the existing literature on PRMs, (ii) we provide comprehensive evaluation criteria for our model, and (iii) our open-sourced code facilitates future exploration of this field.\n\n> Although the automatic labeling method is used, its accuracy still has room for improvement (about 75%), and there may be some incorrectly labeled data, which may interfere with model training and affect the further improvement of model performance.\n\nWhile the automatic labeling method introduces some noise (~75% accuracy), our PRM remains robust to moderate noise levels. This is evident from VersaPRM’s strong performance gains over previous math PRMs, despite being trained on noisy data. In this paper, we used Llama-70B due to budget constraints and as it was sufficient to create a model that outperforms previous math PRMs. That said, to further increase labeling accuracy, we can employ stronger models, such as GPT-4o, or reasoning models such as DeepSeek-R1 or OpenAI o1. This would give us higher accuracy labeling. Given time constraints for this response, we will aim to include an improved model with more accurately labeled data in the final version of the paper.\n\n> Although a variety of evaluation methods are used, these methods are mainly based on existing reasoning strategies and data sets, and may not fully cover all possible reasoning scenarios and fields. For some more complex or challenging reasoning tasks, the performance of the model may need further verification.\n\nWe have evaluated VersaPRM across diverse test-time methods. We acknowledge the importance of broader validation and will discuss and explore additional reasoning scenarios in the final version.\n\n> Although a comparative analysis with a variety of open source mathematical PRMs was conducted, the comparative analysis was mainly based on existing models and datasets. For some more advanced models or methods that have not yet appeared, the comparative advantages of the model may need to be re-evaluated. Exploring Mathematical Extrapolation of Large Language Models with Synthetic Data (Li et al., Findings 2024)\n\nWe will cite  “Exploring Mathematical Extrapolation of Large Language Models with Synthetic Data” in the revised version. This paper introduces a novel arithmetical puzzle task and demonstrates how fine-tuning on large-scale synthetic examples enables precise multi-step mathematical reasoning. The fine-tuned model not only solves the puzzles but also generalizes to harder, out-of-distribution problems involving larger numbers and the composing components of the arithmetical puzzle problem. While this approach is effective for math, we note its applicability to non-math domains remains unclear, as defining comparable structured tasks in those areas presents challenges.\n\n**Final note:** Thank you again for the comments. If you have any remaining questions, please do not hesitate to let us know.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "- Advantages\n1. A new method called VersaPRM (Multi-domain Process Reward Model) is proposed. By training on synthetic reasoning data, the reasoning ability of large language models in non-mathematical fields is effectively improved, filling the gap in the application of existing process reward models (PRMs) in multiple fields.\n2. A novel synthetic data generation and automatic annotation method is designed. The chain of reasoning (CoT) is generated and automatically annotated using LLM, which avoids the high cost and low efficiency of manual annotation, while ensuring data quality and diversity, providing rich materials for model training.\nComprehensive evaluation criteria: A variety of evaluation methods are used, including weighted majority voting (WMV), best N selection (BoN), beam search (Beam Search) and Monte Carlo tree search (MCTS), etc., to verify the performance of the model from different angles, which can fully reflect the performance of the model in different fields and different reasoning strategies.\n3. All data, code and models are open source, which facilitates subsequent research and promotes further exploration and application of this field in academia and industry.\n- Disadvantages\n1. Although the automatic labeling method is used, its accuracy still has room for improvement (about 75%), and there may be some incorrectly labeled data, which may interfere with model training and affect the further improvement of model performance.\n2. Although a variety of evaluation methods are used, these methods are mainly based on existing reasoning strategies and data sets, and may not fully cover all possible reasoning scenarios and fields. For some more complex or challenging reasoning tasks, the performance of the model may need further verification.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Although a comparative analysis with a variety of open source mathematical PRMs was conducted, the comparative analysis was mainly based on existing models and datasets. For some more advanced models or methods that have not yet appeared, the comparative advantages of the model may need to be re-evaluated.\n\n[Exploring Mathematical Extrapolation of Large Language Models with Synthetic Data](https://aclanthology.org/2024.findings-acl.55/) (Li et al., Findings 2024)", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nupiFyQYs1", "reviewer": "Reviewer_pcYc", "summary": "This paper introduces VersaPRM, a multi-domain Process Reward Model (PRM) designed to enhance reasoning capabilities across diverse domains beyond mathematics. The authors identify that current PRMs are predominantly trained on mathematical data and demonstrate poor generalization to non-mathematical domains. To address this limitation, they develop a synthetic data generation and annotation pipeline that uses Llama-3.1-8B-Instruct to generate Chain-of-Thought (CoT) reasoning steps and Llama-3.1-70B-Instruct to automatically label these steps. Using this synthetic data, they train VersaPRM, which shows performance gains across multiple domains of MMLU-Pro compared to math PRMs. The authors contribute their multi-domain PRM, the MMLU-Pro-CoT-Train (Labeled) dataset, and open-source their implementation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful feedback. We've addressed your concerns below.\n\n> Multiple test-time computation methods (MV, WMV, BoN, Beam Search, MCTS) but computational costs across these methods are not controlled\n\nWe clarify the computational costs of test-time methods: WMV and BoN scale with the number of generated CoT solutions $N$. Beam Search’s number of beams $B$ is equivalent to $N$, and MCTS scales with the branching factor times the number of iterations. Figure 6 compares MCTS and Beam Search in terms of computational cost for an equivalent number of generated CoT solutions.\n\nHowever, our goal is not to compare inference-time methods against each other, but to show that VersaPRM consistently outperforms math PRMs across all these methods.\n\n> 30 examples is insufficient to establish confidence in labeling quality\n\nWe expanded our sample to 60 questions. The revised analysis shows that 78% of responses marked correct by the autolabeler were accurate (95% CI: 0.68–0.89), and 70% of those marked incorrect were accurate (95% CI: 0.62–0.78). Based on this, we estimate 74% of CoT responses in the training set are correctly labeled. We will label more examples for the revision.\n\n>  Using Llama-70B directly as an evaluator at test time compared to VersaPRM\n\nThank you for your valuable suggestion. We provide an [additional experiment](https://drive.google.com/file/d/1rwev0yBT0scwyMj1Tgc6nXaUGy1EsFad/view?usp=sharing) comparing VersaPRM against Llama-70B directly as a judge on MMLU-CoT-Eval. VersaPRM outperforms Llama-70B across all values of $N$.\n\nRecall the training data for VersaPRM is labeled by Llama-70B with access to the correct answers during prompting, ensuring high-quality step-wise labels (Section 5.3). At inference, however, Llama-70B lacks prior knowledge of the correct answer, limiting its judging effectiveness as a judge. Thus, the gains stem from our PRM methodology, not merely Llama-70B’s knowledge.\n\n> The experiments do not adequately assess the cost-effectiveness of the approach; Using a 70B model to generate training data is expensive\n\nWe used Llama-70B via AWS Bedrock batch inference to label ~85K CoTs in our training data, costing ~$50. This is far cheaper than manual labeling or stepwise rollouts. We will clarify this in the final paper.\n\n> Lack discussion of several important related areas\n\nThank you for suggesting potential discussion on important related areas. We will add in detailed discussion of these areas in the final paper.\n> Limited discussion of potential biases inherited from the generator and labeler LLMs\n\nThank you for raising this issue. Since both generator and labeler LLMs are Llama models, VersaPRM may perform better on Llama-based inference. A detailed study of the bias would be important, especially for alignment. We will add this discussion in the revision.\n\n> The reason behind moderate improvements in some domains (e.g., History) compared to other domains\n\nManual inspection of questions in domains with moderate improvements (e.g., history and health) reveals that these primarily test factual recall over reasoning. We posit that aggregation methods like BoN and WMV are less effective in these domains, as success depends largely on the LLM’s factual knowledge–not reasoning quality–and whether the PRM can recognize the correct fact. Therefore, we suspect PRMs struggle in these domains due to weaker pretrained knowledge of health/history.\n\n> No ablation study about the synthetic data generation pipeline.\n\nSection 5.3 presents an ablation on auto-labeling prompts (Lines 275–282). We also tested removing ground-truth answers, which dropped agreement rates from 83% to 60% for CoT originally autolabeled as correct and 70% to 40% for those autolabeled as incorrect. We are conducting further end-to-end PRM evaluations using ablated training data and will update results accordingly.\n\n> A more detailed error analysis showing specific examples where VersaPRM succeeds but math PRMs fail; Visualizing what constitutes \"good\" versus \"bad\" reasoning steps across different domains\n\nWe provide mislabeled CoTs from math PRMs that VersaPRM correctly labels [here](https://drive.google.com/file/d/1IWVXEgDCpdhDHZ_zixL3ZHtbhq9xWGQq/view?usp=sharing). We will include examples of VersaPRM errors.\n\n> VersaPRM for iterative refinement of reasoning (rather than just reranking)\n\nSection 6.3 tests MCTS and Beam Search, which achieve strong results with significantly fewer computations.\n\n> Open-ended generation tasks extension\n\nWe include an [experiment on open-ended law questions](https://drive.google.com/file/d/1wYJO1daWMX-hngTnNg0hnrXX79Ggnhvt/view?usp=sharing).\n\n**Final Note**: We appreciate the in-depth feedback and hope our comments and additional experiments address the questions.  If our responses have addressed your concerns, please kindly consider raising the score.", "claims_and_evidence": "While the paper makes several reasonable claims, there are some gaps in the evidence presented. The core claim that VersaPRM generalizes better across domains (VersaPRM outperforms math PRMs) is supported by comparative evaluations, but a fundamental question remains unanswered: Is the improvement primarily from the distillation of a stronger model's knowledge (Llama-70B) into a smaller model, rather than from the PRM architecture or training methodology itself? The authors do not adequately control for this variable.", "methods_and_evaluation_criteria": "The evaluation using MMLU-Pro is appropriate and provides a solid testbed with reasoning problems across 14 domains. \n\nThe comparison across multiple test-time computation methods (MV, WMV, BoN, Beam Search, MCTS) shows the robustness of their approach across different inference-time techniques. However, authors do not adequately control for computational costs across these methods, making efficiency comparisons difficult.\n\nThe evaluation of auto-labeling quality through manual inspection provides a reasonable sanity check, though a larger sample could strengthen confidence in the dataset quality. For a dataset spanning multiple domains with thousands of examples, 30 examples is insufficient to establish confidence in labeling quality.\n\nHowever, what's fundamentally unclear is whether the paper's approach is simply knowledge distillation from Llama-70B to a smaller model. The authors do not include a crucial baseline: using Llama-70B directly as an evaluator at test time. This would help determine whether training a separate PRM provides any advantage over directly using the stronger model's judgments.", "theoretical_claims": "The paper does not make significant theoretical claims requiring proof verification. The formal definitions provided in Section 3 for PRMs and aggregation methods are straightforward and align with established concepts in the literature.", "experimental_designs_or_analyses": "The experimental design is comprehensive and sound:\n\n1. The authors use appropriate baselines, including multiple open-source math PRMs and majority voting.\n\n2. The evaluation across all domains of MMLU-Pro with consistent metrics allows for fair comparison.\n\n3. The experiments with DeepSeek-R1 (Figure 7) provide preliminary evidence (limited to the law subset) that the approach benefits even stronger reasoning models.\n\n\nHowever, Several issues exist in the experimental design:\n\n1. Lack of a proper baseline using Llama-70B directly as an evaluator at test time, which would help isolate whether the benefit comes from distilling Llama-70B's knowledge or from the PRM training approach.\n\n2. The experiments do not adequately assess the cost-effectiveness of the approach. Using a 70B model to generate training data is expensive, and it's unclear if this cost is justified by the performance gains.\n\n3. The validation of auto-labeling quality uses a small sample size (30 questions), which could be expanded for more robust validation of the dataset quality.", "relation_to_broader_scientific_literature": "This work relates to:\n\n1. It extends work on Process Reward Models (Lightman et al., 2024; Wang et al., 2024b; Luo et al., 2024) by addressing their domain generalization limitations.\n\n2. It builds on test-time computation techniques for LLM reasoning (Snell et al., 2024; Yao et al., 2024; Wan et al., 2024) by proposing a practical method for constructing efficient PRM evaluators.", "essential_references_not_discussed": "The paper covers parts of the relevant prior work comprehensively (areas covered in related work section). However, the paper lacks discussion of several important related areas:\n\n1. Knowledge distillation literature, which is essentially what the authors are doing when using Llama-70B to generate training data for a smaller model.\n2. Literature on model alignment through AI feedback (e.g., Constitutional AI approaches), which uses similar techniques of having larger models provide feedback to smaller models.\n3. Recent work comparing the cost-effectiveness of using larger models at test time versus training specialized models, which would provide important context for evaluating their approach.\n4. Work on zero-shot evaluation capabilities of large models, which would be relevant for understanding the baseline capability of Llama-70B as a direct evaluator.", "other_strengths_and_weaknesses": "**Strengths:**\n- The problem addressed is significant and practical, as it enables more effective use of test-time computation across diverse domains\n- The synthetic data generation pipeline is effective\n- The comprehensive ablations provide valuable insights about what factors matter for multi-domain PRM effectiveness\n- The open-sourcing of data, code and models will benefit the research community\n\n**Weaknesses:**\n- Fundamental ambiguity about whether improvements come from knowledge distillation or PRM training\n- The manual evaluation of auto-labeling quality uses a relatively small sample\n- Limited discussion of potential biases inherited from the generator and labeler LLMs\n- The improvements in some domains (e.g., History) remain modest compared to other domains, but reasons for this variability aren't deeply analyzed\n- No ablation study about the synthetic data generation pipeline. This appears to be the main contribution of the paper but the process is not dissected well.", "other_comments_or_suggestions": "- A more detailed error analysis showing specific examples where VersaPRM succeeds but math PRMs fail would enhance understanding of the model's strengths\n- Visualizing what constitutes \"good\" versus \"bad\" reasoning steps across different domains could provide interesting insights\n- Exploring more efficient inference techniques that require fewer candidate solutions would enhance practical applicability\n- Investigating whether the approach could be extended to open-ended generation tasks beyond multiple-choice questions would be valuable", "questions_for_authors": "1. Have you evaluated using Llama-70B directly as an evaluator at test time compared to your VersaPRM? This would help isolate whether the benefit comes from distilling Llama-70B's knowledge or from your specific PRM training methodology.\n\n2. Have you analyzed the specific types of reasoning errors that VersaPRM is better at identifying compared to math PRMs across different domains? This could help explain the varying degrees of improvement observed across domains.\n\n3. Have you explored using VersaPRM for iterative refinement of reasoning (rather than just reranking), where feedback from the PRM guides the generation process itself? This could potentially lead to higher quality reasoning with fewer total generations.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fa1MjzJpIX", "reviewer": "Reviewer_EnQL", "summary": "The paper introduces VersaPRM, a multi-domain Process Reward Model (PRM) designed to improve reasoning abilities across diverse domains beyond mathematics. Traditional PRMs have been primarily trained on mathematical reasoning tasks and fail to generalize effectively to other disciplines such as Law, Philosophy, and Biology. To address this issue, the authors propose a synthetic data generation and annotation pipeline that produces step-wise labeled reasoning data across multiple domains. VersaPRM is trained on this synthetic multi-domain dataset, leading to improved generalization and performance across various test-time inference strategies (e.g., Weighted Majority Voting (WMV), Best-of-N (BoN), Beam Search, and Monte Carlo Tree Search (MCTS)). Empirical evaluations on MMLU-Pro-CoT-Eval demonstrate that VersaPRM outperforms existing math-focused PRMs in non-mathematical domains, with a notable 7.9% improvement in Law compared to a baseline.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your meaningful feedback and recognizing that (i) our paper presents strong empirical results and robust experiments, (ii) it addresses an important gap in PRM research, (iii) it provides thorough comparison between math PRMs and VersaPRM, and (iv) it facilitates open science with open-sourcing.\n\n> The synthetic data generation process produces high-quality multi-domain reasoning labels. This claim is not fully supported, as the precision is around 70-75%, lower than 80% claimed from OpenAI.\n\nOur synthetic labeling achieves ~90% of the accuracy of human-labeled PRM800K (the target benchmark), despite requiring no manual effort. The slightly lower precision (70-75%) reflects the cost-effectiveness trade-off: OpenAI’s 80% required expensive human annotation, while our method is fully automated.\n\n> The impact of synthetic data noise is not fully explored. Although manual evaluation suggests 75% accuracy, additional breakdowns on how mislabels affect model performance would be beneficial.\n> Auto-Labeling Reliability The auto-labeling method for reasoning steps, while cost-effective, introduces potential noise. The manual evaluation (75% accuracy) suggests that a portion of the training data may contain incorrect annotations, which could impact PRM reliability.\n\nWe did not attempt to further decrease the noise in the training dataset, as we found 75% label accuracy sufficient for VersaPRM to generalize robustly across domains and inference methods. \nFrom a preliminary analysis, we observe that many mislabels stem from a failure to identify faulty reasoning due to incorrect factual information.\n\nFurther noise reduction (e.g., via stronger labeling models) is promising but deferred to future work.\n\n> The choice of backbone LLMs (Llama vs. Qwen) is not fully explored—for example, the effect of using even larger models (e.g., GPT-4 or DeepSeek-R1) remains an open question.\n> Impact of larger-scale LLMs on PRM performance (e.g., GPT-4 vs. Llama-3).\n\nIn Section 6.4, we provide initial results with DeepSeek-R1 (700B parameters) in the law domain. In addition, we are currently testing this approach in the biology domain and will include these results. Broader exploration of large models for PRM initialization/auto-labeling remains future work.\n\n> More examples of mislabeled CoTs to analyze failure cases.\n\nWe provide examples CoTs mislabeled by math PRM but not by VersaPRM in [this link](https://drive.google.com/file/d/1IWVXEgDCpdhDHZ_zixL3ZHtbhq9xWGQq/view?usp=sharing). We will include more diverse examples of errors that VersaPRM also makes.\n\n> There are few typo and words repeated. Please check carefully.\n\nThank you for noting this—we will thoroughly proofread the manuscript.\n\n**Final note:** Thank you again for the comments and we appreciate the thorough review!", "claims_and_evidence": "Overall, the paper presents strong empirical support for its claims, but there are some areas where additional clarification could be beneficial.\n\n1. PRMs trained only on mathematical data fail to generalize to other domains.\nThe results in Tables clearly demonstrate that existing math-trained PRMs (e.g., Math-Shepherd, Qwen-2.5-Math-PRM) perform poorly in non-math domains, supporting this claim.\n\n2. VersaPRM improves reasoning performance across multiple domains through synthetic multi-domain training.\nThe authors provide multiple ablation studies showing that PRMs trained on diverse reasoning data consistently outperform math-trained PRMs across Law, Philosophy, and Biology.\n\n3. The synthetic data generation process produces high-quality multi-domain reasoning labels.\nThis claim is not fully supported, as the precision is around 70-75%, lower than 80% claimed from OpenAI.", "methods_and_evaluation_criteria": "The proposed methodology and evaluation criteria are aligned with the research goals.\n\n1. Benchmark Datasets:\nThe authors evaluate VersaPRM on MMLU-Pro-CoT-Eval, which covers 14 diverse domains (Math, Physics, Law, Biology, Philosophy, etc.), ensuring a comprehensive evaluation.\n\n2. Evaluation Metrics:\nThe use of WMV, BoN, and search-based methods (MCTS, Beam Search) effectively captures both accuracy and inference-time performance.\n\n3. Baselines & Comparisons:\nThe comparisons with four open-source math PRMs (Math-Shepherd, Qwen-2.5-Math-PRM, etc.) establish a strong benchmark.", "theoretical_claims": "The paper does not include formal theoretical proofs but provides empirical justifications for its findings.", "experimental_designs_or_analyses": "Yes, the experimental design is robust, but a few areas could be strengthened.\n\nStrengths:\n1. The synthetic data generation pipeline is clearly described, using Llama-3.1-8B to generate Chain-of-Thought (CoT) reasoning and Llama-3.1-70B as an auto-labeler.\n2. Multiple reranking methods (WMV, BoN) and search strategies (Beam Search, MCTS) ensure that results are not limited to a single inference-time approach.\n3. The ablation studies confirm that performance gains come from multi-domain data rather than additional training data volume.\n\nPotential Weaknesses:\n1. The impact of synthetic data noise is not fully explored. Although manual evaluation suggests 75% accuracy, additional breakdowns on how mislabels affect model performance would be beneficial.\n2. The choice of backbone LLMs (Llama vs. Qwen) is not fully explored—for example, the effect of using even larger models (e.g., GPT-4 or DeepSeek-R1) remains an open question.", "relation_to_broader_scientific_literature": "The paper builds upon and extends prior research in reward modeling and reasoning in LLMs, particularly in the following areas:\n\nProcess Reward Models (PRMs) vs. Outcome Reward Models (ORMs):\n1. Prior works (e.g., Uesato et al., 2022; Lightman et al., 2024) established that PRMs outperform ORMs in math reasoning.\n2. VersaPRM expands PRM utility beyond math, demonstrating improved performance in law, biology, and philosophy.\n\nTest-Time Compute and Self-Improvement Methods:\n1. The paper aligns with test-time inference strategies like Tree of Thoughts (ToT) (Yao et al., 2024) but enhances them with PRM-guided reranking.\n2. Demonstrates that process reward models remain useful even for strong reasoning models (e.g., DeepSeek-R1).\n\nSynthetic Data for PRM Training:\n1. Prior works (e.g., Wang et al., 2024; Zheng et al., 2024) used synthetic process data for math reasoning.\n2. VersaPRM generalizes this approach to non-math domains, demonstrating broader applicability.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "Weakness\n1. Auto-Labeling Reliability\nThe auto-labeling method for reasoning steps, while cost-effective, introduces potential noise. The manual evaluation (75% accuracy) suggests that a portion of the training data may contain incorrect annotations, which could impact PRM reliability.\n\nStrength\n1.Novelty & Relevance\nThe paper addresses an important gap in PRM research by extending process supervision beyond mathematical domains.\nThe synthetic data generation approach, which includes automated reasoning step labeling, is an innovative method to scale PRM training.\nEmpirical Contributions\n\n2. The study provides thorough comparisons between math-focused PRMs and VersaPRM, showing clear performance improvements across multiple domains.\nThe ablation studies and test-time inference analyses validate the generalization capacity of VersaPRM.\n\n3.Reproducibility & Open Science\nThe authors commit to open-sourcing all datasets, model checkpoints, and code, facilitating reproducibility and further research.\n\n4.Strong Experimental Setup\nThe evaluation includes rigorous baselines, including open-source math PRMs and multiple test-time inference techniques (e.g., majority voting, beam search, Monte Carlo Tree Search).\nThe inclusion of large-scale reasoning models such as DeepSeek-R1 strengthens the credibility of the findings.", "other_comments_or_suggestions": "There are few typo and words repeated. Please check carefully.", "questions_for_authors": "no", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this work, the authors propose VersaPRM, a novel Process Reward Model (PRM) that is trained on synthetically generated data beyond math word problems. The paper includes a few key contributions: 
1. the authors first demonstrate that the current PRMs perform poorly in non-mathematical domains (because they are trained exclusively on math word problems); 
2. the authors propose a novel synthetic data generation pipeline to generate and label reasoning chains from MMLU-Pro tasks (multi-domain). Notebly, they use a smaller LLM (8b) to generate the reasoning chains, and use a larger LLM (70b) to label.
3. the authors train a multi-domain PRM using the above data, and show that fine-tuning with synthetic multi-domain data provides PRMs better generalizability. 
4. the authors promise to open-source data, code and models.

The authors and reviewers had a productive discussion. Below are some of the salient discussion points:
1. Reviewers had concern that since the reasoning chains are collected from MMLU-Pro, the current evaluation may provide insufficient evidence on VersaPRM's generalizability (because test tasks are in-domain). The authors acknowledge that time does not allow to run experiments on new datasets. However, they proposed a new setting which exludes one domain category from training and assess whether the resulting PRM could generalize to that held-out domain. Results show that the resulting PRM is still comparable to the full model shown in the paper, this suggests the generalization comes from domain-generl reasoning capabilities rather than having tasks in-domain.
2. Reviewers had concern that the size of the data subset used to  assess the quality of the auto-labeled data was small (30 data points). The authors double the size and show the same results. 
3. The authors provide additional experiment results that compares VersaPRM against using Llama70b directly as a judge. Results show that because the lack of ground-truth labels, directly using Llama70b as a judge does not provide as good performance. This weakens the assumption that VersaPRM's good performance may due to the distillation from stronger model (70b), and suggests that the performance gain is really from the PRM methodology.
4. The authors and reviewers discussed the potential of using even stronger models in data labelling, which could potentially further improve the data quality and thus the performance of the fine-tuned PRM. 

The authors did a great job addressing the reviewers' concerns. In fact, the authors have successfully convinced two reviewers to increase their scores, this is rare in my batch. 

Additional AC comments: 
Thanks for integrating R1 experiments and analysis so fast! I acknowledge that R1 was released one week before the ICML deadline. 

Overall, this is a strong paper with clear contribution to the ICML community. The fact that the authors promise to open-source everything (data, model, code) makes the work even valuable to researchers and practitoners in the sub-field of LLM reasoning, LLM-as-a-judge and verifier training. I recommend to accept this paper.
Decision: Accept (oral)