Entropy-Regularized Process Reward Model
Authors: Hanning Zhang
Pengcheng Wang
Shizhe Diao
Yong Lin
Rui Pan
Hanze Dong
Dylan Zhang
Pavlo Molchanov
Tong Zhang
Venue: Accepted by TMLR ()

Abstract: Large language models (LLMs) have shown promise in performing complex multi-step reasoning, yet they continue to struggle with mathematical reasoning, often making systematic errors. A promising solution is reinforcement learning (RL) guided by reward models, particularly those focusing on process rewards, which score each intermediate step rather than solely evaluating the final outcome. This approach is more effective at guiding policy models towards correct reasoning trajectories. In this work, we propose an entropy-regularized process reward model (ER-PRM) that integrates KL-regularized Markov Decision Processes (MDP) to balance policy optimization with the need to prevent the policy from shifting too far from its initial distribution. We derive a novel reward construction method based on the theoretical results. Our theoretical analysis shows that we could derive the optimal reward model from the initial policy sampling. Our empirical experiments on the MATH and GSM8K benchmarks demonstrate that ER-PRM consistently outperforms existing process reward models, achieving 1% improvement on GSM8K and 2-3% improvement on MATH under best-of-N evaluation, and more than 1% improvement under RLHF. These results highlight the efficacy of entropy-regularization in enhancing LLMs' reasoning capabilities.

Published in Transactions on Machine Learning Research (06/2025)
Entropy-Regularized Process Reward Model
Hanning Zhang
Pengcheng Wang
Shizhe Diao +
Yong Lin
Rui Pan
Hanze Dong
Dylan Zhang
Pavlo Molchanov
Tong Zhang
Universily of Nlinois Urbana- Champaign & Princeton University
4 Universily of Toronlo Salesforce Research
NVIDIA
Reviewed on OpenReview:
Abstract
Large language models (LLMs) have shown promise in performing complex multi-step rea- soning, yet they continue to struggle with mathematical reasoning, often making systematic errors. A promising solution is reinforcement learning (RL) guided by reward models, par- ticularly those focusing process rewards, which score each intermediate step rather than solely evaluating the final outcome. This approach is more effective at guiding policy mod- els towards correct reasoning trajectories. In this work, we propose an entropy-regularized process reward model (ER-PRM) that integrates KL-regularized Markov Decision Processes (MDP) to balance policy optimization with the need to prevent the policy from shifting too far from its initial distribution. We derive novel reward construction method based on the theoretical results. Our thcoretical analysis shows that could derive the optimal re- ward model from the initial policy sampling. Our empirical experiments on the MATH and GSMSK benchmarks demonstrate that ER-PRM consistently outperforms existing process reward models, achieving 1% improvement on GSM8K and 2-3% improvement on MATH under best-of-N evaluation, and more than 1% improvement under RLHF. These results highlight the elficacy of entropy-regullriizttoo in enhancing LLMs' reasoning capabilities.
1 Introduction
Large language models (LLMs) have demonstrated remarkable performance across numerous tasks, partic- ularly in the challenging area of mathematical reasoning (OpenAl et al.] 2024 Yang et al. 2024 Dubey
et al. 2024h. key factor behind these successes the use of synthetic data, as explored in works such as (Toshniwal et al. 2024b). Open-source models fine-tuned on these synthetic datascts have shown significant improvements test accuracy on standard mathematical reasoning benchmarks such as MATH (Hendrycks al. 2021 1 and GSMSK (Cobbe et al. 2021). Following supervised fine-tuning, reinforcement learning (RL) has received considerable attention as a technique for further improving models' reasoning abilities by aligning the model's behavior with human values and goals, especially after the release of OpenAl-ol mode!"
Meta--Mathh (Yu et al. 2023). MAmmoTH (Yue et al. 2023). and Open-Math
The RL methods used in the current literature on LLM reasoning can be broadly categorized into two approaches. The first category focuses on training-time alignment, where RL used to fine-tune the model to maximize an RL metric evaluated by a reward model. Examples include PPO (Schulman et al.) 2017). 2024). More recently, scaling inference-time computation has also been shown to enhance model performance by using reward models guide LLM decoding. Techniques such as best-of-n sampling (Dong et al. 2023 and Monte Carlo Tree Search (MCTS) (Xie et al. 2024 are representative
GRPO (Shao et al.
"The first two authors contribute equally. Email: Sahheng.wangmmaln wtwronto.css (adiso, pmolchanev) com. y176900princeton.oo edu, wwwzeeddddgaaleeorrr..ooo https://openai. com/:indee////ttrroocinnngg openai- o1-preiee//
(hanning5, ruip4, shizhuo2, tozhang)0illinois. edu,

---

Published in Transactions on Machine Learning Research (06/2025)
examples of this approach. All these methods require external reward model to provide reward signal to update the model parameters rank the candidate responses. Regardless of the specific RT method, the quality of the reward function remains the most crucial factor, as it sets the upper bound for the algorithm's performance.
Solution: a
aa',a"aa"
Answer: 85
y= X
Outcome Reward
Problem: Half the value S3x-99 is 3x8. the value of Sx.S?
at We the problem the equation St (fracf1){2) (3x y) = x 37S.
Hard-label
2,1 a
a
3,1
a
L1,1
Answer:
y
v
ri max y
Golden Answer: 83
a
2,2
a
3,2
L,2,2
Answar: y:=1
Soft-label
a' the i-th step of the solution a
a
a
3,3
a
L33,3
Answer: y==
X
Entropy regularized label
d d:: the i-th step of the j-th finalized solution
the correcinees of the i-th
Y2
B t.. Mioz
Figure Illustration of ER-PRM, along with other baseline methods construct the process reward data and outcome reward data. The key idea of ER-PRM is to calculate the process reward from the sampling trajectories under entropy-regullaiiattion
One distinct feature of complex mathematical reasoning is that models typically require multiple steps of reasoning before producing the final answer (Wei et al. 2023). Accordingly, there are generally two types of reward models in the mathematical domain: the outcome reward model (ORM) and the process reward model (PRM). ORM provides an overall score for the quality of the final answer, while PRM assigns a score to each step in the reasoning process, providing more fine-grained feedback. Existing works (Wang et al. 2024 Lightman et al.] 2023 show that PRM outperforms ORM by a considerable margin. For instance, in Lightman et al. (2023), we use the PRM to select the final output from the candidate responses, we achieve a test accuracy of 78.2%, compared to the ORM with 72.4% and majority voting of 69.6%. Furthermore, PRM is more interpretable because it mirrors human behavior when evaluating reasoning paths. The PRM model is able to guide the policy model's decoding at each individual step or rerank candidate answers to enhance accuracy and consistency during generation.
Inspired by the superior performance of PRM, rescarchers have spent great efforts recently in the train- ing of PRM. To get the process annolation for the PRM, one can ask humans label the data at each step (Lightman
al.) 2023), which is extremely expensive and inefficient. Since then, researchers have explored alternative labeling techniques. One represcntative example is the automatic labeling technique introduced by Math-Shepherd (Wang et al.) 2024). The main idea of Math-Shepherd is to interpret the score of each step as its potential to deduce the correct final answer. Then, we can sample multiple trajec- tories starting from the intermediate step, and use the ratio of correct trajectories as a proxy for this score. The Math-Shepherd has proved as an effective way to generate high-quality process reward scores and even outperform human-labeled data.
Despite its effectiveness, Math-Shephers considers traditional Markov Decision Process (MDP) framework, which differs from the typical RL practice used with LLMs. Since the introduction of reinforcement learning from human feedback (RLHF), the standard approach has been using rngroiy-regulariize reward relative to reference model (often the initial model) (Ouyang et al. 2022 Bai et al.) 2022). This also applics to RL in reasoning lasks where we formulate the problem as an entropy regrooy-regulrrizeed MDP Shao el al., 2024 Xiong et al. 2024b Zhong et al.) 2024b). The main goal here is to balance optimizing the reward and staying close the original policy, which is crucial because we start with a well-trained LLM, and we do not want the model to deviate too far from this initial strong foundation policy. In recognition of this gap, in this paper, formulate the multi-step mathematical reasoning task under the entropy-regularized MDP framework,
2

---

Published in Transactions on Machine Learning Research (06/2025)
derive the mathematical principles of the process reward construction, and propose practical algorithms. Our contributions are summarized as follows.
We propose the entropx-regularized process reward, a novel method for labeling the process reward score with entropy regularization.
Based on the labeled data, we train a new process reward model to improve the performance. On the MATH and GSM8K datasets, we achieve significant improvements using the best-of-N evaluation method. Furthermore, by employing the reward model with rejection sampling, observe substantial enhancement in the performance of the policy model.
To further boost the process reward model rescarch, we will release all the data, code, and chockpoints to the community.
2
Related Work
hematiil Reasoning and Synthetic Data Generation. Mathematical problem solving is crucial for evaluating LLMs' reasoning abilities. A common approach is Chain-of-Thouue (CoT) prompting, which
guides LLMs to solve problems step-by-step (Wei et al.) 2023 Zhou et al. 2022 Zhu et al., 2022 Tong
ct al. 2024). However, general chatbots still struggle with mathematical reasoning, as shown by benchmarks
(Cobee et al. 2021; Hendrycks et al.] 2021). To improve performance, several works propose generating
synthetic data and fine-tuning LLMs, including Meta- MattaMMatQAAA (Yu et al.) 2023). MAmmoTH (Yue et al. 2023), MMIQC (Liu & Yao 2024), ToRA (Gou et al. 2023), Open-Mathhnntructt (Toshniwal et al. 2024b). and Dart-Math (Tong et al.) 2024). These datasots use teacher models to generate and filter responses, ensuring only correct. answers are retained, process known as rejection sampling or best-of-n sampling
(Dong et al.
2023
Yuan et al.
2023 All these emphasize stepwise reasoning for improved task performance.
Applying RL to Mathematical Reasoning. Inspired by the success of RL in chatbots like Chat-GPT and Claude, researchers have begun applying RL to mathematical reasoning tasks. Some works use deep RL methods like PPO (Schulman et al. (017) and GRPO (Shao al., 2024 to optimize external reward signals.
Others apply direct preference learning algorithms, such as DPO and KTO (Jiao et al. 2024 Yuan et al.
2024), for problem-solving. Online iterative DPO, initially for chat tasks (Xiong et al. 2024a Xu et al. 2023) has been adapted for CoT reasoning (Xie et al.
2024 Xiong et al. 2024b; Pang et al. 2024). Additionally,
inference-time RL methods like best-of-n sampling (Dong et al. 2023 Lightman et al. 2023) and MCTS (
et al. 2024 Chen et al. 2024a Lai el. al. 2024) guide decoding or response selection using external reward models. These approaches, including deep RL, iterative preference learning, and inference-tingg methods, all depend on reward models and can benefit from our study.
Reward Modeling in Mathematical Reasoning. In mathematical reasoning, reward models in RLHF are not trained via pairwise comparisons due to the availability of ground-truth answers (Bai et al. 2022:
Ouyang et al., 2022 Dong et al.] 2024). These models are generally divided into Outcome Reward Models (ORM) and Process Supervision Reward Models (PRM). ORM evaluates the overall quality of a response, while PRM provides feodback at each step of the reasoning process. The success of PRM approaches, like Lightman al. (2023), has fueled interest in studying PRM construction. However, PRMs require extensive human annotation, which is challenging for open-source efforts. To mitigate this, Math-Shepherd (Wang et al.) 2024) proposed method for efficiently acquiring process reward labels by generating parallel completions and using trajectory ratios to evaluate correctness. This has been proved as effective as human annotation to generate high-quality datasets. Building on this, studies have incorporated process supervision and MCTS to improve RLHF pipelines, such as stepwise DPO with MCTS (Chen et al.) 2024b), an MCTS- basod PRM framework (Luo et al., 2024), MCTS Self-Refine (Zhang al., 2024), and process generation from final answers (Chen et al. (2024a]. These efforts have significantly enhanced process supervision efficiency, motivating the use of MCTS for high-quality data generation.
3

---

Published in Transactions on Machine Learning Research (06/2025)
3 Approach
In this section, we present a novel reward formulation from an entropy perspective that utilizes KL- Regularization to derive the reward calculation of PRM, which we refer to as Entropy-Regularized PRM (ER-PRM). This formulation is followed by the derivation of obtaining the optimal policy model through RLHF using our entropy-rogulated process reward calculation approach. The mathematical result essen- tially from the KL-regularized RL literature, which has recently been studied in the context of DPO training in RLHF [Xiong et al. 2024b Zhong et al. 2024a). And move step forward for the reward modeling. Subsequently, we compare our approach with Wang et al. (2024); Luo et al. (2024) and then demonstrate that our methodology is more robust than those proposed Engregation-basee reward calculation due to their dependency on initial policy model.
3.1 Task Formulation and Optimal Policy
We consider training an LLM for reasoning. Given prompt 2, the LLM produces L-step reasoning chain == [a'.... aI.. The reward r(a, r) indicates whether the result of the reasoning chain is correct or not. We want to find LLM, denoted by policy n. (a/z), that optimizes π E II with the KI-regularizod loss function
[(T) = (1, [r(a, x) - =(122)
mo(a|z)
where mo the initial policy model, a pretrained LLM, and æ is the model being fine-tuned. According Lemma I the minimizer for is
The (alx) = To(a(r)))r(aa))
Eg Bawny (nr(a,)) o (0(a)))))
3.2 Multi-step Reasoning
We refer to multi-step reasoning as over partial chains by adopting these definitions in appendix D and our partial reward score assesses the correctness for one step beyond the current partial chain context. Let ..... a!l = =aa.....,de a'), be partial reasoning chain up step and a - [al+1 a'. be the completion of the
partial reasoning chain from step I 1+ 1. We define the intermediate reward by ( a) Ba-Mio e ((, Thus, we assume we have prompt æ and the first a! steps, we get the derivation for the remaining a -1);
To(a To(a--1lp [rr, alll (er(er)))
n. aa [1,, al4))
E, (1 ( ( (ll) ( e(r(()
and given prompt x and we obtain the derivation for "!
n. (a()))/)) Au(al!] (z)
(1)
EA e (al())))
r(al x) = In 1 (7((,))
(2)
The partial reward according to this formula is our definition of entropy regularized process reward. We can optimize partial reasoning step T<all|2) using this process reward.
An important property of this formulation is that process reward model can be computed by using the reference policy mo to generate the completion. In comparison, in traditional RL, the reward depends on the optimal policy that generates the completion. Therefore in the traditional RL, one has to learn reward and policy simultaneously. This is not necessary in our satropy-based approach. As one can see, equation 2 employs soft optimism over paths gencrated by the reference policy. More generally, the reward can be computed using updated policy including the optimal policy. In this case, the equivalent formula becomes
r(a (.) = - 1 -I E, Wiwe, e vr(aa))
(3)
17
4

---

Published in Transactions on Machine Learning Research (06/2025)
where we have used the following fact to express equation 2 using Rx:
To(a -r,, al"1) = T. (a- [r, a (r((a)) E Ba-lixxe e-gr(a,))
Intuitively, equation implements soft pessimism over paths generated by the optimal policy. It can be shown that the reward of this model is equivalent to the reward computed using the reference policy in equation 2 As we have already pointed out, the fact that entropy-regularized process reward model can be computed using the reference policy is a key advantage of entropy regularization approach. In this paper, we will use equation to compute our process reward models, and demonstrate advantage over previous approaches empirically.
We note that the process reward has two formulations: when completions generated from the initial policy To, it is soft-max, and when completions are generated from the optimal policy, it is soft-min. If one trains LLM and reward model simultancously using examples generated by the updated LLM, then one needs to gradually turn from soft max to soft min during the training process. For ease of understanding, we provide intuitive explanations of our approach's soft-max and soft-min perspectives in Appendix E along with a detailed analysis of the thoorctical advantages of entropy regularization for process rewards.
4 Experimental Settings
We incorporated a figure in Appendix F to further clarify experiment settings.
Process Reward Construction We conduct experiments using two widely adopted mathematical datasets: GSMSK (Cobbe ct al. 2021 h and MATH (Hendrycks et al., 2021). Our process reward data is generated and labeled through automatic labeling approach with entropy regularization, similar to Math-Shepherd (Wang et al. 2024), and is inspired by Monte Carlo Tree Search (MCTS).
In this automatic labeling process, the quality of cach step is defined by its potential to deduce the correct final answer. To enhance data diversity, we use a generator to sample 15 solutions per problem, each including step-by-step reasoning. For each step, we employ a completer to generate possible paths leading to the final answer. The quality of a given step then assessed based on the number of sampled paths that produce the correct final answer. The process reward score for each step is computed using Equation based on the initial policy no- Specifically, the reward function r(a,x) assigns a value of if the step leads to the correct final answer and O otherwise.
Process Reward Sampling We conduct experiments using Mistral-7B (Jiang al. 2023 fine-tuned on the MetaMath dataset (Yu et al. 2024) which we denote Mistral-MetaMMtth-77 and DeepSeek-math-77B instruct (Shao et al.] 2024). These models serve as the generator and completer for data collection, using the method introduced in the paragraph above. For each model, we collect approximately 260K data points and 1.5 million step annotations. Our reward models are trained using Llama-3.1-8B (Dubey et al. 2024
The reward models are trained on the following datasets:
Mistral-data:
Process reward training data, collected using Mistral-MettaMtth-7B YYu et al.
2024
both the generator and completer.
DeepSeek-data: Process reward training data collected using Baabbeel--math----nittttee Shao et al.
2024h as both the generator and completer.
In our main experiments (Section 5 we train the reward models using Llama-3. 1-BB on both Mistral-data and DeepSock-data. Additionally, Section 6.1 we investigate reward model training at different scales by using
and -.2----.-BBInsstuuee (Dubey et al. 2024h on Mistral-data.
Process Reward Model Training The reward models are trained with a global batch size of 64, a learning rate of 1e-6, maxinum sequence length of 512, and a single epoch using four H100 GPUs. We perform hyperparameter tuning on the learning rate, exploring values from -62e- -,, During
5

---

Published in Transactions on Machine Learning Research (06/2025)
the automatic labeling process, set 7 2 for Mistral data, and 17 10 for DeepSeek data in main experiments. We also conduct hyperparameter tuning for across the values (0.1, 1, 2, 5, 8, 10, 15).
Process Reward Model Evaluation We evaluate our reward models using the full test set of GSM8K and the MATH500 dataset introduced by Lightman et al. (2023). Our evaluation follows best-of-N strategy. For each problem in the test set, we randomly sample N solutions from the generators. Each step within generated response is scored by the reward model. The final answer is selected as the response with the highest overall score. The score for cach response is determined by the lowest-scoring step within it.
Bascline Scttings and Training Details To evaluate the effectiveness of our method, we compare our models against three baselines: the hard-label process reward model (Hard-label PRM), the vanilla soft-label process reward model (Soft-label PRM), and the outcome reward model (ORM).
Hard-label PRM: A reasoning step is labeled as good if it ultimately leads to the correct final answer. Soft-label PRM: The step score is approximated by the probability of reaching the correct final answer.
ORM: Assigns a single overall score to the entire reasoning process indicating correct or not.
The reward calculation methods for these models are illustrated in Figure [ We train the Hard-label PRM, Soft-label PRM, and ORM using the next token prediction, also known as auto-regressive training, Specifically, PRM is trained to predict the probability of special token '+' after each reasoning step, while ORM predicts this token at the end of the response. The reward models are optimized using cross-entropy loss between the predicted and ground-truth rewards, producing scores in the range of (0, 1). The dataset format follows Math- Math-Shepherd (Wang et al.) 2024). In addition to the next token prediction, we explore rogrossion-based training in Section [6.4 We also explore reward models with various sizes in Section 6.1 and out-of-distributiio (OOD) performance with stronger policy models as generators in Section 6.2 For each experiment settings, we repeat times with different seeds and report the average accuracy along with the standard deviation to ensure the reliability of the results.
Policy Optimization with PRM-Guided Reinforcement Learning We conduct experiments to en- Baateek--Mtth-B---nsa
hance our policy models, Mistral-MettMath-77 and ment learning from human feedback (RLHF). Specifically, we adopt the rejection sampling fine-tuning strat- egy (Dong et al. 2023) on the MATH training set and the OpenMathinstruct22 dataset (Toshniwal et al. 2024b). For each question, we generale 16 responses and rank them using our process reward model (PRM). We then select the highest-scoring responses and fine-tune the policy model on this filtered dataset. The fine-tuned policy models are evaluated on GSM8K and MATH using Top-1 accuracy under a zero-shot chain- of-thought (CoT) setting, Additionally, we compare policy models optimized with our reward model against those guided by baseline reward models. We also report the average accuracy and the standard deviation.
using reinforce-
5
Experimental Results
5.1 Buto-Regressive Training Results
We train the reward model with Tlama-3. 1--BBB as the base model. The results of our model compared with several baseline methods are shown in Figure 2 and Figure 3 which are trained in an auto-regressive way. The details are also listed in Table and Table in the Appendix.
Mistral-MettMath--b as the Generator, Reward Models trained on Mistral-data When using Mistrall-MetaMaathhaa as the policy model, we observe that N increases, the best-of-N accuracy improves for most models. Notably, ER-PRM consistently outperforms all bascline models across all evaluation settings. In general, process reward models (PRMs) outperform outcome reward models (ORMs), demon- strating the effectiveness of stepwise reward modeling. Our method shows consistent improvements, with particularly significant gains on more challenging datasots. Specifically, on MATH, which contains more complex problems than GSMSK, the performance gap widens substantially. This highlights our approach's superior ability to handle difficult mathematical reasoning tasks.

---

Published in Transactions on Machine Learning Research (06/2025)
Mistral-GSMBK
Mistral- -MTTH
Soft lobel PM
41-
Soft lobel PA
09
Hard-acel PRM CAM
Hard-aael PRM
40-
CRM
Best-of-N Accuracy
88-
Best-ofNN Accuracy
39-
30
84
16
32
64
16
32
64
Number of responses per problem
Number of responses per problem
(a) Best-of-N evaluation on GSM8K
(b) Best-of-N evaluation MATH500
Figure 2: Best-of-N evaluation results GSM8K and MATH500 datasets with Nistral-MettaMtth-bb the generator. The reward models are trained on Mistral-dar -ata (GSM8K and MATH).
Deepseek-GSMBK
Deepseek- -MTHH
90.0
05.5
PMM
Hardkabel PSM ORN
-
ORN
Best -FN--O-NN Accuracy
05.0
Best-of-N Accuracy
52-
08.0
87.5
87.0
43
16
32
32
04
Number of responses per problem
Number of responses per problem
(a) Best-of-N evaluation on GSM8K
(b) Best-of-N evaluation on MATH500
Figure 3: Best-of-N evaluation results on GSMSK and MATH500 datasets with SeepSeek--mat-122
instruct as the generator. The reward models are trained on DeepSeek-data.
the Generator, Reward Models trained on DeepSeck-data When trained on DeepSeek-data and evaluated on Trepseee-mmath--BBnnstree the results show significant accuracy boost. ER-PRM consistently outperforms Soft-label PRM and Hard-label PRM on GSMSK and MATH500. However, we observe that on MATH500, the performance improvement from N-4 to N-64 is less pronounced. Additionally, we identify reward hacking issues when using large N. Similar trends are also observed Section 6.2 where 2-luMaath--laaa.... 1-8B used as the policy model. Interestingly, ORM achieves exceptionally strong performance on GSM8K, suggesting that well-trained ORM with high-quality data can be highly effective.
We also find that Hard-label PRM exhibits strong performance in best-of-N evaluation. In most cases, especially for large N, it surpasses Soft-label PRM. We attribute this to the semi-supervised nature of the Hard-label training process. Since the Markov Decision Process (MDP) labels are only approximations of the process reward rather than precise values, the Hard-label setting helps reduce noise during training.
We note that here the Hard-label setting adopts the same labeling strategy with Math-Shepherd (Wang al. 2024), trained with the same base model as FR-PPR to ensure fair comparison. Our model outperforms

---

Published in Transactions on Machine Learning Research (06/2025)
Math-Shepaa Math-Shepherr in nearly all settings GSM8K and MATH, demonstrating the ability to better rank and select candidate responses. These results highlight the consistency and robustness of our reward models, making them more competent than existing open-source alternatives.
5.2 RLHF Results
To verify the effectiveness of our auto-regressive Fntropx-Regullrized Process Reward Model (ER-PRM), we compare it against the Soft-Label PRM and Hard-Label PRM within the context of RLHF using the RAFT algorithu {Dong et al. 2023). We employ Mistral-MetaMath-7b and BeepsSek-matll---inssttu as the initial baseline policy models and adopt the reward models trained on Mistral data and DeepSeek data, respectively.
5.2.1 Manifestation and Mitigation of Overfitting
To clarify our evaluation methodology, all results re- ported in this paper were evaluated exclusively on the test sets of GSM8K and MATH, never on train- ing data. During initial experiments, we observed that when using only the original GSM8K and MATH training sets for RLHF, the models showed limited generalization to the test sets, suggesting po- tential overfitting since these models had already been exposed to these datasets during their origi- nal supervised fine-tuning. Our analysis confirmed that Mistral-MetaMathh-7 was fino-tuned on Meta- MathQA (which incorporates GSM8K and MATH problems), while DeepSeek-Mabh-7B-nntnnne ex- plicitly included these datasets in its training cor- pus.
Models
Methods
GSM8K
MATH
Baseline
77.9
28.6
Soft-Label
78.510.3
30.410.8
Mistral
Hard-Label
79.0.1..5
31.5.0..5
ER-PRM
79.710.3
3.4
Baseline
82.0
42.8
Loft-Label
82.910.4
42.810.4
DeepSeek
Hard-Label
82.4+0.3
43.8
ER-PRM
83.410.33
14.7:10.7
Table 1: Zero-shot CoT evaluation for policy model Mistrrl--MetaMath--b and DeepSeek-mathh-11 instruct improved via Rejection Sampling Fine- tuning. Mistral and DeepSoek in the table do- note Mistral-MetaMath--)) and meepSeek-mauh-71 instruct.
To address this overfitting challenge while preserv-
ing the consistency of model behaviour, we incorporated 12,500 novel questions from the MpthMaathlsstuuttt dataset (Toshniwal et al., (024a)) into our RLHF training data conduct a blended training. This dataset contains synthetic mathematical problems and rationales that are not present in the original training data of our policy models. For each question, the initial policy model generated 16 candidate rationales (the number of candidates was selected by grid search), which were then evaluated by the ER-PRM and baseline PRMs to select the highest-scoring rationale for RLHF training. The Open data was used only for training, never for ovaluation.
5.3 Performance Evaluation
Table 1 summarizes the performance improvements achieved using RLHF enhanced via ER-PRM compared to baseline PRMs. ER-PRM outperforms the baselines, surpassing Soft-Label PRM by 0.8% and 2.7% on GSM8K and MATH, respectively, and Hard-Label PRM by 0.5% and 0.8% for the Mistral model. We observe even larger improvements for the DeepSeek model compared to the baselines, highlighting ER-PRM's superior ability to guide the more potent policy model and its robustness in handling more complex task settings. Additionally, we find that the Hard-Label setting consistently outperforms the Soft-Label setting in RLHF, with the surprisingly strong performance of Label indicating its feasibility. This is notable discovery, differing from the findings [uo et al. (024)).
In general, our RLHF approach achieved superior performance in evaluating the policy model on GSM8K and MATH test sets. This improvement can be attributed to our ER-PRM, which consistently demonstrates 1-2% higher best-of-N accuracy in selecting data from both in-domain and out-of-domain datascts compared the baselines. This highlights the stability and effectiveness of our entropy-regularized reward formulation in selecting accurate rationales from varying quality data sources.
8

---

Published in Transactions on Machine Learning Research (06/2025)
GSM8K
MATH500
Best-of-N
Ours
Soft-label PRM
Hard-label PRM
Ours
Soft-label PRM
Hard-label PRM
N=4
82.120.2
0..
81.920.2
32.310.5
32.040.6
31.520.5
N=8
82.6=0.11
82.3+0.2
82.2+0.4
34.2:t0.5
33.7+0.5
33.5+0044
N=16
83.5-0..3
83.2+0.2
82.8+0.3
37.0=:0.6
35.4++0.6
36.3+0.7
N 32
84.120.3
83.620.5
83, 83.8
36.410.8
35.940.7
36.211.1
N4
84.0
83.820.4
83.920.4
36.31:0.9
35.5:0.9
35.911.3
Topl Stace
78.09
29.4
Table 2: Best-of-N evaluation for policy model Mistral--eetaMathhhb,, The reward models are trained from Baama-3.2-1B-Insruue on Mistral data (GSM8K and MATH).
6 Analysis
6.1 Reward Models with Various Sizes
In addition to training the reward model with Llama-3.1-80 we also explore training on various sizes. this section, we train the reward model with 3.2--..32----nnstructt and Alamm-3.2-1B-Instruus (Dubey all 2024). We hope to increase the performance of smaller-size models we use the instruct-version for them instead of base models. We train both of them using the Mistral data.
The experiment results are shown in Table 2 Although we change into models of smaller sizes, the best-of-N accuracy still outperforms the Top accuracy by large margin. For example, for the 1B size model, the best-of-4 accuracy outperforms Top 1 accuracy by 4% and the best-of-64 accuracy outperforms by 5~6% on GSM8K dataset. We observe that the performance gain from best-of-4 to best-of-64 for the 1B model is much smaller than the 8B model shown in Table which has 4~66% performance gain while 1B model has only 2-33 performance gain. The comparison demonstrates that larger models have higher chances of selecting the gold answer from candidates when N is large. We also discover that the 1B model suffers from severe reward hacking problems on the MATH test set. For ER-PRM and Hard-label PRM, they both achieve an accuracy of 37.0% on best-of-16 evaluation. However, their accuracy drops about 2% on best-of-64 evaluation, showing that they are distracted by the wrong answers and could not effectively select the gold responses. The reward hacking problem becomes less severe for the casy datasct GSM8K, which probably due to the less number of wrong candidate answers.
Experiments with Llama-3.2-1B-Instee
Experiments with Llama-3.2-3B-Instree size model, it consistently oulperforms the 1B size model, showing that training with larger-size reward models would generally achieve better performance. The 3B reward model on MATH shows a competent performance as the 8B reward model (Table 5), demonstrating the fcasibility of training small-size but powerful models. We also discover that the reward hacking problem largely mitigated for the 3B size model, as the best-of-N accuracy keeps increasing on MATH as N increases, and does not drop much on GSMSK from best-of-32 best-of-64. However, we still notice that on the GSM8K test set, the performance gain from N = 4 to N 64 is still little compared to the 8B case. We may conclude that larger model size reward model would generally have larger performance gain when N increases.
The experiment results shown in Table 3 For the 3B
In both 1B and 3B size settings, ER-PRM consistently outperforms the Soft-label PRM and Hard-label PRM, demonstrating the effectiveness of our methods across various model sizes.
6.2 Out-of- Df-Diitribbutio Performance of Process Reward Models
In this subsection, we examine the OOD performance of the PRMs. The DeepSeek- Math--e-k-- (Shao al. 2024) and OpenMatth2-Llamaa.1-5 (ooshniwal al. 2024b) are used as the policy models for generation, and the reward models are used for best-of-N evaluation.
DeepScek-Math-77B.. as the policy model We firstly use DeepSock-Math-1 as the policy model for best-of-N generation. We train the reward model from Llama-3. 1-8h Mistral data. The

---

Mistral Published in Transactions on Machine Learning Research (06/2025)
GSM8K
MATH500
Best-of-N
Ours
Soft-label PRM
Hard-label PRM
Ours
Soft-label PRM
Hard-label PRM
N=4
$3.620.3
83.320.3
83.32033
35.610.6
35.240.4
35.320.7
N=8
84.1=0.4
83.7+00.5
83.5+0.4
37.7+0.88
37.1+0.8
37.4+00.5
N=16
84.5=0.33
84.0+0.2
83.9+0.4
38.9::0.6
38.2+0.7
38.3+0.6
N 32
$5.0
84.620.4
84. 4...5
40.811.1
39.6:0.9
40.12008
N-
85.410.4
85.020.4
84.810.3
41.9111.0
40.2::0.8
41.110.88
Topl Stace
78.09
29.4
Table 3: Best-of-N evaluation for policy model Mistral-MettaMathhhh,, The reward models are trained from Bamma-3.-33BInstruuee on Mistral data (GSM8K and MATH).
results are shown in Figure I and Table 10. The accuracy increases slowly when N increases from 4 to 64 on GSM8K. And the increment the MATH dataset is also slower than in the case where the policy model. We observe the performance drop for ORM reward models, indicaling its instability for out-of-distribution generated data. The reason for the smaller performance gain is that, our process reward data is generated from a weaker model Mistral-Metaaattia behavior the weak model. For certain steps where the weak model fails to produce the correct answer, the reward model is likely to assign these steps a low score. However, a stronger model may be capable of deriving the correct answer in such cases.
The reward model might only understand the
Deepseek-GSMBK
Deepseek-
30.0
ER-PAM
ERAPN
39.5
Hant-label
Hant-label
DRM
CAM
89.0
Best-of-N Accuracy
52
88.5
Best-of-N
88.0
51
87.5-
87.0
15
32
16
Number responses per problem
Number responses per problem
(a) Best-of-N evaluation GSM8K
(b) Best-of-N evaluation MATH500
Figure 4: Best-of-N evaluation results GSMSK and MATH500 datasets with DeepScok-Math-77B.R the generator. The Hard-label PRM is the same setting as Math-Shepherd.
OpenWath2-Llama3.-55 as the policy model We want to examine the reward model performance paired with one of the most powerful open-source policy models fine-tuned from open-source data. Especially, we hope to lift the performance on MATH, which difficult for most of the open-source models. We use OpenMath2-Tlamas..I
(Toshniwal et al. 2024b) as the generator, which fine-tuned from million mathematical problems and excellent in mathematical reasoning. achieves Top accuracy 66.2% of the MATH500 test set. We evaluate the best-of-N performance on the MATH500 test dataset. For each question,
OpenMath2-Llanma 2aa222-llaa..111 1-8B to generate N candidate solutions and use the reward model to select the response. We use the reward model trained from Llama-3.1 3.BBBB on DeepSeck-data.
The results are shown in Table g We first observe a satisfactory performance gain compared to the Top 1 accuracy (66.2%) for all 3 reward models. It could increase the accuracy by more than 4%. The results show that even the reward model trained with data from a weaker model, it could still guide the decoding process during the inference time. We also observe the reward hacking problem when we use the weak reward model to improve the inference of the strong policy model. As shown in the table, there is not much performance
10

---

Published in Transactions on Machine Learning Research (06/2025)
GSMBK
MATH
89
Best-ofN Accuracy
Best-oFN Accuracy
87
0-01
0-1
34-
n=
32
16
32
64
16
32
64
Number of responses problem
Number of responses problem
(a) Best-of-N evaluation on GSM8K
(b) Best-of-N evaluation MATH500
Figure 5: Best-of-N evaluation results on GSM8K and MATH500 datasets with Mistral-MettMathhTb as the generator with different 7). The reward models are trained on Mistral data on GSMSK and MATH separately.
gain from
4 to N 64. And N 64 does not perform the best. We discover that the ER-PRM method still improves the policy model most, demonstrating the feasibility of our method when implemented with strong models.
6.3 Impact of Entropy Regularization
GSMBK
MATH
42
El-PoM--uutufup sooive
Rott-Segresaton
Soft label PAM
-
Hardkabel P3M4
Best-of-N Accuracy
Best-of-N Accuracy
88-
38
30
ER-MIM- ERoIIM Autofligreasiee
ER-PRM Regression
16
32
64
16
32
64
Number of responses per problem
Number of responses per problem
(a) Best-of-N evaluation on GSM8K
(b) Best-of-N evaluation MATH500
Figure 6: Best-of-N evaluation results GSM8K and MATH500 datasets with Mistral-MottaMthh-hh as the generator. The reward models are trained using regression. They are also compared with the auto- regressive training strategy. The reward models are all trained on Mistral data.
To better interpret our autropy-rouularize method, we change the hyper-paramcter to regularize the penalty the entropy during automatic process reward labeling. We train the reward model on GSM8K and MATH dataset separately, instead of training on both datasets the main experiments. We then evaluate our models using the best-of-N strategy on cach datasct. Note that when n 0, the labeling stratogy is vanilla soft labeling. When 7 + oo, the labeling strategy hard labeling. We show the performance change n changes, and intend to find the optimal y for these two datasets.
11

---

Published in Transactions on Machine Learning Research (06/2025)
The result is shown in Figure| 3 for best-of-N performance on each 7). We observe that there is peak performance for each dataset and then the performance drops when y is too large small. The experiments show that = 1 or = 2 is the best for hard dataset MATH. 15 is the best for easy dataset GSM8K. Larger 7) represents a larger penalty on the entropy for the process rewards where they are pushed higher and closer to the hard label estimation, while smaller represents a small penalty on the entropy for the process rewards. We conclude that to achieve the best performance on automatic process reward labeling, a more fine-grained estimation of the reward scores should be applied. Simply applying hard label estimation may introduce too much noise for reward model training. We also notice that vanilla soft label estimation may not be an optimal strategy, as the performance is relatively bad when 1y is small (n = 0.1) for both datasets.
Considering the difficulty the GSM8K and MATH datasets, We may need a larger entropy penalty for the process reward for the hard dataset, and a relatively smaller entropy penalty estimation for the easy dataset.
6.4 Regression Training Strategy
In addition to training the reward model in an auto-regressive way, we also explore the reward models with the regression training strategy, where we train the model to output a score directly given a problem and current steps. The best-of-N evaluation results are shown in Figure
The performance best-of-N increases as N be- comes larger. Our reward model still outperforms the Soft-label and Hard-label scttings. We also iden- tify it as method to train reward models. However, we notice that auto-regressive training could yield better results for reward models than regression training. On both GSM8K and MATH datasots, the best-of-N accuracies are consistently lower than the models trained using the auto-regressive strat- egy. Especially MATH, there is significant gap of the accuracy between the two training methods. One hypothesis is that auto-regressive training inte- grates the steps, reducing the total number of train- ing steps, which in turn leads to less forgetting. For the MATH dataset, since there are more reasoning steps per question, splitting them into multiple training samples increases the difficulty for the reward models learn. It also possible that the reasoning steps in auto-rogrossive training are continuous and coherent, allowing the reward model to better learn the process of assigning rewards.
Best-of-N
Ours
Soft-label PRM
Hard-label PRM
N-4
60.140.4
67.5400.6
69.2
N=8
70.420.6
69.1100.6
69,210.5
N=
70.5+0.5
69.340.8
09.70066
N-32
70.3++0.4
68.8+0.77
60.6+0077
N4
70.41:0.6
63.3100.5
68.810.4
Table 4: Best-of-N evaluation on the MATH500 test set. The reward models are trained from Llama-3.1- 8B on DeepSeck data. The policy model for genera- lion is OpenMatha-Llana33.55
7 Conclusion
In this paper, we demonstrate the potential of process reward models (PRMs) in enhancing the performance of large language models (LLMs)) on complex mathematical reasoning tasks. By leveraging the fine-grained feedback provided by PRMs, we achieve a notable improvement over traditional outcome reward models (ORMs), as PRMs better capture the multi-step reasoning required for such tasks. To enhance the process reward model, we formulate the multi-step reasoning in the entropy perspective and then derive the process reward score by new aggregation method. The effectiveness of our automatic labeling process rewards, coupled with entropy-regularized reinforcement learning techniques, highlights the importance of balancing reward optimization with policy stability. Our model outperforms baseline ORMs and PRMs by a large margin on the MATH and GSM8K benchmarks. Furthermore, we apply our process reward model improve the policy model under RLHF experiments and observe a significant improvement after rejection sampling.
12

---

Published in Transactions on Machine Learning Research (06/2025)
References
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac HalfieldDDodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022. URL https://aaxxa cerrziiv.orggaee 2204/ 05862
Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Alphamath almost zero: process supervision without process, 2024a. URL http:://aaxxivorg//aaa..
03553
Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Step-level value preference optimization for mathe- 10858
matical reasoning, 2024b. URL
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https: /ttp:://axxivvrroi wwwgg//b////..../////// 2210. 14168
abs
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative loundation model alignment. Xiviv preprint arXiv:234.00670 2023.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rihf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankunar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aure- lien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Al- lonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dicuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Geor- gia Lewis Anderson, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Ilyyan Zarov, Imanol Arricta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Gelfert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Ji- awen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika. Tyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala-Yeary Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pa- supuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krish- nan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Caldcrer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Railcanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hos- seini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nic, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Syd- ney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias
13

---

Published in Transactions on Machine Learning Research (06/2025)
Speckbacher, Todor Mihaylov, Tong Xiao, Uijwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ra- manathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xin- feng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaci, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yumus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Elconora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Med- ina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Her- man, Grigory Sizov, Guangyi, Zhang, Guna Lalshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Ge- boski, James Kohli, Japhet Asher, Jean- Baptitte Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shopard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjic Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veer- araghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Mar- tynas Mankus, Matan Hasson Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Maccy, Mike Wang. Miquel Jubert Hermoso, Mo Mctanat, Moham- mad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Raf Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Ro- han Maheswari, Russ Howes, Ruty Rinolt, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswany, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sueha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vitor Albiero, Vlad lonescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The llama 3 herd of models, 2024. URL /stps://arx/// org/ hbs///b/////
abs//407 2407 21783
14

---

Published in Transactions on Machine Learning Research (06/2025)
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen, et al. Tora: A tool-integrated reasoning agent for mathematical problem solving. or Xie preprint or orXim:2309 1 17452. 2023.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the math datasct, 2021. URL https:
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https://arxiv. htttp:://axxiv.or//bbba. 20100.0225200
Fangkai Jiao, Chengwei Qin, Zhengyuan Liu, Naucy F Chen, and Shafiq Joty. Learning planning- based reasoning by trajectories collection and process reward synthesizing. arXiv preprint arXiv:2402.00658 2024.
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms, arXiv preprint arXiv:2406. 186299 2024.
Hunter Lightman, Vineet. Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step, 2023. URL
abs/2305. 20050
Haoxiong Liu and Andrew Chi-Chih Yao. Augmenting math word problems via iterative question composing. arXiv preprinl ar Xiv:22200.000 090000, 2024.
Liangehen Luo, Yinxiao Liu, Rosanne Liu, Samral Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. Improve mathematical reasoning in language models by automated process supervision, 2024. URL https://arxiv. .rgg abs//406. 06592
OpenAT, Josh Achiam, Steven Adler, Sandhiní Agarwal, Lama Ahmad, Igge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Cascy Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Docarcaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Fcoffet, Atty Fleti, Tyna Floundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lppes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanic Lin, Matcusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniy Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mchta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiihhro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe,
15

---

Published in Transactions on Machine Learning Research (06/2025)
Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng. Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alcthea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra. Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarclli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Ben- jamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Tlya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wicthoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL https:://arxiii org/
Long Ouyang, Jell Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddic Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowc. Training language models to follow instructions with human feedback, 2022. URL https: ssttps//arrivy wwwpii..rg//bbbae. org/abs/2203, 02155
Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston Iterative reasoning preference optimization. arXiv preprint 0rXiv:240,.19793, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arxiv preprint vrxiv:17011 1117777773333 06347, 2017.
Zhihong Shao, Poiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv www./bgg///////e... abb//4bs///222/ 03300
Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He. Dart-math: Difficulty-wwaee rejection tuning for mathematical problem-solving. 2024.
Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinstructt Accelerating ai for math with massive open-source instruction data, arXiv preprint 1rXiv:2410001500 2024a.
Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. Opeumathinstruct-l: A 1.8 million math instruction tuning dataset. arXiv preprint arXiv:2402.10176, 2024b.
Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce 1mms step-by-step without human annotations, 2024. URL https: http://rgg/bb//////22 08935
Jason Wei, Xuczhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fci Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thoughu prompting elicits reasoning in large language models, 2023. URL
https://arxiv.org/eeee
Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shich. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint 0r50::2455000451, 2024.
16

---

Published in Transactions on Machine Learning Research (06/2025)
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human leedback: Bridging theory and practice for rlhf under kl-constraint, 2024a. URL
www.////g.......a.a/..// 11456
Wei Xiong, Chengshuai Shi, Jiaming Shen, Aviv Rosenberg, Zhen Qin, Daniele Calandriello, Misha Khalman, Rishabh Joshi, Bilal Piot, Mohammad Saleh, Chi Jin, Tong Zhang, and Tianqi Liu. Building math agents with multi-turn iterative preference learning, 2024b. URL https://arxii http://wwww..gga//////.........
Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprinl. ar Xrx::2331. 16682, 2023.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via sell-improvement, 2024. URL https: org/ abs/ 2409. 12122
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint ar arXiv:2909. 1 12284, 2023.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your mathematical questions for large language models, 2024. URL /ttpss//arxtie htt///xxvv.oro//aaaa// (rrg/abs/2300 12284
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, et al. Advancing Ilm reasoning generalists with preference trees. arXie preprint arXiv:2404.2000 2024.
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint ar Xrr::2230 01125, 2023.
Xiang Yue, Ge Zhang Xingwei Qu, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist. models through hybrid instruction tuning. arXiv preprint or Niw:22009005533 2023.
Di Zhang, Xiaoshui Huang, Dongzhan Zhou, Yuqiang Li, and Wanli Ouyang. Accessing gpt-4 level mathematical olympiad solutions via monte carlo tree self-refine with llama-3 8b, 2024. URL. https:
Serriv.org//bb////0
Tong Zhang. Mathematical analysis of machine learning algorithms. Cambridge University Press, 2023.
Han Zhong, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf, 2024a. URL httts:// wwwpiiiv.rg//b///b/ee/ee/ wttg/a///g/.......///////// 18922.
Han Zhong, Guhao Feng, Wei Xiong, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprint arxiv:2404 18922, 2024b.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe Le, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprinl ar Xrx:222005. 10625, 2022.
Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang, and Yujiu Yang. Solving math word problems via cooperative reasoning induced language models. arxiv preprint 22Xiv:2210. 0. 16257, 2022.
17

---

corresponding to the GSM8K and Published in Transactions on Machine Learning Research (06/2025)
A Ethical Statements and Limitations
Our work enhances mathematical reasoning in LLMs using entropy-regulatized process reward model (ER-PRM), and we have carefully considered ethical concerns in data usage and algorithmic design. For data sources, our experiments rely on publicly available human preference datasets, ensuring compliance with their respective licenses. These datasets do not involve privacy issues, and we adhere to ethical guidelines in their use. Algorithmically, while ER-PRM improves reasoning stability, reward modeling can influence LLM behavior, potentially reinforcing unintended biases in problem-solving approaches. Additionally, not carcfully managed, preference learning could lead to overfitting certain reasoning styles, limiting model generalization. To mitigate these risks, encourage rigorous testing and diverse dataset construction. Lastly, advanced reasoning models pose potential misuse risks, such as academic dishonesty or exploitation in high-stakes decision-making. We recommend ongoing safety evaluations and responsible deployment to cnsure robustness and fairness in real-world applications.
ER-PRM has several limitations. High computational cost may hinder scalability, and overfitting risks require careful tuning, particularly in DeepSeek-Mand MattseeekMMath---ee Maath-7B-Instruct, Our approach is setthmmmtics-foouseer and its generalization to other domains remains untested. Finally, reward aggregation mechanisms could benefit from more interpretability. Addressing these issues will improve the model's robustness and applicability.
B
Technical Lemma
Lemma 1 (Solution of KL-regularized Optimization) Given loss functional with respect to p(11x),
written
Ewwr(()) (ww) + mDD.. (p(D))((()).n))))))) po())
+nDRI (p()).po(-) exp ((())
log (.--po))) exp (www)),
where the minimizer of the loss functional p" p"() Po(w) exp (tU(w)). also known as Gibbs distribu- tion.
See Proposition 7.16 and Theorem 15.3 of Zhang 2023 for detailed proof.
C Best-of-N Evaluation Results in Tables
In this section, we list the performance of the best-of-N evaluation of ER-PRM and other baselines Tables. The results for the reward models with policy model Mistral-MettaMthh-77 which corresponds to Figure 2 The results for the reward models with policy model Deepseek-Mathh-7B
shown in Table 5
Instruct are shown in Table 6 which corresponds to Figure 3
The results for Section 6.3 in Analysis are shown in Table and Table
MATH datasets. The results for Section 6.4 are shown in Table
D Definition of Step, Partial Chain, and Chain
Step:
A step refers to the smallest unit operation in a reasoning process. Each step performs a single operation (o.g., a single calculation or logical deduction) and represents one incremental action within the overall reasoning process.
18

---

Published in Transactions on Machine Learning Research (06/2025)
GSM8K
MATH500
Best-of-N
Ours
Soft-label PRM
Hard-label PRM
ORM
Ours
Soft-label PRM
Hard-label PRM
ORM
N-4
86.0+00.3
85.740.2
86.5+004
84.240.8
34.8+-0.8
33.9+-0.5
34.440.7
34.3+0.5
86.740.4
86,94003
87.0+0.3
86.940.7
36.720.7
36.640.8
36.3=0.6
36820.6
N=16
88.920.4
87 840.4
2203
140.5
39.14111
38.62006
67.520.6
N=
89.3=0033
3..33
88.829.3
87 67
40.52:0.8
38.820.7
39 8:0..6
39.220.5
N-64
0..
88.9.10.3
89.1100.
88.110.6
41.3.10.6
39.710.8
40.1.10.8
39.310.6
Top18noc
78.00
29.4
Table 5: Best-of-N evaluation for policy model Mistral-MetaMatth-7 The reward models trained using auto-regressive strategy from Llama-3.1-8B on Mistral data (GSM8K and MATH).
GSMSK
MATH100
Best-of-N
Ours
Soft-label PRM
Hare-label PRM
ORM
Ours
Soft-label PRM
Hard-label PRM
ORM
N=4
86.720.3
86.3300.4
86,520.4
86.210.3
50.820.4
50.210.9
50.520.6
49.220.5
N-8
8..3:.055
87.0.10.4
87.210.3
87.010.4
52.11:0.5
50.810.77
51.8
50.1
0.5
88.1
58.0+-03
87.940.3
50.+1.0
52.740.3
N-32
80.2+0033
88 94004
88.7+0.6
88.7+0.4
63.1+-0.5
51.2+009
62.5+0.5
51,6+055
N=61
90.0=0.1
G+0.3
89.3+0.4
89.5
53.5+0.5
50.9+0.8
53.1
51.50066
1op1
82.03
42.8
Table 6: Best-of-N evaluation for policy model DeepSeel-Matth- The reward models are trained from Llama-3.1-8B on DeepSeek data (GSM8K and MATH).
GSM8K
Best-of-N
0.11
= 0.1
1 =
2
=
5
=
8
7) = 10
1) = 15
= 20
N-4
83.93
84.69
84.99
84.38
85.22
84.69
84.91
84.53
N=8
85.14
85.82
86.43
86.35
86.73
86.28
86.66
86.43
N=16
85.29
85.75
86.05
87.11
87.57
87.04
87.72
87.04
N=32
86.13
85.52
86.43
87.72
87.87
88.02
88.78
87.11
N=64
86.13
85.90
86.35
88.17
88.25
88.48
88.78
87.19
Table 7: Best-of-N evaluation for different y) values on GSM8K dataset. The reward models are trained from Hlama-3.1-85 solely on GSMSK.
MATH
Best-of-N
= 0.1
n
1)
2
n 7= 5
8
17
10
1)
15
1
20
N=4
35.0
34.0
35.0
33.2
32.4
33.2
33.2
31.8
N=8
35.8
37.2
39.0
35.0
36.0
35.8
35.6
33.8
N=16
39.0
38.6
10.0
37.0
39.4
38.6
39.0
37.4
N=32
39.8
39.8
40.2
38.6
39.2
39.4
38.6
38.8
N-64
38.6
41.0
40.2
37.4
38.2
40.0
40.2
39.4
Table 8: Best-of-N evaluation for different 7) values on MATH dataset. The reward models are trained from Llama-3.1-8B solely on MATH.
GSMSK
MATH500
Best-of-N
Ours
Soft-label PRM
Hard-label PRM
Ours
Soft-label
PRM
Hard-label PRM
N=4
84.84
84.60
84.60
33.6
31.4
31.8
N-8
86.80
86.05
86.73
33.6
33.0
N=16
87.95
87.41
87.41
36.2
35.0
35.6
N=32
88.40
87.87
87.71
37.0
35.0
36.6
N-64
88.40
88.24
88.17
36.8
34.2
36.4
Table 9:
Best-of-N evaluation for policy model Mistral-MetaMMthhTTh The reward models trained using regression stralegy from Alama-3.1-8) on Mistral data (GSM8K and MATH).
19

---

Published in Transactions on Machine Learning Research (06/2025)
GSM8K
MATH500
Best-of-N
Ours
Soft-label PRM
Hard-label PRM
ORM
Ours
Soft-label PRM
Hard-label PRM
ORM
89.39
88.77
52.6
51.4
52.6
N
89.91
89.61
80.09
54.8
55.0
N=16
90.22
90.22
89.76
89.01
55.6
54.8
55.4
55.2
N=32
90.45
90.45
89.84
89.01
56.0
53.0
55.0
55.0
N-61
90.67
90.45
89.84
88.10
57.0
53.4
53.6
53.4
Topc
88.17
51.0
Table 10: Best-of-N evaluation for policy model DeepStek-Mathh-7B-R. The reward models are trained using auto-regrcssive strategy from Llama-3.1-8B on Mistral data (GSM8K and MATH).
Chain:
A chain represents the complete series of operations that culminate in the final result or target solution. It consists of multiple sequential steps, forming the full reasoning process.
Partial Chain:
A partial chain is subset of a complete chain, consisting of sequence of intermediate steps leading toward (but not necessarily reaching) the final result. Partial chains are used in our work to compute intermediate reward scores that guide the reasoning process step-by-step.
In our paper, when we refer to "step-level derivation," " we are specifically describing the last step reward score associated with partial chains. These scores are step-level reward scores and not chain-level reward scores, as they evaluate the contributions of individual steps (or intermediate sequences) rather than the final output of the entire chain. The distinction is important because step-level scores are critical for guiding the model each reasoning step, while chain-level scores evaluate the overall success of the complete reasoning process.
E Intuitive Understanding and Theoretical Advantages
E.1 Intuitive Perspective Rngraay-Regullriized Process Rewards
We provide intuitive explanation of entropy-regularized process rewards:
When using initial policy model To (a weaker SFT model like 7B parameter LLM) to generate reasoning paths, our formulation essentially answers the question: 'If we continue from this step using our current knowledge, what's the likelihood reaching a correct answer?" The entropy regularization term (controlled by 17) determines how optimistic this assessment should be. Higher 7 values make the model focus more on the highest-reward paths, creating a 'soft-optimmstic" evaluation that weights multiple possible futures according to their potential.
The Soft-Max View:
The Soft-Min View: Conversely, when using optimal policy model т. (a strong model like GPT-4o), our formula addresses the question: 'What's the smallest penalty we would need to apply to make this step as good as other alternatives?" This creates a 'soft-pessimistic" evaluation that focuses on the worst-case scenarios but with some forgiveness. This approach employs a more cautious strategy to align with the preferences of a stronger model.
Mechanics of Reward Aggregation: To illustrate how reward aggregation works in practice, consider the step a in Figure From this step,
three possible continuation paths reaching LbI, L222 L3.3, We caleulate the correctness of each path individually, and get y1 = =1, 3/2 1, and y/3 0 since the first two paths derive the correct final answer. Once we get the value of different we are able to calculate the expected reward score using the equation provided in Figure I
20

---

Published in Transactions on Machine Learning Research (06/2025)
E.2
Intuitive Understanding and Theoretical Advantages
Building on this intuitive foundation, now articulate the key theoretical advantages of our entropy- regularized approach:
(1) Dual Formulation Flexibility:
Our process reward approach offers a significant theoretical advantage through its dual formulation. When sampling completions from the initial policy no, our process reward employs soft-max aggregation equation equation - Conversely, when sampling from the optimal policy T, it implements soft-min aggregation equation equation 3
This duality allows our method to adapt smoothly depending on the sampling policy, When sampling from a weaker initial policy To, the soft-max formulation acts as an optimistic estimator that amplifies high-reward paths, creating stronger learning signals. When sampling from a stronger optimal policy тx, the soft-min formulation provides conservative estimate that focuses on the most reliable paths while maintaining diversity.
(2) Reward Signal Amplification:
Traditional methods calculate rewards as simple expectations under the initial policy not r(a-") Ex. [aaaa"]] While straightforward, this approach can result in weak signals when un has a low prob- ability of sampling high-reward trajectories, which common in complex mathematical reasoning tasks where correct solutions form small subset of solution space.
Our regropy-regnllarized formulation addresses this through the exponential transformation @ which amplifies the reward signal before expectation. This transformation effectively redistributes probability mass toward high-reward trajectories, making the learning signal stronger and more informative. The subsequent logarithmic transformation maintains the relative ordering of rewards while controlling the scale.
(3) Independence of Optimal Policy with Single Hyporparameter:
A key theoretical advantage of our method is that it requires tuning only single hyperparameter y that controls the entropy penalty, providing an elegant way balance the exploitation between high-reward trajectorics and the brcadth of policy space.
This contrasts with previous approaches that often require complex weighting schemes or multiple parameters to achieve similar effects. Furthermore, our reguaazee process reward can be computed using only the initial policy Tu, as demonstrated in equation equation This removes a dependency present in previous RI. methods, where the optimal policy needed.
21

---

Published in Transactions on Machine Learning Research (06/2025)
F Figure of Experimental Setting
Phase 1: Process Reward Data Construction
Phase 2: Process Reward Modal Training
Phase Evaluation
Generator Completer Models
Datasete
Dase Model Selection
Bent of Dvakuation
Sample solutions, select highest scoring response using trained reward model. Using accuracy directly
Mistral- MctaMath-22
Expecimente), Llama-3.2 10/30 (Scale Analysia Experiment)
Deepseek Math- Instuccttuuurte
Selution Generation
Trakiin Configuration
RUHF
Batch Slas: Learning Rate: 1a-6, Deepseek
Optimize policy model using rejection sampling fine- tuning with trained Reward Medels. Using per performance gain of pulicy model evetuate the reward modilts
problem step by step ressoning
Completinn
Analysis
n (0.1 1, 15),
For each step, generate poesible completion patha final answer
Learning Rate (3a-6, 2a-8, 1e-8, 5e-7) Using Orid Search
Test models: Seep Marteek---atht---o
OpenMath2 -Llamaa..1118 Evalliate performance across
sizes (18,
Reward RewardCaloubbn Caloutation
Banelines
Using to calculinee reward and select appropriate
ORM - Outcome Roward Model
rationale answer candidate
All experiments are evaluated on the test ents GSMMK and
MATH
Hard-Laabll PRM Label
results
deviation
Mistrall Dota
Deepanmii- Data
Collected using Miettal-ieecaMath-7888 Completer
c
Soft-Label PRM - Label by probability of reaching
Figure 7: Figurative Illustration of Experimental Setting
22


Reviews:
[{"review_id": "hXFCaUsGVo", "reviewer": "Reviewer_74eA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n--------------------\n- The paper introduces an entropy-regularized process reward model (ER-PRM) that builds on existing process reward models by incorporating KL-regularized MDPs. This approach effectively balances reward optimization with policy stability, ensuring that the model does not deviate excessively from its initial distribution. The theoretical grounding and mathematical derivations provide a solid foundation for this method.\n\n- The proposed method is evaluated on two widely used mathematical reasoning benchmarks, MATH and GSM8K, demonstrating consistent improvements over baseline process reward models. The results indicate that ER-PRM enhances reasoning performance, particularly on complex mathematical problems, with 1% improvement on GSM8K and 2-3% improvement on MATH under best-of-N evaluation. The additional experiments with RLHF fine-tuning further highlight the model’s ability to improve reasoning accuracy in LLMs.\n\nWeaknesses or Concerns:\n--------------------\n- The proposed ER-PRM method lacks significant contribution and novelty. It is essentially a variant of the soft-label PRM, incorporating KL-regularized MDPs to prevent the model from deviating excessively from its initial policy during optimization. Furthermore, the percentage improvement on the MATH and GSM8K datasets is not substantial.\nThe paper should also compare the results of the proposed method with those of existing state-of-the-art approaches on MATH and GSM8K. Additionally, providing a comparison with current state-of-the-art methods and highlighting how close the proposed results are to them would offer valuable insights for readers.\n\n- Sections 4 and 5 require substantial improvement in terms of structure and clarity. The authors should create distinct subsections (or use paragraph headings) to separately discuss the different generator models and reward models used. There are inconsistencies regarding which LLM was used as the reward model (see the point below). Additionally, Section 4 does not justify the choice of LLMs. Without a clear justification, it remains unclear whether the proposed ER-PRM method is generalizable to other LLMs.\n\n- The paper contains grammatical errors, such as in the sentence: \"When the models are trained on deepseek data and evaluate on deepseek-math-7b-instruct...\" Additionally, some sentences are poorly constructed, such as the first two sentences in the second paragraph of page 7.\n\n- The paper states: \"The reward models trained on data from Mistral and deepseek are denoted as...\" In Section 5, the authors repeatedly refer to \"deepseek data.\" It is recommended to cite appropriate sources, as \"deepseek data\" is a vague term given DeepSeek's recent release of multiple models and datasets.\n\n- In Section 4, the authors state: \"We train our reward models using Llama-3.1-8B.\" However, Table 2 and Table 3 present results for reward models trained on Llama-3.2-1B-Instruct and Llama-3.2-3B-Instruct, respectively. This discrepancy should be clarified.\n\n- In Section 2, the authors mention: \"However, PRMs require extensive human annotation, which is challenging for open-source efforts.\" In this paper, the authors use an LLM as the reward generator. However, how reliable is this reward modeling? Since the reward generator is not inherently sound, it may assign low rewards even to high-quality responses. This also raises concerns about generalizability, as noted earlier.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces an Entropy-Regularized Process Reward Model (ER-PRM) for improving mathematical reasoning in LLMs. It builds on process reward models (PRMs), which assign rewards to each reasoning step rather than just the final answer. The core idea is to use KL-regularized Markov Decision Processes (MDPs) to ensure that the model does not deviate too much from its initial policy while optimizing for reasoning correctness. The reward formulation uses entropy regularization, balancing reward maximization with policy stability.\n\nThe paper evaluates two generator LLMs viz., Mistral-7B fine-tuned on MetaMath dataset and DeepSeek-Math-7B-Instruct. The reward models used are Llama-3.2-3B and Llama-3.2-1B. The results demonstrate superior reasoning accuracy over baseline reward models like hard-label PRM, soft-label PRM and outcome reward model (ORM), on the MATH and GSM8K datasets. The authors also use ER-PRM in RLHF training, yielding further gains in reasoning accuracy. The models are evaluated on MATH and GSM8K datasets, w.r.t. the best-of-n and top-1 accuracy. Overall, the proposed method achieves 1% improvement on GSM8K and 2-3% on MATH under best-of-N evaluation. It is observed that ER-PRM consistently outperforms these baselines, especially for complex reasoning tasks. This approach strengthens LLMs' stepwise reasoning capabilities by ensuring a stable optimization process and providing fine-grained step-level supervision using entropy-regularized reward functions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Refer weaknesses.\n\nThe paper introduces an entropy-regularized process reward model (ER-PRM) with a solid theoretical foundation and some improvements in mathematical reasoning tasks, though the gains are small. However, it lacks originality as it primarily builds on existing methods with KL-regularized MDPs. Additionally, the paper needs stronger comparisons with top-performing models, clearer organization in the experimental and results sections, and a better justification for the choice of models. These issues should be addressed in the paper.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wsotlC41dr", "reviewer": "Reviewer_kNgM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nThis paper proposes a novel method to train PRMs based on entropy regularization.\nThe method is tested on various LLMs ranging in size, tested on various datasets ranging in difficulty, and compared to other traditional baselines. Overall the proposed method seems to outperform all other methods, yielding strong PRMs for both inference time filtering and training time filtering during rejection sampling rlhf cycles.\nThe method is also tested on “unseen” models where the policy model is different than the model that generated training data for the PRM.\nEventually, the authors analyze the effectiveness of their regularization in an ablation study and discuss the auto-regressive -vs- regression training strategies.\n\n# Weaknesses\n\n1. The proposed ER-PRMs outperform other baselines by 1 or 2% points, however without any confidence intervals or standard deviations across multiple runs with different random seeds, this outperformance can be due to random noise. Results must include confidence intervals to see if the difference is statistically significant.\n\n2. The claim (made in Section 5.1) that the proposed method is especially effective on complex tasks is not well supported by evidence. The authors refer to figures 2b and 3b to support this claim. While Figure 2b seems to go in that direction with ER-PRMs outperforming all other baselines (gains from ~39% to ~42% when N=64), the numbers in Figure 3b are much more nuanced. ER-PRM only marginally performs better than Hard-label PRM with gains from ~52.5% to ~53% when N=64.\n\n3. Minor: While it is interesting to see best-of-n approaches performing better than top1 accuracy, the experimental section should not focus too much on this as it is generally expected that any best-of-n approach will yield better performance than top1 accuracy. Since this paper introduces a PRM meant to judge the top N responses, the readers should focus on this. When we focus on other best-of N baseline models, the proposed method is only marginally better (from 0.5 to 2% overall).\n\n# Questions\n\nIn section 5.2, the authors mention that they observed that the baseline policy model was overfitting to the training data, resulting in only marginal performance gains after RLHF on the original training set. My question is why did they evaluate on the training set and not the test set?\nThe authors then highlight that they added 12,500 questions from OpenMAthInstruct-2, but this dataset is not listed in the result table 1. Did they added these examples to the train or test set?\n\nOverall, this makes me question the rest of the results in the paper, was everything evaluated only on the training set and not on hold-out test sets? Further clarifications is needed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces entropy-regularized process reward models (ER-PRMs), which are trained similarly to PRMs but have an additional KL-regularized term in their loss function to balance optimizing the reward and staying close to the original policy.\n\nThe ER-PRM are evaluated in two scenarios: as inference-time judges to pick the best-of-N response from existing policy models (Mistral-MEtaMath-7B and deepseek-math-7b-instruct); and as training-time reward models to improve policy models in RLHF cycles. The final policy models are then evaluated on GSM8k and MATH and compared to models guided by different baseline reward models. \n\nOverall the proposed method achieves slightly better performance compared to other baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Add standard deviations to results in all tables, and confidence intervals to results in all figures for Claims And Evidence to be evaluated as \"yes\"\n\n- Be more nuanced in some claims with words like “significantly” being justified only by a few percentage points.", "broader_impact_concerns": "no concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HtRc0OaX4o", "reviewer": "Reviewer_KBAn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**:\n\n1. The integration of entropy regularization into process reward modeling is a fresh approach, addressing the stability of policy optimization in multi-step reasoning tasks.  Derives reward formulations from KL-regularized RL principles, grounding the method in established theory.\n\n2. The paper presents empirical evidence of the effectiveness of ER-PRM. The results on the MATH and GSM8K benchmarks show consistent improvements over existing methods, highlighting the efficacy of the proposed approach. Comprehensive experiments across datasets, model sizes, and training strategies (e.g., auto-regressive vs. regression) demonstrate robustness.\n\n3. The authors provide a solid theoretical foundation for their method, deriving the reward construction method from KL-regularized MDP. This theoretical analysis strengthens the credibility of the proposed model.\n\n4. The paper includes a wide range of experiments, including best-of-N evaluations, RLHF results, and out-of-distribution performance tests. These experiments provide a comprehensive evaluation of the model's capabilities and robustness.\n\n**Weakness**:\n\n1. Section 3 assumes familiarity with KL-regularized RL, which may hinder accessibility. Intuitive explanations or visualizations of the reward aggregation process (soft max/min) are sparse.\n\n2. The paper mentions signs of overfitting in some of the experiments, particularly with the deepseek-math-7b-instruct model. A more detailed analysis of overfitting and strategies to mitigate it would be beneficial.\n\n3. The experimental results are not significant. Could the advantages of ER-PRM be further explained at a theoretical level?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method for enhancing the mathematical reasoning capabilities of LLMs through an entropy-regularized process reward model (ER-PRM). The authors integrate KL-regularized MDP to balance policy optimization while preventing the policy from deviating too far from its initial distribution. The key contributions include:\n\n1. Entropy-Regularized Process Reward Model (ER-PRM): A new method for labeling process reward scores with entropy regularization, derived from theoretical results.\n\n2. Empirical Validation: Experiments on the MATH and GSM8K benchmarks demonstrate that ER-PRM outperforms existing process reward models, achieving significant improvements in reasoning accuracy.\n\n3. Theoretical Analysis: The paper provides a detailed derivation of the reward construction method and shows that the optimal reward model can be derived from initial policy sampling.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical**:\n\n- The authors should add a section discussing the ethical implications of their work, including potential biases, fairness, and the broader impact of using entropy-regularized process reward models in LLMs.\n\n- Explain the advantages of the method from a theoretical perspective.\n\n**Recommended**:\n\n- A more detailed analysis of overfitting, including strategies to mitigate it, would strengthen the paper. This could involve discussing regularization techniques or providing additional experiments to demonstrate the model's robustness.\n\n- Expand the Limitations section to discuss computational overhead, scalability, and generalization to non-mathematical tasks.", "broader_impact_concerns": "No concerns. The authors mention the potential for reward hacking and the challenges associated with training reward models, but an explicit Broader Impact Statement would be helpful to highlight these issues.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission received three detailed reviews, all of which asked for clarifications and highlighted strengths and weaknesses of the paper. The authors responded in detail to those and updated the paper accordingly. All three reviewers finally recommended the acceptance of the submission. Nevertheless, two of the reviewers still remarked on the limited novelty and significance of the submission which is, however, not a central decision criteria.

Hence, I am suggesting acceptance of the paper with minor revisions, which are detailed below and are based on the reviewers' comments but also my own reading of the submitted paper:
* Although grammar of the manuscript improved a lot already, another careful update of the paper in that regard should be conducted.
* Update section 3.2 to be easier comprehensible and have a clear logical structure (e.g., currently the authors start by saying "we define the reward by ..." just to say we obtain "..." (essentially the assumed reward function). Later they say that any policy can be computed using any policy and state an equation for the reward. Please show how you arrive at this equation. Also clarify, and this is important as you extensively remarked on this in the discussion with the authors, how the difference in the equations for the reference policy and optimal policy come about. Also explain what happens in the mentioned case that "More generally, the reward can be computed using any policy including the optimal policy" or adjust this statement.
* Clarify the computation of the confidence intervals for the best-of-n approach.
* Include confidence intervals in all results tables (in particular, also in Tables 1 and 4), and, if possible, also in the figures.
* The clarity of the experimental setting could be improved by showing a chart or similar depicting how all components and models come together.
Decision: Accept (poster)