Let's Verify Step by Step
Authors: Hunter Lightman
Vineet Kosaraju
Yuri Burda
Harrison Edwards
Bowen Baker
Teddy Lee
Jan Leike
John Schulman
Ilya Sutskever
Karl Cobbe
Venue: ICLR 2024 poster (2024)
Keywords: LLMs
chain of thought
verifiers
human feedback
reasoning
Primary area: generative models
TL;DR: We use step-level human feedback to build a robust verifier of LLM reasoning that gets 78% accuracy on a subset of the MATH test set.

Abstract: In recent years, large language models have greatly improved in their ability to perform complex multi-step reasoning. However, even state-of-the-art models still regularly produce logical mistakes. To train more reliable models, we can turn either to outcome supervision, which provides feedback for a final result, or process supervision, which provides feedback for each intermediate reasoning step. Given the importance of training reliable models, and given the high cost of human feedback, it is important to carefully compare the both methods. Recent work has already begun this comparison, but many questions still remain. We conduct our own investigation, finding that process supervision significantly outperforms outcome supervision for training models to solve problems from the challenging MATH dataset. Our process-supervised model solves 78% of problems from a representative subset of the MATH test set. Additionally, we show that active learning significantly improves the efficacy of process supervision. To support related research, we also release PRM800K, the complete dataset of 800,000 step-level human feedback labels used to train our best reward model.

Published as a conference paper at ICLR 2024
LET'S S VERIFY STEP BY STEP
Hunter Lightman", Vineet Kosaraju", Yura Burda", Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Nya Sutskever & Karl Cobbe"
OpenAl San Francisco, CA, USA
kar1@openai.com
ABSTRACT
In recent years, large language models have greatly improved in their ability perform complex multi-step reasoning. However, even state-of-the-arr models still regularly produce logical mistakes. To train more reliable models, can turn either to outcome supervision, which provides feedback for a final result, process supervision, which provides feedback for each intermediate reasoning step. Given the importance of training reliable models, and given the high cost of human feedback, it is important to carefully compare the both methods. Recent work has already begun this comparison, but many questions still remain. We con- duct our own investigation, finding that process supervision significantly outper- forms outcome supervision for training models to solve problems from the chal- lenging MATH dataset. Our process-supervised model solves 78% problems from a representative subset of the MATH test set. Additionally, we show that active learning significantly improves the efficacy of process supervision. To sup- port related research, we also release PRM800K, the complete dataset of 800,000 step-level human feedback labels used to train our best reward model.
INTRODUCTION
Large language models are capable of solving tasks that require complex multi-step reasoning by generating solutions in a step-by-step chain-of-thought format (Nye et al., [002]] Wei et al. 2022 Kojima et al.] 2002).. However, even state-of-the-an models are prone to producing falsehoods they exhibit a tendency to invent facts in moments of uncertainty (Bubeck et al. 2023). These hallucinations (Maynez et al. (020)) are particularly problematic in domains that require multi-step reasoning, since single logical error is enough to derail a much larger solution. Detecting and mitigating hallucinations is essential to improve reasoning capabilities.
One effective method involves training reward models to discriminate between desirable and unde- sirable outputs. The reward model can then be used in reinforcement learning pipeline (Ziegler et al. 2019 |tiennon et al. 2020 Nakano et al.) 2021 Ouyang et al. 2022) or to perform search via rejection sampling (Nichols al.) 2020. Shen al. 2021] Cobbe al.) 2021). While these tech- niques are useful, the resulting system is only as reliable as the reward model itself. It is therefore important that study how most effectively train reliable reward models.
In closely related work, (eesato et al. (2022) describe two distinct methods for training reward mod- els: outcome supervision and process supervision. sutcome-supprriiee reward models (ORMs) are trained using only the final result of the model's chain-of-thought, while process-supervised reward models (PRMs) receive feedback for each step the sons to favor process supervision. It provides more precise feedback, since it specifies the exact location of any errors that occur. also has several advantages relevant to Al alignment: is eas- ier for humans to interpret, and it more directly rewards models for following a human-endorsed thotuggh. Within the domain of logical reasoning, models trained with outcome supervision regularly use incorrect reasoning to reach the correct final answer (Zelikman et al.) 2022) Creswell
There are compelling rea-
ct al. 2022). Process supervision has been shown to mitigate this misaligned behavior (Uesato et al.
2022
PPiimry authors. Correspondence to: Karl Cobbe <karl @openai.com>

---

Published as a conference paper at ICLR 2024
Despite these advantages, Uesato et al. (2022) found that outcome supervision and process super- vision led to similar final performance in the domain of grade school math. We conduct our own detailed comparison of outcome and process supervision, with three main differences: more capable base model, we use significantly more human feedback, and we train and test on the more challenging MATH dataset (Hendrycks al. 2021
use
Our main contributions are as follows:
1. We show that process supervision can train much more reliable reward models than out- come supervision. We use our state-of-the-art PRM to solve 78.2% of problems from representative subset of the MATH test set.
2. We show that large reward model can reliably approximate human supervision for smaller reward models, and that it can be used to efficiently conduct large-scale data collection ablations.
3. We show that active learning leads to 2.6x improvement in the data efficiency of process supervision.
4. We release our full process supervision dataset, PRM800K, to promote related research.
2 METHODS
We perform comparison of outcome and process supervision, following a similar methodology to Uesato al. 2022). Outcome supervision can be provided without humans, since problems in the MATH dataset have automatically checkable answers. In contrast, there is no simple way to auto- mate process supervision. We therefore rely on human data-labelers to provide process supervision, specifically by labelling the correctness of each step in model-generated solutions.
We conduct experiments in two separate regimes: large-scale and small-scale. Each has its own advantages, and they offer complimentary perspectives. At large-scale, we finetune all models from GPT-4 OpenAl 2023). We focus on advancing the state-of-the-art by training the most reliable ORM and PRM possible. Unfortunately the training sets for these reward models not directly comparable. for reasons we will discuss in Section These models are therefore not ideal for mak- ing an apples-to-apples comparison of outcome and process supervision. To address this flaw, also train models at small-scale, where we can conduct a more direct comparison. In order to re- move dependence costly human feedback, use large-scale model to supervise small-scale model training. This setup enables us to conduct several important ablations that would otherwise be infeasible.
2.1 SCOPE
At each model scale, we use single fixed model to generate all solutions. We call this model the generator. We do not attempt to improve the generator with reinforcement learning (RL). When we discuss outcome and process supervision, we are specifically referring to the supervision given to the reward model. We do not discuss any supervision the generator would receive from the reward model if trained with RL. Although finetuning the generator with RL natural next step, intentionally not the focus of this work.
We instead focus exclusively on how to train the most reliable reward model possible. We evaluate reward model by its ability to perform best-of-N search over uniformly sampled solutions from the generator. For each test problem we select the solution ranked highest by the reward model, automatically grade it based on its final answer, and report the fraction that are correct. A reward model that is more reliable will select the correct solution more often.
2.2
BASE MODELS
All large-scale models are finetuned from the base GPT-4 model OOenAA 2023). This model has been pretrained solely to predict the next token; it has not been pretrained with any Reinforcement Learning from Human Feedback (RLHF) (Christiano et al. 2017). The small-scale base models similar in design GPT-4, but they were pretrained with roughly 200 times less compute. As
2

---

Published as a conference paper at ICLR 2024
The denominator fraction less than times the numerator. the fraction equivalent to 2/5, what the numerator of the fraction? (Answer:
14
Let's call the numerrtorr
P
So the denominator is 3x-7.
xo
5x Bx 14.
x.
Figure 1: A screenshot of the interface used to collect feedback for each step in a solution.
additional pretraining step, finetune all models dataset of roughly 1.5B math-relevant tokens, which we call MathMix. Similar to Lewkowycz et al. (2022). we find that this improves the model's mathematical reasoning capabilities. Details on how this dataset was constructed can be found in Appendix
2.3
GENERATOR
To make parsing individual steps easier, train the generator produce solutions in newline delimited step-by-step format. Specifically, we few-shot generate solutions to MATH training prob- lems, filter those that reach the correct final answer, and finetune the base model on this dataset for single epoch. This step is not intended to teach the generator new skills; it is intended only to teach the generator to produce solutions in the desired format.
2.4 DATA COLLECTION
To collect process supervision data, we present human data-labelers with step-by-step solutions to MATH problems sampled by the large-scale generator. Their task to assign each step in the solution label of positive, negative, neutral, as shown in Figure [ A positive label indicates that the step is correct and reasonable. negative label indicates that the step either incorrect or unreasonable. A neutral label indicates ambiguity. In practice, step may be labelled neutral if it is subtly misleading, or it a poor suggestion that technically still valid. We permit neutral labels since this allows to defer the decision about how to handle ambiguity: at test time, neutral labels as either positive or negative. A more detailed description of the labelling instructions provided in Appendix [D]
treat
We label solutions exclusively from the large-scale generator in order to maximize the value of limited human-data resource. We refer the entire dataset of step-level labels collected as PRM800K. The PRMS00K training set contains 800K step-level labels across 75K solutions to 12K problems. To minimize overfitting, we include data from 4.5K MATH test problems in the PRM800K training set, and we therefore evaluate our models only on the remaining 500 MATH test problems. More details about this test
be found in Appendix
During data collection, we must decide which solutions to surface to data-labelers. The most straightforward strategy to uniformly surface solutions produced by the generator. However, we surface solutions that make obvious errors, the human feedback we get is less valuable. We would prefer surface solutions that are more likely fool our best reward model. To that end, we attempt to strategically select which solutions to show data-labelers. Specifically, we choose to surface convincing wrong-answer solutions. We use the term convincing to refer to solutions that are rated highly by our current best PRM, and we use wrong-answe incorrect final answer. We use this slightly verbose phrasing to emphasize the fact that correctness determined solely by checking the final answer, process which occasionally leads to misgraded solutions. We expect to gain more information from labeling convincing mrong-answes solutions, since know the PRM is mistaken about at least one step in each such solution.
to refer to solutions that reach an
3

---

Published as a conference paper at ICLR 2024
Lee
Lot
where each nnt-ronntnnt polynommia 662; is monile with beeege contieintte and connor be fictored furtlerr over the integgess Compute 2i0 + Ued0 ----- peD)
wheme wath son-romatant polynombal su-)) montc with intager confiiciintts and
connet he ficorrd further over the Intogorr Compute pc0) + 5x00 +- poll)
Lnstie that the palyacural has digne and Dely
mtae a sclettrtte to stspiity an
Das poodille stretagg to kask be to aessaal mone the palcossiin roing reliina theomei
sopood equation)
can quadeatic oppactio an 0KS 33 gral polmmonnal
Hoween, that world ievavv thockig posstlle Sactoce and there guarsstio that the polnortt toe say oxtional mose all
Now loo Soor ondh Clese Sittes Tuober prpoott bo over the lengees
aP Ses
Solls
by cagnel
Aoptring itas Messtt with
socce the the poleronzial sous press lappeses
snlotttate yowarf
tve the asceed Batter p sos wos the defirmnco of syuame forerts wittc waps thee [c.e --880
Tlax the pelsoooiaal lecuras 5f By stick to gusinn equatioss ia
Austring (tas fenedaa
tookingg
mostiiityy
apuee agait
GAE M.
(o 1300
Tooo mumblrr
Now: can sebutttutt back pet of 3y an (efi s-ggeet--
Theesee www.eeteeeeeeoo Bie fameeeeeeeeeee the fategge,
looko prostisong
1620)= =1 505e1 K and sutel
css try to Factor ouch those fictors fottee, using the affocrsc of sqgares feeeddi
1s the Batr valan
fedee and sill
Iin apply tie ditemeee of reparte fomrats spin
Setter and
and (611 1-100
Ry 2000 1300 11)
Now haw Sutieed the pelenomrral comptteety Isto momt treer and gpadnee suttie Imaoo codticooos
Awowe
Thew t96 add peotlem weltnng
To ans valaas si Jos ano plg in
up
Stapktiig pil1) 1963
112053001
Miltiestin oe p000 pottl p
Figure 2: Two solutions to the same problem, graded by the PRM. The solution on the left is correct while the solution on the right is incorrect. A green background indicates a high PRM score, and red background indicates low score. The PRM correctly identifies the mistake in the incorrect solution.
In addition to using this selection strategy, we also iteratively re-train our PRM using the latest data at several points in the data collection process. At each iteration, we generate N solutions per problem and surface only the top K most convincing wrong-answer solutions to data-labelers. We experiment with either applying this top-K filtering at problem level (K solutions per problem) or globally across the dataset (K solutions in total, unequally distributed among problems). Since the data collection process is expensive, it was not feasible to conduct at-scale ablations of these decisions. However, we perform several surrogate ablations in Section 4 using our largest PRM as a labelling oracle for smaller PRM. More details about data collection can be found in Appendix B
2.5 OUTCOME-SUPERVIEE REWARD MODELS (ORMs)
We train ORMs following a similar methodology to Cobbe et al. 2021). We uniformly sample a fixed number of solutions per problem from the generator, and we train the ORM to predict whether each solution is correct or incorrect. In practice, we usually determine correctness by automatically checking the final answer, but principle these labels could be provided by humans. At test time, we use the ORM's prediction at the final token as the overall score for the solution. We note the automatic grading used to determine ORM targets is not perfectly reliable: false positives solutions that reach the correct answer with incorrect reasoning will be misgraded. We discuss additional ORM training details in Appendix E
2.6 PROCESS-SUPERVISTR REWARD MODELS (PRMS)
We train PRMs to predict the correctness of each step after the last token in each step. This prediction takes the form of single token, and we maximize the log-likelihood of these target tokens during training. The PRM can therefore be trained in a standard language model pipeline without any special accommodations. To determine the step-level predictions at test time, it suffices to perform a single PRM forward pass over the whole solution. We visualize large-scale PRM scores for two different solutions in Figure 2 To compare multiple solutions, it is necessary to compute a single score for each solution. This is an important but straightforward detail: we define the PRM score for solution to be the probability that every step is correct under the PRM. We implement this as the product of the correctness probabilities for each step. We describe other possible scoring strategies and additional PRM training details in Appendix F
4

---

Published as a conference paper at ICLR 2024
ORM 72.4
PRM 78.2
Majority Voting
%
Solved (Best-of- 1860)
69.6
78
76
%6 Problems rrblemm Solvod [Best af
74
Process Supervised RM
Dutcome- Dutcome-Supervisin RM Maj ority Vating
62
N number of solutions per problem
Figure 3: A comparison of outcome-supervised and process reward models, evaluated by their ability to search over many test solutions. Majority voting shown as strong baseline. For N < 1000, we visualize the variance across many subsamples of the 1860 solutions generated in total per problem.
When we provide process supervision, we deliberately choose to supervise only up the first in- correct step. This makes the comparison between outcome and process supervision more straight- forward. For correct solutions, both methods provide the same information, namely that every step is correct. For incorrect solutions, both methods reveal the existence of at least one mistake, and process supervision additionally reveals the precise location that mistake. we were provide additional process supervision beyond the first mistake, then process supervision would have an even greater information advantage. This decision also keeps the labelling cost similar for humans: with- out relying easy-to-check final answer, determining the correctness of solution is equivalent to identifying its first mistake. While most MATH problems do have easy-to-check final answers, expect this to not remain true in more complex domains.
LARGE-SCALE SUPERVISION
We train the large-scale PRM using the step-level labels in PRM800K. To ensure the large-scale ORM baseline is as strong as possible, we train on 100 uniform samples per problem from the generator. This means the ORM training set has overlap with PRMS00K, and it is order of magnitude larger. Although these two training sets are not directly comparable, each represents our best attempt to advance the state-of-the-at with each form of supervision. We note that training the ORM solely on PRM800K solutions would be problematic, since our active learning strategy has heavily biased the dataset towards wrong-answe solutions. We did explore training the ORM superset of PRM800K solutions, by mixing in uniformly sampled solutions, but we found that this did improve ORM performance.
Figure 3 shows how the best-of-N performance of each reward model varies function of N. Since majority voting is known be strong baseline (Wang et al., 2022, Lewkowyez al. 2022). we also include this method as point of comparison. While the ORM performs slightly better than the majority voting baseline, the PRM strongly outperforms both. Not only does the PRM reach higher performance for all values of N, but the performance gap widens N increases. This indicates that the PRM more effective than both the ORM and majority voting at searching over a large number of model-generated solutions. We experimented with using RM- RM-weighted voting (Li et al. 2022; Uesato et al. 2022) to combine the benefits the PRM and majority voting, but this did noticeably improve performance. We specific subset of the MATH test set for evaluation,
5

---

Published as a conference paper at ICLR 2024
60
60
(Bed at-))
Problems Selved (Best-of 5000
(55
50
50-
45
45
40
35
35-
PRM Active Learning
30
PRM (PRMargs supervised)
PRM (PRMuge supervised)
ORM (PRMuge Supervised
ORM (PRM supervised)
ORM (fiwal-rnswer supervised)
25
ORM (final-nnwwer supervised)
10P
10:
10
10
10a
10a
16'
N number of solutions problem
Number solutions labelled per prablem
(b) Three reward models trained 200 sam- ples/problem using different forms of supervision, compared across many test-time compute bud- gets.
(a) Four series reward models trained using ferent data collection strategies, compared across training sets of varying sizes.
Figure 4: comparison different forms of outcome and process supervision. Mean and standard deviation is shown across three seeds.
which describe in Appendix Appendix G
We further break down these results by problem difficulty in
SMALL-SCALE SYNTHETIC SUPERVISION
We find that the PRM outperforms the ORM at large-scale, but this result alone paints an incomplete picture. To better compare outcome and process supervision, there are two confounding factors that must be isolated. First, the training sets for the ORM and the PRM are not directly comparable: the PRM training set constructed using active learning, biased towards amswer-incorrect solutions, and is an order of magnitude smaller. Second, the final-answer grading will provide positive labels spurious solutions that reach the correct final answer despite incorrect reasoning. This could damage ORM performance, an effect we may or may not want to attribute to outcome supervision more generally.
Due to the high cost of collecting human feedback, we cannot easily ablate these factors using human labelers. We instead perform the relevant ablations by using the large-scale PRM supervise smaller models. This setup enables us to simulate large amount of data collection at a modest cost. For the remainder of this section, we refer the large-scale PRM from Section as PRMaage
4.1 PROCESS VS OUTCOME SUPERVISION
We now conduct direct comparison of outcome and process supervision. We first sample between I and 200 solutions per problem from small-scale generator. For each dataset, we provide three forms of supervision: process supervision from PRMargee outcome supervision from PRMaage and outcome supervision from final-answer checking. The choice of supervision is the only difference between these three series of reward models, which are otherwise trained on identical datasets. See Appendix for more details about how PRMaag is used for outcome and process supervision.
In Figure 4a we evaluate each reward model by its best-of-500 selection. We see that process supervision significantly outperforms both forms of outcome supervision at all data collection scales. In Figure 4b
evaluate the best reward model from each series by its best-of-N performance across different values of N. We see that using PRMage for outcome supervision is noticeably more effective than final-answer checking. This be explained by the fact that PRM provides better supervision for solutions that reach the correct final answer using incorrect reasoning.
It is not clear whether supervision by PRMaag or final-answr checking represents the more ap- propriate outcome supervision baseline. While final-answer supervision is more explicitly outcome based, its main weakness the existence of false positives is arguably over-emphasized in the

---

Published as a conference paper at ICLR 2024
Table 1: We measure out-of-distrriutiion generalization using recent STEM tests. We evaluate the outcome-supervised RM, the process-supervise RM, and majority voting using 100 test samples per problem.
ORM
PRM
Majority Vote
# Problems
AP Calculus
68.9%
86.7%
80.0%
45
AP Chemistry
68.9%
80.0%
71.7%
60
AP Physics
77.8%
86.7%
82.2%
AMC10/12
49.1%
53.2%
32.8%
84
Aggregate 63.8%
72.9%
61.3%
234
MATH dataset. Outcome supervision by PRMare better represents outcome supervision in domains that are less susceptible to false positives. We consider outcome supervision by PRMar to be the more relevant baseline, but we encourage the reader draw their own conclusions.
4.2 ACTIVE LEARNING
Finally, we investigate the impact of active learning. We train a small-scale reward model, PRMaelocter on single sample from each problem, and we use this model to score 1000 samples per problem. To train each of our larger reward models, we select N samples per problem such that 80% are the most convineing (according to PRMsslcctrr wrong-answer samples, and 20% are the most convincing samples that remain (right- or wrong-answer).. We score the selected samples with PRMarge and train those scores. This process ensures that all samples relatively convincing under PRMselector. that a large fraction are known to contain at least one mistake, and that our overall dataset is not too heavily biased toward wrong-answer solutions. Performance of this data labelling scheme is shown in Fa By comparing the slopes of the line of best with and without active learning, we estimate that this form of active learning approximately 2.6x more data efficient than uniform data labelling. We note that the model trained on the largest active learning dataset (200 samples per problem) appears slightly underperform the expected trend line. Our best explanation for this observation is that 200 samples represents a significant fraction of the overall selection pool (1000 samples) and that this relative lack of diversity limits the possible upside from active learning.
We also performed a preliminary investigation into the impact of iteratively retraining PRMMselector throughout data collection. Between iterations, we re-trained PRM using all currently labeled data. Unfortunately, we observed instability in this process which were unable to diagnose. The resulting reward models performed no better than the models described above. We expect some form of iterative retraining to be beneficial in active learning, but currently have no concrete evidence to support this claim. We consider this compelling direction for future research.
5 OOD GENERALIZATION
To get some measure of PRM on a held-out set 224 STEM questions, pulled from the most recent AP Physics, AP Calculus, AP Chemistry, AMC10, and AMC12 exams. Since these tests were released after the pre-training dataset was compiled, we can have high confidence that the model has not seen these problems. We report the best-of- 100 performance of the ORM, PRM and majority voting in Table We observe results similar to those Section 3 the PRM outperforms both the ORM and majority voting. This shows that the PRM can tolerate modest amount of distribution shift and that its strong performance holds up on fresh test questions.
generalization, we evaluate our large-scale ORM and
6 DISCUSSION
6.1 CREDIT ASSIGNMENT
One clear advantage of process supervision that it provides more precise feedback than outcome supervision. A reward model trained with outcome supervision faces difficult credit-assigmment

---

Published as a conference paper at ICLR 2024
task- to generalize well, it must determine where incorrect solution went wrong. This is par- ticularly difficult for hard problems: most model-generater solutions contain an error somewhere, so the marginal value of negative label from outcome supervision is low. In contrast, process pervision provides a richer signal: it specifies both how many of the first steps were in fact correct, as well as the precise location of the incorrect step. Process supervision makes credit assignment easier, and we believe that this explains its strong performance.
6.2 ALIGNMENT IMPACT
Process supervision has several advantages over outcome supervision related to Al alignment. Pro- cess supervision is more likely to produce interpretable reasoning, since it encourages models to follow process endorsed by humans. Process supervision is also inherently safer: it directly re- wards an aligned chain-of-thought rather than relying on outcomes as a proxy for aligned behavior (Stuhlmüller & Byun 2022). In contrast, outcome supervision is harder scrutinize, and the prefer- ences conveyed are less precise. In the worst case, the use of outcomes as an imperfect proxy could lead to models that become misaligned after learning exploit the reward signal (Uesato et al. 2022 Cotra 2022
Everitt et al.) 2017).
In some cases, safer methods for Al systems can lead to reduced performance (Ouyang et al. 2022 Askell et al., (202)) cost which is known as an alignment tax. In general, any alignment tax may hinder the adoption of alignment methods, due to pressure to deploy the most capable model. Our results show that process supervision in fact incurs a negative alignment tax. This could lead to increased adoption of process supervision, which we believe would have positive alignment side- effects. It is unknown how broadly these results will generalize beyond the domain of math, and we consider it important for future work to explore the impact of process supervision in other domains.
6.3 TEST SET CONTAMINATION
The test set of the MATH dataset contains problems that are discussed in several online venues, and it likely that some of these problems appear in the pretraining dataset for our models. We attempted to remove all MATH problems from our MathMix dataset using atrnngmatching heuristics, but since humans can post guarantees about the overlap between MathMix and the MATH dataset.
rephrasings of a problem online, it is difficult to make any strong
In our experience inspecting model moner--gnerated solutions, we saw no clear signs our models mem- orizing MATH problems. However, it is impossible rule out subtle forms of memorization that would slip past manual inspection, and it is still possible that some degree of contamination has slightly inflated our performance the MATH test set. Even in that case, would expect any contamination to manifest similarly across all methods, and that the relative comparisons made throughout this work would remain mostly unaffected.
We also note that the PRM regularly surfaces correct solutions to MATH problems that have a low single-digit percentage solve-rate under the generator, some examples of which can be seen in Ap- pendix [ The generator's 's low solve-rate is an additional indication that it has not encountered such problems via test set contamination. Our generalization results from Section 5 further strengthen our claim that test set contamination has not significantly impacted this work, since we observe qualitatively similar results on problems that are guaranteed be uncontaminated.
7 RELATED WORK
7.1 OUTCOME VS PROCESS SUPERVISION
In work closely related to our own, Uesato et al. 2022) compare the impact of outcome and process supervision in the domain of grade school math. They found that both methods led to similar final- answer error rates, and that process supervision achieved those results with less data. While our core methodology is very similar, there are three main details that differ. First, we use a more capable model to collect PRM800K dataset and to perform large-scale experiments. However, our scall-scale results in Section suggest that large-scale models are not necessary to observe benefits from process supervision. Second, we evaluate the MATH dataset, which is significantly
8

---

Published as a conference paper at ICLR 2024
more challenging than GSM8K. Third, we collect much larger quantity of process supervision data.
On the surface, the results from Uesato et al. 2022) may to conflict with claim that process supervision leads to better performance. However, we believe the apparent conflict can be explained by the difference in the scale of the supervision. The data scaling trend in Figure 4a suggests that small amount of process supervision and a large amount of outcome supervision do in fact lead to similar performance, consistent with the results from [esato al. 2022). The trend also shows that process supervision beats outcome supervision when scaled up, even when judged based solely on outcomes. This is consistent with our results in Section 3 case for using process supervision.
We believe these results make strong
7.2 SYNTHETIC SUPERVISION
Similar to our work in Section Gao et al. (2022) use a large reward model to supervise the training of smaller models. They study the over-optimization that occurs during RLHE, with experiments that require large quantities of human preference data. To work around this challenge, they use a gold-standard reward model to replace human feedback. Our use of large-scale reward model to supervise smaller reward models shares similarities with their approach.
7.3 NATURAL LANGUAGE REASONING
Several recent studies that have examined the reasoning ability of large language models are im- plicitly relevant to our work. Lewkowyez et al. (2022) showed that finetuning models on a large
corpus of technical content led significantly improved performance showed that self-consistency leads to remarkably strong performance on many reasoning bench-
MATH. Wang et al. (2022)
marks, notably without requiring any additional finetuning. /Wei al. (2022) and Nye et al. (2021)
demonstrate the importance of explicitly performing intermediate reasoning steps via chain of thought or a scratchpad in order solve tasks that require multi-step reasoning. Kojima et al.) 2022) show that models are able to perform this behavior zero-shot, conditioned only on a simple prompt.
CONCLUSION
We have shown that process supervision can be used to train much more reliable reward models than outcome supervision in the domain of mathematical reasoning. We have also shown that active learning can be used to lower the cost of human data collection by surfacing only the most valuable model completions for human feedback. We release PRMS00K, the full dataset of human feedback used train our state-of-the- the-art reward model, with the hope that removing this significant barrier to entry will catalyze related research on the alignment of large language models. We believe that process supervision is currently under-explored, and we are excited for future work to more deeply investigate the extent to which these methods generalize.
ACKNOWLEDGGENNTS
We thank Joshua Achiam, Mark Chen, Jonathan Gordon, Dan Hendrycks, Lukasz Kaiser, Oleg Murk, Ben Sokolowsky, Francis Song, and Jonathan Uesato for valuable feedback and thoughtful discussions: Giambattista Parascandolo and Daniel Selsam for their contributions to the MathMix dataset: Jonathan Ward for contributing to the data collection interface; Wojeiech Zaremba for en- couraging us to scale up data collection; Peter Hoeschele and Aris Kostantinidis for supporting our data collection; the research acceleration and supercomputing teams OpenAI for providing infrastructure support; and the team at Scale and the many data-labelers who created PRM800K.
REPRODUCIBILITY STATEMENT
To assist reprodueibility and further research we are releasing all of the labels that we gathered over the course of this project. Appendix(Bcontte information about the dataset, what data was used for training, and a link to the repository containing the raw labels. Appendix C exeepliiss our evaluation methods. Finally, Appendix R and Appendix explain how we trained ORMs and PRMs.
9

---

Published as a conference paper at ICLR 2024
REFERENCES
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as laboratory for alignment. arXiv preprint
2021.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka- mar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12711: 2023.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- tems, 30, 2017.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arxi:2110.14168. 2021.
Ajeya Cotra. Without specific counter the easiest path transformative Al likely leads
http::/wwww...mooomoooooooooooo/ wi -ithoou-speci oama::/eeeeeccco..ooooommmmmmeee
AI takeover.
2022.
Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inferennce, Exploiting large lan- guage models for interpretable logical reasoning. arXiv preprint arXiv:2205.09771, 2022.
Tom Everitt, Victoria Krakovna, Laurent Orseau, Marcus Hutter, and Shane Legg. Reinforcement learning with a corrupted reward channel. arXiv preprint arXiv: 2017.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. arXiv preprint arXiv:2210. /0760, 2022.
Dan Hendrycks. Collin Burns. Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2 103. 003003774 2021.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205. //916, 2022.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag. Theo Gutman SSloo, et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv: 14858, 2022.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. On the advance of making language models better reasoners. arXiv preprint (xXix:226.002::5 2022.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. arXiv preprint arXiv:2005.0000 2020.
Rejichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted ceession-answering with human feedback. arXiv preprint
2021.
Eric Nichols, Leo Gao, and Randy Gomez. Collaborative storytelling with large-scale neural lan- guage models. In Proceedings of the 13th ACM SIGGRAPHI Conference Motion, Interuction and Games, pp. 1-10, 2020.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski. Jacob Austin, David Bieber, David Dohan, Aitor Lewkowyez, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:21 2021.
OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,2000 2023.
10

---

Published as a conference paper at ICLR 2024
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXh:2203.021555 2022.
Jianhao Shen, Yichun Yin, Lin Li, Lifeng Shang, Xin Jiang, Ming Zhang, and Qun Liu. Generate & rank: A multi-task framework for math word problems. arXiv preprint arXiv:2109.0303: 2021.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Andreas Stuhlmüller and Jungwon Byun. Supervise process, outcomes, https://ought. ottp/pppaates/22222---00---peeee 2022.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211. 2022.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances Neural Information Processing Systems, 35:15476-15488 2022.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909 190..09999999, 2019.
11

---

Published as a conference paper at ICLR 2024
A MATHMIX
Similar to Lewkowycz et al. 2022 we construct a large-scale dataset of high-quality math-relevant tokens for use in lightweight pretraining stage, before finetuning on comparably smaller datasets like MATH and PRM800K. This dataset, which we call MathMix, has two main differences com- pared to the one used to train Minerva. First, it is smaller and more aggressively filtered high- quality math problem-se mobliem-solving content, and second, it does not explicitly mix in general language data.
Minerva was trained on 38.5B tokens of arXiv documents and webscrape pages with LaTeX content, while MathMix consists of smaller set of 1.5B tokens containing individual math problems and their solutions, free-form text discussing math problems and concepts, and synthetic data (Table 2 While Minerva was pretrained on dataset with 5% general natural language data, we chose not to mix in any natural language data explicitly, primarily because MathMix already contains plenty of natural language data.
Table 2: MathMix dataset components.
Data type
Token count
Present in pretraining?
Math problems and solutions
275M
No
Free-form math discussion text (1)
430M
No
Free-form math discussion text (2)
450M
Yes
Synthetic data (1)
30M
No
Synthetic data (2)
100M
Yes
Critiques grading data
500M
No
Note that when training smaller models, as in Section 4 use slightly smaller variant of MathMix that excludes the critiques data and only consists of TB tokens. For our large models experiments, train MathMix for roughly 3B tokens (2 epochs). For our small models experiments, train for 6 epochs (roughly 6.6B tokens).
We apply set of decontamination checks MathMix against the test split of the MATH dataset, including stripping out LaTeX and searching for matching n-grams, but we can make no strong guarantees on the efficacy of this decontamination. As discussed in Section [6.3] we would expect the relative comparisons made throughout this work to be significantly impacted by test set contamination.
12

---

Published as a conference paper at ICLR 2024
B PRM800K
We collected 1,085,590 step-level labels over 101,599 solution samples. We present the whole unfil- tered dataset PRM800K. During training we discard labels used for quality control, as well as any step-level labels for which the labeler was unable to complete the task. The filtered dataset contains about 800,000 step-level labels over 75,000 75,000 solutions. The full PRM800K dataset is available at http://tthuub.cooopppa//////
The data collection was split into two separate phases. phase we collected labels for multiple alternative completions at each step of solution. This seeded our dataset but was cumbersome for many steps the alternatives were repetitive, and we found labelers spent a lot of time supervising long uninteresting solutions. As result, the step-level labels we collected in this phase are more repetitive than those collected later. In total, phase represents about 5% PRMS00K, or about 40,000 step-level labels.
The majority of our labels were collected as part of phase 2, during which we scaled up and stream- lined the data collection process. Phase 2 data collection is split into 10 generations. For each gen- cration, we sample N solutions per problem from the generator. We rank these solutions with our current best PRM and surface the highest scoring wrong-answan solutions this PRM between each generation using all the latest data. This active learning strategy changes sometimes surfaced correct solutions (either by
labelers. We retrain
the balance of our data considerably. Though manually injecting correct solutions or because of errors in our automatic grading), the vast majority of the labels we collected in this phase are for incorrect solutions. Table 3 breaks down the balance of correct/incorret steps and solutions between the different phases of data collection. Though we mostly collected labels on incorrect solutions, we still collected many labels for correct individual steps. In fact, our small-scale ablations in Section[4.2 suggest that this active learning strategy, which favors labelling high-scoring wrong-answe answer solutions, improves performance despite the resulting imbalance in the dataset.
suggggss
Table 3: Distribution of positive/negative steps/solutions.
phase 1
phase 2
combined
% end in correct solution
85.1
13.2 74.1
14.2
% correct steps
58.6
73.1
Some our phase 2 questions are intended for quality control. For a quality control question, researchers mark which steps are reasonable to label as incorrect. Then we assess that labelers are able to consistently mark those steps as incorrect. Prior to starting on phase we required all labelers to label 30 quality control questions. This served as screening test, and we only admitted labelers that agreed with our gold labels at least 75% of the time.
We then designated 10-20 problems per generation as additional quality control questions, and we randomly served them to labelers as they worked through the task. We used the results of this continuous quality control to remove labelers whose quality slipped too far, as well as to prepare ed- ucational material on common mistakes in order to improve labeler alignment with our instructions.
C EVALUATION
As we scaled up the project, we began having to collect labels on multiple solutions for the same training problem. In order to avoid the risk of fitt-fitting the 7,500 MATH training problems, we expanded the training set to include 4,500 MATH test split problems. We therefore evaluate models only on the remaining 500 held-out problems. We selected these 500 test problems uniformly at random. In Figure [ we show that the distribution difficulty levels and subjects we used can
in this subset is representative of the MATH test /ow.///ppa/n0000 We leave it for future work to explore how many distinct training problems are actually necessary, and how quickly our methods overfit to the training set.
whole. The specific
be found at httips//gittubb.c com wttpbbooooooooop////iioo
13

---

Published as a conference paper at ICLR 2024
MATH subject distribution
MATH difficuity level distribution
lest
our lead
MATH test
MATH test
0.15
3.15
0.10
3.10
3.55
8.00
Interneilaxe Algebro
Alpettra
Number Theaty
Preeigebira
Caremetry
Counting Pobbtiniiy
Level
Figure 5: Two histograms comparing the distribution of problem difficulty levels and subjects in both the original MATH test and in our 500 problem test subset.
D LABELLING INSTRUCTIONS
Labelers were tasked to look at steps in a solution and label each one as positive, negative, or neutral. A step is considered neutral if it is appropriate in context, reasonable, correct, and contains only computations that can be verified easily. A step is positive if is neutral and also progresses towards the solution. All other steps considered negative. Labelers were given reference solutions, but they were given the ground truth final answers. We chose not to provide reference solutions avoid biasing them towards one particular path to the solution. We chose to provide ground truth final answers since this information can sometimes help labelers resolve their own misunderstandings.
In phase 1. labelers were permitted to enter their own steps in the case that all candidate steps were negative. Then the solution would progress from a randomly selected positive step (or neutral their were no positive ones). This often resulted in trajectories that got stuck in endless sequences of neutral steps that said reasonable things but made frustratingly slow progress towards a solution or negative steps that needed constant human supervision. In phase 2, we pre-generate whole solutions and end the task as soon as the first negative step is encountered. The full instructions given to labelers can be found at
E ORM TRAINING DETAILS
We train nutcome-superviise reward models in the same manner as token-level verifiers from Cobbe jtt al. (202I). with a few subtle differences to hyperparameters. In particular, we only train for single epoch each dataset of model samples and reward model labels, without dropout, and without jointly learning a language modeling objective. We find that performance is not sensitive to most other hyperparameters, within reasonable range.
To collect model samples, we simply sample uniformly from the generator at a temperature of 1.0 without applying any rebalancing of positives or negatives. At training time, the reward model makes predictions for every token in the context. The target for each token in a solution is the same, based whether the solution is labelled correct incorrect. At test time, simply use the score of the final token in the completion as the overall score of the solution. We note that this setup is identical the way token-level verifiers were trained in Cobbe al. (202I).
14

---

Published as a conference paper at ICLR 2024
F PRM DETAILS
F.1 TRAINING
We train our PRMs by fine-tuning the MathMix model to predict the probability of positive, nega- tive, and neutral labels given a solution prefix ending in one of our labeled steps. We sweep over hyperparameters using a dataset containing the first ~ 10% of PRM800K. Fine-tuning an LLM from its ordinary language modeling task to classification task like this is a large distribution shift, and we found low learning rates were important to stable PRM training.
All of our PRMs are trained for 2 epochs. On smaller datasets (such as in phase 1 and the first few generations of phase 2) this improves the final performance over training for just epoch. Additional epochs, up to some point, don't it noticeably help or hurt performance. On larger datasets, the benefits of epoch training diminishes, but we continue doing for consistency.
F.2 SCORING
There are multiple ways of using the PRM score solutions. In general, we produce single solution-level score by performing a reduction over step-level scores, where the step-level score is the probability that the step's label is positive. This involves two specific implementation decisions. First, when determining a step-level score, we either consider a neutral label to be positive neg- ative. Second, when determining a solution-level score, we either use the minimum or the product over step-level scores as a reduction.
We show results from four scoring strategies in Table The best performing strategy to take the product of step-level scores and to consider the neutrals positives, but the difference in performance between all strategies is minor. Throughout the rest of this work, we consider neutral steps to be positive, and define the solution score be the product of step-level scores. Using the product instead of the minimum as the reduction does create a slight bias against solutions with larger number of steps.
Table 4: Best-of- 1860 test performance using the PRM with four different scoring strategies.
product
minimum
neutral positive
78.2% 77.4%
77.6% 77.8%
neutral negative
15

---

Published as a conference paper at ICLR 2024
G DIFFICULTY BREAKDOWN
We show performance our ORM and PRM on each quintile of the MATH dataset. We determine quintiles based on the pass under the generator. is interesting note that the performance gap is not only apparent on high difficulty problems: it is in fact apparent across all difficulties. For the lowest difficulty problems, we see that is possible find adversarial examples that fool the ORM, since the ORM's performance slightly decreases as the number of samples increases. In contrast, the PRM remains highly robust over this same set of samples.
We also see that increasing the number of samples has the largest positive effect on the highest difficulty problems. This to be expected, since a large number of generator samples may be required to find true and convincing solution to hard problem.
Quintile 1 leasiest)
Quirtlle
Quintile
Quintlle 4
Quintile 5 (hardest)
0.8
02
10
10"
10
10a
207
30*
10'
10
solutions
Figure A breakdown of ORM vs PRM performance by problem difficulty.
16

---

Published as a conference paper at ICLR 2024
H SYNTHETIC SUPERVISION DETAILS
We can use PRMaee to provide either outcome or process supervision for smaller models. We determine the labels for individual steps based on the step-level probabilities outputted by MRM To do this, we set an arbitrary threshold: any step that PRM assigns a negative label with greater than 20% probability is considered incorrect. We choose this threshold based on the observation that PRMage is slightly miscalibrated in the direction of favoring positive labels.
To provide process supervision for a solution, we directly return the step-level labels (positive or negative) provided by PRMaree up until the first step that is marked as negative. This mimics our true human data collection process. To provide outcome supervision, we mark the solution as correct and only if PRM considers every step to be correct (using the same thresholding logic).
17

---

Published as a conference paper at ICLR 2024
I
PRM VISUALIZATIONS
All examples shown come from the large-scale generator (GPT-4). We note the pass-rate under the generator give some sense of the difficulty these problems.
I.1 TRUE POSITIVES
These cherry-picked examples show the best-of- 1860 solution from the generator ranked by the large-scale PR...
Problem Generator pass-rate: 0.1%. This challenging trigonometry problem requires applying several identities in not-at-all obvious succession. Most solution attempts fail, because it is hard to choose which identities are actually helpful. Though successful solutions to this problem are rare, the reward model correctly recognizes when a valid chain-of tf-thught has been found.
Simplity tan 100" +-4ot 100"
noticc expression:
100 degrees
will need
Vleasiiiies simpliff
recall Chot Lun and sin 2sin cur
use
expression coon
can combine the terms by finding common demominator of eow 100%!
can simplity the anmerattoc by using the double angle identiry agaain, but in reverse
1igrt star st
nottcc thas 200 degrress is the eame 180 desrovee plus 20 degrees, so can use the sum angple ideasit for sine
ggt sell oojen ree mess tart anaata
recall that sis 180" D and cok 180" 90 can simpliif further
get 22
Lodace Unat 100 degres the same 90 degross plus 10 degeoss, the ungle ideutiity foe sine and
free aill
recall that sia 1 and costi" Ul so con simplify furshes
con simplity the sign by multiplying the numerator and denominstor by
Lget Zete aod
notice thas 201 dogroes is the same as 30 degives minr 10 degros, an cann use the difference ungle ldcatity for sine.
1ggt 2ienash eee cos ses antt
simplity
counedd
terma comenctoos
can cancel out the sio 10" fecmos the numerator and denominator:
<
Ansowe::
18

---

Published as a conference paper at ICLR 2024
Problem 2. Generator pass-rate: 5.8%. In step 7 and the generator starts performing guess- and-check. This a common place the model might hallucinate, by claiming a particular guess is successful when it isn't. In this case, the reward model verifies each step and determines that the chain-of-thought is correct.
Find the unique odd integer such than O 28 and the inverse of modulo 23.
integer
comprastion
To that, I con multipty book sides by and ses 1 28 (moo 293)
Then I can complete the square by adding and subtracting (2/21 the Tefs side and (eood 23
Simplliying, squaaee
Emod 2035
modiulo
can try some valsos of and wen If they works For exomple, IF then (F+
(mod 25)
25)
So ose possible value
But this is not odd, so nood to Rid another
moddll
con
and ged
But this is negative, so need to add 20 get positive walne)
amother possible value of congrumsce, unsque
Answer:
Problem Generator pass-rate: 1.7%. The generator successfully applies several trigonometric identities to simplify the expression.
Find the degrew measure the lease positive angle for which 20"
D 20P
sin f1 cas 201 sin eas RI + cas sin 201 cas ain AIP"
easine
angles up degress degrass
Thke sumpess that I might be able to use eomoo sttovoomettii identities to simplity the expreseton
example, recall that cosld B) A
Aein and einc B)
apply idconitiess
terme cumerattor cos 006
cost5" 20") coegas 300) coe
Stnetleryy If apply these sdentiiies to the terms In the desominater, get sti cos 20" 351 cos 50 cost" slato"
(IP)
80%)
New have tan
wonder if simplify this fraction further
tesonnmmetric idmutity says
dbill desll
If apply this silntity the numerntor, 25" 85" 2ow wolaum
Zere 55" (ox(-)00))
cos
000)-30")
Stmaleriy, recall another triqunometric ideatia that says sin 2os destt
get 25"
also -i) wind, wn write win( 307)
Now dow coe
apny devonn
simpliff
also know that tan is negative in the mocond and Soorth quodramte and that tand -V3 whom 120" 300%
angle
Answer 120
19

---

Published as a conference paper at ICLR 2024
Problem 4. Generator pass-rate: 4.5%. Here, the generator successfully performs complex series of polynomial factorizations, The use of the Sophie-Germaln identity in step is an important step that could be considered insightful.
Lot
---3'- (-
monic integer
integeess Compute (.()) +P(()) +-.++(())) +p))))
polenomist
than polymomiial beasstee
whids quaitastic mpsstion
Fean forer thes quodieette eqposiior in Cy 420 origiton polpmoonial to a0paa
Now nest to Sictor each these Sactom Brrshre posotble over Inteoim
For the last factor, x eecell Uhe Soptii Geamonts stently, whick sags thet (o 25 Bobie 23" Z0bi
Applring ldeeetty with
Tie
difffrenne aquares fommdda
Appbiin
So original palyromtal
There and all trretteette factors the intogers, Sale) (ile)
p012)
Ii tndt the sm thatr valow
ared tn Surtor them
Tiot 20(3) with pha 2:2 2: 2.2
the soswer i6 ++1+2+2-000 4
1.2 TRUE NEGATIVES
Problem 5. Generator pass-rate: 4.5%. The generator attempts to the difference of squares formula in step 12 on an expression that isn't in fact a difference of squares. The reward model catches this mistake.
non-constant polyy nomtonl pále Integor coctticlents, integens. Compute pi21) + ()))
that would involer checking possible of theee
notional
Anether pooiible srratey
alyyheate mride the poyyoottt medorn lis degnee
Enttinn thet the polenomial has only
can vobetrrtt
what happres
Thes the polyncontal brcomes
whild quadiaati equantio
Lso to tector this quodeacte expaatinn by looking for two mumber: that matttppy -o- and add to Thoae mimbers am and -1, ran welteng of 3g
+00gg" -15
Now cas suteriuute back a
got Bet www.teeeppeee gypa
Ilo lodes peooniting beqause hove redrood the degres mectors from to
Econ by to fcttor each three Eactore fumber, mein ditlemace squazes formala
East
2012 2002 1y0a
Eoun apply
squares Docmuta sgain to
20/a" Lifa Ilfa
polsmonnal completely
Integer
Uhess
Simpliiiig (10::0 000
Mulliteing,
20

---

Published as a conference paper at ICLR 2024
Problem 6, Generator pass-rate: 93.5%. In step 7, the generator makes incorrect attempt to simplify an expression. The reward model catches this mistake.
A class of 30 students recendly took 6est. If 20 students scored 80, studeats scored 90, and 2 studente scored 100, then what was the class (the mean) this test?
To class emenage,
up wouets una by
can use shortest to add up the scores, since they are grouped by valus
example, Instoad of adding
times,
soultiply
resulls
Similaciyy mullippy 90 by 8
2 get sums (boer grogge
So, the socal sum of the scores is 80 20 + 100
distributive property couttiplication simplity
equivalent to 120 +
100)
Thar 80 720 1001
arithmmti
60 1890, total mum is 1630 + 720 100, which 2500)
Now, to Bind the average, need to divide 2600 30, the munalee of students
long
soup!
2500 divided by 30 83,33, repeating)
This marane Lind Use class 53.33, 83 and one-thidd
Problem 7. Generator pass-rate: 48.0%. In step 11, the generator makes simple calculation emror. The reward model catches this mistake.
positive
pelme
notine that Bill rt so lts poeitive divisors are of the form 2": where e (0, 1) and [0,1,2.33
To find the of thew divisos, can formala that involow the gooetric seriess
prime number and
nonoegative integer, then the sumn of the powers of from O is
8o, the som the powsres of from
and the sum of the of 5 from O to dim 156
distributive property. sum
prodoct
sumn
Now; I need to find the sum of the distinct peims divisors 2l
coun try to Buntor A trbos primos by trial direision
divisible
by get 510)
reppeat this process and divide by again, gesting 273)
Now. 278 to not divistble 2, so try the noxt poimo,
S,
pet
divide by again, getting 3u
Now. 31 prime number, so have found all the prinse factors of
prime
Answor:
21

---

Published as a conference paper at ICLR 2024
Problem 8. Generator pass-rate: 5.8%. The justification in step 8 is strange, but the reward model lets it slide. In step 9, though, the model incorrectly factors the expression. The reward model catches this mistake.
Find the unique odd integer such that 0 < < 23 and 1 2+2 is the inverse of modulo 23,
nead to find an odd integer between 0 and 23 such that is the inverse of modulo 23:
That means that it and + multiply to give 1 modulo 23;
can write this ss d(t + 2) (mod 23)
can try to guess and chock values of until find one that works, but that might take long time
can also try to use some properties of modular arithmetäc to simplity the equation. For excmmple, I can subtract 23 from 2+2 2 and still get the same remalader modulo 23,
So can write o(t 100--2 2) M(2 21) (mod 23)] simpler,
now
difference two multiples
Now divide both sides long (mod 23
get Bi 21)
moltiple of 28,
This gives 21 21 (mod 23)
simplify this further adding 21 to both sides
This me (mod 23)
This mosns that has the same remainder as when divided by 23
The smallest positive value of that satishes this is
This is also an odd integer, 8o it meets the criteria of the problem.
can check that 8 works by plugeing it into the original equation.
get I(1 2) (mod 23), which is trie
Pberefore, unique odd inleger (hul satisles the given conditions is
Answer:
1.3 FALSE POSITIVES
Problem 9. Generator pass-rate: 18.5%. The generator makes subtle counting in step 9. On the surface, it appears reasonable to claim that there are 5 ways to exchange the same colored ball since there colors. However, this undercounts by factor of since Bob has 2 choices for which ball to return to Alice. The reward model is fooled by this mistake.
Bob and Alice euch hove bag that cuntains one ball of wacch of the colors, blue, bag puts
and Bob's bag. then randomly
violet. Alice randomly selects selects one ball from bag and puts into Alice's bag. What the probability that after this procese the contents of the two bags are the same?
find probibility
nosd divide mumber of fsvorable outcomes mumber of possible outcomes.
Uhis coee, the evenl
contents two bags are the vome afler the prucses
So seed to couns how many ways this can happes, and bow many ways the process can happen in total
with the total mumber ways the
happen.
Alice can thoosr any five balls her lug, and Bob can choose any of the six bails in his bag.
So the total mumber of ways the process can happen is 5 tinses 6, which is 30)
will
many ways
lagg
prusoee
This moans thas A and Bob must have exchanged the same volor of ball
There are five colors, so there are five ways they can eschange the axme color of ball ERROR Therefory,
fnsorablee
So the profsbillty the event is divided 30, which simplifee to 1/6
Answer: 1/6
22

---

Published as a conference paper at ICLR 2024
Problem 10. Generator pass-rate: 17.6%. In step 13, the generator attempts to simplify the equation by combining like terms. It correctly moves and combines the linear terms to the left-hand side, but then mistakenly leaves the right-hand side untouched. The reward model is fooled by this mistake.
walking bustiness pald each dog unaeeigned responstble dogs. Under companny' 's policy, they assigned groupe
responsible
pay Nate's pay dogs reassigned
Denall walking
dogs stayed dogs
Valee
Denilile pop Nate's
Rayy salli
witter eeuastio Bar the Bist aomaiii, where Dimali walle: de mumu duge and Maie stogs
The cesto Denill's dop Note's dose taat
Now let me witte an aguaction for the second somanto where it Naterss dags am meselgmed to Denalll
The cutoo of Denill's dop to Nate's degg
I want theee two rottoe be equal, so can not them equal to each odher and socr fox
multiyy and smppiity equation
Subseaoting from both
Rearsngnn the get 200 120 ERROR
inlticacing 12x foom both sitee Be
raw burter out ti from brohh tomms and get -ttes This moans that cftice
Solsing for 1 got Morara
Howmer, the problem sape that an lynam that saletion
Thoetees, only socution =2
Problem 11. Generator pass-rate: 13.4%. The generator attempts to perform long division, but in step 16, it forgets to include the leading zeros in the repeating part of the decimal. The reward model is fooled by this mistake.
When converting to u docimall, the docional turns out to be repeating docimal, How many digits repeat in this repeating docimal?? For example, if you gei the repoating decimal 0,121, (hen your unswer ansver should be
should be i, and if you get 0.4367,
To contect coecton dectnaal, conn divide the ounseator by the Genominator usiing long divitoo
Elll start by wiiting 31 over 11111 and being doen semi
Then I'll cok, low many times does 11113 go 2007
Well dowen's go momby oo watee neno whore the dectmal point brang drmes another zwens
Now haoe 31000
How many times does to 31007
Sell
above decimal point bring sees)
Stow does
sertie showe the decimal potns and wibtreer VaLa2 from 31000 31000!
that
Bring deam her and sak hove timos done TEEE imto S77800
B1 goos acrcsn tisss, wilte a above Uhe coctanal point and autiineet 27777 from 87780 wos 10003)
1M living deen aeee seryy and sol: hope many times doww 11111 toty 1000000
Is gooe in mine times vn waste a alhow the decinal point and subtrrace ioter from 100000 to are 2013
Wait a misute, Tun back to wlace vtusteat That areus the desimat will supeat from Iace ont
So I'll pat a bar over the 279 to indicate that It reposts indetintety ENROR
The decimal 0,000799
209)
The question aaks many digats repoat repootting desimall
khat thrre thene diigne repesti R. and
23

---

Published as a conference paper at ICLR 2024
Problem 12. Generator pass-rate: 9.1%. In step 4, the generator falsely claims that the sequence repeats itself every 12 terms, when it's in fact every 10 terms. This sort of counting mistake occa- sionally fools the reward model.
Given that
525, and
when 2.8,
find the value of zai
problem involving
sequence,
depends previous terme
One way to approach this probiom to look for a pattern or a Sormulaa that can help us fidd any serms the sequence without biving to compute all the poevious termat
seguence
woything
523
420
211
267
207
321
420
370
211,
-21
020 3755
-376
(-211)
267
420,
39
420
(-376
+ (-211) - 020
330
21
-
-523
(-420)
[-575] (-211) 267.
211
310
24
2.
267
(-523
+ (-420) 576) 211,
332
331
R1o
3,
(-267)
(-523) (-420)
R11
(-267) (-523)
(-267)
sequencee repeats every teemns, Boirinn ita
ERRON
This moans that we can find any term the soquenoc by finding Its remainder when divided by 12, and then looking the corne sporriigg term tn the first cycle
ecample,
by pos
This means that Tes is the same as Ra, which is 420)
Similiury, can Bin and aaas dividing them 12 and lookingg thetr remaiideess romaindee
Shy -529)
And Zuro has a remalnder 3, the same as whichh is (20)
Phersfore the value of
aggs 420-4 (-n23)
3172
24


Reviews:
[{"review_id": "IupAcBr0py", "reviewer": "Reviewer_DxcT", "summary": "This paper investigates the effectiveness of process supervision compared to outcome supervision for training reliable reward models in large language models, focusing on the challenging MATH dataset. The authors conduct a detailed comparison of these two types of supervision for training reward models, using a more capable base model and significantly more human feedback.\n\nThe main contributions of the paper are\n  * demonstrating that process supervision can train more reliable reward models than outcome supervision, with their model solving 78.2% of problems from a representative subset of the MATH test set\n  * showing that active learning can lead to a 2.6× improvement in the data efficiency of process supervision\n  * releasing their full process supervision dataset, PRM800K, to promote related research.", "questions": "1. Who are the human data-labelers?  Are they MTURK crowdsource workers, math teachers, or? What is their proficiency level?\n2. \"MathMix consists of a smaller set of 1.5B tokens containing individual math problems and their solutions\" ---> where are these math problems from?  textbooks or scraping the web?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Assigning rewards for intermediate steps is a novel and intuitive idea. Compared to assigning rewards for the outcome alone, judging intermediate steps better leverages problem structures.\n2. The empirical results are very strong.\n3. The released dataset can support the research community to further explore this direction.", "weaknesses": "1. The reproducibility for this work is concerning. It is hard to understand under what conditions one can successfully train an effective process reward model.  The authors did not have sufficient details for both models and data.  In the paper, the authors stated, \"The small-scale base models are similar in design to GPT-4, but they were pretrained with roughly 200 times less compute.\" However, the paper neither reveals the size of the small-scale base models nor the large-scale base models.  It is unclear whether the data gets the PRM to work or the scale of the model that gets it to work. I understand that work done at industry has its confidentiality rules to follow, but the authors can maybe conduct experiments on open models on their pretraining and fine-tuning datasets? Due to the reproducibility issue, I think this work would be a better fit for a blog post than a main conference paper.\n2. It is unclear whether PRM would work for other types of reasoning problems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OhRcpsKPHR", "reviewer": "Reviewer_HKpB", "summary": "The authors release a new dataset with process supervision, which provides human feedback for each intermediate reasoning step. The authors label the complex reasoning steps for solving MATH dataset. The authors find that process supervision significantly outperforms outcome supervision, whether only human feedback on the final answer, for training models. The authors also find that active learning significantly improves the efficacy of process supervision.", "questions": "Conducting RLHF would be more impressive.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1, A new dataset with humans verifying each reasoning step.\n\n2. The process supervision is important and the labeled dataset is useful for further research on reasoning with math.\n\n3. The authors have also done experiments with active learning to improve the efficacy.", "weaknesses": "1. The work only explores the math problem. It would be better to explore different tasks.\n\n2. The authors haven't applied it to RLHF. It is still not clear how process supervision and outcome supervision affect the generation model performance. Actually, if outcome supervision is large and diverse enough, the model trained with outcome supervision can also do process supervision by feeding the reason path step by step.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kZo04kDRJE", "reviewer": "Reviewer_4gd2", "summary": "The paper proposes to provide language models with intermediate supervision on the step level instead of providing a single reward based on the whole generation. They collect step-level feedback with human data-labelers in the math domain and compare outcome-supervised reward models and process-supervised reward models on this dataset. They demonstrate that providing step-level feedback can achieve better performance and also show that active learning can be used to lower the data labeling cost.", "questions": "1. Under the same budget, is it better to provide step-level rewards on a small set or outcome-level rewards on a large set?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The idea of providing fine-grained feedback for learning models is intuitive and technically sound.\n2. They collect a dataset PRM800K that contains step-level labels across different solutions to math problems, which is helpful for future research in this direction.\n3. The paper is well-written and easy to follow.", "weaknesses": "1. It is a well-known problem in the RL community that providing dense rewards can be better than providing sparse rewards, and there is a research direction on reward shaping that is very relevant to this idea. Therefore, the idea of providing intermediate rewards for training can lack novelty.\n2. The paper mainly conducts experiments in a math dataset, where it is easy to provide step-by-step intermediate rewards. However, there are many other tasks such as essay writing and story generation where it can be hard to provide such feedback. Therefore, it is unclear if the method is generalizable.\n3. It is hard to draw insights from their experiments. It would be better to see the advantages of using step-level rewards in more dimensions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Yib7DkqJwy", "reviewer": "Reviewer_5YZq", "summary": "In this work, the authors conducted an investigation to determine whether the outcome supervision method or the process supervision method is more effective for training reliable models. They found that process supervision produces more reliable models, and that a large reward model can accurately approximate human supervision for smaller reward models. Additionally, they discovered that active learning can improve the data efficiency of process supervision by 2.6 times. The authors have also released their comprehensive process supervision dataset, named PRM800K.", "questions": "Do the authors have more results on significantly different datasets?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.\tThis work offers an independent and thorough investigation into the superiority of outcome supervision versus process supervision methods, yielding new and valuable conclusions.\n\n2.\tThe authors introduce an active learning method to enhance the data efficiency of process supervision, making it more practical for real-world applications.\n\n3.\tThe authors have released their extensive human-labeled dataset.", "weaknesses": "The experiments were conducted solely on examination datasets. For more robust conclusions, it would be beneficial to conduct experiments on a broader range of significantly diverse datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)