Solving Math Word Problems with Process-based and Outcome-based Feedback
Authors: Jonathan Uesato
Nate Kushman
Ramana Kumar
H. Francis Song
Noah Yamamoto Siegel
Lisa Wang
Antonia Creswell
Geoffrey Irving
Irina Higgins
Venue: Submitted to ICLR 2023 (2023)
Keywords: language models
reasoning
reward models

Abstract: Recent work has shown that prompting language models to generate reasoning steps improves performance on many reasoning tasks. When moving beyond prompting, this raises the question of how we should supervise the finetuning of such models: outcome-based approaches which supervise the final result, or process-based approaches which supervise the reasoning process itself? Differences between these approaches might naturally be expected not just in final-answer errors but also in reasoning errors, which can be difficult to detect and are problematic in many real-world domains such as education.  We run the first comprehensive comparison between process- and outcome-based approaches trained on a natural language task, GSM8K. We find that pure outcome-based supervision produces similar final-answer error rates with less label supervision. However, for correct reasoning steps we find it necessary to use process-based supervision or supervision from learned reward models that emulate process-based feedback. In total, we improve the previous best results from 16.8% $\rightarrow$ 12.7% final-answer error and 14.0% $\rightarrow$ 3.4% reasoning error among final-answer-correct solutions.

Under review as a conference paper at ICLR 2023
SOLVING MATH WORD PROBLEMS WITH PROCESS-BASED AND OUTCOME-BASED FEEDBACK FEEBACCK
Anonymous authors Paper under double-bling review
ABSTRACT
Recent work has shown that prompting language models to generate reasoning steps improves performance on many reasoning tasks. When moving beyond prompting, this raises the question of how we should supervise the finetuning of such models: outcome-base approaches which supervise the final result, or process-based approaches which supervise the reasoning process itself? Differences between these approaches might naturally be expected not just in final-answer errors but also in reasoning errors, which can be difficult to detect and are problematic in many real-world domains such as education. We run the first comprehensive comparison between process- and outcome- outcome-based approaches trained on natural language task, GSM8K. We find that pure outcome-based supervision produces similar final-answer error rates with less label supervision. However, for correct reasoning steps we find it necessary to use process-basee supervision or supervision from learned reward models that emulate process-based feedback. In total, we improve the previous best results from 16.8% + 12.7% final-answer error and 14.0% >> 3.4% reasoning error among amal-answer-corre solutions.
INTRODUCTION
Recent work has shown that asking language models use step-by-step reasoning improves perfor- mance on reasoning tasks (Shwartz et al., 2020; Nakano et al., 2021; Cobbe et al., 2021; Wei et al., 2022; Kojima al., 2022: Lewkowyez et al., 2022). While these works have primarily focused on prompting language models, prior work suggests that finetuning should outperform prompting alone (Stiennon et al., 2020; Perez et al., 2021; Ouyang et al., 2022). This raises the question of how best to supervise such models. Two natural approaches are outcome-basse approaches, which supervise the final result, and process-based approaches, which supervise each step of the reasoning process, including the last step outputting the final result.
In this work, we conduct the first comprehensive comparison between process- and outcome-based approaches trained (Cobbe et al., 2021) of math word problems. In all cases, we generate a sequence of reasoning steps leading the final answer, but vary whether not supervision is provided only the final answers suttcom--based or on individual reasoning steps pproeess-based). For process-based approaches we consider supervision provided by both offline Human--geneatedd reasoning traces from the GSM8K dataset itself, as well as online human correctness annotations, which we collect for each step of genee-ggeneraed samples. We compare these approaches in the context of number of different modeling and training components, including: few-shot prompting, supervised the-tuning, reinforcement learning (RL) via expert iteration, and reward modeling for both reranking and RL.
natural language task. For this,
the recently proposed GSMSK dataset
Throughout, we consider two primary metrics: trace error rate, which measures how often the model makes any mistake in its reasoning trace according to human annotators, and final-answer error rate, which only considers the model's 'ss final answer and ignores the reasoning trace. By "reasoning trace" we refer to all steps reasoning, including the last step which in GSM8K the final numeric answer. While process-based approaches may provide multiple benefits (discussed further in Appendix B). including encouraging human understanding of the problem domain, here concentrate investigating their effect the trace rate is directly measurable and of interest in many settings. For example, in educational settings, an answer without an (understandable) explanation may often confuse more than it explains. Recent
rate. We do so because trace error

---

Under review as a conference paper at ICLR 2023
findings suggest that outcome-basee approaches often lack in this area. For example, work natural-laguuage-baaaaa reasoning (Zelikman et al., 2022; Creswell et al.. 2022) suggests that models optimized exclusively for annwer correctness often produce the correct final answer, when their generated reasoning traces are incorrect.
Key results We find that our best approach, which combines supervised learning with reward- model-based reinforcement learning, significantly improves the state-of-the-art for both trace error rate, from 14.0% - 3.4%, and final-answer error rate, from 16.8% - 12.7%. Final-answer error rate is further lowered to 2.7% when the model allowed abstain on 30% of questions. Our key findings regarding process- and outcome-based feedback are as follows:
Outcome-based and process-basee approaches lead to similar final-answer error rates, Both without reward models (23.5% vs. 22.2%) and with reward models (16.6% vs. 14.8%), language models supervised with final-answer correctness attain nearly the same final-answer error rate as those trained to imitate human- human-providee provided solutions.
Both process- and outcome-supervised reward models learn to emulate process-based feed- back. Somewhat surprisingly, find that even reward models trained with outcome-based labels (indicating whether the final answer correct), result in predictions that agree more closely with the process-based labels (indicating whether each reasoning step correct) than they do with the outcome-based labels themselves. While this effect may be dataset-specific, as discussed in Section 3, it helps explain the effectiveness of reward models for improving trace error, and we hope that it is investigated further in future work.
Low trace error requires either process- feedback, or reward model that emulates it. All models using reinforcement learning directly against final-answer correctness resulted in high trace eтror, with a best trace eтror of 12.4%, compared to only 3.8% for our best process-basee based method. Building on our previous finding, most of this gap closes when using reinforcement learning against a reward model rather than final-answer correctness directly, which reduces trace error to 5.5%.
METHODS
This section describes the dataset, evaluation metrics and the different modelling components evalu- ated in this paper. See Fig. for an overview of how they all fit together.
2.1 DATASET AND EVALUATION METRICS
We conduct all experiments the GSM8K dataset (Cobbe et al., 2021), composed of grade school math word problems. We chose GSM8K because it is a competitive benchmark, and contains natural language reasoning traces. We focus on single dataset, since the need to recruit human annotators with the domain expertise to accurately evaluate reasoning traces imposes a large up-front cost. See Section 4 for discussion of other datasets we considered. Table 2 and Appendix I show several example problems. We split out our own validation set of 256 examples from the original training set, which leaves us with 1118 training and 1319 test examples.
We report two main metrics for all methods evaluated on the GSM8K test set. Final-answer error rate the fraction of problems for which the method does not produce the correct final answer. Because all final answers on GSM8K are integers, this can be measured with exact string matching. Trace error rate the fraction problems with correct final answers for which the method produces at least one incorrect reasoning step. We estimate this via human annotations of the correctness of each reasoning step, which is detailed Section 2.7.
We report final-answer and trace errors as two separate metrics because are particularly interested in errors which remain undetected after checking easily verifiable metrics (in this case, final-answer errors). For example in educational setting it is important show student the correct steps get the answer, and we can easily filter out incorrect traces that lead to the wrong answer, but is much more difficult to filter out incorrect traces that lead the correct answer.
2

---

Under review as a conference paper at ICLR 2023
coveK Full Aensorss
Macsely Ducoding
Prizared Nodal
Cota Arrstaziinn
inper Rnowar
Base Largage
FeoiShel
Beneking Mookd
Prompting
Stene Sampleo
Mocil Iro Bonatonn
COMES
Filand Samples iPp. 22
Haman Annoadicn
Ouique blerec
Figure 1: Method Overview. This schematic provides overview of the various modeling and training components considered and how they fit together. Some details (covered in the text) are ommitted for readability.
2.2 TRAINING: OVERVIEW
Our goal is to train system for the ceeuueece--o-seqeeee task (Sutskever et al., 2014) of taking the text problem as input and generating the text of an answer as output. For math word problems, the answer is full reasoning trace: newline-separatee sequence of steps, where the last step is expected to provide the final answer. For GSM8K, the final answer is always an integer.
Our approach broadly follows prior work on RL for language models (LMs) (Ziegler et al., 2019; Nakano et al., 2021; Menick et al., 2022). We use an LM as a policy, which maps the problem statement and steps-so-far to next step. In the RL formalism, this treats each step and the observation is provided by all the tokens so far. The policy can be obtained through any of few-shot prompting, supervised finetuning (Section 2.3), RL (Section 2.6). We also train LMs as reward models (Section 2.4), which score proposed full or partial completions from the policy, and can be used both for reranking samples from the policy, or the source of rewards during reinforcement learning. All of our models are based on a 70 billion parameter pre-trained LM. In the following subsections, we describe how we train and assemble these components.
action,
2.3 SUPERVISED FINETUNING
In supervised finetuning (SFT), we finetune an LM to maximize the log log-likelihood of a sequence of target tokens, given sequence of input tokens. In our paper, we use SFT as a process- based approach by taking the reasoning traces provided the GSM8K dataset as the target tokens (as opposed to the outcome-based approach of using only the final answer as the target), with the problem statement as the input tokens.
Training details We finetune using AdamW (Loshchilov and Hutter, 2017) with learning rate of 2 x 10-6 and a batch size of 256. We stop finetuning once the language modeling loss begins to increase on the validation set. For our SFT model, this happens after 70 steps, amounting to slightly more than 2 training set epochs.
2.4 REWARD MODELS
We evaluate two main approaches to training reward models (RMs) (Christiano et al., 2017: Ziegler et al., 2019; Menick et al., 2022), also known as verifiers (Cobbe et al., 2021). both approaches, we implement the RM token after each step. In the snccome-supervised RM (ORM), the binary label for each step indicates whether the resulting final answer of that full sample matched the reference final answer, proposed by Cobbe et al. (2021). A policy which maximizes the ORM score at each step thus maximizes the RM-estimated probability at each step of eventually reaching the correct final answer. For the process- supervised RM (PRM), the binary label after each step indicates whether the steps so far are correct. Because we lack reliable programmatic means for determining the correctness of intermediate steps, we use human annotations for these labels, as described in Section 2.7. A policy which maximizes the PRM score thus selects each step to maximize the RMMestimated probability the steps so far being correct. If the steps so far are correct, this typically means such policy minimizes the probability of introducing a mistake on the current step. As reported in Appendix A, we find this outperforms the
LM, trained to predict a binary label as either 'correct' or 'incorrect'
We withhold the exact reference to the hase model for the duration of the double blind review period to
preserve anonymity.
3

---

Under review as a conference paper at ICLR 2023
Final Answer RL
Eaickon Sampla
Filsed Sanper
Hour Quedien
GOVER
Generited Samples
ORM-RL
Find
Proteens
Sanple
PRM- RL
Soudinn Sumpis
Fiaardd Futl Saidion Sampl
VOVEK
Problars
Figure 2: Policy improvement. This schematic summarizes the different policy improvement operators for each of Final-Answer RL, ORM-RL, and PRM-RL, described in the main text.
approach from Li et al. (2022), which is similar to our PRM but replaces human evaluations with a heuristic based on exact string matching with the results of the intermediate calculations.
Unless otherwise noted, for all approaches which include an ORM, we train the ORM using samples from the policy for that approach, taking K = 96 samples with temperature 1.0. We follow Cobbe et al. (2021) and regularize with dropout, with a dropout parameter of 0.1, and otherwise reuse the hyperparameters used for SFT from Section 2.3. To speed up learning in the SFT-based approaches, we initialize the ORM training using the SFT model parameters, while for the few-shot based approaches we initialize from the base pretrained LM. For the PRM, we annotate samples per problem from the SFT policy, restricting to problems where the SFT majority prediction (see Section 2.5) was incorrect, in order to make the most of our human annotation budget. Due to the small size of our human-annotate dataset (1560 full solutions), we initialize the PRM parameters to the ORM parameters and lower the learning rate to x 10-? The RM loss curves have some fluctuation, and so we select the RM with the best validation loss before 2000 steps.
Training details
2.5 DECODING
For test-time decoding, we first sample K = 96 full solutions from the policy, and then select the best sample, either by ensembling or with an RMM. We sample with temperature T 1.0, and use the syntax from Cobbe et al. (2021) to allow the model to decide when to use calculator. In early experiments, we also tried RM reranking after each generated step (rather than the full solution), but found that this led to slightly worse performance, increasing final-answer error by 1-2%.
We use two approaches to select the best sample. When no RM is available, we use majority voting. For this, we first select the most common final answer from the K samples, then select random sample from among those yielding this selected final answer. This is called coniisenst by Wang et al. (2022), and is similar to more general techniques like Minimum Bayes Risk decoding (Kumar and Byrne, 2004). Otherwise, we use RM-weighted decoding, also called verifier-voting by Li et al. (2022). Here, we weight each sample according to the RM-estimated correctness probability, select the final answer with the largest total weight, and then select the sample with the highest RM score from those yielding the selected final answer. More formally, we select the final answer f = arg max, fi. fg..tfnal_ann(yy)== ans(y.)=; rm htmmmmppp_____ where y1..... VK are the model samples, then select the best sample according y² arg max,; yyfnnllLa = rm_ anm_rrob(y:). This works slightly better compared to simply selecting the sample with the highest RM score (improving final-answer error rate about 1 1% with the SFT model, and slightly more with the RL models). However, we note that both majority voting and RM-weighted decoding are slightly less general due to their reliance on exact string-macchin between final answers.
2.6
RL VIA EXPERT ITERATION
All RL experiments use expert iteration (Silver al., 2017: Anthony et al., 2017). As meta- algorithm, expert iteration alternates between two high-level operations. In policy improvement, we combine base policy with a search procedure to produce samples from a so-called expert policy. Then in distillation, we perform supervised learning on these expert samples to improve the base policy towards the expert policy. We use 5 epochs and select the best model of the 5, based on final-answer test error with RM-weighted decoding, or majority voting if no RM is available.
SFT-based vs. New-shot-based implementations The initial base policy can be either the SFT policy, 5-shot prompted version of our base L.M. We particularly note that, aside from the
4

---

Under review as a conference paper at ICLR 2023
Error rate (%)
Approach
Base model
Trace
Final-answer
Few-shot (Wang et al., 2022; Wei et al., 2022)
PaL.M-540B
14.0
25.6
Few-shot (Lewkowycz al., 2022)
Minerva-540B
21.5
Few.shot+Final-Annwerr RL (Zelikman, 2022)
GPT-J-6B
89.3
Few-shot, ORM reranking (Li al., 2022)
Codex-175B
16.8
Zero-shot (Kojima et al., 2022)
InstructGPT-175B
59.3
SFT, ORM reranking (Cobbe al., 2021)
GPT-175B
45.0
Few-shot, Majority Voting
Our Base-70B
41.5
No RM
Few-shot--Final-nnsee RL, Majority Voting
Our Base-70B
19.8 (7.9-31.7)
23.5
SFT+Final-Answer RL, Majority Voting
Our Base-70B
12.1 (4.6--99.)
20.2
SFT, Majority Voting
Our Base-70B
(4.8-188)))
22.3
Few-shot, ORM reranking
Our Base-70B
27.8
RM Rerank
Tewn-shot-Finnl-Annwer RL, ORM reranking
Our Base-70B
(2.1-222))
16.6
SFT+Final-Answer RL, ORM reranking
Our Base-70B
3.7 (0.5-6.9)
14.2
SFT. ORM reranking
Our Base-
4.4 (0.6-8.3)
14.8
SFT. PRM reranking
Our Base-70B
3.5 (0.5-6.5)
14.1
RM-RL
Few-shot+ORM-PR ORM reranking
Our Base-700 70B
5.5 (2.6-88)))
13.8
SFT+ORM-RL, ORM reranking
Our Base-70B
3.4 (0.0-6.8)
12.7
SFT+PRM-RL, PRM reranking
Our Base-70B
3.8 (0.5-7.1)
12.9
Table 1: Results overview. We show trace and final-answer table alongside or after the description of key results in Section 3. Trace error rates are averaged across raters. In parentheses, we provide min-max range, depending on whether errors require both raters to agree (min) or just a single rater (max). While there is significant noise in the trace error rates, we can still observe general trends. Within each group, we list approaches in order from most outcome- based (top) to most process-basse based (bottom), other than the few-shot model, which has no finetuning procedure to supervise.
rates. We suggest reading this
random training examples used for the prompt, none of the few-shot-based approaches ever use the intermediate reasoning steps provided in the GSM8K dataset, our human annotations, or any models derived from this data. When initializing from the SFT model, we follow Polu and Sutskever (2020) and reuse expert samples from each iteration, so that our training set grows each epoch. We do not do this with few-shot approaches because in that setting, the samples from the early epochs have many trace errors which we do not want the RL model to imitate. Correspondingly, there are several minor implementation differences between the two cases, which we note throughout our detailed descriptions of the policy improvement and distillation procedures.
Policy Improvement We consider three versions of the policy improvement procedure (Figure 2). In the Final- anal-answer RL approach, also called Self Self-t Reasoner and proposed by Zelikman et al. (2022), we generate K full traces per problem and filter by inal-answer correctness. For the few-shot version, we select all traces yielding the correct final answer, while for the SFT-based version, we only use randomly chosen sample per problem. In the ORM-RI. approach, generate K full traces per problem, and select the sample with the highest score according to the ORM model. In the PRM-RL approach, we instead treat each step an individual episode. each step, we generate K candidate steps, select the candidate with the highest PRM score, and continue from the selected step until the model outputs step with the final answer indicator text, or a maximum of 15 steps. We set K = 96 across all experiments. For few-shot sew-shot-based approaches, we retrain the RM after every expert iteration. For SFT-based approaches, we skip this step and use a fixed RM, since somewhat surprisingly, this did not make a significant difference in preliminary experiments.
For distillation, we use the same hyperparameters SFT. As with SFT, we apply early stopping by validation loss, where our validation set is constructed from expert policy samples on the validation set. For SFT-based approaches, we initialize with the SFT parameters at each distillation step, while for few-shot-based approaches, we initialize with the base model parameters.
Distillation
2.7 DATA ANNOTATION
As discussed Section 2.4, the PRM trained on stepwise labels indicating whether the steps so far correct. To collect this data, present human annotators with the problem statement, the
5

---

Under review as a conference paper at ICLR 2023
ORM
byse
ORM
ORM
PRM
4-
1.00
.88
0.77
0.87
0.85
PRO
Policy type
Feve shot
Few Few-stoo
0.95
0.76
Thace error (%6)
10
ORH
0.90
PRM
ORM
0.77
0.76
12-
0.85
14-
1
Vlang et-il, 2022
16-
PRM
16-
20
27.5
25.0
22.5
20.0
17.5
15.0
12.5
Figure 4: Agreement matrix between RMs and RM labels We focus on the ORM predictions (yel- low box), though provide the full agreement ma- trix for completeness (which is symmetric by con- struction). Notably, the ORM model has higher agreement with the PRM labels compared to the ORM labels. Results are for steps on the PRM validation set.
Final onswer (%)
Figure 3: Trace and final answer errors. This summarizes Table I in more visual format. Bet- ter results are up and right. Vertical lines are used where trace eтror is not available. Approaches that off the chart (due to large error rates) have their final-answer error rates shown in brackets.
reference solution from GSM8K, and the generated model solution, and ask them to indicate the first model step with major mistake, if any exist. Our instructions define major mistake as "a step where the information expressed is incorrect, or it would no longer be possible to reach the correct solution without undoing that step". From these annotations, we can label every step with binary label indicating whether the steps so far are correct: all steps before the first major mistake are labeled 'correct', while the remainder are labeled 'incorrect'.
We applied small amount of dataset cleaning by removing samples from annotators with low inter-annotator agreement (measured on the 20% of solutions where we used duplicate labelling). as well as those from GSM8K problems flagged by annotators as ambiguous. This removed about 20% of our data, leaving annotations for 1560 model samples across 530 training set problems, corresponding to 9856 step-level binary labels. For the validation set, we used the same procedure, but added duplicate labelling and a manual pass by the paper authors to resolve inter-annotator disagreements. Our validation set contained 162 model samples, with 913 total steps. For evaluation, we used 200 problems with correct final answers per model. This was done for each the 10 models in Table 1, again with duplicate labelling. We describe full details of our data collection procedure in Appendix F.
RESULTS
Our results are summarized in Table 1 and Fig. 3. The ORM-RL and PRM-RL models achieve final-answer error rate below 13%. improving on the 16.8% final-answer eπror for the current state-of-the-arr model (Li et al., 2022). This is despite the fact that Li al. (2022) use base model which is better suited to math (Codex- 41.5% for our few-shot base model. Our final-answer error rate is further reduced to 2.7% when the model is allowed to abstain only 30% of questions. The ORM-RL and PRM-RL trace error rates, of 3.4% and 3,8% respectively, significantly improve on the 14% reported by the best prior work (Wang al., 2022; Wei al., 2022). Beyond these quantitative results, highlight four key takeaways:
reporting 23.3% few-shot final-answer error, compared
Supervising final-answer correctness alone suffices for low final-answer error rate. The SFT and Few-shot+Final -nnsser RL models attain similar final-answer error rates both without an RM (22.3% vs. 23.5%) and with an ORM (14.8% vs. 16.6%). ). This is notable, Few-shot+Final-Ansee Answer RL only requires supervision of the final answers, rather than the full reasoning traces. Put another way. Few-shot+Final-Ansse
RL uses 1-4 tokens of label supervision per question, while SFT uses hundreds. This suggests that in cases where inal-answer correctness is sufficient, outcome- based approaches can provide label-efficient approach with competitive performance.

---

Under review as a conference paper at ICLR 2023
Tyrion changes his face mask two times every time he goes out. If he goes out three times day. how many face masks does he use every 2 days?
Problem:
Model answer (Trace incorrect, final answer correct): Tyrion goes out times every day and changes his face mask 2 times every time he goes out so that's 's a total of 322=63*2=6m face masks every day.
->
Tyrion goes out 6 face masks every day and there are 2 days in day that's 2=======111 12 face masks every days.
Final answer: 12
Table 2: Trace error example. In this hand-picked example from the Few-shot+-inal-Annse RL model, the final answer is correct, but the reasoning steps are not. The - symbol indicates the incorrect step. Because Final-Answer RL only optimizes for final-answer error, the intermediate steps need not be understandable to a human evaluator, or correct under standard English semantics.
ORM-supervised reward models approximate PRM labels. Despite the fact that ORMs are only trained to predict whether the final answer is correct, we can see in Fig. 4 that ORM predictions tend to agree more with the PRM labels than with the ORM labels themselves (85% vs. 77% averaged over all steps). 2 We suspect this is because it is simpler for the ORM to learn to recognize when steps are correct, than it is to check the answer by internally computing the final answer itself. This is further supported by that fact that, even though trace error is measured only on samples with the correct final answer, ORM reranking significantly improves trace error relative to SFT alone (4.4% vs. 11.4%). This suggests that RMs are checking the reasoning steps, and not just the final answers. However, we caution against over-generalizing: the fact that the ORM model approximates the PRM labels may be domain-specific. This may depend on both the relative difficulty for the model to compute the correct answer with and without reasoning traces, and the lack of spurious solutions (Goldman al., 2017) in math problems, where incorrect reasoning steps are unlikely to lead to the correct final answer.
Low trace error requires either process-based feedback or a reward model that emulates it. Fig. 3 shows that despite similar final-answer error rates, there is a significantly higher trace error rate for the outcome-based Few-shot+Finnl- Finnl-----al- RI vs. the process-based SFT model (19.8% vs. 11.4%). This discrepancy persists with RM reranking: New-shot+Final-Answe RL with ORM reranking underperforms SFT with ORM/PRM reranking (12.4% vs. 4.4%/3.5%) However, we find that when train the few-shot RL model using ORM (Few-shot+ORM-RL. rather than training directly against final-answer correctness, the trace error drops significantly from 12.4% to 5.5%, closing much of this gap. We believe this results from the previous finding, i.e. that the ORM is basically learning to emulate the PRM allowing the model to learn from emulated process- based feedback and resulting in relatively low trace error rates.
Selective prediction greatly reduces final-answer error, particularly for models with low trace error. In many practical applications, it is possible to abstain. For instance, if ML system was used to explain a problem to a student, or assist in calculation, it would be preferable for the model to abstain rather than produce an incorrect output. This motivates the selective prediction setting (El-Yaniv et al., 2010; Geifman and El-Yaniv, 2017), where the model is allowed to abstain on r% of inputs, and selective error rate is measured on the inputs where the model does not abstain. To determine which inputs to abstain on, we set a threshold with the threshold determined by r. Figure 5 shows that by abstaining on r = 30% of inputs, we reduce final-answer error rate from 14%->2.7% which can be further reduced to 1.5% r 50%. Further, at r 30,, selective prediction with SFT and the ORM or PRM yields a 5x final-answer error reduction, compared to 3x in the www-shottFinnl-Anweee RL case. This may be related the improved trace error for SFT: when trace error is lower, the RM can more reliably use intermediate step correctness to determine its confidence. However, further investigation would be necessary to properly understand this effect.
the RM score of the selected sample,
The results in Table also lead a few additional conclusions (discussed in detail in Appendix A):
Finetuning significantly improves performance over prompting alone, as expected.
Reward model reranking significantly boosts both trace and final answer accuracy.
RL significantly improves few-shot performance, but provides more modest gains on top of SFT.
Optimizing against an RM outperforms optimizing final-answer correctness directly, on both trace and final-answer error.
AAppendix F shows similar results when considering just the final step.
7

---

Under review as a conference paper at ICLR 2023
12
Selective error (%)
SFT, ORM
SFT. PRM
reduction factor
10
SFT, GRM
SFT, PRM
12
Answer RL. ORM
Fowt-shot Final-
Arswer RL.
Few-shot
3-
0.0
0.2
0.4
0.0
0.1
0.3
0.5
Abstain rate
Abstain rate
(a) Selective error rate
(b) Error reduction factor from selective prediction
Figure 5: Selective error. By abstaining, we can greatly decrease the error rate on answered questions. (Left) For the SFT policy with PRM reranking, final-answer error falls from 14% to 2.7% when abstaining on 30% of inputs, a roughly 5x reduction factor. (Right) Models trained with SFT are significantly more amenable selective prediction compared those trained via Final-Answer RL.
Finally, in other experiments, we found that models trained on GSM8K generalize well out of domain, with our O MRM-RL model with majority voting achieving a final-answer error rate of 64.6% on the pre-algebra split of the MATH dataset (Hendrycks et al., 2021). See Appendix G for more details.
RELATED WORK
Concepts related to process- and outcome-based feedback In this work, we focus on different forms of supervision available for training LMs, framed in terms of spectrum between process- and outcome-bare bosoee approaches. This framing has been used in blog posts (Stuhlmüller and Byun, 2022) and informal discussion, though to our knowledge this the first empirical paper to discuss it. Similar distinctions from past work include that between approval-based and reward-based reinforcement learning (Knox and Stone, 2008) and between strong and weak supervision (Reed and De Freitas, 2015). Appendix D discusses these in relation to process- and outcome-bass based feedback.
Solving math problems with LMs Math word problems have been popular domain for studying reasoning LMs (Kushman et al., 2014; Ling et al., 2017; Amini et al., 2019; Miao et al., 2020; Hendrycks et al., 2021; Cobbe et al., 2021). Several recent papers have demonstrated that few-shot prompting alone can lead to impressive performance on GSM8K (Chowdhery et al., 2022; Lewkowycz et al., 2022: Wei et al., 2022: Wang et al., 2022). All of these papers include the reasoning traces in the few-shot prompts which encourages the model to generate verbalized reasoning steps (also referred to self-talk (Shwartz et al., 2020) and chain-of-thought prompting (Wei al., 2022)). Few-shot prompting benefits from training on mathematical content (Lewkowycz et al., 2022), as well as finetuning for instruction following (Ouyang et al., 2022). Kojima et al. (2022) and Li al. (2022) show that while few-shot final-answer error rate is 81.1% for GPT-3 (Brown et al., 2020), this improves 41.8% for InstructGPT (Ouyang et al., 2022) and 23.3% for Codex (Chen al., 2021).
We focus on finetuning because we are interested in the effects of different feedback procedures, and because significantly outperforms prompting alone for our base LM. The original GSM8K paper (Cobbe et al., 2021) demonstrated significant improvements from reward models or verifiers, and we use their ORM approach. et al. (2022) also study RMs and propose heuristic-based step-aware RM, which slightly degrades performance on GSM8K, but boosts performance on a wide range of other benchmarks. We find that human evaluations each step provide an improvement. We also use STaR (Zelikman al., 2022) (referred to as Few-shot+ Fiw-shoot-innl-Anseer RL throughout this paper), and show its GSM8K final-answe answerr error can be reduced from their reported 89% to 23.5% through the of better base model and further reduced to 13.8% by using RM--dsed RL instead of their final answer RI. procedure. In contrast to the above prior work, we not only show improved performance. but also provide comprehensive comparison across different types of feedback, with focus trace error rate in addition to final-answer error rate.
Multistep reasoning in LMs Moving beyond math problems, a large body of work studies mul- tistep reasoning for LMs. While full review is beyond the scope of this work, discuss a few representative categories approaches. Prior work has suggested improvements to the base model (Lekkowycz al., 2022; Ouyang al., 2022), as well prompt-based approaches (Perez al.,
8

---

Under review as a conference paper at ICLR 2023
2020; Shwartz al., 2020; Wei et al., 2022; Kojima et al., 2022). Our focus supervision techniques for finetuning is complementary to such improvements.
Past work has focused either exclusively the process-basee side, Wu et al. (2021) summarize full-length books by recursively composing summaries of shorter segments. Creswell et al. (2022) and Nye et al. (2021) process-basee SFT in synthetic settings where reasoning traces can be synthesized programatically. On the outcome- based side, Zelikman et al. (2022) answer multi-step reasoning questions. In contrast to all of this work, we directly compare outcome-basee and process-based techniques on a natural language task, and include detailed analysis of trace error rates.
outcome-based or process-baset approaches. On
To our knowledge, the prior work which most directly compares process- and autcome-based feedback for multistep LM reasoning WebGPT (Nakano al., 2021). In WebGPT, intermediate steps are web browser interactions, which they supervise either via SFT, or via outcome-based RL on resulting answers. Similar to us, they observe significant improvements over SFT from outcome-basee RM reranking, but adding full-solution RL on top has minimal effect. However, we more comprehensively explore both process- and outcome-basse supervision, additionally evaluating process- process-supervised RMs, the PRM-RL approach, and purely outcome-supervisel RL policies (without SFT). This allows us to draw broader conclusions regarding the effects of supervision on both final-answer and trace error rates compared to prior work.
Comparisons between process- and outcome-based approaches to algorithmic tasks. Most prior work on head-to-head comparisons of process- and outcome-based approaches has been on algorithmic tasks such as sorting numbers, which avoid working with human data. Early outcome- based approaches such as Neural Turing Machines (Graves al., 2014) are trained end-to-end to predict the final answer. In contrast, Neural Prgrrammer-nttepreeeess (Reed and De Freitas, 2015; Li et al., 2016; Cai et al., 2017) are trained to imitate execution traces, requiring stronger supervision but improving generalization. Iterated Amplification (Christiano et al., 2018) uses a bootstrapping procedure to train models to approximate a potentially exponentially sized tree of reasoning steps. Our work extends these results natural language, where neither programmatic execution traces nor decomposition procedures are available, and these must be learned from human feedback.
Related datasets In this work we chose to work with the GSMSK dataset, because it provides natural language reasoning traces that allow for detailed comparison between process- and outcome- based approaches without requiring us to collect the traces ourselves. Alternative datasets which also contain full reasoning traces include EntailmentBank (Dalvi et al., 2021), Strategy (Geva et al., 2021), ProofWriter (Tafjord et al., 2020), and CLUTTR (Gontier et al., 2020). However, compared to GSM8K, these datasets either contain templated problems (ProofWriter and CLUTTR) or are significantly smaller in size (Entailmen/Bank and StrategyQA), Furthermore, working with multiple datasets would be expensive because of the need to train human annotators for each task, and to collect a significant number of human feedback annotations for each dataset.
5 CONCLUSION
In this work, we the first comprehensive comparison between process- and sutcome-bbseed supervision on a natural language task. We find that process- and outcome-based feedback have different strengths, and the appropriate choice will often be context-dependent. In our results, both types of supervision lead to similar final-answer error rates, with our best models improving the state- of-the-art final-answer error on GSM8K from 16.8% to 12.9% when using process-based supervision and to 13.8% when using srrcome-based supervision. In contrast, obtaining low trace error requires either process-basee based supervision, or a reward model that emulates A purely process-baas based approach of SFT with PRM reranking reduces the state-of-the-arr trace error rate from 14.0% to 3.8%, while analogue achieves 12.4% trace error. However, somewhat surprisingly, we find
its outcome-base that reward models trained with outcome-based labels result in predictions that agree more closely labels than they do with the outcome- based labels themselves. By using this
with the process-basee reward model during RL training, close most of this gap, reducing trace error from 12.4% to 5.5%. While some of these conclusions may be specific to our setting of math word problems, we hope that future work explores the extent which they generalize to other domains.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
A. Abdolmalcki, J. T. Springenberg, Y. Tassa, R. Munos, N. Heess, and M. Riedmiller. Maximum posteriori policy optimisation. arXiv preprint arXiv: 180.00690000 2018
A. Amini, Gabriel, Lin, R.
Y. Choi, and Hajishirzi. Mathqa: To- wards interpretable math word problem solving with operation-based formalisms. arXiv preprint 1rXi::1905.13399 2019.
T. Anthony, Z. Tian, and D. Barber. Thinking fast and slow with deep learning and tree search. Advances in Neural Information Processing Systems, 30, 2017.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners, Advances in neural information processing systems, 33:1877-1901, 2020.
Cai, R. Shin, and Song. Making neural programming architectures generalize via recursion. arXiv preprint arXiv: 1704.06611, 2017.
M. Chen, J. Tworek. H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, Tang. I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.033744 2021.
URL https:::pp httpt:://aa.iiiii... /aaxivv.vv/////////// orgg abs 220/ 03374.
A. Chowdhery, Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint WwXiv:200.002311, 2022.
P. Christiano, B. Shlegeris, and D. Amodei. Supervising strong learners by amplifying weak experts. arXiv preprini arXiv: 111.00881000855. 2018
F. Christiano, J. Leike, Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
K. Cobbe, V. Kosaraju, M. Bavarian, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.1414 1466.8 2021.
A. Cotra. formative ai likely leads //www/ www.:raaggnnnnnteeeemm forum without specificc
Without specific countermeasures, the easiest path to trans-
ai takeover, 2022.
URL https:
counte
easiestesssttttee pptt-to.
A. Creswell, M. Shanahan, and Higgins. Selection-inferenee. Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09971. 2022.
B. Dalvi, P. A. Jansen, O. Tafjord, Z. Xie, H. Smith, L. Pipatanangkura, and P. Clark. Explaining answers with entailment trees. ArXiv, abs/2104.08661, 2021.
M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and E. Kaiser. Universal transformers. arXiv preprint 180iv: 180.008807.0., 2018.
R. Yaniv al. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(5), 2010.
T. Everitt, V. Krakovna, I.. Orseau, M. Hutter, and S. Legg. Reinforcement learning with corrupted reward channel. arXiv preprint 1rXiv:1705.08417. 2017.
Y. Geifian and R. EI- Yaniv. Selective classification for deep neural networks. Advances in neural information processing systems, 30, 2017.
10

---

Under review as a conference paper at ICLR 2023
M. Geva, D. Khashabi, E. Segal, T. Khot, D. Roth, and J. Berant. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Transactions of the Association for Computational Linguistics (TACL), 2021.
O. Goldman, V. Lateinnik, U. Naveh, A. Globerson, and J. Berant. Weakly- semantic parsing with abstract examples. arXiv preprint arXiv: 2017.
N. Gontier, K. Sinha, S. Reddy, and C. Pal. Measuring systematic generalization in neural proof generation with transformers. Advances in Neural Information Processing Systems, 33:22231 22242, 2020.
A. Graves. Adaptive computation time for recurrent neural networks. arXiv preprint vrXiv 2016.
A. Graves, G. Wayne, and I. Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014.
A. Guez, M. Mirza, K. Gregor, R. Kabra, S. Racanière, T. Weber, D. Raposo, A. Santoro, L. Orseau, T. Eccles, et al. An investigation of model-free planning. In International Conference Machine Learning, pages 2464 2473. PMLR, 2019.
D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2 103.03874 2021.
G. Irving, P. Christiano, and D. Amodei. Ai safety via debate. arXiv preprint 11XXv:880500089) 2018.
Z. Kenton, T. Everitt, L. Weidinger, I. Gabriel, V. Mikulik, and G. Irving. Aligmment of language agents. arXiv preprint 100i::2003.16552 2021.
W. B. Knox and P. Stone. Tamer: Training an agent manually via evaluative reinforcement. In 2008 7th IEEE international conference development and learning, pages 292-297, 297. IEEE, 2008
T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models zero-shot reasoners. arXiv preprint arXiv:200..11116 2022.
V. Krakovna, J. Uesato, ton, J. Leike, and S. Legg. Specification gaming: the flip side of ai inge- nuity, 2020.
Mikulik, M. Rahtz, T. Everitt, R. Kumar, Z. Ken-
http:///dddeppraafffeeeeeeeeeee medi um com/ speci fi caaaiinnngaa gami ng-the-flis wiwe-odd-----ee a www---of---iiiinninee ayl--ingentttttttobbeeeee
URL
R. Kumar, J. Uesato, R. Ngo, T. Everitt, V. Krakovna, and S. Legg. REALab: An embedded perspective on tampering. arXiv preprint arXiv:201 1.00020, 2020.
Kumar and W. Byrne Minimum bayes-risk decoding for statistical machine translation. Technical report, JOHNS HOPKINS UNIV BALTIMORE MD CENTER FOR LANGUAGE AND SPEECH PROCESSING (CLSP), 2004.
N. Kushman, Y. Artzi, L. Zettlemoyer, and R. Barzilay. Learning to automatically solve algebra word problems. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics ( (Volume I: Lonn Pupers), pages 271-281, 2014.
A. Lewkowyez, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, Ramasesh, A. Slone, C. Anil, Schlag, T. Gutman-Solo, Y. Wu, B. Neyshabur, G. Gur-Ari, and V. Misra. Solving quantitative reasoning problems with language models, 2022. URL 14858.
C. Li. D. Tarlow, A. L. Gaunt, M. Brockschmidt, and N. Kushman. Neural program lattices. 2016.
Y. Li, Z. Lin, S. Zhang, Q. Fu, B. Chen, J.-G. Lou, and W. Chen. On the advance of making language models better reasoners. arXiv preprint D.Xii:22066023344 2022.
W. Ling, D. Yogatama, C. Dyer, and Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv: arXSiv/705.0000 1705.04 1466, 2017.
11

---

Under review as a conference paper at ICLR 2023
1. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint 17110017 2017.
J. Menick, M. Trebacz, V. Mikulik, J. Aslanides, F. Song, M. Chadwick, M. Glaese, S. Young, L. Campbell-Gilliggaan G. Irving, et al. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203. 11147, 2022.
S.-Y. Miao, C.-C. Liang, and K.-Y. Su. A diverse corpus for evaluating and developing english math word problem solvers. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pages 975-984, 2020.
A. Mirhoseini, A. Goldie, M. Yazgan, J. W. Jiang, E. Songhori, S. Wang, Y.J. Lee, E. Johnson, O. Pathak, A. Nazi, et al. A graph placement methodology for fast chip design. Nature, 594(7862): 207-212, 2021.
R. Nakano. J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, Jain, V. Kosaraju, W. Saunders, et al. Webgpt: Browser-assisted guestion-answeree with human feedback. arXiv preprint arXiv:2112.09332, 2021.
M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, D. Dohan, A. Lewkowycz, M. Bosma, D. Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.011 2021.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, Schulman, J. Hilton, Kelton, L. Miller, M. Simens, A. Askell, Welinder, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback, 2022. URL https: / http//aavvvv...///e//eee/...a..a org/ abs 22003.20222 02155.
E. Perez, P. Lewis, W.-t. Yih, K. Cho, and D. Kiela. Unsupervised question decomposition for question answering. arXiv preprint arXiv:2002.09758, 2020.
E. Perez, D. Kiela, and K. Cho. True few-shot learning with language models. Advances in Neural Information Processing Systems, 34:1 10111105--1700 11070, 2021.
Polu and I. Sutskever. Generative language modeling for automated theorem proving. 2020. URL https: arxivv. org/ als///////////////// 200ds/200 2009. 03393.
M. Rauh, J. Mellor, J. Uesato, P.-S. Huang, J. Welbl, L. Weidinger, S. Dathathri, A. Glaese, G. Irving, I. Gabriel, et al. Characteristics of harmful text: Towards rigorous benchmarking of language models. arXiv preprint arXiv:2206.083225 2022.
Reed and N. De Freitas. Neural programmer-interpretnn arXiv preprint arXiv:1511.06279, 2015.
J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre,, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, Graepel, et Mastering atari, go, chess and shogi by planning with a learned model, Nature, (88(7839):004-699) 2020.
V. Shwartz, P. West, R. I.. Bras, C. Bhagavatula, and Y. Choi. Unsupervised commonsense question answering with self-talk. arXiv preprint wrXiv:2004.05488, 2020.
D. Silver, J. Schrittwieeer, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676); 354 359, 2017.
N. Stiennon, Ouyang. J. Wu, D. Ziegler, Lowe, C. Voss, A. Radford, Amodei, and Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
A. Stuhlmüller and J. Byun Supervise process, not outcomes. 2022. URL https://ought: org/updates 04 04-00-proc6see--
I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27, 2014.
12

---

Under review as a conference paper at ICLR 2023
O. Tafjord, B. D. Mishra, and P. Clark. Proofwriter: Generating implications, proofs, and abductive statements over natural language. arXiv preprint arXiv:2012.13048, 2020.
J. Uesato, R. Kumar, V. Krakovna, T. Everitt, R. Ngo, and S. Legg. Avoiding tampering incentives in deep RL via decoupled approval. arXiv preprint arXiv:201 2020.
X. Wang, Wei, D. Schuurmans, Q. Le, Chi, and D. Zhou. Self-consisteney improves chain of thought reasoning in language models. arXiv preprint UrXiv:2203.1117/ 2022.
J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting clicits reasoning in large language models, 2022.
J. Wu, L. Ouyang, D. M. Ziegler, N. Stiennon, R. Lowe, J. Leike, and P. Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109. 10862, 2021.
Z. Yang, P. Qi, S. Zhang, Bengio, W. Cohen, Salakhutdinov, and D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering, arXiv preprint (rXiv:1809.09600, 2018.
E. Zelikman, Y. Wu, and N. D. Goodman. Star: Bootstrapping reasoning with reasoning. arXiv preprint 22..::22003.44666. 2022.
D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, Christiano, and G. Irving. Fine-tuning language models from human preferences. arXiv preprint 199.00999, 2019.
13

---

Under review as a conference paper at ICLR 2023
Error rate
Model
Greedy
Majority
RM-weighted
Few-shot
54.3
41.4
27.8
Few-shot + Final-answer RL
36.4
23.8
16.6
Few-shot ORM-RL
31.5
18.8
13.8
SFT
41.1
22.3
14.1
SFT + Final-answer RL
35.0
20.2
14.2
SFT + ORM-RL
31.2
17.8
12.7
SFT PRM-RI
34.4
17.6
12.9
Table 3: Final-answer error with different decoding strategies. This repeats some results from Table but includes all decoding strategies. RL always provides significant boost in the few-shot setting (top), and with greedy sampling (first column), which is important for latency-sensitiy applications. ORM-RL and PRM-RL are notably better than Final-Answer RL all settings.
A ADDITIONAL FINDINGS
Table 3 shows the final-answer error rates for various models using different decoding strategies.
Finetuning improves performance relative to prompting alone. Our focus different ap- proaches supervising LMs assumes that, provided sufficient data, finetuning outperforms prompting alone. We first validate this assumption for GSM8K. We find that the 5-shot prompted policy alone (Few-shot) achieves a 41.5% final-answer eπrоr rate with K 96 majority voting (and 54.3% error when using single greedy sample). While this is impressive considering that the few-shot policy requires no additional finetuning data, it leaves significant performance on the table compared to both the www-sshotFFii---AAe
RL and SFT models, thus validating our initial assumption.
Reward model reranking significantly boosts both trace and final answer accuracy. We find that RM reranking significantly improves trace error, reducing it from 11.4% to below 5% for SFT. RM reranking also benefits the Few-shot+Final-Ane significantly higher than in the SFT case. As found by Cobbe et al. (2021), our results also show that RMs decrease final-answer error, from 22.3% below 15%. We also experimented with the "step-level voting verifier" from Li et al. (2022). This works similarly to the PRM approach, but computes labels using heuristic based on intermediate numeric results. This resulted in slightly worse 15.9% final-answer error.
RL model, though trace eπrоr remains
RL significantly improves few-shot performance, but provides more modest gains on top of SFT. We from Table that when starting from few-shot model, RL cuts the final answer error rate by half, regardless of the decoding method. In contrast, when starting with an SFT model, RL has very little effect top of using RM for decoding, though does provide significant improvement when using greedy decoding (41.1 % to 31.2%).
Optimizing against an RM outperforms optimizing final-answer correctness directly. both the shw-shot and SFT settings, ORM-RL and PRM-RL outperform Final-Answer RL across all three decoding strategies. On face, this may be surprising given that ORM-RI optimizes an approximation (the ORM full-solution scores) of final-answer correctness. However, earlier RM analysis (Fig. 4) suggests that the ORM approximates process-base feedback, and checks reasoning steps rather than the final answer directly. Thus, one potential explanation is that Final-Answer RL only checks that solutions reach the correct final answer, whereas PRM-RL and ORM-RL check for solutions which reach the right answer for the right reason.
B WHEN TO USE PROCESS- VS. OUTCOME-BASED FEEDBACK?
Process- and outcome- auteo feedback have different strengths, and the appropriate choice will often be context-dependent. As general rule, outcome-based feedback tends to be appropriate when
14

---

Under review as a conference paper at ICLR 2023
reliable and complete evaluation metric is available, while process-based feedback is most appropriate otherwise. Here we discuss the considerations that lead us to this view. We start by discussing relative strengths and weaknesses with respect to final-answer and trace error, before moving to motivations for process-based feedhack that are yet to be empirically validated.
B.1 FINAL-ANSWER VS. TRACE ERROR
Our results suggest that when low final-answer error is sufficient, outcome- approaches provide label-efficient method for obtaining this, whereas when low trace error is desired, it is helpful to use
feedback, or an approximation of it. The relative importance of these metrics is context-dependent In context where desired outcomes are easy to evaluate and the evaluation process is robust, final-answer error is appropriate. For example, if final answers can be quickly validated, either programmatically by quick user inspection, then low final-answer error a fairly complete performance measure. In contrast, Menick et al. (2022) provide question-answert example where low trace error necessary, since even if the final answer correct, is very difficult for users to rely on this answer without also seeing the sources which led to that answer. In other domains such as education, the reasoning steps themselves may also be of direct interest.
as an
B.2 PROCESS BASSED APPROACHES BOTH REQUIRE AND FACILITATE HUMAN UNDERS STANDNG
Relative outcome- approaches, prasess-based approaches tend require greater human understanding (Krakovna et al., 2020). For example, outcome-based feedback based on power- consumption, chip area, and other metrics can be used to optimize computer chip layouts (Mirhoseini et al., 2021), while a process-based approach would require detailed expert knowledge on designing chip layouts. In order to be competitive in general, process-based approaches require us to improve human understanding, either aided by ML systems such in Amplification Debate (Christiano et al., 2018; Irving et al., 2018) or through broader means, such as by working with experts (Rauh et al., 2022), training people (Stiennon et al., 2020), providing them assistive tools.
Second, process-basse basesd approaches may facilitate human understanding because they select for reasoning steps that humans understand. By contrast, to-understand strategies, and result in less understandable systems, if these strategies are the easiest way to achieve highly-rated outcomes. For example in GSM8K, when starting from SFT, adding Final-Answer RL decreases final-answer error, but increases (though not significantly) trace error.
optimization may find hard-
B.3 PROCESS-BASED APPROACHES AVOID TAMPERING INCENTIVES
common concern within the AI safety literature is from RL agents which tamper (Everitt et al., 2017), i.e., corrupt their feedback mechanisms in order to receive positive feedback. As hypothetical example, consider an assistive agent which repeatedly interacts with users. If optimized for total user satisfaction, such an agent may influence users towards preferences which are easier to satisfy (e.g. easier to predict, or generally more amenable to ML-generated proposals), in order to increase user- reported satisfaction (Kenton et al., 2021). A similarly long-term concern is with agents which gain power and take complete control of their feedback procedures in order to ensure positive feedback (Cotra, 2022). contrast, consider training from process- based feedback, using user evaluations of individual actions, rather than overall satisfaction ratings. While this does not directly prevent actions which influence future user preferences, these future changes would not affect rewards for the corresponding actions, and so would not be optimized for by process-based feedback. We refer to Kumar et al. (2020) and Uesato et al. (2020) for a formal presentation of this argument. Their decoupling algorithms present a particularly pure version of process- based feedback, which prevent the feedback from depending directly on outcomes.
basseds-based
An alternate approach to avoiding tampering is to continually improve the outcome- based metrics, by monitoring for sequences of actions leading to tampering, and penalizing these sequences. However, this approach only scales to tampering we can detect: in other cases, effects can be difficult to observe
Note that the outcome-basee vs. process-basse spectrum applies in general to supervising any sequence of actions based on the resulting outcomes. or based on each action individually. Because the only actions in GSM8K are reasoning steps, typically refer directly to reasoning steps in this paper, but the different approaches supervision apply more generally.
15

---

Under review as a conference paper at ICLR 2023
measure (e.g.. how would we determine if system influences preferences?) understand (e.g., ianocuous-lokiing decisions may still have large effects, particularly in aggregate). Broadly, risk with the incremental outcome- based approach is addressing the most casily-noticed problems, without addressing root causes or subtler cases.
C LIMITATIONS TO GENERALIZABILIT OF OUR RESULTS
We generally expect process-based and outcome-based feedback to align more closely for math compared to other domains. For math problems, incorrect traces typically lead to incorrect final answers. This matches our carlier finding that outcome-supervise RMs approximate process-based feedback. In contrast, in other domains, undesirable behaviors may be helpful for highly-rated outcomes, e.g., manipulation may increase reported user satisfaction. As a result, we believe optimizing for outcomes (final-answer correctness) for math problems has stronger effect on inducing a correct process than it would in other domains.
D CONCEPTS RELATED TO PROCESS- AND AUTCOME-BASER FEEDBACK
In this work, we focus on different forms supervision available for training LMs, and discuss these forms of supervision in terms of distinction between process- and outcome- based approaches. Here, we discuss similarities and differences to other related distinctions used throughout the literature.
Supervised learning vs. reinforcement learning Broadly, supervised approaches tend to be more process-based, and RL approaches tend to be more outcome-based Indeed, the most process-based approach we consider is purely supervised (SFT), while the most nntcome-basee approach pure RL (Few-shot++Finll Finoot--ii---nswerr RL). However, RL approuches can be more or less process- process-based, such as when comparing PRM-RL to ORM-RL. or Final-Answer RL. Conversely, supervised imitation of reasoning traces filtered by outcomes (e.g., traces which led to highly-rated user interactions) blurs together with RL approaches based on supervised learning high-return trajectories (Silver et al., 2017; Anthony et al., 2017; Abdolmaleki et al., 2018). The process- and outcome-based categorization thus acknowledges that the resulting model depends on the broader nata-providing procedure, which often will not be fully described by code.
Strong vs. weak supervision While the meanings of strong vs. weak supervision can vary de- pending context, approaches which supervise intermediate steps often referred to strongly supervised (Yang et al., 2018; Perez et al., 2020). Similar to the above, while process-based ap- proaches tend towards strong supervision, and nutcome-based approaches towards weak supervision, strongly supervised approaches can be either process- or outcome- For instance, a person evaluating intermediate reasoning steps could either be directly evaluating those reasoning steps, or deferring to evaluations of their resulting outcomes.
Verbalized reasoning Verbalized reasoning traces do not necessarily imply process-based ap- proaches. Indeed, whereas all approaches in this work use verbalized reasoning traces, they use both process- and outcome-brat necessarily represent the model's ''s internal reasoning process, with the bulk of the reasoning happening inside of the large neural network activations, except in strictly modular approaches (Creswell et al., 2022). This caution holds particularly for domains with the potential for deception (Kenton et al., 2021).
feedback. Additionally, the verbalized reasoning traces do not
Nonetheless, verbalized reasoning still helps enable process-based feedback. In contrast approaches based on iterative computations performed in activation space (Guez et al., 2019; Dehghani et al., 2018: Graves, 2016; Schrittwieser al., 2020), humans can directly supervise iterative reasoning steps in natural language.
E DATA ANNOTATION DETAILS
Participants and pay The full details our study design, including compensation rates, were reviewed and approved by our independent ethical review committee. All participants provided
16

---

Under review as a conference paper at ICLR 2023
Few-shot
ORM labels
PRM labels
1.00
ORM
0.89
0.87
0.86
0.9
Fewi Few--hot Shot ORM
0.89
0,82
0.88
0.87
0.95
ORM labels
0.87
0.82
0.86
0.93
0.90
PRM
0.86
0.88
0.86
0.91
0.85
PRM labels
0.87
0.93
0.91
Figure 6: Agreement matrix between RMs and RM labels on the last step. Computed only on the last step rather than all steps. Results are similar, with slightly higher agreement everywhere.
informed consent prior to completing tasks and were reimbursed for their time. It is our policy that researchers must pay workers/participants least the living wage for their location.
Training dataset problems For constructing PRM training dataset, we used samples from SFT model. Due to limited annotation budget, only annotate problems where the SFT majority voting prediction is incorrect, since this focuses training on difficult problems, and still includes a mix of correct and incorrect samples due to annotating model solutions per problem.
Quality assurance Since evaluating the accuracy of solutions mathematical problems is special skill, we ran a preliminary qualification study before using annotations from participants for training evaluation. For the qualification annotation tasks, we selected model solutions where three authors unanimously agreed on the first major mistake, and required participants to annotate at least 3 out of 4 such solutions correctly. In total, 21 /91 candidates were included in our annotator pool.
Additionally, we used duplicate annotations for 20% of the training problems. We removed ratings from annotators who had an inter-annotator agreement rate below 75% on doubly-rated problems. On manual inspection, we found that these annotators had typically made errors in this disagreeing cases. This removed data from 21 annotators, amounting 21 % of our originally labelled training data, leaving us with 530 annotated problems (compared to 675 originally).
After quality assurance steps, using the same of duplicated problems, we measure inter-rater agreement rate of 92% and Cohen's 's x of .915 for the task of predicting the first incorrect step. Note that this estimate will slightly biased upwards as we filtered raters based on this same set, but as the inter-rater agreement rate is fairly bimodal across raters, this effect should be relatively small.
For the evaluation, we gather duplicate annotations for all ratings, and see an inter-rater agreement of 87%, with Cohen's of .34 the binary task of labelling the full trace correct. Inter-rater agreement is significantly lower than on the training set. We attribute this to the fact that for evaluation only annotate traces with the correct final answer, and observe that when the final answer is correct, mistakes in the reasoning can be quite subtle, while when the final answer is incorrect the reasoning mistakes often more clear cut.
17

---

Under review as a conference paper at ICLR 2023
John and Gary are playing a game. John spins a spinner numbered with integers from to 20. Gary then writes list of all of the positive factors of the number spun except for the number itself. Gary then creates a new spinner with all of the numbers on his list. John then spins this spinner, and the process continues. The game is over when the spinner has no numbers it. If John spins 20 on his first spin, what is the maximum number of total spins (including the one he already made) that John can make before the game is over?
Problem:
Solution If John spins a 20, then Gary's list contains the numbers 1, 4, 5, 10. Thus, these are the numbers on the second spinner. John spins a I, then Gary's list will empty because there are no positive factors of I besides itself. Thus, the game will be over. This yields a maximum of additional spin. If John spins a 2, then Gary's list will only contain the number 1. Then on John's next spin, we will have same scenario as above. This yields maximum of 2 additional spins. If John spins 4, then Gary's list will contain As we have already found above, spinning a yields more additional spins than I, so the maximum additional spins in this case is 3 spins. If John spins 5, then Gary's list will only contain the number 1, As above, this will yield a maximum of 2 additional spins. Finally, if John spins 10, then Gary's list will contain the numbers maximum number of additional spins, so this case has a maximum of 3 additional spins. Thus, of all of the possibilities, spinning 4 10 next could result in 3 additional spins, so the maximum total number of spins is 4 These would be achieved by spinning 20, 10, 2, 1 20, 10, 5, 1 20, 4, 2, 1.
the numbers
and 5. Of these numbers, 2 and have the highest
Table 4: An example question and answer from the Pre-algebra split of the MATH dataset.
Before Regex Transformation:
We are given that $$54+ (4))53eeo 17) 17)-200-3122ddiv 6)=200SS Now, let's 'ss remove the parentheses: 1554+98ddiv44+23kccoo 17-200-312ddiv 6.5 What does this expression equal?
After Regex Transformation:
We are given that 54+(98/1 (0000///22221122200022222------0
Now, let's 's remove the parentheses: 54499/11+23391-2200-332 What does this expression equal?
Table 5:
Example question from the MATH dataset before and after cleaning.
F ADDITIONAL RM ANALYSIS
Figure 6 shows the agreement matrix between RMs and RM labels on the last step. Similarly to Fig. 4 (which averages over all steps, rather than just the last step). we see that the ORM has higher agreement with the PRM labels, despite being trained to predict the ORM labels. Note that on the last step, the ORM and PRM labels will exactly coincide, except for the case where the final answer is correct, but the trace is incorrect (by construction, it is impossible for the trace to be correct with an incorrect final answer). Thus, this result indicates that the ORM tends more towards predicting whether the full trace correct, and not just whether the final answer is correct.
G OOD EVALUATION DETAILS
To evaluate ditt---distribution generalization, we evaluate our models zero-shot on the pre-algebra split of the MATH (Hendrycks et al., 2021) dataset. We limited to problems without diagrams, leaving 633 of the original 871 problems, and removed some LaTeX formatting as discussed below. Overall, we see noticeable OOD generalization, with final-answer error rate of 64.6% for our SFT+ORM-RL model with majority voting (74.2% error with filtering, assuming the model fails all questions with Asymptote diagrams). This is significantly worse than the 29% error on pre- pre-Alebrra questions from Lewkowyez et al. (2022), which uses much larger base LM and trains more math data, but significantly better than the previous best result of 92.3% error for GPT-3 from Hendrycks et al. (2021). An example question and answer from the dataset can be see in Table 4.
18

---

Under review as a conference paper at ICLR 2023
Regex Pattern for Latex Command
Replacement
Simple Replacements
(Mgeq/Nge)
(lgeqNlgee)
(Nleqlille)
(lneqNine)
Nimplies
(Nddoss\NccdorssNldotts)
Ntimes
Nrightarrow
->
Veedot
Iddiv
Api
pi None
Vquuad
Remove text formatting:
12 (2nd capture group)
r' 12",, inp) Neemp//lmmoookmmmammmttti/llnnnn (167MddP//M6HHaa=2A-2227/ r' 12', inp)
12 (2nd capture group)
Remove whitespace:
\s?(ss++\)ss?
(Ist capture group)
Wallowbreak NhspaceM/.* [
None None
Remove math mode:
M (Ist capture group)
(Ist capture group)
Convert fractions:
(Mfrrac\MfiracNNdfirraa)
12/3
12/3
Table 6: Latex Regex Transformations. This table shows all the regex transformations we performed on our data in the order they are applied.
Latex Transformations We used simple regular expression (regex) based transformation to convert Latex math expressions to plain texts, since the GSMSK data not in Latex format and were more interested in OOD generalization across problems rather than formats. See Table for an example before and after conversion. Our goal was not to cover all Latex commands and formatting edge cases in the MATH dataset, but instead wanted address the commonly used commands for which we could write regular expression transformations. See Table for the full of transformations we applied.
Extracting the final answer In the MATH dataset, the final answer is always in box, by formatting
the answer with the "Iboxed" command, e.g. 4.5 To obtain the final answer from the solution text, we find the boxed expression and extract its argument.
H NEGATIVE AND PRELIMINARY RESULTS
Here, we include various observations we made while running experiments. These are much less carefully checked than the results reported in the main paper, and some may be specific details
19

---

Under review as a conference paper at ICLR 2023
Model
Final-answer error
Rew-shott-Finll-nssee RI.
65.7
SFT
67.6
SFT, ORM reranking
65,4
SFT, PRM reranking
67.7
SFT+Final-Anse RL
63.3
SFT+ORM-RL
63.2
Table 7: Final-answer error MATH Pre-algebra. Unless RM reranking is specified, all numbers are reported with majority voting. Final-answer error fairly similar across all approaches.
or our training setup. However, we include them as we believe they may nonetheless be helpful for future researchers:
Training the ORM required roughly 20 times more steps than SFT training. We believe this is likely due to a combination of sparser token supervision (one token per step, rather than hundreds of tokens per solution), requiring dropout regularization, and being able to train multiple generated samples per problem.
We experiment with step-level reranking. by applying argmax over K options each step, rather than solution-level reranking. However, we found this increased final-answer error by about 1% with the PRM, and 3% with the ORM. From manual inspection of problems solved by solution-level reranking, but not step-level reranking, this seemed to largely be due to insufficient entropy in the policy. In these failure cases, there would often be step where valid next step existed, but none of the K continuations found it (and often, all K continuations would make the exact same mistake). In contrast, with solution-level reranking, slight variations in the earlier generated steps would yield more variation across the corresponding completions. This could potentially be addressed by, for example, conditioning the model steps rejected by the RM, but for simplicity we used solution-level reranking throughout this paper.
In preliminary experiments, retraining the ORM between expert iterations during SFT+ORM-RL training did not help. In contrast, for the few-shot-basee expert iteration implementations, we do retrain the ORM every iteration, in part avoid imitating trace errors in early epochs.
During training set collection, we initially asked annotators to provide a corrected version of the mistaken step. However, it was difficult to communicate this task in sufficient precision: for example, corrected steps would include calculation outputs before the calculations (which are difficult for an autoregressive LM handle), or would try to stay too close to the reference solution even when the earlier model steps were part of a different (but still valid) approach. Additionally, unlike for the labelling task, we couldn't 't easily compute inter-rates or researcher-rus agreement rates.
I EXAMPLE GSMSK PROBLEMS AND SOLUTIONS
We include several examples to provide qualitative sense of the task and learned model behavior. Table 8 contains 10 randomly sampled problems, and the output of the SFT+ORM-RI model with ORM reranking. Table 9 contains 5 trace errors, where the final answer is correct, but at least one of the steps was marked as incorrect by our raters.
20

---

Under review as a conference paper at ICLR 2023
Table
The glee club ordered 20 pizzas and 70% of them. The football team ordered twice many pizzas and ate 80% of them. How many pizzas are left?
Problem:
Model answer (final answer correct): The football team ordered 20+2=<20=2<40m:0 pizzas. So, the total number of pizzas ordered 40+20=<40+20=600-6 pizzas.
The glee club ate 20*0.7==20007=144 On 14 pizzas. The football team atc 80.0:/0000000....... Therefore, the total number pizzas eaten is 32+ 32-14=42+144444466 Hence, the number of pizzas left is 60- 46=w60-46=14w14, Final answer:
pizzas.
Problem:
Johnny took allowance $20 and added an extra $10 to it. then invested this sum of money, which tripled in year. How much money did he have after year? Model answer (final answer correct): Johnny had $20 + $10 $<20+10=30.00w// He then had $30 $<30=33-90.00-90.
Therefore, he had $90 after year. Final answer: 90
Problem: A jellyfish tank has numerous jellyfish in it. A fifth of the jellyfish large, and third of the large jellyfish change color from green to blue under UV light. The other jellytish are small and always stay blue. When a UV light turned on, jellyfish changed color. How many jellyfish are the tank? Model answer (final answer correct): If third of the large jellyfish change color, and there are fifth of all the jellyfish that are large, there are 1/3* 1/5= all the jellyfish change color. If jellyfish changed color under the light, and 1/15 of all the jellyfish change color, then there 15((1/15)=90 jellyfish the tank. Final answer: 90
Problem: Farmer Brown's farm is 200 acres, and Farmer Smith's 't farm is 100 more than twice that. How many acres do the two farms have, together?
Model answer (final answer correct): Twice the size of Farmer Brown's 200-acre farm is 2"200==2"200=0000-< acres.
Farmer Smith's farm is 100 more acres than twice the size Farmer Brown's farm, or acres.
100+400==<10+4400050000
Therefore, together, the two farms 500-2000=4000~22007700000000 acres.
Final answer: 700
Problem:
Castle bought 3 boxes of Coco Crunch and boxes of Fruit Loops this week. Last week she bought 4 boxes of cereal. How many more boxes cereal did she buy this week than last week?
Model answer (final answer correct): Last week, Castle bought 4 boxes.
This week, she bought 3 + 5 <345=8>8 boxes.
So, she bought 8 4 48-4=4w4 more boxes of cereal this week than last week. Final answer: 4
Problem: A basket of green food costs $25 and basket of red food costs $18. If you buy 3 baskets of each basket red green food and red food, how much will you have to pay in total if you get food?
Model answer (final answer correct): you buy baskets green food, you will pay $25 x 1+22533775000~5.000
If you buy 3 baskets of red food, the original cost $18 x
Since you get $2 off for each basket of red food, your total savings is $2 3 $=223=6.00~6000 The new price for the baskets of red food is 554 S6 144-6-48.000488..
So, you will pay $75 $48 65.75+48=1233 23.00 in total.
Final answer: 123
Continued on next page
21

---

Under review as a conference paper at ICLR 2023
Table 8 Continued from previous page
Problem: A bird decides to leave her summer home in the north and travel her winter home in the south. To make the journey, the bird leaves her home flying in southerly direction for 10 hours at speed 30 miles per hour. Then, the bird turns direction and flies towards the north for 2 hours per hour. Finally, the bird changes direction and flies toward the south for 5 hours at a speed of 22 miles per hour before landing her southern home. What is the distance, in miles, between the hird's northern and the southern homes?
speed of 18 miles
Model answer (final answer incorrect): The bird flies in south direction for 10:30==<100:3::300:::0
miles.
Then she turns and flies in northem direction for 2=18= miles.
Finally she turns again and flies in southerly direction for 5*22=x5*22=1 10>1 10 miles.
So, the bird travels a total 300+36++1 10=x300+33++ 10=446x446 miles to get from her northern her southern home.
Final answer: 446
Kim has started his own housekeeping business and is calculating how much profit he will make from his clients. He already has clients, but talking to another 5 potential clients and feels confident enough to include them his calculations. Each client's 's home will need bottles of bleach and a pack of cloths to clean. Bottles of bleach will cost $2 each and packs of cloths will cost $5 each. These his only expenses. Hc calculates that his total income each week will be $92. Profit is the difference between total income and total expenses, so how much profit, dollars, will Lucas make each week?
Problem:
Model answer (final answer correct): He will need total of 2 bottles of bleach per client 8 clients w2*8=16= bottles bleach each week.
This will cost 16 botdle x $2 per bottle S* S=16+2=32.00-33.
will also need a total
clients pack of cloths 1=88 packs of cloths each week.
This will cost 8 packs $5 per pack 508$5=40.000+0.
So his total expenses each week will be $32 + $40 $+32440=72..00.77
This means his profit each week will be $92 $72 $+92-72=200000 Final answer: 20
Problem: A hospital sees 500 people day. Each patient is seen for an average of 24 minutes. The doctors charge $150 an hour the hospital and the hospital charges the patients $200 hour. How much profit does the hospital make from these visits?
Model answer (final answer correct): Each day the doctors see 500:244=:500022441100000 12000 minutes worth of patients So they are paid 12000660=4120000-2200000 hours
So they are paid 1000110055522000555::00000000000
So the hospital makes 200:000-55.000-00000000000000000
So they make 4000-000055440000000000000000000 in profit
Final answer: 10000
22

---

Under review as a conference paper at ICLR 2023
Problem: Lorraine and Colleen trading stickers for buttons. Each large sticker is worth large button or three small buttons. A small sticker is worth one small button. A large button is worth three small stickers. Lorraine starts with 30 small stickers and 40 large stickers. She trades 90% of her small stickers for large buttons. She trades 50% of her large stickers for large buttons and trades the rest of them for small buttons. How many buttons does she have by the end?
Model answer (final answer incorrect): Lorraine has total of 30 40 = <30+40=70m stickers. She trades
small stickers <900 "30=27.0Mu 2.027 small stickers for large buttons.
So she has 30 27 <30-27=3s3 small stickers left.
She trades
large stickers <50 #40=20.0w20 large stickers for large buttons.
So she has 40 20 +40-200 2000200=2>200 large stickers left trade for small buttons.
That means she trades 20 3 small stickers =00600 small stickers worth large stickers for small buttons.
So she has a total of 3 20 60 <3+20+60=83w83 buttons by the end.
Final answer: 83
Table 8: Random problems and model solutions. We include 10 randomly sampled problems and model solutions from the SFT+ORM-RI. model with ORM reranking, which has 12.7% final-answer error overall.
Table
Julia's boat sprang leak while she was out on the lake. The was taking on two liters of water for every ten feet she rowed back towards shore. It took her sixteen seconds to row twenty feet. The shore was seconds away. How much water the boat taken on by the time Julia reached shore?
Problem:
Model answer: Julia rowed 20 feet in 16 seconds.
So she rows a rate 20 /16 <20 /16=111115550 1.2> 1.25 feet /second.
The shore is 64 seconds away.
So the distance to shore is 64*1.25 *64*1 1.....2. 1.25=80.0wa 0=80.0 fect.
The boat taking in 2 liters of water for every 10 feet.
-> So for every 10 feet, the boat was taking in 2 /10 = <2 /10=0.2x0.. liters water.
The boat took water a of 0.2 liters /sccond.
So for 80 feet, the boat would take in 80 = <80* 1.00*02==1... 02=11.>= 1.6 liters of water.
Julia's boat had taken in 1.6 liters of water by the time she reached shore.
Final answer: 16
Problem: Carl buys ten packs of cookies. Each pack of cookies has six cookies inside. Each cookie cost $0.10. How much change does Carl receive pay with $10 bill?
Model answer: Carl bought ten packs of cookies, which means he bought 60 cookies because 6 10=e6* 10=60+60
->
Carl spent 60x x $0.10==60*.10000 00000000000000000000000000000000
Carl paid S10 so the change was $10 - $6 = <10-6=4>4
Carl received $4 change.
Final answer: 4
Problem: Anakin and Locsin went to the beach today. Anakin caught 10 starfish, 6 sea horses, and clownfish. While Loesin caught fewer starfish than Anakin, fewer sea horses than Anakin, and more clownfish than Anakin. How many fish were they able catch?
Model answer: Anakin caught starfish Loesin caught 5 fewer, so Loesin caught starfish. Anakin caught 6 sea horses and Locsin caught 3 fewer, so Locsin caught 3 sea horses.
Anakin caught 3 clownfish Loesin caught 2 more, so Locsin caught 5 clownfish.
Thus, Anakin caught 10+6+3=el (0+6+3=199 19 fish and Locsin caught total of 19+13==19+13=32: fish.
fish for
Final answer: 32
Continued on next page
23

---

Under review as a conference paper at ICLR 2023
Table 9 Continued from previous page
Sam sells bread. He has target of selling 120 crates of bread in week. One week he was closed on Monday and Friday. Over the weekend he sold 20 crates. On Tuesday he sold crates, on Wednesday 12 crates, and Thursday 18 crates. By how many crates Sam off from his target for the week?
Problem:
Model answer: Monday he was elosed, so that's 120 crates- 20 crates 15 crates
crates 18 crates
-> 120 crates 55 crates a 20-55=665565 crates
He needed sell 120 crates but only sold 65 crates, he off by 120 65 a <1200-5555555 crates. Final answer: 55
Problem: If classroom has 3 times many girls they do boys, and /10 many nongendered children as they do boys, and the classroom has 30 boys. How many total children does have?
Model answer:
-> /10 the boys the same as saying that there is boy for every 10 boys so there is boy for every 10 boys
There are 30 boys so there are 30 /10 = <30 /10=3.0x boys
times many girls as boys there are 3 times many girls as boys
There are 3 boys so there are times as many girls as boys
There are 3 boys there are times many girls boys
There are 3 times as many girls as boys so there are 3 times as many girls boys
There are 3 boys so there are times as many girls as boys
There are 3 boys so there are times as many girls as boys There are boys so there are times as many girls as boys
There are 3 times as many girls as boys so there are 3 girls for every boy
There are 3 boys so there are girls every boy
There are 3 girls for every boy so there are 3 girls for every boy
There are 3 girls for every boy so there are #33==33*0=0000 girls
There are 90 girls and 30 boys and nongendered children there are 90+30+3 <90+30+3=1233122
children in the classroom Final answer: 123
Table 9: Incorrect Problems and Solutions. We include 5 randomly selected solutions where the New-shot+Final-Asswe RL model generates correct final answers but raters mark the traces as incorrect. The first step rated as incorrect is labeled with -. The authors would consider the third example correct, but was marked incorrect by an annotator due to the blank step before the final answer.
24


Reviews:
[{"review_id": "kpJLCuTtQZ", "reviewer": "Reviewer_ytw5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nMost of the recent work on language models for solving math word problems relies on prompting engineering. Instead, this paper studies how to finetune the language models for solving math word problems with outcome-based and process-based approaches. The paper develops two metrics: race error rate and final-answer error rate. The paper implements multiple outcome-based and process-based approaches and conducts experiments on a recently proposed dataset, GSM8K. The best model could improve the previous best results from 16.8% to 12.7% final-answer error and from 14.0% to 3.4% trace error.\n", "summary_of_the_review": "The paper is interesting to me, but there are two main concerns: the methods have limited modeling novelty, and all the experiments are conducted on only one dataset. It would be helpful to see revisions regarding these two concerns.\n", "strength_and_weaknesses": "**Strengths**\n\nThe paper comprehensively compares process- and outcome-based approaches on the math word problem dataset, GSM8K. It reports many impressive results of outcome-based and process-based approaches on final-answer error rates and trace errors. These results are meaningful for the follow-up work to develop models with both low error rates and trace errors. Figure 3 is very informative and shows that the trace errors are associated with final answer errors, especially when final answer errors are low.\n\n\n**Weaknesses/Feedback**\n\n**1. The methods have limited modeling novelty.** \n\nIt is good to see that the paper conducts the *first* comprehensive comparison between process- and outcome-based approaches trained on the math word problem-solving task. The paper conducts extensive experiments on the two types of approaches, and there are a lot of interesting findings. However, it seems that the paper does not propose a new method.\n\n**2. All the experiments are conducted on only one dataset.** \n\nI understand that there are some limitations for other datasets, and conducting experiments on new datasets leads to extra costs. However, whenever the annotation pipeline is set up, the additional costs of conducting experiments on new datasets are acceptable. It is worthwhile to conduct the experiments on smaller datasets. A recently proposed MWP dataset, TabMWP (https://arxiv.org/abs/2209.14610), might be a good benchmark for the experiments in the revised paper, which is annotated with multi-step reasoning steps.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "huJRuAzCjf", "reviewer": "Reviewer_pkRc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The papers study the finetuning of LLMs for solving maths problems. In particular, comparing training partial vs overall correctness. While the evaluation shows that both produce similar global accuracy, partial correctness requires feedback of that kind. The paper considers multiple existing alternatives for both kinds of signals. A group of them manage to get significant improvement, but there is not a clear combination that helps, perhaps because some of the methods are similar. ", "summary_of_the_review": "The paper considers the kind of supervision for MATH reasoning problems using LLM. The line of work is important, but I'm afraid it's hard to get concrete lessons from the paper. ", "strength_and_weaknesses": "Strengths\n- Scholarship on methods for training for correctness.\n- Some combinations of methods produced significant improvements.\n- Good attempt to synthesize the empirical observations\n\nWeaknesses\n- The organization of sections 2.3 to 2.5 is a bit confusing. Perhaps there are too many combinations in each block of Table 1. Even when I attempt to re-read, I remain confused about how the vertical caption \"RM rerank\" is related to the RM using majority, ORLM or PRM. \n\t- The legend of Fig 3 is also revealing. There are three axis combines in partial combinations, but only some combinations appear. \n\t- I'd consider working backwards from the findings, and reducing the number of experiments to show. That might imply moving some results to the appendix and keeping only short comments in the main body on how other methods give similar results.\n- I find it hard to know if the multiple decisions/hyper-parameters per method could be affecting the validity of the results.\n- The observation that the models get away with correctness while failing at intermediary steps might be explained by an easier chance to introduce artifacts/shortcuts that improve the accuracy.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2iIMAqg2kk", "reviewer": "Reviewer_GLbN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates different approaches to solve math word problems (MWPs), specifically on the GSM8K dataset. The authors compare training with 1) final answer correctness only (ORM), 2) each intermediate step's correctness (PRM), and 3) with different reward models for ORM and PRM. The authors find that 1) ORM and PRM achieve similar final answer correctness; 2) PRM and PRM fine-tuned models learn to emulate step-by-step generated solutions; 3) PRM (or similar approaches) is required to improve the correctness of each intermediate solution step; and 4) allowing the model to abstain significantly improves performance (on the not-abstained solutions).", "summary_of_the_review": "The depth (both technical and experimental aspects) of the paper feels more like a workshop paper (or can be written more succinctly to summarize the main findings). If the authors intend to present their work as a scientific investigation, I encourage the authors to more systematically consider the scientific questions and experimental evaluations. I look forward to deeper insights that the authors can bring on the different approaches that current/existing methodologies are tackling MWP solving and how these approaches compare to each other. ", "strength_and_weaknesses": "### strengths\nI believe the paper is on an important direction. A systematic evaluation of different approaches for MWP solving is very much appreciated and needed.\n\n\n### weaknesses\nI wish the results and analyses can go a bit deeper than what have been presented. The main 4 results appear in the paper multiple times and, along with the detailed experiment setup (which I appreciate), can be more succinct (or move some to appendix to highlight the major findings). The authors investigations seem to center around two approaches to fine-tune the model (e.g., output vs. process-based rewards). I think a few other settings can also be considered, such as whether there is a prompt for fine-tuning/inference (similar to few-shot) and the number of examples used in the prompt. Or compare different decoding schemes. \n\nAs a scientific investigation paper, I expect deeper analyses, in addition t o simply presenting the observations, to enable finding more insights (at present, most of the results in the paper are somewhat expected). For example (a few suggestions):\n- ORM and PRM give roughly the same final-answer correctness. Could the authors design an experiment to investigate why this is the case, e.g., empirical evidence to support the hypothesis it is \"simpler for ORM to recognize when steps are correct than it is to check the answer by internally computing the final answer itself\"?\n- The authors claim ORM seems to learn to emulate the PRM. How does it do that, even without explicitly being supervised on the correctness of each intermediate step? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject