Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision
Authors: Zhiqing Sun
Longhui Yu
Yikang Shen
Weiyang Liu
Yiming Yang
Sean Welleck
Chuang Gan
Venue: NeurIPS 2024 poster (2024)
Keywords: easy-to-hard generalization
scalable oversight
AI alignment
Primary area: natural_language_processing

Abstract: Current AI alignment methodologies rely on human-provided demonstrations or judgments, and the learned capabilities of AI systems would be upper-bounded by human capabilities as a result. This raises a challenging research question: How can we keep improving the systems when their capabilities have surpassed the levels of humans? This paper answers this question in the context of tackling hard reasoning tasks (e.g., level 4-5 MATH problems) via learning from human annotations on easier tasks (e.g., level 1-3 MATH problems), which we term as easy-to-hard generalization. Our key insight is that an evaluator (reward model) trained on supervisions for easier tasks can be effectively used for scoring candidate solutions of harder tasks and hence facilitating easy-to-hard generalization over different levels of tasks. Based on this insight, we propose a novel approach to scalable alignment, which firstly trains the (process-supervised) reward models on easy problems (e.g., level 1-3), and then uses them to evaluate the performance of policy models on hard problems. We show that such easy-to-hard generalization from evaluators can enable easy-to-hard generalizations in generators either through re-ranking or reinforcement learning (RL). Notably, our process-supervised 7b RL model and 34b model (reranking@1024) achieves an accuracy of 34.0% and 52.5% on MATH500, respectively, despite only using human supervision on easy problems. Our approach suggests a promising path toward AI systems that advance beyond the frontier of human supervision.

Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision
Zhiqing Sun'r Longhui Yu²; Yikang Shen", Weiyang Liu
Yiming Yang Sean Welleck Chuang Gan 3.6f
Carnegie Mellon University, PPkingg University, MMIT-IBM Watson Al Lab
Universitty of Cambridge, Maa Planck Institute for Intelligent Systems, "UMass Amherst
Code: Edward-Sun Sunneaaunneeeeeaeeeeeeeee easy-to to-tttooarrrr
Abstract
Current alignment methodologies rely on human-provice demonstrations or judgments, and the learned capabilities of AI systems would be upper- spper-bounded by human capabilities as result. This raises a challenging research question: How can we keep improving the systems when their capabilities have surpassed the levels of humans? This paper answers this question in the context of tackling hard reasoning tasks (e.g., level 4-5 MATH problems) via learning from human annotations easier tasks (e.g., level 1-3 MATH problems), which we term as easy-to-hard generalization. Our key insight that an evaluator (reward model) trained on supervisions for easier tasks can be effectively used for scoring candidate solutions of harder tasks and hence facilitating easy-to-hard generalization over different levels of tasks. Based on this insight, we propose a novel approach to scalable alignment, which firstly trains the pprocess-supervisee problems (e.g., level 1-3), and then uses them to evaluate the performance of policy models on hard problems. We show that such easy-to-hard generalization from evaluators can enable easy-to-hard generalizations in generators either through re-ranking or reinforcement learning (RL). Notably, our process-supervisee 7b RL model and 34b model (reranking@ @ 1024) achieves an accuracy of 34.0% and 52.5% on MATH500, respectively, despite only using human supervision on easy problems. Our approach suggests promising path toward AI systems that advance beyond the frontier of human supervision.
reward models on easy
Introduction
Rapid advancements in large language models (LLMs) indicate that in the near future, highly sophisticated AI systems could surpass human capabilities in certain areas, significantly enhancing our capabilities in solving harder problems beyond the levels we can currently solve [47, 49]. Since the current AI alignment methods mostly rely on either supervised fine-tuning (SFT) with human- provided demonstrations [59, 78, 14] or reinforcement learning from human feedback (RLHF) (97, 68, 50], their capabilities would inherently limited as humans cannot always provide helpful demonstrations or supervision on the hard tasks beyond their expertise [64].
In order build future AI systems for tackling complex challenges, such as advancing scientific knowledge, it is crucial to develop new approaches for scalable oversight challenge, i.c., to supervise the AI systems that can potentially outperform humans in most skills [9]. The key question is:
Can we limit human supervision to easier tasks, yet enable the model to excel in harder tasks?
'Equal contributions as leading authors.
'Equal contributions as senior authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Scalable Alignment (Superalignment)
Traditional Alignment
5^5=?
3^3=?
2x2=4
3x3=?
humans cannot reliably supervise superhuman models on the hardest tasks
humans supervise strong models hard tasks
Burns' Analogy on Weak-to-Strong Generalization
Our Analogy on Easy-to-Hard Generalization
2x2=3?
3x3=?
1+1=2
3x3=?
weak models unreliably supervise strong models on hard tasks that humans can evaluate
humans reliably supervise strong models on easy tasks and evaluate them on hard tasks
Figure 1: Illustration of different alignment scenarios: traditional alignment relies on human demon- strations or judgements [50]: scalable alignment [9] assumes that humans cannot reliably supervise amarter-than-humarl models; weak-to-strong generalization [11] focuses using weak models with unreliable labels supervise strong models; Our proposed easier-to-general generalization focuses the transfer of rewarding policies from weak models to harder tasks.
We refer to this scenario Easy-to- -ard Generalization [63, 95, 11, 29]. This setting requires human supervision on the harder tasks, which differs from existing work that either enhances humans' ability to verify the outputs of Al systems [81, 60, 57] enables weak-to-strong generalization via a teacher that only offers unreliable or noisy supervision [11].
The most basic form of easy-to-hard generalization can be achieved by training the policy models (i.e., generator) using supervised fine-tuning (SFT) or in-context learning (ICL) on easy tasks [55, 10], and expect this will unlock the ability to perform well hard tasks. However, it has been observed that SFT or ICL training of generators on easy tasks often fails to generalize to hard tasks [71, 24, 95]. We hypothesize and show that methods beyond these enable stronger degrees of casy-to-hard gencralization. Our intuition guided by the observation that evaluation easier than generation [34, 46], evaluator may offer degree of easy-to-hard generalization that is useful for improving generator. If that true, we can first train a verifier on easy tasks, then make of its generalization ability to supervise the generator hard tasks.
Complex tasks can often be broken down into smaller steps [95] and verified by validating the individual steps strategy that is commonly employed in solving mathematical problems (74,40, 73]. Inspired by this, we train outcome-supervised and process-superviise reward models [74, 85, 75, 40] our sasy-to-hare evaluators. The training dataset is often comprised of set of labeled easy tasks, each with a question and high-quality solution', paired with a set of unlabeled hard tasks that represented only by their questions. This simulates the practical setting of having numerous problems with known solutions, as well as significant unresolved challenges, such as the Millennium Prize Problems [12], which present challenging open problems. The pivotal aspect of easy-to-hard generalization thus lies how we effectively leverage the capabilities of easier-level models in solving harder problems.
Our investigation includes to training policy and reward models on the easy (i.e., level 1-3) portion of the PRM800K [40] dataset, and comparing the performance of majority voting with the policy model
We assume that human supervision is of high quality the easy tasks in general.

---

Easy (train): Domain of Human Supervision Hard (test): Domain Beyond Human Supervision
Failed Generalization in Easy-to-Hard Generation
Successful Generalization in Easy-to-Hard Evaluation
1+1
Sampling Solutions
x
Verifying Solutions
Easy-to-Hard Generators
Easy-to-Hard Evaluators
optimized against
trained on process supervision
easy-to-hard evaluators
We first train the evaluator with process supervision or outcome supervision (which simulates the process supervision) to enable easy-to-hard evaluation, and then use facilitate easy-to-hard generation via re-ranking or RI.
Figure 2:
only and weighted majority voting with the policy model and PRMs (Process-superviiee Reward Models). We also introduce the Outcome & Process Reward Model (OPRM), which harnesses the complementary strengths of outcome reward models (ORMs) and process reward models (PRMs): judging if each step in reasoning is correct (like PRMs do) and deciding if the final answer is right (like ORMs do). Our findings reveal marked performance improvement with the inclusion of reward models, especially on the hard (i.e., level 4-5) portion of the MATH500 test set. This improvement indicates that easier-level evaluators can maintain their effectiveness on harder tasks. We have similar observations in our experiments on the MetaMath dataset [86] and the Math- Math-Shepherd dataset [75].
We further investigate the use of the easy-to-hard evaluator reward model in reinforcement learning, where the evaluator provides targeted, step-by-step guidance in solving hard problems. We have an intriguing finding that training with human supervision only on the easy tasks (i.e., training with Level 1-3 problems and answers) can outperform both SFT and Final-Answer RL training on the filll dataset (Level 1-5). This finding underscores the potential of using easy-to-hard evaluation improve easy-to-hard generators, particularly when dealing with varied levels of task complexity.
2
Related Work
2.1 Scalable Oversight
While present-day models operate within the scope of human assessment, future, more advanced models may engage in tasks that are beyond human evaluation capabilities. This raises concern that such models might prioritize objectives other than maintaining accuracy (Andreas 3, Perez et al. 53, Sharma al. 64, Wei et al. 80). To address this, branch of research develops techniques to enhance the human capacity to supervise such models, such as via using AI to evaluate the work of other AIs [1, 38, 60, 9]. Our setting differs from enhancing human oversight; instead, we focus on enabling models to excel in hard tasks where human supervision may not be available. This also differs from weak-to-strong generalization [11], where human supervision may be available, reliable, on hard tasks. However, our framework aligns with the "sandwiching" concept proposed for measuring progress in scalable oversight, which involves domain experts evaluating the outputs of Al-assisted non-experts [18, 9, 57].
not
3

---

MATH-farr (Level 4-5) and Final Answers"
MATH-Fayy (Lavel 1-3) Problems Final Answers
MATI-Easy (Level 1-3) Solutions
Easy-to-llrd Model
Solution
Majority Vatiigg
Base Language Model
In-Context Learning (ICL) Model
Weighted Voting. Best-of-N
MATI--Cayy (Level 1-3) Solution Samples
Reinforcement Learning: ReST-EM, DPO,
Eaily Mony-to-HHrd Hard Reward Model
Figure 3:
The overview diagram of our methods: the different components of modeling and training and how they interconnected.
2.2 Compositional Generalization
Compositional generalization is fundamental aspect of how language works [13]. It refers to the ability to understand and utilize novel combinations based on the understanding of basic concepts and limited number of their combinations [23]. Recently, least-to-most prompting [95, 20] teaches language models how to solve a complex problem by reducing it to series of easier sub-problems, achieving casy-to-hard generalization perfect generalization accuracy. In addition, least-to-most prompting has also been successful in mathematical reasoning tasks, specifically in datasets like GSM8K [16] and DROP [21], by teaching language models to solve problems more difficult than those seen in the prompts. This success not only underscores the capacity of language models effectively break down complex tasks into simpler sub-tasks Perez et al. [51], but also demonstrates their generalization capability in solving these sub-problems.
semantic parsing tasks like SCAN [37] and CFQ [35] with
2.3 Easy-to-Hard Hard Generalization
Past work has evaluated easy-to-hard generalization by training easy-to-hard generators easy tasks using supervised finetune-tuning (SFT) or in-context learning (ICL. [55, 10]. Nevertheless, Swayamdipta et al. [71] showed that the BERT model performs poorly common-sense reasoning when only trained on easy data. Fu et al. [24] showed similar results for ICL on reasoning tasks like GSM8K [17]. In concurrent work, Hase al. [29] evaluate the performance of easy-to-hard generators on more datasets and models, and find that ICL or SFT on easy tasks is strong baseline multiple-choice tasks like ARC [15] and MMLU [30]. contrast, we evaluate the easy-to-hn tt-ddard generation performance on the more challenging MATH dataset [32], and show that easy-to-hard evaluation can improve generator's easy-to-hard generalization beyond ICL and SFT. Iterative machine teaching [43] gives theoretical justification to show that training classifiers from easy to hard
examples yield better generalization.
3 Methodology
We study the casy-to-hard generalization problem: how can we enable capabilities beyond human su- pervision? Specifically, we explore the efficacy and scalability of various easy-to-hard methodologies on competition-leve mathematical
sobblem-solving problems (MATH; Hendrycks et al. 32). This dataset suitable for our study since explicitly categorizes problems across five difficulty levels. We consider levels 1-3 as "easy" tasks, encompassing both the problems and their respective solution demonstrations, along with the correct answers. Conversely, levels 4-5, characterized by their more complex nature, treated as "hard" tasks and are represented solely by their questions. The MATH

---

dataset's ' difficulty distribution roughly follows a 1: 2: 2 3 ratio across levels 1 to 5. So our division maintains balanced number of easy and hard tasks.
The remainder of the paper aims to answer following research questions:
RQ1: How do generators generalize from easy to hard?
RQ2: How do evaluators generalize from easy hard?
RQ3: evaluators generalize better than generators, how can we take advantage of this to enable stronger easy-to- generalization in generators?
3.1 Setup
Dataset MATH [32] is dataset of 12,500 challenging competition mathematics problems, where 7,500 them are training problems and 5,000 are originally used for testing. Following Lightman et al. [40], Wang al. [75], we use the identical subset of 500 representative problems (i.e.. MATH500) as our test set, uniformly sample another 500 problems for validation, across all five difficulty levels, and leave the rest 4,000 MATH test split problems combined with the original 7,500 MATH training split problems as our training set.
Simulated Human Demonstrations While the original MATH dataset provides full step-by-step solutions, these solutions typically skip many thain-of-thoughe steps [79], which can be hard for language models to directly imitate?. Instead, [86] as our SFT training data: the former is generated by a Minerva-style base GPT-4 model using few-shot prompting after filtering the correct answers [39, 48), while the latter is generated by ChatGPT [47]. We keep all the GSM8K data in the MetaMATH dataset since they are typically easier than the problems in MATH. PRMS00K comes with human annotated process labels, while for Meta.aath we use Math-Shepherd as the corresponding process labels [75].
consider filtered PRM800K [40] and MetaMATH
3.2 Generators
For given dataset (e.g., variant of MATH), we consider the following generator models:
Full & Hard ICL Full in-context learning (ICL) is base model prompted with exemplars sampled from all difficulty levels, or only from the level [24].
Hard ICL This model is prompted with exemplars from easy problems. This baseline evaluates the degree to which a model can solve problems more difficult than those seen in the prompts [95].
Full SFT As prior work suggests that finetuning should outperform prompting alone [68, 52, 50], the full supervised fine-tuning (SFT) model is typically considered as a ceiling that a model can achieve on a type of task [11, 29].
Easy-to-Hard SFT This generator model is trained only on the easy tasks. Prior work suggests that it can generalize hard tasks but with some degeneration in performance [71].
The generator models are evaluated greedy decoding and self-consistency (also known as majority
voting) settings [76].
3.3 Evaluators
Similarly, consider the following evaluator models that can be trained either the easy tasks only, or on the full dataset. Notably, unlike final-answer rewards, reward models trained on easy tasks can be applied to evaluate solutions to hard problems.
Final-A Answer Reward symbolic reward that provides a binary reward based on the accuracy of the model's final answer. The matching is performed after normalization".
Hendrycks [32] found that having models generate MATH-style step-by-step solutions before producing an answer actually decreased accuracy. htttps:// /tte::///ittt tiub com/ http://////mooo/ooo////////////a
5

---

Table I: Easy-to-hard generalization of generators. We compare generator performance under various decoding settings. PRMS00K and METAMATH indicate the SFT training data and ICL exemplars. Evaluations performed on the same MATH500 test set.
PRM800K MAI 16
METAMATH MAJ@ 16
GREEDY
MAJ@256
GREEDY
MAJ@
FULL ICL EASY-TO-HARD ICL
12.8
15.6
20.8
16.4
18.4
25.6
12.6
18.0
27.0
16.6
19.0
27.0
LLEMMA-7B
14.0
17.6
24.4
14.2
17.4
26.8
FULL SFT EASY-TO-HARO SFT
20.6
32.0
36.2
31.4
40.2
41.6
19.8
31.6
36.0
30.0
38.6
42.4
FULL ICT.
18.6
23.6
36.0
20.6
28.8
39.2
HARD ICL
15.8
21.4
34.2
21.8
26.4
38.6
LLEMMA-34B
EASY-TO-HARD ICL
18.2
25.2
36.8
19.8
26.8
37.2
FULL SFT
25.6
41.8
46.4
35.4
44.2
45.6
EASY-TO-HARD
24.8
40.8
32.2
42.6
Outcome Reward Model (ORM) trained on the Final-Answer rewards. Following Cobbe et al. [16], Uesato et al. [74]. Lightman al. [40], train the reward head to predict every token whether the solution is correct, in a similar sense to a value model [85]. At inference time, we use the ORM's prediction at the final token as the reward of the solution.
Process Reward Model (PRM) is trained to predict whether each step (delimited by newlines) in the
reasoning path is correct. The labels are usually labeled by humans [74, 40] or estimated with rollouts [65, 75].
Outcome & Process Reward Model (OPRM) Building on the distinct advantages of ORMs and PRMs, we introduce the Outcome & Process Reward Model (OPRM), which harnesses the comple- mentary strengths of both. OPRM is trained on the mixed data of ORMs and PRMs. Specifically, it evaluates the correctness of each intermediate reasoning step, akin PRMs, while also assesses the overall solution's 's accuracy at the final answer stage, mirroring the functionality of ORMs.
3.4 Optimizing Generators Against Evaluators
Finally, given a generator model (i.e., policy model) and evaluator model (i.e., reward model; R.M,, optimize the generator against the evaluator using either re-ranking reinforcement learning.
Best-of-n (BoN), also known rejection sampling, is reranking approach that sample multiple solutions from the generator and selects one with the highest RM score.
Weighted Voting is similar majority voting self-consistency [76], but weights each solution according to its RM score [74].
Reinforcement Learning (RL) We consider three online/offtine RL variants, Reinforced Self- Training (ReST) [28, 67], Direct Policy Optimization (DPO) [56], and Proximal Policy Optimization (PPO) [62]. Due to the space limit, please find their detailed description in Appendix B.
3.5 Evaluation Metrics
In this study, we have chosen not to establish terms analogous to the weak-to seck-to--ttong performance gap recovery (PGR) discussed in Burns al. [1]] or the easy-to-hard supervision gap recovery (SGR) highlighted by Hase et al. [29]. This decision is based on our observations that sometimes, models trained exclusively
simpler pasks-particularlyy when employing RL training- outperform those trained across the entire spectrum of problem difficulties. Therefore, we mainly focus on the absolute and relative performance of generators (optionally optimized by the evaluator) on the MATH500 test set [40].
3.6 Implementation Details
Base Language Model Llemma is large language model mathematics [6], which continue pre- trained from Code Llama [58] LlaMA-2 [72]. We use both 7b and 34b variants in experiments.

---

5F1-7b + GPRM-70 ian PRMBOoK]
SFT-7b + OPPM-7 (on PRMBOOK]
SFT-7b OPRM-70 for FRMBOOK]
Accuracy on Easy (Leval 1-31 Problems
Accuracy on Hard (Leval 4-5) Problems
Accuracy on All (Level 1-5] Problems
aogless Solves
60
22.5
1/ Problems Solved
40
33-
20.0
35
50-
17.5
45
15.0
30
Majerty Veting
&
Majority Viriing
25
Majoeity Vating
15
Weigited Voding RM
10.0
Woighted Votting wi
Weighted Vediing wi RM Dest-af-ly we HMM
Sest-ffh oy 1M1
7.5
Best-dfit in IM
25
32 b4 128 z5e sizing m number of solutions per problem
5.0
be 125256
15
8 1b 32 126 zos sizznze N number of so solutions per problem
N number of solutions per problem
SFT-34b + GPRM.34b lon PRMBпсK) Accuracy on Easy (Level 1-31 Problems
SFT-34b + 3PRM (on PRM DOK)
SFT-34b OPRM lon PRMBOCK) Accuracy on All (Level 1-5) Problems
Accuracy on Hard (Level 4-5) Problems
75
70-
Pret mnd Solved
30
56 Probl errra Salved
65
45.
40
55-
20
50
35
45
Majority Veting
Weigited Vosing RM
%%
16
Weigited wiingg wi RM Best of w RM
Majorit Vinting
30
Majoriy Vettig
Weighted Velling wi RM
Bost ot RM
Best of we RM
35
18 17 M 126 5121074 - number of solutions per problem
5
M 128 258 5121074 N number solutions problem
20
16
12n 358 5171674
N number solutions problem
Figure 4: Easy-to-hard generalization of 7b (upper) and 34b (lower) evaluators. Both SFTs and RMs are trained on the easy data. We found that PRMs trained on easy tasks can significantly improve the re-ranking (i.e., weighted voting) performance on hard tasks. The shaded margin of the curve plot in this paper represents the performance variance.
SFT / RL/ Reward Model We fine-tune all models in full fine-tuning with frozen input-output embedding layers and normalization layers. RMs initialized from the base model, and have an added scalar head to output the reward. In PPO training, we initialize the value model from the reward model.
Hyper-parameters Due the space limit, our training hyper- hyper-parameters can found in Ap- pendix. C.
4 Main Results
4.1 Easy-to-Hard Generalization of Generators
In Table 1, we compare the easy-to-hard generalization performance of the generators under various decoding settings:
Supervised Fine-Tuning (SFT) outperforms In-Context Learning (ICL): This consistent with prior work [68, 50, 74]. We also find that the performance of ICL has larger variance than SFT with respect data ordering (or random seeds) [19, 93].
SFT data quality impacts easy-to-hard generalization: PRM800K data is generated by a base (unaligned) GPT-4 model through few-shot prompting and is thus of lower quality than well-aligned ChatGPT-gena -e-eerttee MetaMATH data. We find that only MetaMath MetaMath-trained models have certain easy-to-hard gaps (e.g., 16.6 v.s. 14.2 in MetaMath-7b-ICLL. while such gaps in PRM800K-traine models are very small (less than 1%), or even inverted in the ICL setting. We hypothesize that low-quality SFT data may only teach the model the format of the task [59, 78, 76), while high-quality (imitation) SFT data can teach the model the principles of solving the task [70, 27]. Nevertheless, the strongest performance achieved by full SFT on the high-quality MetaMath data (35.4), showing an unignorable difference, with gap of up to 3.2, compared to its easy-to-hard SFT counterpart (32.2).
4.2
Easy-to-Hard Generalization of Evaluators
The primary metric we use to assess the effectiveness our process reward model is not the average accuracy of verifying each step in a solution but rather the overall performance achieved through re-ranking methods (See discussion in Sec. 3.5). We first use re-ranking to evaluate the easy-to-hard generalization performance of evaluators.
7

---

SFT-7b + OPRM-34b (en PAMBOOK)
5FT-7b OPRM-34b (on PRMBOOK)
SFT-7b + OPRM-346 fon PRMBOOK)
Accuracy on Easy (Leval 1-31 Problems
Accuracy on Hard (Leval 4-5) Problems
Accuracy on All (Level 1-5) Problems
65
% Problems Solved
Problems Solved
t
6 55-
60
E50 55
20
a5
45
30
40
Majorty Veting
Majoett Veting
25
Majoetty Varing
35
Weighted Voding RM Sest-of-h my 191
16
Weighted withig wi RM Best-dihi my IM
Weighted Vetting wi RM Dest-af-ll my HMM
30
20
25
32 b4 128 25e sizing m number of solutions per problem
16 32 bi 128 25e sizzeze N m number of solutions per problem
2
15
4 # 1b 32 b4 126 zoe sizznze
N number of so solutions per problem
SFT:34b + OPRM-76 (on PRMBOOK) Accuracy on Easy (Level 1-31 Problems
SFT-34b GPRM-76 (on PRMBOOK) Accuracy on Hard (Level 4-5) Problems
SFT:34b OPRM-76 (on PRMBOCK]
Accuracy on All (Level 1-5) Problems
75
eme Salved
70-
5% Pret emm salved
30
9% Prorrleems errre Salved
65-
60-
40
55
20
50
35
45
Majority Vetinn
Weiglted Voding RM
16
Weightee wiingg wi RM Best of w/ RM
Majority Vitting
30
Majoniy Vettin
Weighted Velling wi RM
Bost of RM
Best of we RM
35
18 17 M 126 256 5121074 N number of solutions per prablem
M 128 258 5121074 N number solutions problem
20
16 12n 398 5171674
N number solutions problem
Figure 5: Easy-to-hard generalization of evaluators applied to generators of different sizes. We evaluated 7b generator + 34b evaluator (upper) and 34b generator 7b evaluator (lower). Both SFTs and RMs are trained on the easy data.
4.2.1 Re-ranking
We consider two re-ranking strategies: Best-of-n (or rejection sampling) and Weighted Voting. In our easy-to-hard generalization setting, both SFT models and Reward Models (RMs) are trained on easier tasks (levels 1-3), but evaluated on all difficulty levels (1-5). We compare the performance between majority voting (SFT only) and re-ranking (SFT OPRM) on the PRM800K dataset in Figure 4-5, and the performance of different reward models (PRMs, ORMs, & OPRMs) on the PRM800K dataset in Figure 8-9. Specifically, we min the reward aggregation function for best-of-n and prod for weighted voting'. The figures illustrate the performance of different decoding strategies or reward models under the same number of sampled solutions per problem. We have the following findings:
OPRMs outperforms ORMs and PRMs This confirms our hypothesis that Process Reward Models (PRMs) and Outcome Reward Models (ORMs) capture different aspects of task-solving processes. By integrating the strengths of both PRMs and ORMs, Outcome & Process Reward Models (OPRMs) demonstrate superior performance. However, follow-up experiments conducted on the MetaMaah//attt-Shpphee datasets do not demonstrate significant improvements from incorporating additional ORM training examples. This lack of enhancement may be attributed to the fact that Math-Shepherd is already generated from final-answer rewards. This suggests that there remains a substantial difference between process rewards labeled by humans (e.g., PRM800K) and those generated automatically (e.g., Math-Shepherd).
Weighted voting outshines Best-of-n This finding diverges from past research where minimal performance differences were observed between weighted voting and Best-of-n [40, 74]. Our hypothesis is that this discrepancy arises from specific experiment, which involves training less powerful base model (Llemma; Azerbayev et al. 6) on more difficult tasks (MATH; Hendrycks et al. 32). This setup might diminish the effectiveness of the reward model, potentially leading to an over-optimization of rewards [25]. Given these insights, weighted voting is preferred as the primary re-ranking method for further discussions. Nevertheless, Best-of-n still achieves competitive performance to majority voting when producing only one full solution. In Figure 5, we also find that the 34b evaluator can significantly improve the 7b generator, while the 7b evaluator can still improve the performance of the 34b generator.
Greater effectiveness of re-ranking on harder tasks: Weighted voting not only consistently surpasses majority voting but also shows more pronounced advantage harder tasks. This
'See more detailed analysis of reward aggregation functions in Appendix. I..
8

---

Table 2: Comparing reinforcement learning (RL) approaches for easy-to-hard generalization. All
methods are of 7b size and evaluated with greedy decoding.
REWARD
ACCURACY
RI. FINAL-ANSWER
PROCESS R.
EASY (LEVEL 1-3)
HARD (LEVEL 4-5)
ALL
(SFT / PRM trained on level 1-3 of PRMB0OK)
SFT
28.2
12.2
19.8
REST-EM
EASY
12.6
ITERATIVE DPO
EASY
EASY
42.0
12.2
26.4
PPO
EASY
RASY
42.0
14.1
27.4
PPO
ALL
EASY
45.4
14.9
29.4
(SFT WFT/PRO PRM trained on level 1-5 of MetaMath/ / Math Shepher))
LEEMA-BASEE SFT SoTA (OCRS)
51.7
13.7
31.4
PREVIOUS RL SoTA [75]
33.0
(SFT/ PRM trained on level 1-3 of MetaMath Math-Shepherd)
SFT
14.9
28.8
REST-EM
EASY
EASY
50.4
14.5
31.6
ITERATIVE DPO
EASY
EASY
53.8
16.0
34.0
ITERATIVE DPO
ALL
EASY
49.6
10.7
29.2
PPO
EASY
EASY
50.8
15.3
32.2
PPO
ALL
EASY
53.8
16.0
34.0
observation leads to the conclusion that evaluators demonstrate better easy-to-hard generalization capabilities in comparison to generalors. This motivates to explore RL approaches that optimize the generator against the evaluator to further improve the performance of easy-to-hard generation.
4.2.2 Reinforcement Learning (RL)
Given the conclusion above, an important question arises: how can evaluators once again assist generators in achieving enhanced easy-to-hard generalization capabilities? We further investigate the enhancement of policy models through RL, utilizing easy-to- ho-hard evaluators as reward models. Similar to re-ranking, SFT and PRM are only trained on easy data. For fair comparison between PRM800K and MetaMath, we only use vanilla PRMs in the RL training. All the RL methods use the validation accuracy for selecting the best checkpoint Our comparison spans offline (ReST & DPO) and online (PPO) RL algorithms under two training conditions:
Easy Questions & Easy Final Answers. The SFT model samples from easy questions and receives the corresponding Final-Answer and optional PRM rewards.
All Questions Easy Final Answers. This assumes access to range of easy and hard problems for RL training, with rewards for hard tasks solely provided by the easy-to-hard hard evaluator.
Based on the results reported in Table 2, we have the following findings:
DPO and PPO excel over REST. Among the RL algorithms trained on the PRM800K dataset, PPO emerges the most effective, significantly surpassing both ReST and DPO. On the MetaMATH dataset, PPO and DPO achieve top performance, while ReST shows only marginal improvements the SFT baseline. The comparative analysis between DPO and PPO across the PRM800K and MetaMATH datasets indicates that while DPO's efficacy on par with PPO given a high-quality SFT model initialization, PPO's effectiveness is less contingent the quality of the underlying SFT model [50, 56].
PRM rewards are more beneficial than Final-Answer rewards for hard tasks. Notably, models trained with PRM rewards with human supervision on the easy tasks (achieving top performance of 34.0) outperform the previous state-of-thh offte-of-theart art model trained across all task levels (33.0). This highlights the effectiveness of leveraging easy-to-han to-hard evaluations improve generator performance across varying task difficulties.
'This includes stopping iterations in ReST-EM and iterative DPO, and stopping online steps in PPO.

---

Table 3: Easy-to-hard generalization of evaluators coding problems (APPS). Both SFTs and RMs are trained on the easy (Introductory) data. We found that ORMs trained on easy tasks can improve the re-ranking (Best-of-N) performance hard (Interview & Competition) coding problems.
SFT / ORM
AVERAGE ACCURACY (%)
STRICT ACCURACY (%)
TRAIN DATA
INTRO.
INTER.
CоMP.
ALL
INTRO.
INTER.
CОMP.
ALL
31.4
15.5
17.0
5.2
EASY
GREEDY
26.8
14.1
9.5
15.7
11.0
3.0
0.0
4,0
CODE LLAMA
EASY
BEST-OP-1
25.4
12,0
0.1
13.5
16.0
2.7
4.8
EASY
BEST-OB-4
27.1
13.8
8.1
15.3
14.0
4.0
0.0
5.2
EASY
BEST-OF-16
29.7
16.3
11.3
18.0
19.0
5.0
7.4
37.6
21.7
22.0
EASY
GREEDY
33.9
19.4
8.5
20.1
21.0
6.0
1.0
8.0
CODE LLAMA 34B
EASY
BEST-OP-1
28.5
14.5
15.3
21.0
3.3
6.2
EASY
BEST-OP-4
36.3
21.3
10.5
22.1
24.0
8.7
10.2
EASY
BEST OF 16
45.9
25.8
10.0
26.6
30.0
10.7
13.0
4.3 Easy-to-Hard Generalization on the Coding Domain
We conduct further experiments in the coding domain with the APPS dataset [31]. Similarly to Lightman et al. [40], we sub-sampled 500 questions from the original test set of APPS as our test set. Specifically, we sub-sampled 100 Introductory questions, 300 Interview questions, and 100 Competition questions, following the original distribution the test set.
In Table 3, we compare the performance of SFT-trained Code Llama [58] (7b & 34b) with greedy decoding and best-of-N approach. the latter, an Outcome Reward Model (ORM) of the same model size is trained to select the best coding one from N sampled solutions.
We found that while the reward model only trained the outcome supervision of easy (Introduc- tory) data, significantly improves the model performance on hard (Interview & Competition) data. These findings extend the premise of easy-to-hard generalization beyond the confines of mathematical reasoning, suggesting its applicability across diverse domains.
5
Conclusion
Our study advances the field of AI alignment by demonstrating the potential easy-to-hard gen- eralization, where models trained simpler tasks can be guided to solve more complex problems without direct human supervision on these harder tasks. Through the use of pprocess-supervisee reward models for evaluating and enhancing policy models, show that evaluators this form generalization, outperforming traditional training methods. Our findings highlight the effectiveness of re-ranking strategies and reinforcement learning (RI.) in leveraging evaluators for performance gains on difficult tasks. This approach presents a promising direction for developing AI systems capable of surpassing human problem-solving capabilities, suggesting scalable alignment method that could enable AI to independently advance knowledge in complex domains.
facilitate
While our study provides valuable insights into easy-to-hard generalization and the potential of sropess-uuperviee reward models, there are limitations to consider. These include the focus on specific model sizes and datasets, the domain specificity of reasoning tasks, and the need for further research on the long-term implications and robustness of the method.
6 Acknowledgement
This work is supported by OpenAI Superalignment Fast Grants and Microsoft Accelerate Foundation Models Research (AFMR) Initiative. Additionally, 7S thanks Google PhD Fellowship; SW thanks NSF SCALE (NSF DMS 2134012) and Convergent Research.
References
[1] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint 1rXiv11606.06555, 2016. 3
10

---

[2] Shengnan An, Zexiong Ma. Zeqi Lin, Nanning Zheng, Jian LaanGG Lou, and Weizhu Chen. Learning from mistakes makes Ilm better reasoner. arXiv preprint arXiv:2310.20689 2023. 18
[3] Jacob Andreas. Language models as agent models. Findings of the Association for Computa- lional Linguistics: EMNLP 2022, pages 5769-5779, 2022. 3
[4] Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017. 18
[5] Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm understand learning from human preferences. arXiv preprint arXiv:2310. 12036, 2023. 18
[6] Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. arXiv preprint arXiv:23 10. 10631, 2023. 6, 8, 18, 24
[7] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05860 2022. 18
[8] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer EI Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel Bowman, Zac Hatfield- Radfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback. 2022. 18
[9] Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamile Lukosuite, Amanda Askell, Andy Jones, Anna Chen, et al. Measuring progress on scalable oversight for large language models. arXiv preprint arXiv:2211.08 03540, 2022. 1, 3
[10] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
2, 4
[11] Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschen- brenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong gener- alization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.099290 2023. 2, 3,
[12] James A Carlson, Arthur Jaffe, and Andrew Wiles. The millennium prize problems. American Mathematical Soc., 2006.
[13] Noam Chomsky, On the representation of form and function. 1981,
[14] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra. Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling cnstuctiionfinetunee language models. arXiv preprint arXiv:2210.11416 2022.
[15] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv:1803.054555 2018. 4

---

[16] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jeny Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv: 10. 14168, 2021. 4, 6, 18
[17] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021. 4
[18] Ajeya Cotra. The case for aligning narrowly superhuman models. Al Alignment Forum, 2021.
3
[19] Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002..06355, 2020." 7
[20] Andrew Drozdov, Nathanael Schärli. Ekin Akyürek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, and Denny Zhou. Compositional semantic parsing with large language models. In The Eleventh International Conference on Learning Representations, 2022. 4
[21] Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume / (Long and Short Papers), pages 2368-233 2378, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 00.18653/v1/N19-11 1246. URL hatpp:/s///aaaaahoooooo/o 2tg/N19-1246 4
[22] Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305.14387 2023. 20
[23] Jerry A Fodor and Ernest Lepore. The compositionality papers. Oxford University Press, 2002. 4
[24] Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity- prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations, 2022. 2, 5, 18
[25] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835-10866. PMLR, 2023. 8
[26] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang. Minlie Huang, Nan Duan, Weizhu Chen, et al. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint arXiv:2309.17455 2023. 18
[27] Arna Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary 1lms. arXiv preprint 21X15 2023.
[28] Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahem, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint urXiv:2308 300.98998.8 2023. 6, 18
[29] Peter Hase, Mohit Bansal, Peter Clark, and Sarah Wiegreffe. The unreasonable effectiveness of easy training data for hard tasks. arXiv preprint 200702240,.0671, 2024. 2, 4, 5, 6
[30] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 20200 4
[31] Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.0900 09938, 2021. 10
12

---

[32] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprini arXiv:2 2021. 4, 5, 8, 18, 24
[33] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 2023. 18
[34] Richard M Karp. On the computational complexity of combinatorial problems. Networks, 5(1): 45-68, 1975. 2
[35] Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashu- bin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, et al. Measuring compositional generalization: A comprehensive method on realistic data. In International Conference on Learning Representations, 2019. 4
[36] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916 2022. 18
[37] Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of requence-to-sequeen recurrent networks. In International conference on machine learning, pages 2873-2882. PMLR, 2018. 4
[38] Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv: 1.10077,, 2018. 3
[39] Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206. 14858, 2022.
5, 18
[40] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step, arXiv preprint 20X5v:2305.20050, 2023. 2, 5, 6, 8, 10, 18, 19, 30
[41] Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation:
Learning to solve and explain algebraic word problems. In Proceedings of the 55th Anmual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers),
pages 158-167, 2017. 18
[42] Bingbin Liu, Sebastien Bubeck, Ronen Eldan, Janardhan Kulkarni, Yuanzhi Li, Anh Nguyen, Rachel Ward, and Yi Zhang. Tinygsm: achieving> 80% on gsmök with small language models. arXiv preprint arXiv:2312.09224,, 2023. 18
[43] Weiyang Liu, Bo Dai, Ahmad Humayun, Charlene Tay, Chen Yu, Linda B Smith, James M Rehg, and Le Song. Iterative machine teaching. In International Conference on Machine Learning, pages 2149-2150 PMLR, 2017.
[44] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical rea- soning for large language models via reinforced evol-instruct. arXiv preprint MXXv:2308.00588 2023. 18
[45] Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback. arXiv preprint arXiv:2312.00888 2023. 18
[46] Moni Naor. Evaluation may be easier than generation. In Proceedings of the twenty-eighth annual ACM symposium on Theory of computing, pages 74-83, 1996. 2
[47] OpenAl. OpenAl: Introducing ChatGPT, 2022.
com contbblo/ccchaggp........ chatgpt.
5.5
13

---

[48] OpenAl. Gpt-4 technical report, 2023, 5, 18
[49] OpenAl. OpenAl: GPT-4, 2023. URL httpt:://pppeaaaaee htt.://aa...coooeeeeeerr///////a 4.
[50] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155. 2022. 1, 2, 5, 9,
18, 20
[51] Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. Unsupervised question decomposition for question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8864-8880, 2020. 4
[52] Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. Advances in neural information processing systems, 34:1 1054-11070, 2021. 5
[53] Ethan Perez, Sam Ringer, Kamilé Lukošiūté, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:2212.09255, 2022. 3
[54] Stanislas Polu and lla Sutskever. Generative language modeling for automated theorem proving. arXiv preprint arXiv:2009.0339; 2020. 18
[55] Alec Radford, Karthik Narasimhan. Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. 2, 4
[56] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305. /8290, 2023. 6, 18, 20
[57] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proot q&a benchmark. arXiv preprint 2rXiv:2311.12000 2023. 2,
[58) Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. /2950, 2023. 6, 10
[59] Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, et al. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2021. 7,7
[60] William Saunders, Catherine Ych, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.00 05500, 2022. 3.
[61] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.024330 024388, 2015. 20
[62] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 107.0.1707.06347, 2017. 6, 18
[63] Avi Schwarzschild, Eitan Borgnia, Arjun Gupta, Furong Huang, Uzi Vishkin, Micah Goldblum, and Tom Goldstein. Can you learn an algorithm? generalizing from easy hard problems with recurrent networks. Advances in Neural Information Processing Systems, 34:6695-6706, 2021. 2
[64] Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Seafield-Dodds. Scott R Johnston, et al. Towards understanding sycophancy in language models. arXiv preprint 2xXiv:2310.33548 2023, 1,
14

---

[65] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre. George Van Den Driess- che, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mas- tering the game of Go with deep neural networks and tree search. Nature, 529(7587):484 2016.
[66] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, al. Mastering the game of go without human knowledge. nature, 350(7676):354
2017. 18
[67] Avi Singh. John Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil. Peter Liu, James Harrison, Jachoon Lee, Kelvin Xu, Aaron Parisi, et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.06:: 2023. 6, 18, 19, 20
[68] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances Neural Information Processing Systems, 33:3008-300 -020,, 2020. 1, 7, 18
[69] Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Salmon: Self-alignment with frinciple-followiig reward models. arXiv preprint arXiv:2310.05911 2023. 18
[70] Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. arXiv preprint arXiv:2305.030% 2023. 7
[71] Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9275-9293, 2020. 2, 4,5
[72] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.0922 2023. 6, 18
[73] Trieu H Trinh, Yuhuai Wu, Quoe V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476-482, 2024. 2
[74] Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback. arXiv preprint arXiv:2211.14277 2022. 2, 6, 7, 8, 18, 29
[75] Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui. Math shepherd:: Verify and reinforce ilms aeep-by-step without human annotations. CoRR, abs/2312.08935., 2023. 2, 3, 5, 6, 9, 19, 20, 26
[76] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instruc- tions. arXiv preprint 22t.:22222.11566 2022. 5, 6,
[77] Zihan Wang, Yunxuan Li, Yuexin Wu, Liangchen Luo, Le Hou, Hongkun Yu, and Jingbo Shang. Multi-step problem solving through a verifier: An empirical analysis on model-induced process supervision. arXiv preprint arXiv:2402.026658 2024. 30
[78] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2021. 1,7
[79] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS, 2022.
5, 18
15

---

[80] Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoe Le. Simple synthetic data reduces sycophancy large language models. arXiv preprint vrXiv:2308.03958. 2023. 3
[81] Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109. 10862, 2021. 2
[82] Jing Xu, Andrew Lec, Sainbayar Sukhbaatar, and Jason Weston. Some things more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312. 16682, 2023. 18
[83] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:22 10.00369,, 2022. 18
[84] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305. 2023. 18
[85] Fei Yu, Anningzhe Gao, and Benyou Wang. Outcome-supervised verifiers for planning in mathematical reasoning. arXiv preprint arXiv:2311.09724, 2023. 2,
[86] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 2023. 3,5, 18, 19, 20, 26
[87] Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. Scaling relationship learning mathematical reasoning with large language models. arXiv preprint 2331v:2308..0185, 2023. 18
[88] Xiang Yue, Xingwei Qu, Ge Zhang. Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05663 2023. 18
[89] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning, Advances in Neural Information Processing Systems, 35:15476-15488, 2022. 18, 19
[90] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprini arXiv:2210.03493, 2022. 18
[91] Yao Zhao, Mikhail Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Salch, and Peter J Liu. Calibrating sequence likelihood improves conditional language generation. In The Eleventh International Conference Learning Representations, 2022. 18
[92] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic- hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023. 18
[93] Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pages 12697-12706. PMLR, 2021. 7
[94] Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. Progressive- prompt- ing improves reasoning in large language models. arXiv preprint arXiv:2304.09797, 2023. 18
[95] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe V Le, et al. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations, 2022. 2, 4, 5, 18
[96] Yongchao Zhou, Andrei Toan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. arXiv preprint arXiv:221.001910 2022. 18
16

---

[97] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford. Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprinl arXiv: /90.08839, 2019. I
17

---

A Additional Related Work
A.1 Rationale-Augmented (Mathematical) Reasoning
Ling et al. [41] pioneer the work of solving math word problems by generating step-by-step solutions before the final answer. Cobbe et al. [16] extend this work by constructing much larger dataset to finetune a pre-trained large language model to solve math word problems, and a outcome-suppervsse verifier rank candidate solutions. Wei
[79] demonstrate that the reasoning ability of language model can be elicited through the use of prefixed rationales. Subsequent research [36, 83, 39, 96, 84] in tasks requiring human-level reasoning skills has also highlighted the efficacy of rationale augmentation.
Among all the reasoning tasks, select mathematical reasoning to evaluate easy-to-hard generaliza- tion ability, given that mathematical reasoning serves as valuable assessment for complex reasoning abilities and features clear delineation of difficulty levels. Recent research efforts focus on prompt design [79, 95, 24, 90, 94] to elicit the intrinsic reasoning capabilities of models, or data engineering for fine-tuning [44, 87, 88, 86, 26, 42, 2, 6]. which draws on experts provide high-quality training datasets. Our work is categorized as fine-tuning based work. However, unlike previous work, our focus lies in exploring how to generalize more challenging mathematical problems when only provided with easy mathematical data.
A.2 Outcome Reward Models & Process Reward Models
For some multi-step complex reasoning tasks, such as generating highly complex code, it may be challenging for humans to fully grasp the outputs produced by an advanced AI system. In such scenarios, process-supervisee reward models (PRMs) present a promising solution [74, 40]. These models operate by supervising each step in the reasoning generation process, rather than focusing solely on the end result. They are particularly effective in tasks where the reasoning process itself is important the final outcome [32, 33].
Uesato al. [74] find that process-supervivn than outcome-superrieed reward models (ORMs) when re-ranking sampled solutions from the policy model, but their performance is similar during reinforcement learning (RL) via expert iteration [66, 4, 54, 89, 28, 67]. Lightman et al. [40] compare ORMs and PRMs with more capable base model [48] and significantly more human-labeled process feedback on the more challenging MATH dataset, and also find that PRMs significantly outperform ORMs in the reranking setting. In contrast to these works, which only study the effectivenss PRM in an independent and identically distributed (IID) domain, study the utilization of PRMs in the casy-to-hard generalization scenario, and show that easy-to-hard evaluators instantiated by PRMs can enable easy-to-hard generation policy models.
reward models (PRMs) achieve better performance
B Reinforcement Learning Algorithms
Reinforeed Self-Training (ReST) is an offline RL algorithm, which alternates between generating samples from the policy, which are then used to improve the LLM policy with RM-weighted SFT [28, 67]. Its variants include expert iteration [4] and rejection sampling fine-tuning [72, 87].
Direct Policy Optimization (DPO) is a class of offline RL algorithms [56] that consider both positive and negative gradient updates. It fine-tune the policy model preference dataset consisting of paired positive and negative samples. The variants include NLHF [45], IPO [5], and SLiC [91, 92]. Recent work shows that iteratively applying DPO leads improved performance [82].
Proximal Policy Optimization (PPO) is an online RL algorithm which samples from the policy during fine-tuning [62]. It is widely used in RLHF [68, 7, 50] and RLAIF [8, 69].
18

---

C Hy Hyper-parameters
C.1 Supervised Fine-Tuning & Reward Modeling
For the PRM800K dataset [40], the SFT model is trained using steps that are labeled as correct. For the MetaMath dataset [86], given that the original dataset can contain upwards of ten solutions for the same question, potentially leading to over-fitting, we implement filtering process. This process ensures that, during any given epoch, more than three solutions per question retained, thereby mitigating the risk of over-fitting.
The PRMs trained on the corresponding released dataset [40, 75]. For generating solutions to train ORMs, we sample 32 solutions for each question from the language model using top-K sampling with K=20 and temperature of 0.7. We also ensure that the ratio between positive and negative samples for each question is between 1:3 3:1.
See Table 4 for list of training hyper-parameters used in the training jobs. We use full fine-tuning for SFT/RM training.
Table 4: Hyper-parameters in our SFT/RM training jobs
PRM800K
METAMATH
SFT
PRM
ORM
OPRM
SFT
PRM
LEARNING RATE
2B-5
2E-5
2E-5
8E-6
2E-5
EPOCHS
3
2
2
3
2
LLEMMA-7B
BATCH SIZE
128
128
128
128
128
128
MAX SEQ LEN
768
768
1024
1024
1024
768
DTYPE
BF16
BF16
BF16
BF16
FP32
BF16
LEARNING RATE
IE-5
1E-5
1E-5
1E-5
5E-6
EPOCHS
3
2
3
LLEMMA-344
BATCH SIZE
128
128
128
128
128
MAX SEQ LEN
768
768
1024
1024
768
DTYPE
BF16
BF16
BF16
BF16
FP32
C.2 Re-Ranking
For majority voting, weighted voting, and best-of-n, we sample from the language model using top- K sampling with K=20 and temperature of 0.7. At time, we use the ORM's prediction at the final token as the overall score for the solution, and use the PRM's prediction at each intermediate step (denoted by the new line symbol) and the final token as the process reward scores.
C.3 Reinforcement Learning
We use full fine-tuning during the RL stage.
ReST-EM Following Singh et [67], we sample solutions for each question from the language model using top-K sampling with K=40. We also used cut-off threshold of 10 for the maximum number of solutions per problem [89, 67]. We performed iterative ReST training for two epochs, and observed performance degeneration starting from the third epoch. For PRM800K, we used temperature 1.0, while for MetaMath, we used temperature of 1.2. The rest training hyper- parameters are the same in SFT training.
Iterative DPO We sample 8 solutions for each question from the language model using top-K sampling with K=20 and temperature of 1.0. We use the process reward model to assign score between 0 and to each solution, and use final-answer reward to assign an additional 0/1 score to each solution. A preference training pair is constructed only when the score difference between positive and negative solutions greater than 1.0. We used a cut-off threshold of for the maximum number of preference pairs per problem.
19

---

Table 5: Full results of comparing reinforcement learning (RL) approaches for easy-to-hard gener- alization. All methods are 7b size and evaluated with greedy decoding. i indicates the model is trained with additional final-answer labels on hard tasks (similar Singh al. [67]), which is not strictly a easy-to-hard generalization setup.
REWARD
ACCURACY HARD (LEVEL 4-5)
DATA
FINAL-ANSWER
PROCESS RM
EASY (LEVEL 1-3)
ALL
(SFT/ PRM trained on level 1-3 of PRM800K)
SFT
28.2
12.2
19.8
REST-EM
33.2
12.6
REST-EM
HARD
HARD
31.9
8.0
19.4
REST-EMi
ALL
ALL
35.7
8.8
21.6
ITERATIVE DPO
EASY
EASY
V
42.0
12.2
26.4
ITERATIVE DPOS
V
38.2
11.5
24.2
PPO
EASY
EASY
42.0
14.1
27.4
PPO
HARD
HARD
34.0
9.2
21.0
PPOT
ALL
ALL
42.0
10.7
25.6
45.4
14.9
PRM trained level of MetaMath )hephhr))
LLEMMA-BASED SFT SOTA (OURS)
51.7
13.7
31.4
PREVIOUS RL SOTA [75]
33.0
(SFT/ PRM trained level 1-3 of MetaMath Math- Shepher)
SFT
44.1
14.9
28.8
REST-EM
EASY
EASY
50.4
14.5
31.6
EASY
EASY
53.8
16.0
34.0
TEERTTIVE DPO
ALL
EASY
49.6
10.7
29.2
ITERATIVE DPO*
ALL
ALL
47.9
12.2
29.2
PPO
EASY
EASY
50.8
15.3
32.2
PPOt PPO
ALL
50.8
13.4
31.2
ALL
EASY
53.8
16.0
34.0
For all DPO training [56]. we used learning rate of 2 10-6, batch size of 64, and a DPO training epoch 1. We set 0.1 for all DPO experiments, and performed at most 5 DPO iterations (i.c., sampling new solutions and performing one DPO epoch).
PPO We follow Dubois al. [22] on the implementation of the PPO algorithm, which a variant of [50]s. Specifically, we normalize the advantage across the entire batch of rollouts obtained for each PPO step and initialize the value model from the reward model.
We clipped the gradient by its Euclidean norm at limit of 1. Our training spanned 500 PPO steps on the RL data (MATH questions except MATH500 and our 500 validation questions). For generalized advantage estimation (GAE; Schulman et al. [61]). both À and y were set at 1.
For PRM800K, we used batch size of 512 for each PPO step. This comprised 8 epochs of gradient steps, each having 64 rollouts. We applied a peak learning rate of 2 x 10-5 with cosine decay. We opted for constant KI regularizer coefficient of 0.01, and sampling temperature of 0.7.
For MetaMath/Mathh Shephhrd,, we used a batch size of 512 for each PPO step. This comprised epochs of gradient steps, each having 256 rollouts. We applied peak learning rate of 5 x 10 with cosine decay. We opted for a constant KL regularizer coefficient 0.002, and a sampling temperature of 1.2.
D Re-ranking Results on MetaMath
Similar to Sec. 4.2.1, we assess the effectiveness of process reward models on the MetaMath/Mathh Shepherd dataset [86, 75J. From Figure 6, we can see that PRMs are also more effective harder tasks when trained on MetaMath/Matt--Shppher
'nttps://github https://ggtaab..oooooaaa/ltatta wwwp//ooaa/ooononiiooee
prefeenceee
20

---

Table 6: Pass@N scores (upper bound of Best-of-N) coding problems (APPS).
SFT ORR
ACCURACY (%) INTER.
DECODING
TRAIN DATA
INTRO.
COMP.
ALL
EASY
PASS
11.0
CODE LLAMA 7B
EASY
PASS @ 4
20.0
4.7
2.0
7.2
EASY
PASS @ 16
39.0
9.3
6.0
14.6
EASY
PASS @ 1
19.0
5.3
2.0
7.3
CODE LLAMA 34B
EASY
PASS @ 4
32.0
10.7
3.0
13.4
EASY
PASS @ 16 44.0
18.3
8.0
21.4
MetaMath-7b Math-Shephard-72
MetaMath-76 76 Sheh-hhephed--7b Accuracy on Hard (Lovel 4-5) Problems
MotaMath-76 Math- Sath-Shepherd-J1 Accuracy on (Level 1-5) Problems
Accuracy Easy (Leval 1-3) Probloms
88 Problems Solved
65
60
40.0
37.5
55
35.0
50
Majoe By Voting
Weighted vetting RM
Majoely Voting
Weighted Vetting w/ RM
30.0
32.5
Hajority Voding
Weightted Votiing wi RM Best-offt AM
Bust-of-h
Best-afhh
27.5
10
1262563120024
number of Solutions problem
N
number solutions problem
number of solutions problem
Figure Easy-to-hard generalization of evaluators on MetaMath. Both SFTs and RMs are trained on the easy data.
21

---

E
Qualitative Comparison
QUESTION
The value of y varies inversely as v and when x 24, y 15. What when 3?
y varies inversely as vx. then = Vz Plugging in the given values, we have V/4 or = 15v21
for some constant k:
A
15v6
24
Now we can find x when = 3 by plugging in these values into the equation
METAMATH-SFT
VB
We have 3 Cross-multiplying, get 3 4 4/0 15v6, 12 15v6.
1bv6
or 15vG
4/r
Dividing both sides by 12, we find vx
15v6
12.6 2"
Squaring both sides, we get x = (g) 6.25
y varies inversely as vz. then we can write the equation as y vr for some constant We given that when 24, y 15, into the equation to solve for k: 15 k
substitute these values
//4
Multiplying both sides by 24 gives 15
METAMATH- PPO
Simplifying the left side gives 15. 2 = k, which further simplifies to 30 = this value of l in the equation to find when y 3:
Now
30/6
Multiplying both sides by v gives 3 30/6
Dividing both sides by gives / 10v6
Squaring both sides gives x (10v6)² = 100(6) 600
Table Qualitative comparison between MetaMath- FF-FT and MetaMath-7b-PPP as well as the first error detected in MetaMath-SFT greedy decoding by Math-Shepherd. Examples are cherry- picked for better illustration.
F Performance of DPO Iterations
Iterative DPO on PRM800K
Iterative DPO on MetaMath
Problems 5olved
30
polmss Solved
40
28
Train on Level 1-3
Train on Level 1-3
26
Train on Level 1 5
36
Train on Level 5
24
34
20
30
18
28
16
2
2
Num iterations
Num iterations
Figure Test performance as a function of DPO iterations.
22

---

QUESTION
What is the total surface area, in square inches, of a cube with volume of cubic foot?
The volume of a cube is given by = 83 where 8 the length of one side. cubic foot, so 83
We are given that V Taking the cube root of both sides, we find that =.
PRM800K-SEP
The surface area of a cube is given by A 6s
so the total surface area of this cube is A = 6(13) Answer
square inches.
So the volume of cube s And we know the volume is So 3 1.
So 80 The surface area of a cube is 6s?.
PRM800K-PPO
So the surface area is 6 1 6.
So the answer
# Answer
Table 8: The distribution shift before and after PPO training of a Llemma-7B model trained PRM800K. We believe such a big shift could explain why does PPO work exceptionally well on PRM800K, where the policy is initialized from weak SFT model.
23

---

G Re-ranking Performance Analysis of PRMs, ORMs & OPRMs
We compare the re-ranking performance of Process Reward Models (PRMs), Outcome Reward Models (ORMs), and our proposed Outcome & Process Reward Models (OPRMs). Figure 8 shows the results on 7b models and Figure 9 on 34b models. We find that in our setting of Llemma [6] + MATH [32], PRMs and ORMs perform similarly, with PRMs slightly outperforming ORMs on hard tasks. But the OPRMs that trained on the mixed data of PRMs and ORMs significantly outperforms both of them.
5FT-7b + PRM-7b (on PAMBBOK) Accuracy Easy (Leval 1-3) Probloms
SFT-7b ORM-76 (on PRMBOOK] Accuracy on Easy (Level 1-3) Probloms
SFT-7h OPRM-70 (an FRMBOOK]
Accuracy on Easy (Level 1-3) Probloms
probleme Solved
% Problems Solved
550
55
5s
50
45
45
40
Majo by Veltng
Weighted vottigg RM
Majonit Veiing
Weighttd vettig w/ RM Best-ffh w
35
Weighted votiing w/ RM Best-af-h RM
Majenit Voling
30
Best-oth RM1
22 64 N number of Solutions problem
siziele
125 256 5121224 N number solutions problem
25
16 N number of solutions problem
sizinge
SFT-7b + PRM-76 (on PRMBOOK) Accuracy Hard (Leval 4-5) Problems
SFT:7b ORM:76 fon РBМBоок)
SFT-7b OPRM-7 (on PRMB0OK) Accuracy Hard (Level 4-5) Problems
Accuracy on Hard (Level 4-5) Problems
25.0
25.0
25.0
B 22.5
20.0
20.0
20.0
17.5
6 17.5
5 17.5
15.0
E 15.0
E 15.0
12.5
12.5
8 12.5
10.0
Mejority Verting
Waighitd Vottig RM
é 10.0
Majority Vrrting
Waighted Veting w/ RM
10.0
Majority virtigg
Weignted Voting wi RM
7.5
Seat-aly
7.5
Seabsfff
7.5
Best-alMt HM
5.0
5.0
128258staeze
32 et Di22ssstasee number of solutions problem
number solutions problem
number solutions problem
SFT-7b PRM-7b fon D0K)
SFT-7b ORM-7b (on PRMBOOK) Accuracy on All (Level 1-5) Problems
SFT-7b OPRM-70 fon PRMBOOK) Accuracy (Level 1-5) Problems
Accuracy All (Level 1-5) Problems
45
45
40
Selved
35-
35
30
30
Majerity Veting
Majofty Vetting
Majerity Votiigg
20
Best
Weggtied wiingg RM
Best
Weigtted wiingg RM
20
Weighted wiiing wi
number solutions problem
number solutions problem
number solutions problem
Figure 8: Comparing process reward models (PRMs, left), outcome reward models (ORMs, middle), and outcome & process reward models (OPRMs. right) on 7b models trained on the PRM800K dataset. Both SFTs and RMs are trained on the easy data.
24

---

SFT-344 + PRM-34b (an PRMBOOK) Accuracy Easy (Level 1-3) Problems
SFT-24b ORM:34b len PRMBOOK)
SFT-34b OPRM-34b (on PRMBOOK)
Accuract on Easy (Level 1-3) Problems
Accuracy on Easy (Level 1-3) Problems
moleems selve?
70
Prodies solvea
Problems Salved
70
60-
60
60
55
50
50
Majonty Vating
Majoelyy
Majerity
45
Weeghted wiing
Weigtted Wring
Weighted Wiinng
Best-oth
Bestofhk wy
Best-atk RM
35
16 32 14 126 256 siz122e number of solutions per problem
32 M 128 256 s11122e number solutions per problem
35
1b 126 255 s12152e number of polutions per problem
SFT-34b + PRM-34b (an PRMB0оK) Accuracy on Hard (Level 4-5) Problems
SFT-34b ORM:34b ion PRMBOOK)
SFT-34b OPRM-34b (on PRMBOOK) Accuracy on Hard (Level 4-5) Problems
Accuracy on Hard (Level 4-5) Problems
Solved
30
% Troblens Solved
%% Problems Solved
25-
25
20-
20
15
Majority Veiing
Majoiit
15
Majonit Vating
Weighted withig RM Best ht wi
Weigtted Wiingg
Weighted witigg ad Best of RM
Best of wi
32 64 128 256 5121024
16 32 64 128 259 5121024 N number solutions per problem
8 16 32 64 128 256 5123024
N- number of solutions per prablem
N number of lolttions per prablem
SFT-34b e PRM-34b lan PRMBOOK)
SFT-34b GRM-34b ton PAMBOOK) Accuracy on All (Level 1-5) Proaloms
SFT-34b OPRM-34b (on PRM600K) Accuracy on (Lovel 1-5) Problems
Accuracy on All (Level 1-5) Problems
Hooblems Solved
Problems Solved
50
ds
40
30
Majorit Veiing
Majenity Veting
30
Majoeity Voling
Welghted votiig RM
Weighted wittin w/
Weighted Vodiing wi Besk-affl wi RMM
Busteffy st 191
Bast-ffh ai
25
20
22 64 126 256 5121024
20
54 328 259 512102e number solutions per problem
20
16 54 128 256 Sizange number of solutions per problem
4 8
number of Solutions per problem
Figure 9; Comparing process reward models (PRMs, left), outcome reward models (ORMs, middle), and outcome & process reward models (OPRMs, right) on 34b models trained on the PRM800K dataset. Both SFTs and RMs trained the easy data.
25

---

H Re-ranking Results on MetaMath
Similar to Sec. 4.2.1, we assess the effectiveness of process reward models on the MetaMath/Math-
Shepherd dataset [86, 75]. From Figure 10, we can see that PRMs are also more effective on harder tasks when trained on
MetaMatth-o 7b Sah--Shephed-76
MetaMathh7b ORM-7b
MetaMath-7b Shath--Sppher--OPMM
Accuracy Easy (Level 1-3) Problems
Accuracy Easy (Level 1-3) Problems
Accuracy Easy (Level 1-3) Problems
70
70
Proaleme Selved
Problems Salved
problems Selved
80
8 15 &
55
Majority Veting
Majoriy Voting
Majority voting
45
Weggteed wiing RM
% Gn
Weigfted wiing RM
45
Wiighted wiing
Best-ahk
Bestafhh
Best-aht
M 12n 5ibic9e
ii 290 51218ze
number solutions problem
number solutions problem
number lutionn problem
MetaMath-76 Shephere
MetaMath-7B ORM Hard (Level 4-5) Problems
MetaMath-7D Math Shepherd OPRM-7
Accuracy Hard (Level 4-5) Problems
Accuracy Hard (Level Problems
24
N Problems Solved
24
% Problems Solved
24
12
20
20
18
18
16
26
14
Majorit Veting
Majority Veting
Majority viting
Weighted wiing RM
Weighted Wring
Weighted Wiing
Best
Best-ofh
5121624 problem
number solutions problem
number
number
problem
Math-Shwpherd-70
MetaMail- Matt-Shephhed-
Accuracy All (Level Problems
(Level 1-5) Problems
Acturacy (Level Problems
45.0
45.0
45.0
42.5
42.5
42.5
44.00
40.0
8
40.0
37.5
37.5
37.5
"
35.0
35.0
35.0
32.5
32.5
8 32.5
Majorit Wiing
Majerty Weighted
E 30.0
Waing
Weighttd Voting
Weighted Viting
27.5
27.5
8 27.5
number 1252563121026
123226312222
25.0
129256
problem
per problem
number solutions per problem
Figure 10: Comparing process reward models (PRMs, left, trained on Shehh-Shepherd,,, outcome reward models (ORMs, middle), and outcome & process reward models (OPRMs, right) on 7b models trained on the MetaMath dataset. Both SFTs and RMs are trained on the easy data.
26

---

I More Comparisons
SFT-7b + RM-7h (Weighted Varing on PAM600K)
SFT-7b RM-77 (Weighted Vating on PAMBOOK]
SFT-7b + RM-76 (Weighted Vating on PAMROOK]
Accuracy on Easy (Level 2-3) Problems
Accuracy on Hard (Level 4-5) Problems
Accuracy on All (Leve 1-5) Problems
Salves
60
22.5
40
20.0
50-
17.5
35-
5 15.0
35
40
Ourcome RM
Process RM
10.0
12.5
Process RM
Gurcome RM
20
25
Cuncome RM
Process RM
Ouscome 16 Process RM
Ouscome 14 Process RM
Outcome Vs Process RM
25
2 "2"" 16 32 6i sis 2ss 522
5.0
2 "'""" St tå 128 256 5120004
15
2 "2" i5 22 64 64'12a 256 5120:24
N number of solutions per problem
- number of solutions per problem
number of solutions per problem
SFT-7b + RM-76 (Best-ofN on PRMB00K) Accuracy on Easy (Leval 1-31 Problems
SFT-7b + RM-7a (Bast-ofN on PRM600K) Accuracy on Hard (Leval 4-5) Problems
SFT-7b RM-76 (Best-ofN on PRM600K) Accuracy on All (Level 1-5) Problems
60
22.5
Ay Problems Solved
40.
55-
20.0
Us Pracess
a5
50-
17.5
45
15.0
35
Outcome RM
Process
Outcome ia Process HMt
E 10.0
12.5
7.5
25
20
Proness
Outcome to Procees FM
Outcome RMt
25
22 b4 128 25a sizia2e
5.0
126258512224
15
4 8 15
126 256 Sizznze
m number of solutions per problem
N number of solutions per problem
N
number of solutions per problem
Figure 11: Comparing different reward models with Weighted Voting (upper) and Best-of-N (lower) on 7b models trained on the PRM800K dataset. Both SFTs and RMs are trained on the easy data.
SFT-346b + RM-34b (Weighted Voting on PRMBOOK) Acouracy on Easy (Lavel 1-3) Problems
SFT-34b + RM-34b (Weighted Voting on PRMBOOK)
SFT-34b RM-34b (Weighted Voting on PRMBOOK) Accuracy on [Lavel 1-5) Problams
Accuracy Hard (Lavel 4-5) Probimm
75
35
55
Problems Solved
de problems Soloved
16
16
55
20
40
50
45
Preeess SM
Outcoms RM
Outcoms la Procass W
10
15
Dutcoms RM
Dutcome Prosau MX
Process RM
25
35
Dutcoms RN
Frecess RM
Dutcome la trocass M
35
16 22 per
5121024
25e 5121026
20
15 32 64 12s 256 stzuoge solutions
problem
SFT-34b IBest-ofN PRMBOOK)
RMb34b (Best-otN PRMB0Oo)
RM-34b (Best-of-N PRMBOOK) Accuracy (Level 1-5) Problems
Accuracy Easy Pro
Hard (Level 4-5) Problems
75
55
Solvea
Problems ene Salved
Problems Solved
30
45
80-
25
40
55-
50
35
Process HI
Process
30
Process IM
Outcome RM
Outcome
40
64 51z1024
5121024
number solutions per problem
number solutions problem
30 solutions per problem
Figure 12: Comparing different reward models with Weighted Voting (upper) and Best-of-N (lower) on 34b models trained the PRM800K dataset. Both SFTs and RMs are trained the easy data.
27

---

SFT-34b + RM--7 (Weighted Vating on PRMBOOK)
SFT-34b + RM 7b (Weighted Vating PAMBOOK)
SFT-34b + RM-7b (Weigghte Vating on PAMBOOK)
Accuracy on Lasy (Level 1-3) Problems
Accuraty on Hard (Level Problems
Accuracy on All (Level 1-51 Problems
55
70-
50
45-
55-
% Probleer
20
35
30
8440
Pocess
Process
number solutions problem
numberre
problem
number solutions problem
SFT-34b RM-7b [Best-ofN PAMа0OK)
SFT-34b RM.7b (Best-ofN on PAMADOK)
SFT:34b RM-7b (Best-ofN on PRMaOCK) Accuracy on All (Leval 1-5) Problems
Accuracy Easy (Loval 1-3) Problems
Accuracy on Hard (Level 4-5) Probloms
75
55
Progleme Selve
65
% Problemm solved
Outcomm VA Proeess AM
Outcome RM
Process
Problems Selved
50
Process
Outcome VA Prceess FM
Outcome RM
60-
55-
20
50
25
Process RM
Outcome RM
40
Outcome Process RM
siz102e
siz1ze
number solutions problem
number solutions problem
number solutions problem
Figure 13: Comparing different reward models with Weighted Voting (upper) and Best-of-N (lower) on 34b SFT model and 7b reward model trained on the PRM800K dataset. Both SFTs and RMs are trained on the easy data.
7b (Weighted
SFT-7b R04-7b (Weighted on
(Weighted MetaMath) (Level Probiems
Accuracy Easy (Level 1:3) Problems
Accuracy Hard 4:5) Problems
45.0
Protles Selved
65
1 22
42.5
60
8 20
3 37.5
40.0
35,0
50
16
32.5
14
30.0
12
27.5
226 S121004
51200p4
25.0
2"' 32 64's2s 5iza
number of solutions problem
N number solutions problem
nummer of solutions problem
SFT-7b RM-7b (Best-of-N MetaMath) Accuracy Easy (Level 1:3) Pro bloblems
SFT-7b RM-7b (Best-ofNN MetaMath) Accuracy on Hard (Level 4-5) Problems
SFT-7b RM-7b (Best-oFN on MelaMath) Accuracy on All (Level 1-3) Problems
70
45.0
Process RM
Problene Solved
65
Salved
Outcome RM
20
Outcome Proseus RM
3
37.5
40.0
55-
18
35.0
16
3 32.5
45
Process RN
Guicome Process RM
Outcome RM
27.5
30.0
Dutcome RM
Dutcome % Pracess PM
Process RM
64 5121024
128 5121024
25.0
32 64 1252563120024
number solutions problem
number solutions problem
N number of solutions per problem
Figure 14: Comparing different reward models with Weighted Voting (upper) and Best-of-N (lower) on 7b models trained on the MetaMath dataset. Both SFTs and RMs are trained on the easy data.
28

---

Table 9: Results of Full, Hary, & Hard-to-Easy SFT training of the Llemma-7b model
PRM800K
METAMATH
TRAINING DATA
ALL
HARD
ALL
HARD
FULL SFT EASY-TO-HARD SFT HARD-TO-EASY SFT
ALL EASY HARD
20.6
9.9
31.4
13.7
19.8
30.0
14.9
18.4
13.0
30.4
15.3
PRMBOOK/ / LLemma-7b
PRMB0OK LLemma-77
MetaMath
/Leemma77
Finel-Answer
ORN
0.95
1.00
0.91
0.98
ORH
0.95
0.95
0.90
PRM
0.81
1.00
0.81
0.90
0.91
1.00
0.00
PAM
0.76
1.00
0.75
Fina-leewerr
0.85
Rnal-rnswer
0.92
0.89
0.88
1.00
0.90
-075
Figure 15: The agreement between the prediction from the Llemma-77 7a-aaa--b-beedd reward model when trained on ORM and PRM data, and their agreement to ground-truth final-am finl-answwe labels.
Hard-to-Easy Generalization
From Table 5, it is evident that reinforcement learning training on hard tasks alone significantly underperforms compared to training the model on easy tasks or on all tasks. This difference is especially pronounced for PPO on the PRM800K dataset. This raises a crucial question: does training on hard tasks only generalize to easy tasks?
To address this, fine-tuned the Llemma-7b model using all data (easy and hard), only easy data, and only hard data. As shown Table 9, we found that training on all data consistently yields the best performance. Conversely, the generator's performance deteriorates when transitioning from easy- to-hard and hard-to-easy tasks. This suggests that language models face difficulties generalizing in both directions.
is also worth noting that while Full SFT underperforms Easy-to-Hard SFT and Hard-to-Easy SFT on hard test questions, it eventually outperforms Easy-to-Hard SFT and Hard-to-Eas SFT when evaluated on all test questions. We believe that this because by exposing the model to a wider variety of unique questions and difficulties, gains better understanding of the problem space in general, as measured by the accuracy on the full distribution.
On ORM's Approximation of PRM Labels K
From Sec. G, we observe that in PRM80OK, PRMs and ORMs exhibit similar performance levels, with OPRMs outperforming both. This raises the question of why ORMs also demonstrate strong easy-to-hard generalization ability. A straightforward explanation is that ORMs are trained to approximate PRM labels [74]. Specifically, ORMs trained predict the correctness of the entire solution through value estimation. As Uesato et al. [74] state, "it is simpler for the ORM to learn recognize when steps are correct than is check the answer by internally computing the final answer itself."
Nevertheless, people may argue that the conclusion from Uesato al. [74] is based GSM8K's experimental results, so the conclusion may not transfer to the more challenging Hendrick's MATH dataset. To show the universal existence of "ORM's approximation of PRM labels", further conduct evaluation agreement between different rewards on two variants the MATH dataset: PRM800K and MetaMath.
The results are shown Figure 15. Similarly to the findings from Uesato al. [74], we see that the ORM has higher agreement with the PRM, despite being trained to predict the Final- Annwer rewards.

---

Thus, "this result indicates that the ORM tends more towards predicting whether the full trace is correct, and not just whether the final answer is correct."
Overall, this shows easy-to-hard generalization is not exclusively linked to reward models trained on explicit step-wise annotations. It also applies to ORMs that are trained to perform value estimation and practically evaluates each solution step.
We also perform DPO training RL setting, re-ranking the output pairs with ORM also gives similar performance to re-ranking with PRM (29.2 30.4 & 31.2 v.s. 34.0).
MetaMath-initializn Llemma-7b model. We find that in this
L Analysis of Aggregation Functions in PRMs & OPRMs
We explored different methods to consolidate step-wise prediction scores into a single score value, process describe as employing an aggregation function, during the use of the evaluator. Lightman et al. [40] report comparable performance when using min (minimum) and prod (product) as the aggregation function to reduce multiple scores into single solution-level score. Note that when training PRMs on PRM800K [40]. we have already considered neutral steps to be positive as training labels.
Following Wang et al. [77], given {pi, .2.....Pri. as a list of predicted correctness probability of each step (including the final answer), considered the following aggregation functions:
min min P2,P,, P
Pr.
(1)
max max (pI, P2........
(2)
prod = II ps
(3)
mean EiPr Pt
(4)
Pt Σ, log I-pi
mean Lannt
(5)
72
Σ, Pr.
I-Ps
mean_odd = RcLU
(6)
last = Pr.
In Figure 16-18, perform analysis of aggregation functions PRM800K and Math-Shepherd (from MetaMath) datasets with weighted voting and best-of-n decoding and PRMs or OPRMs. In general, we find prod works universally well in weighted voting and min works well in best-of-n. So we adopt these two strategies in our main experiments.
One interesting finding is that for reward models trained the human annotated process reward (e.g., PRM800K), the last strategy does not perform very well, but last works much better on OPRMs and pseudo PRMs (c.g., Math-Shepherd) This could partially explain why OPRMs does not further improve the performance on the Math-Shepherd dataset.
30

---

SFT.7b PA89-7 Weighted Votiig Accurssy
SFT.7b P80977 Weighttd Votiig
SFT.b. + PRM6-7 Weighted Veting
1-53
mmiiee me Sobond
20-
18
16-
30
min
14
max
mean
mesn
metn
mean logit
moon legt
menn logit
mean.00 odd
me33_0000 ood
20-
mean, ood
5121026
of solutions problem
number soludion problem
number soluatuu problem
SFT-7b + PRM 7b Best
SFT.b PRM 7b
ofst
9FT.76 PRM 7b Best ofN
Accunccy Easy
Accurary
16.
Piotleeme Salved
Prablama Salwa
prod
min
min
max
mccn
mobn
meen
mesn_bogt
mete mean_kott
mean, Jogh
mezn odd
mean odd
ood
18
128
number solutiess 256 3121024
15
N solultone probiem
N number solidiins
SFT-34b PRWW-440 + Weighted Veriing Accuracy Easy (Lavel 1-31 Prodiems
SFT:34b - PRM--34 Weighted Voting Accuracy on Hard (Lavel 4-5) Proclems
SFT:34b + PRU5-346 Weighted Voting Accuracy on Jaaeel 1-5) Problems
70
30.0
50
% Tobreee Souedd
45.
3 25.0
45
22.5
prod
40
prod
min
min
max
max
25
max
mean
17.5
mean
mean
mezn logt
moon loglt
mean logit
40
12.5
15 32 54 511122
soluttore problem
problem
N number soluttons
SFT 34b + PPN 340 Best oth
SFT 34b PRM 342 Best.oth,
5FT PRM 343 Best ofN
Accuracy Exsy (Level 1-31
Accuracy handd (Lave) 431 hotlems
Accunity en J.evel 1-5) Proddems
24
% Frobleems Solved
Probiumm Solved
40
prod
prod
prod
min
18
min
min
max
maa
mean
mebn
mean
mean
mein logi
logi
10
M 128
18 128 256 5121024 per probiem
18 32
258 5121024
pe
Figure 16: Analysis of aggregation functions in process reward models (PRMs) on the PRMS00K dataset with Weighted Voting and Best-of-N. Both SFTs and RMs are trained on the easy data.
31

---

SFT.7b -FRM Te Weighted Votng
5F7.76 GPRM To Weighted Wrttig
5FT-7b OPRM To Weighted Voting
Accuracy on Easy [Lavel 1-31 Prosiems
Accuracy [Lavel Proolems
Accurasy on Jaaell 1-5) Problems
S.hheere Eoloed
20.0
max
prod
mozn
max
mean
17.5
15.0
12.5
mean c30
maen loght
meon
30-
mean logi
min
metn
maa
2 10.0
20
7.5
M
15 32 54 problem
5121524
SFT.7b OPRMS 7b Best
SFT.7b OPRM 76 Best
SFT.70 OPRN To Best of
Accurary Eaey (Level Proteess
Accuracy hand [Level Protlems
Accuntry on Cleel 1-5) Proelems
55-
% Probleen Solves
Probilum Solvodd
prod
prod
prod
max
min
mebn
13
max
min
me2n
27.5
min
max
mann
22.5
17.5
M 128 5121076
16 258 5121076
M 128256 5171026
per
SFT-346 -PRM Weighted
SPT-346 OPRM Weighted Vottig
SFT-345 OPRM-346 WWlghhted
Accuracy on Easy (Lavel Preblems
Accuraicy hard (Lavel 4-51 hoteeess
Accurasy on Juevel 1-5j
hobhemm Solved
grod
prod
45
prod
min
min
min
max
max
max
50
you
55
logit
10
16 24 12s 5121026
16
54 12s 25e s121026
16 24
128 20 5121024
problem
per problem
SFT
34b OPRN 34b
34b OPRN 34; ofN
Accuracy on Evsy (Leve) 1-31 Protiams
Accurwty hand [Level Prodeers.
Accursty on Jueval 1-53 Problemss
70
65-
60.
prod
prod
40
prod
14 Prodeene
min
min
min
15.0
12.5
moon
last
last
16.0
18 32
M 128-256330 S121026
18 32 M 128 $121024 number solurions per problem
18 32 M4 128 256 5121024
mum solutions per problem
solddion per problem
Figure 17: Analysis of aggregation functions in outcome & process reward models (OPRMs) on the PRM800K dataset with Weighted Voting and Best-of-N. Both SFTs and RMs are trained the easy data.
32

---

MetaMarh-Th Math: Shephero-72 Weighted Voting
MetsMorth-T: Moth-Shhppher-- + Weighted Voling
MetaMatth-T Math Shephert To Wiegttte Voting
44
Protteen Solved
Poolll SiAved
20
610d
plod
mn
min
5121026
FALT diors probiem
number soludions probimm
N number solutions probiem
MetteMath-7 Nathechagheed-1 a Besiidil Accuracy Essy (Leve Probloms
MataMatth-F a Meithchhoghers-7 Besiolll
Mate/Meth--/ Metbechepherdd/ Bestsa4i
Accuracy Problems
Accuracy on Prociems
-robleme Sahvel
Sabuad
54
12-
pro-
-2
prod
100
128
250
soludionn probimm
soludions probiem
solidioss probiem
Figure 18: Analysis of aggregation functions in psuedo process reward models (PRMs) the Math-Shepherd (from MetaMath) dataset with Weighted Voting and Best-of-N, Both SFTs and RMs are trained on the easy data.
M Societal Impact
Our work on easy-to-hard generalization has the potential for both positive and negative societal impacts. On the positive side, this approach could enable Al systems to tackle increasingly complex problems in domains such as scientific discovery, healthcare, and education, potentially leading to groundbreaking advancements that benefit society. However, the development of Al systems that can operate beyond human supervision also raises concerns about the transparency, accountability, and potential misuse of such systems. It is crucial to carefully consider the ethical implications and establish robust safeguards to mitigate the risks of unintended consequences or malicious applications. Ongoing research and public discourse the responsible development and deployment of these technologies will be essential to ensure that their societal benefits outweigh the potential drawbacks
33

---

N
Analysis of OPRMs' Re-ranking strategies
OPAN-78 for
5P776 SPRM-5RD for FRNB0310
SP--340 OPRI676 fan Accurase Levell Pntrarrs
On534-46 ron PAMBBOK)
Accurasc Lero'l Pob lone
AcclEacy Leve Prbllans
Accur Levell Protrens
40-
770-
10
50
60
60
Mairty letigg
Macrty laing
Maprky leto
Besketh a/
number solaiins per proMinn
numbero" soliiess preticm
sumber of scleloos per probium
sumber sestuters per prebicm
5P1/7 e gees for 19080330
SFT72 grep ses for FA40030)
5F1-300 e oress 73 poreone
SFT-243 i crige 340 to Pasecok)
Acctracy
3.6-
70-
2.55
co-
65
Morry ling
Manrty king
Monkylsieg
75.
Weightd aga
Wegntad wergget
15
Wegrtad
Ne grad letin
20
mwnumber ber sulaiina probium
numbere soliiion prebium
sumber sokalers predicm
sumber sekuuiirss prebiam
Accuracy on Leve Pob lone
Accuracy on Love Podiorrs
Accur on Leve Phollens
Accus 3cy on Levol Prorenn
70-
3
500
60
3.40
R20
15
245
10-
We ve
25
10 Jwaaznnne
S-rizb opea for PR0B0010
SFT/23 oPee ses jor Horone))
SFT
340 + oreos za for exeraos))
SFT-343: oria 54b oon Paassco))
330
35-
Mairit letin
20
Macrly saing
Netrtylettn anstet
Macrity loting
20.
15
TN numbee solation per preliem
It number of soliiins pe preMium
sumber of siluaioss per prodium
Nt sumber of scluuil per probium
SmTTT optatrt tor raog021)) Accuracy Lave Prob aee
SPT:70 arSM HaMB001)
51734t oo PoM000
SPT.346 om16 Maaacoii
Accuraty Lave Problerss
Accuracy on Leve Protiems
Accuracy Lavel Prebiema
20.0
17.5
15.0
12
10.0
5.
75
Wrigh Boogte
3 35 24 133 533024 -ummbrrr solriin protierr
3f X 24 1507330000
25.
3f
35
& 1200000000 problen
numbero
Sumber of parpotiie
number
Figure 19: Easy-to-hard generalization for different difficulty levels' data. Both SFTs and OPRMs are trained on the levell-3 data. Each row compares the performance of different OPRMs' reranking strategies across different levels' data.
As shown in Figure 19, both the Best-of-N and Weighted Voting strategies demonstrate strong perfor- mance across all levels, which leverage the advantages of OPRM methods and thereby underscoring OPRM's effectiveness. Furthermore, despite the SFT models and OPRM models being trained on level 1-3 data, re-ranking strategies enhanced by OPRMs continue to perform well on the unseen and more challenging level 4-5 data. This indicates the feasibility of generalizing from easier to harder tasks using OPRMs.
34

---

OPAN-10 for
SF7/78 OPW in ACCurICY Algob Pratles
ACOUTICY Alyebra Protlns
AccunICy Agoioo Notloms
Alcunncy yobro mebiens
10-
740-
255
60
245
25-
Masrt/ietny keing
Macrty laing
Matrty/ittg
65
Maprty be
10
10
anilll W
15
40
numbere solaiios per prodian
numberel saliiess pretiim
sumber aiailii per predium
cuuberalsssiiie per presism
SiT-7 coapeTa for PR0003Ki
51777 a ars jon PaM000I
57733e a c0RM677 for PaMeooe
577.382 - ono.-34 ron PSMICOK:
Courting
Accurssy on Courting
Courting Trobiees
Counting By
10-
45-
25-
20
15.
Wiigttdd yes
Marty lin
15
Minrty Min)
Wegrad vergurt
Wegnadd hetr
Monkylste
Ne grad letggeel
20
15
24
-..mbrrr soleiins per pretierr
numberra coartins per prohiem
Surbbr of perprati
N "umber of eckution per perprebiem
Pasposi Accunicy on Geonneryy Pollens
Accuriicy or beuniey Postens
Accunicy or Goumery Pobiems
Acoracyy on Gcomeitl moblens
10-
20
340
45
3 ec-
3.5-
nunttng 3
8.
We pe ki
a se peaaazzei
15 se 328 aevaznne
ser it 26 for 190801100
557i7h gPew aes byr Hweand)
5F 340 + oress 23 for ewrann
3F1-34) - coise 54b un Pasecce))
interned Prosiemt
Algebra Prozlams
Alpebra Prablem
Accurccy
Algebra Problemn
10
10-
20-
25
20
25
10.
Martiy fly lin
Marrly Wini
Migelt Seiing
1555522000
number siliion pe preMinn
11 nunber of scliiins po preMiam
sumber of salatons poi probium
R sumber sisuuaioos per probium
smm-7 optae for rRM80710 Accuaay on Nuriad thery MoMMers
577-70 areM Sio for H340020
5P7.345 O0tg 34h Passsook)
Accrray Hurra theury hobiers
Accursoo on Mambic thesy hnbrons
Accusscy on Munierr Theary eni
50
30-
40
20
15.
Mairity leing
25
20
25-
Magrit leing
30
26
Maprty/etay
30
Haarity jolog
D s4 324 pentztoo
D. se 12e mantrne
N "unumbrrrl soaeiee prehlerr
N: numberrr eckkiinss
N: "umberat
opae-20 for
Acoracy mollybio mosiems
Acou an Proilyter Posleno
Acorocy heclyder Podlens
Accurde on rrcolyebr Potlens
10-
65
60
40
10-
Mairtty leing
Majrit lsing
Maprit jiting
wi
45
40
Maorky letrg
WIS
25
Besi ay
numberr' of solation per probian
numbero" solaiens per pretiom
sumber sslalors per prodium
sumber of selutors pel predicm
ST777 oPRN-7 tor mRM00sKI
57777 arRM for Fa48001
577345 - comMi for PaNDosi:
577:34 - org--344 con Paaxoak))
Accurscy Precaicuass
Accumny
Accursc
20
Feu 20
25
200
Morty
2.0
10
255 www.mrrroooooooo C0l prchiem
w..rbbrro colrinn per pretier
Sumber wtw.brrrrlllllppppppeeep prraaiem
sumber of eckucbrr www.sitreppppppooooo perprecien
Figure 20: Easy-to-hard generalization for different type's 's data. Both SFTs and OPRMs are trained on the easy data. Each row compares the performance of different OPRMs' reranking strategies across different types' data.
To determine which types of data benefit more from OPRM's easy-to-hard generalization and which types still struggle with this challenging generalization, we compare OPRM's generalization abilities on different problem types in Figure 20. Among Algebra. Counting & Probability, Geometry, Intermediate Algebra, Number Theory, Prealgebra, and Precalculus problems, OPRMs generalize best on Algebra, Intermediate Algebra, and Precalculus problems. Conversely, OPRMs generalize worst on Counting & Probability problems. These findings are highly valuable for practical system design, allowing us to decide when to use OPRMs to enhance performance based on the downstream data type.
35

---

O Pass@N Analysis for Different Math Questions
SFT-7b (on PRM80OK)
SFT-34b (on PRM800K)
Pass@N on Problems by Level
Pass@N on Problems by Level
100
100
% Problems Solved
% Problems Solved
80
80
60
170
60
Level 1
Level
40
45
Level 2
40
Level
2u
Level 3
Level
20
Level 4
20
Level
Level 5
Level
1
2
16 32 64 128 256 5121024
4
8
16
32 64 128 256 5121024
N
number solutions per problem
number of solutions per problem
SFT-7b (on PRM800K)
SFT-34b (on PRMB0OK)
Pass@N on Problems by Category
Pass@N on Problems by Category
100-
100
% Problems Solved
% Problems Solved
80
60
Prealgebra
60
Presigebra
Counting Probability
Counting & Probability
40
Ngebra
40
10
Algebra
Number Theory
Number Theory
Geometry
Geometry
20
46
Intermediate Algebra
20
Precaiculus
Precalculus
Intermediate Algebra
1 2 4 16 32 64 128 256 5121024
4 8 16 32 64 128 256 5121024
N = number of solutions per problem
N = number of solutions per problem
Figure 21: The increase of P metrics with the number of samples, divided by level and category. As the number of samples increases, the @ass@N N rate tends improve. However, not all problems will eventually yield a correct solution, meaning that even as N approaches infinity, the Pass @N metric may not reach 100%.
shows the changes in Pass @ass@N the number of samples increases. As shown in the figure, not all problems reach a 100% P rate, even with a larger N. Specifically, three key observations be made: First, Pass@N N is strongly correlated with problem difficulty. As depicted in Figure 21, for Level problems, the Pass rate nearly saturates with more samples, while for Level 5 problems, many instances still do not achieve correct solution. Second, larger models (e.g.. 34b) perform significantly better on Pass compared to smaller models (e.g.. 7b). However, as N increases, the advantage of the larger model diminishes on metrics like Pass 1024. Third, the variations in Pass @ N across different categories of math questions are different, which is expected, the difficulty of different types of math questions varies for large language models.
Figure 21
36

---

P
Case Studies for OPRM
Case Study 1: rohot moving forward at constant speed takes 2.5 hours travel kilometer. Moving forward at this same constant speed, it takes the robot 90 seconds to travel the length of particular hallway. How many meters long the hellway? Level: GT: 10
Voting Reward: (A nswer: 36, Voting times: 130,
11.3), (A mswwer: 10, Voting times: 99, reward: 23.73
Majority Solution:
ORR Solution:
Lets zumber
hallway
1o solve pmolen, distante, and time.
fiad the reistioncip between the robot's
0.57
know the robot moved. I km in 2.5 hours.
0.88
foruua distance speed time
equation
That moans moved a meters in 30 seconds.
0.9
Tat tite the robot's speed meters per hour.
write
equation S (x2
0.98
0.11
Then the ftrst case gives 1000 = 2.0, oz' 1030/ 255 600 meters per houe
salve for s. we oan mulliply both sides by 90 to get $36mus.
0.08
length hellway 90/ 36000 length
So the hallway is 36 meters leng.
90J 3030 10 mettrs.
0.79
#t inal Reward: 0.7e5
10
#Final Reward: 0.74
Figure 22: Case Study 1. The evaluator can help identify and reduce the confidence hallucinations caused by misleading information.
Case Study 2: In a rogular polygon, the moasure of an intorior angle is 6.5 times the moasure of an extenor angle. How many sides does the polygon have?
Level:
Voting V.S. Reward: (Answer: 0, Voting times: 132, OPRM reward: 0.03). (nswer: 15, Voting times: 94, OPRM reward: 17.3)
Majority Solution:
dReward
ORRM Solution:
#Reward
So wi know that the mexsum of an interoo or ingle is 6.5 times the m meveum of on ettatoon
know that the sim of the interor undles of wes polopen with n sides (n-21 times
angle:
degreas, the sum the ectofoor anglos is degmas.
0.50
0.90
Ad we know that the sum the meseures of the froooor angntorangles gles of pelyges is given formula (1-2]1 where the rumber sides.
cun write equition relding the inserioe and ectarior anglos of regular polygonn 360, when is the ber
(a-2)
0.01
Right So we con say that 200022m00-211 80.
pun simplity this equat on by dividing everything by 180: n- =6.5 2, which
gives
And that simplifles 6.5+1100 which is Galse.
0.97
0.12
oun add to both sides to isolato ro n- 13 2, which glvss me 115
such polygon.
0.02 #Amnwer
#Ansuer
15
#Finol Reward:
Figure 23: Case Study 2.
The evaluator can assist in reducing the confidence of solutions that misuse mathematical theorems or formulas.
We have included more case studies in Figures 22 and 23, We find that evaluator can help generalize to harder ones in the following ways:
The evaluator can help identify and reduce the confidence of hallucinations caused by misleading information in problems, As demonstrated in Case Study I, the solution selected by majority voting with an answer of 36 is misled by the different units of measurement in the problem (2.5 hours and 90 seconds), resulting in an incorrect solution. Then, the ORPM model successfully gives this solution a low score.
The evaluator assist in reducing the confidence of solutions that misuse mathematical theorems. In Case Study 2, the majority solution incorrectly applies the theorem "the sum of the exterior angles of polygon is 360"". leading erroneous reasoning, and low confidence by the ORPM model.
37

---

Step and Outcome ROC Curve
To assess the accuracy of reasoning step judgments of different reward models, we conducted additional experiments using the PRM800K-tess data, which includes correctness annotations for each step, to test our model's ability to distinguish cor- rect reasoning steps. We randomly se- lected portion of PRM800K-test data to balance positive and negative samples. The accuracy of the reasoning steps for the three models shown in Table 10. This table demonstrates the effectiveness of trained PRM, showing that PRM has significantly greater ability to distinguish steps compared to ORM. Additionally, in Figure 24, we present the Step ROC curves of three models, where PRM and OPRM exhibit better step discrimination abilities compared to ORM.. However, is important to note that a stronger ability to distinguish steps does necessarily indicate that the evaluator is more helpful for generation. We then also present the Outcome ROC curves of three models on discriminating the final outcome. We collect data generated on MATH500 test set from our 7B policy model. According to the final outcome and groundtruth, we label each data and select a OPRM exhibits better outcome discrimination abilities compared ORM and PRM. The above table also shows the effectiveness of OPRM on Outcome discrimination ability.
Model
Step ACC (%)
Outcome ACC (%)
ORM-PRM800K 7BB
64.3
71.7
ORM-PRM800K-1
63.5
OPRM-PRM8OOK-711
79.8
74.4
Table 10: The accuracy of the reasoning steps for differ- ent models.
balanced set to plot the Outcome ROC curves, where
Step ROC Curve
Outcome ROC Curve
1.0
1.0
0.8
0.8
True Positive Rate
True Positive
0.6
0.6
0.4
0.4
Random Guess (AUC 0.50)
Random Guess (AUC 0.50)
0.2
DRM-PRM8GOK (AUC 0.55)
0.2
PRMMPRMBOOK (AUC 0.67)
PRM-PRMBODK (AUC 0.79)
PRM-PAMBOOK (AUC 0.62)
OPRM-PRMBOOK 0.76)
OPRM-PRMBOOK
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
False Positive
False Positive Rate
Figure 24: The Step and Outcome ROC Curve show the ability of discriminate the reasoning step and the whole reasoning solution, illustrating OPRM's effectiveness on identifying both steps and outcomes.
38

---

R Few-Shot Prompt in In-Context Learning
We sample with temperature =.9 for ICL-related experiments. We list our few-shot sample for the In-context learning experiments in Table 1.
4-shot example for PRM800K Full ICL:
Problem:
In right triangle SPQRS, we have 8\ Vanngl Q = Vangle R$ and $PR = 6 61)qrt-(2)$.
What is the area of WVltriangle PQRS?
Solution:
I see that this is an isosceles right triangle, since langle Q Vangle R 45-Mce An'nThat means that $PQ PR 61)sqr(()) \n\nTo find the area of a triangle, I need to multiply the base and the height and divide by 2. AnAnI this case, the base and the height An'nI can simplify this expression by using the property that $1 Milsqrt(a)) Songgrtaaalsgrttfe (sgrt(()> = Vlsqrt(ab)s.. An'nSo, the area is $(Vsgrt (2)Nsgrr((2)) (6 (edot 6)/2 2\oodot 36/2 = 36$.
both $61 $6Vlaqrt(28S,
the
S(6\)
Final Answer: The final
is $36$. hope it correct.
Problem:
The length of the longer side of rectangle SR$ is $10$ percent more than the length of side of square $8.$ The length of the shorter side of rectangle $R$ is $10$ percent less than the length of side of square $S.$ What is the ratio of the area of rectangle SR$ to the area square $S?$ Express your answer as common fraction.
Solution:
Let side the square $s8. \nAnThen the longer side of the rectangle is $1.1 a$. AnAnAnd the shorter side the rectangle $0.9 $0.9e$ \n\nThe area the square is $s-2$. \n\nThe area of the rectangle is $1.1s * 0.9s 0. An\nSo the ratio of the area of rectangle SR$ to the area of square $S$ is filfrac (1)boxed() (fare
{0. (3r") 0.99$. \nYnTherefore the
Final Answer: The final answer $11ffra(999)<11033. hope it correct.
Problem:
Compute $11sin(-60-1lcire))
Solution:
I know the sine function is periodic with period 1360r,, so (0 + 360"\\circ)
) = =\lsin(-00-llcii
(3a0 \nnII also know that the sine function is symmetrical about the origin, $1)sin(-x) ( (1iin() An\aTo I can use the special right triangle with angles $30 "Mcirct, $60-\lcirch, and $90-\ccire$ and sides in the ratio $1: (3: nlnThe sine of an angle is the ratio of the opposite side to the hypotenuse, so $ (11ein(600(eiire) (ggracc() InlnTherefore, NNeein(60-Vicirr) (
for any angle $x$. \nlnTherefore, $11eei((-0-1\eirr) find (11sin(60
Final Anawer: The final answer is $- [al]fracd(\aggrrr3333333333 hope it is correct.
Problem:
Simplify Slisin (x - y) \lcos y \loo y) Visin y.$
Solution:
'0k, so have $isin (x y) Vlcos y Vocos (x Main ys'n that $\\ ain (x - y) \lcos y Vlcos (x y) Ilin y$ in the form of $(lsin(\lalpha) (aaappa)llssi((1bebeeleaa
\lcos (1bbet))++ (Icha))lcos(( (WWbbbt)+)))tteee (ooott))(cos(((aapppp (ain(( baait(bbeeaapssssiaaaaaaaaaaaaa (nnSo,,, $ Wsin (x \lcos \lcos (x y) Vlinn (an )=llsin(x)
AnlnYes and we know that $Visin(1)
Final Answer: final answer $1leinx8. hope it correct.
39

---

4-shot example for PRM800K Hard ICL:
Problem:
In right triangle SPQRS, we have Salangle q = Vangle R$ and $PR ())qqr((2)8
What is the area of Miltriangle PQRS?
Solution:
I see that this is an isosceles right triangle, since $1 langle Q = Vangle R 45-Vicirc$. An'nThat means that $PQ PR = 6\lsqrt(2)s AnAnTo find the area of a triangle, I need to multiply the base and the height and divide by 2. AnAnII this case, base and the height are both $61 (6)ger((() so (e
8(6\)
AnAnI can simplify this expression by using the property that $1 Milaqrt((a) Ceagrr(aaalaqrteen laqrt(b} = Vlaqrt(ab)S.. An'nSo, the area is $(Vagrt (2H)sqrt((2)) (6))cdor 6)/2 2\oodt 36/2 = 36$.
Final Answer: The final answer is $36$. hope it is correct.
Problem:
The length of the longer side of rectangle SR$ is $10$ percent more than the length of side of square $9.$ The length of the shorter side of rectangle $RS is $10$ percent less than the length of side of square $8.$ What the ratio of the area of rectangle SR$ to the area square $S?$ Express your answer as common fraction.
Solution:
Let side
square $s$. \n\nThen the longer side s$. AnAnAnd the shorter side of the rectangle is $0.9s$. \n\nThe area of the square is $s An\nThe area of the rectangle is $1.1s * 0.9s 0.99s AnAnSo the ratio of the area of rectangle SR$ to the area of square $S$ is $1\frac $1\boxed (11))
rectangle $1.1
(2-=)) 0.99$. \n\nTherefore, the Final Answer: The final answer is $(\frac((9))((100335 hope it correct.
Problem:
Suppose that $y varies inversely with [3) (z)$. If $y=2$ when $z=1$, find the value of $z$ when $y=4$. Express your answer simplest fractional form.
Solution:
I know that inverse variation means that the product of the two quantities is write equation of the form $y-3lledot\ [3] [z}=k$,
constant, where $k$ is some constant AnAnTo find $*$, can plug the given values of $y$ and $2$: 32-3Vicdot (3) (1)-k$, which simplifies to $8-k$. AnAnNow I can use this equation to find $2$ when $y-4$: $4-31\edot\ [sqrt [3] (333 (2)-8$, which implies that cube root, I cube both aides: $z= 1ee(( (riighh) (riggtt)=1]]) 33//ffac {1H512)8.
AnAnTo get rid of the
Final Anaver: The final answer is $1)frac(1)(612)$ I hope it is correct.
Problem:
Let $d$ positive number such that when $109$ divided by $d$, the remainder is $4.$ Compute the sum all possible two-digit values of sd$.
Solution:
This problem involves finding the divisors of given number, well using the concept of remainders. \ninone way to approach this to write the division as quotient and a remainder, like this: $109 = qd + 4$, where $q$ is the quotient. \ninThen, I can rearrange this equation to get $105 qd$, which means that $d$ is divisor of $105$. \n\ntow, need to find all the two-digit divisors $105$. \n\nI can use prime factorization to help me do this, An\mI notice that $105 3 Vtimes Atimms 7$, which are all prime numbers. An nTherefore, any divisor of $105$ must be product of some combination of these three factors. AnAnThe possible products are $1, 3, 5, 7, 15, 21, 35, 1058. An\ nHowever, not all of these nThese are the only possible values of $d$ that satisfy the given condition. \n) nTo find their sum, I just add them up: $15 + 21 + 35 71$. Final Ansver: The final answer is $71$. I hope it is correct.
two-digit numbers. Only $15, 21, 35$ are. \n)
40

---

4-shot example for PRM800K Easy-to-Hard ICL:
Problem:
How many 4-letter words with at least one consonant can be constructed from the letters $AS, $B$, $C$, SD$, and $E$? (Note that SB$, $C$, and $D$ are consonants, any word is valid, just English language words, and letters may used more than once.)
Solution:
To count the number of 4-letter words with at least consonant, I can use the complement principle and aubtract the number 4-letter words with consonants from the total number 4-letter words. \n\nThe total number 4- letter words is simply $5*4$, since each letter has choices and the order matters. \n Ine number of 4-letter words with no consonants is just the number of ways to choose letters from the vowels SAS and $E$, which is $2"48. \n\ uTherefore, the number of 4-letter words with at least one consonant is 85~4 2-48. 625 16 609
Final Answer: The final answer is $609$. I hope it is correct.
Problem:
Compute the integer Sk for whichlal\/AWoog..t000 2) \Mlog_(10> (k 1)
+ 2 2 WVlog_(10} k!. AM
Solution:
recognize that this equation involves logarithms of factorials, which are products of consecutive integers. \n\nI also know that logarithme have some useful properties, such as $1llog_ (10) a [Nlog_(10} b \olog_ {10} (ab) and
Sc \llog_ (10)
\Mlog_ d An/nUsing these properties, can simplify the equation follows: \MANoog (k 2)! (1log_(10) (k 1)! +
\llog_(10) \iimllee \ilog_ WileftI
2) (k 1)! 100 \\right]
1og_[10} (k!) Wilnnnnince the bases of the logarithms are equal, I can
conclude that the arguments must also be equal, 1.e., WI(k 2)! (k - 1)! 100 (k!)-2. XV) \n'nloow I have simpler equation to solve for Sk8. AnAnI notice that left-hand side has a factor of $(k 28, which also factor of $ (k- 1)!$ and =(k!)-2$. An\nTherefore, get WI[k - 1)! 100 = k! (k 1)/
can divide both sides by $(k - 2) to
further, I get \M100 = k !. WWWnnnThis means that $k$ is the smallest integer such that its factorial is greater than equal to 100. An\nI
right one. AnAmIf $ 38, then $k! 3! try 41 24 1008. \n\nIf $k 5$, then $k! 5! 120 > 100$. AnlaTherefore, $k =
values of Skš to find the 1008. \n\nIf
48, then Sk!
$ is the smallest integer that satisfies the equation.
Final Answer: The final ansver is $5$. I hope it is correct.
Problem:
What is the largest perfect square factor of 1512?
Solution:
know that
divisible by so can try to factor out powers of 2 until I get an odd number. An in/nDividing by 2 once, I get 756. An\nDividing by 2 again, I get 378. \nlnDividing by 2 one more time, I get 189. Annnoo 189 is odd, so I can 't divide by 2 anymore. \nAnI notice that 189 is divisible by factor out powers of 3. \n\nDividing by 3 once, get 63. \n'nDividing by again, I get 21, \nAnDividing by 3 one more time, get 7. Am\nNow 7 is prime, so I can't factor it anymore. AnAnI have factored 1512 as 2 * 3 * 7. An'aTo find the largest perfect square factor, I need to use the highest even powers of each prime factor.. \n\nThe highest even power of 2 that have is 2-2. \nAnThe highest even power
can try to
that I have 3-2. An\nThe highest even power of that I have is 7-0, which is just 1. An'nSo the largest perfect square factor is 2-2 * 3-2 * 7"0, which is 36.
Final Answer: The final answer is $36$. I hope it is correct.
Problem:
If $3p+4q and $4p+3g-138, what is $6$ equal to?
Solution:
41

---

I notice that the coefficients of $på and $q$ are reversed in the two equations, so elimination get rid of of them. \n\aI try
I wonder if I multiplying the first equation by 4
second equation by 3, and then subtracting them to eliminate $p8. An'nI get $12p+16q-12) 12---122---9--955 which simplifies to $7q=-7$. \n'nDividing both sides by 7, I get $q Final Answer: The final answer is $- 1$. hope it is correct.
42

---

cdot 100 (k Meedot (k-1) Wedot !.-2))!. AnCancellling out the common k Wedot (k-1) Meedot (k-2) \VJ \nFactoring, we find that 4-shot example for MetaMath Full ICL:
Problem:
In right triangle SPQRS, we have Maaangle q = Vangle R$ and $PR 6) (2)qrt((2) What is the area of Wiltringlle PQRS?
Solution:
Since lalangle Q = Mangle R$ we have $1 \triangee PQR$ is an isosceles right triangle. \nTherefore, SPQ - PR 6 6Vlsqrt(2)$ AnThe area of Siltriangle PQR$ is (1)frac(1))) Meddo PQ Weddot PR (\frac(1)22) Wedot 61/sqr((2) Viedot 6\\ sqrt(2) Wboxed(36)SS
Final Answer: The final answer is $36$. hope it correct.
Problem:
The length of the longer side of rectangle SR$ is $10$ percent than the length side square $8.$ The length the shorter side rectangle $R$ $10$ percent less than the length of side of square $S.$ What is the ratio of the area of rectangle $R$ to the area of aquare $S?$ Express your answer as a common fraction.
Solution:
Let the side length of square $S$ be $x$. \nThen the longer side of rectangle $8$ is $1. 11.1x$ and the shorter side is 20.92$. \nThe area of square SS$ is $x"2$ and the of rectangle $R$ is (0 (0.9x) 0.99x-2$. AnTherefore, the ratio of $11ffac(0.99x 99x
area rectangle SR$ to area square \boxed(() (9))
Final Answer: The final answer is (11)).. hope it is correct.
Problem:
Compute the integer 8k for Whichla\\/A/oog..t00 (k 2) \Mlog_11) (k 1)
2 = 2 Allog_110> k.
Solution:
Using properties logarithms,
reurite the equation as\n\\/\\log_.f (10}
\llog_ {10} (x using the fact that $ ((k-2) W cdot (k-1)! Aledot 100) = \Mlog_ {10) (k!) 121 \ilinnlll, using the fact that $ (alog_ Valog_fa} c$ if and only if $b c$, have\n\\((k-2) Wcdot (k -1)! Viddot 100 (2 A.AAV InSimliffing we have\m\\[(k-2) Veddot (k-1)!
((k-2)! Wedot (x-1)!)
Walg_[a> (a) c \ilog_(a) (bc)$, we havelal\/VVlog..u
factors, we have
that $100 Weddot 5 Medot 4 2-2 Vledot 5-28. \nSince $k 2$, Sk (5boxed(5)$.
Final Answer: The final answer is $5$. hope is correct.
Problem:
If 43 and $4p+3q=138, what is $6$ equal to?
Solution:
We can solve this system of equations using the method of elimination. \nMultiplying the first equation by and the second equation gives us: 1e: subtract the first equation from the second equation wns7==28fnnDividitii
n$16p+122=52$$mmle
eliminate the
both sides
by 7 gives ua:/aabp=$//uuussstttttt equations gives us: (n44 3ns6sssnn11+111333 33 (ngn-3//x@(((bxxx((--11) Final Answer: The final answer is $-1$. hope it is correct.
this value of into either of the original
43

---

4-shot example for MetaMath Hard ICL:
Problem:
In right triangle SPQRS, we have Maaagl q = Vangle R$ and $PR 6) (2)qrt((2) What is the area of Wiltringll PQRS?
Solution:
Since lalangle Q = Mangle R$ we have $1 \triangee PQR$ is an isosceles right triangle. \nTherefore, SPQ - PR 6 6Vlsqrt(2)$ AnThe area of Siltriangle PQR$ is (1)frac(1){) Meddo PQ Weddot PR (\frac(1)22) Wedot 61/sqr((2) Viedot 6\\ sqrt(2) Wboxed(36)
Final Answer: final answer is $36$. hope it correct.
Problem:
The length of the longer side of rectangle SR$ is $10$ percent than the length side square $8.$ The length the shorter side rectangle $R$ $10$ percent less than the length of side of square $S.$ What is the ratio of the area of rectangle $R$ to the area of aquare $S?$ Express your answer as a common fraction.
Solution:
Let the side length of square $S$ be $x$. \nThen the longer side of rectangle $8$ is $1. 11.1x$ and the shorter side is 10.928. \nThe area of square SS$ is $x"2$ and the of rectangle $R$ is $(1.1x) (0.9x) 0.99x-2$. AnTherefore, the ratio of $1lfrac(0.99x 9x
area rectangle SR$ to area square \bboee(() (9r)c
Final Answer: The final answer is (100)) hope it is correct.
Problem:
Suppose that $5 varies inversely with [3] If $y=2$ when $2=1$, find the value of $2$ when $y=4$. Express your answer in simplest fractional form.
Solution:
Since $y-38 varies inversely with s\ (1legrt(3] (z)s, can write the equation $y (2)) where $k$ is a constant. Anile are given that Sy-2$ when $2=1$ so we can substitute these values into the equation to solve for $k$:\n$2 \n$2-311 sqrt (1)= cddot 1=ldo can use this value of $1$ to find har $z$ when (BB100//hhacccd/BBBBee///eEetEe- when $y-48, $z- (=x(ooodd(((rrrc(((111)))
\agrt (2)= (atrrggin)))a//aaaar((/11552 (frac(1)551)))) -3=)
Final Answer: The final answer is $1 (5)))a I hope it is correct.
Problem:
$d$ positive number such that when $109$ divided by $a$, the remainder is $4.$ Compute the sum of all possible two-digit values of sds.
Solution:
$109S divided by $d$ nThe prime factorization of $105$ is $3\\cdot5) \ddot7$. \nSince $d$ must be a factor of $105$, the possible values of $d$ are $1, 3, 5, 7, 15, 21, 35, 105$. \ nOut of these, only the digit values are $15, 21, 35$. AnThe sum of these values is
remainder is $4$, then $a$ divides $109-4=1055.
Final Answer: The final answer is $71$. hope it correct.
44

---

4-shot example for MetaMath Easy-to-Hard ICL:
Problem:
How many 4-letter words with at least one consonant can be constructed from the letters $AS, $B$, $C$, SD$, and $ES? (Note that SB$, $C$, and $D$ are consonants, any word is valid, not just English language words, and letters may used more than once.)
Solution:
There total of $5^4 625$ possible 4-letter words that be constructed
from given letters. AnTo count number words with no consonants,
can use the principle complementary counting. AnSince there are vowels ($A$ and $E$) and 3 consonants ($B$, SCS, and $D$), there are $2"4 = 16$ words with only vowels. AnThereorrr the number of words with at least one consonant is $625 16 (boxed(660988.
Final Answer: The final answer $609$. hope correct.
Problem:
Compute the integer Sk 28 for whichlal\//Nlo..t00 (k 2) + \llog_ (k 1)!
2 Wllog_(01 (10) A!
Solution:
Using the properties of logarithms,
rewrite the equation Wllog
using the fact that $ ((k-2)! Wedot (k-1)!) (alog_(a (bc)$, we haveln\V(VVlog.11
(alog[ \llog_
((k-2)! cdot (k-1)! Viedot 100) (1log_10) (k!)-2, W.J \Wnnnnaly,, using the fact that $ Walog_(a b \llog_ fa} c$ if and only if $b c$, we have\n\\((k-2) Weddot (k -1)! Mledot 100 -2. \ ISSimpiii, cdot 100 (k Viedor (k-1) Vledo !. Annannaclllnn out the Wedot (k-1) Wedot (k-2) VnFacttrin, we find Meddot 5 Modot 4 = 2 Vlodot 5-28. \nSince $k > 28, we see that
have\m\\[(k-2) Aledot (k-1)! \
factors, we haveln\\(100 that $100 $k - (1boxed(6)$.
Final Answer: The final
is $5$. hope is correct.
Problem:
If $n$ is $1$ less than multiple of $50$, then what is the remainder when 22 +3$ is divided by $50$?
Solution:
Since $n$ is $1$ less than a multiple of $508, we can write $n-50k-15 1$ for some integer $k$. AnThen 23 (50k-1) (24 2 250k (0000-0)3-22000000 22-9000 WV so the remainder when $n is divided by $50$ is $1 (2)8.
Final Anawer: The final
is $2$. I hope is correct.
Problem:
If $3p+4q=8$ and $4p+3q-13$, what is $4$ equal to?
Solution:
can solve this system equations using the method of elimination. www.ltpllyn the first equation by 3 and the second equation by 4 gives us: us:\n$9p 12 1816p+124-52$Anile eliminate the q term: (0)
subtract the first equation from the second equation to 24$Snmm==8S/mDii
both sides into either the original 10o--xx Final Answer: The final answer is $-1$. I hope it is correct.
by gives us: husuuup-///uuussstttiiig this value equations gives us: (B84)))444)) 13
45

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
They are supported by the experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:[Yess Yes]
Justification:
We discuss this in the section of "Conclusion Limitations".
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
46

---

Justification:
We have no theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have open-sourced the code and released the model.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
47

---

Answer: [Yes]
Justification:
See our supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
wwwiisssssoolooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. wttpccccpppccccceeccec.eccccccct tttpa///ggiieeeeeeuuuuuiiiiii
/CodeSubmiiieeee ssinnn onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We do that in our experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have error bars in our curves.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
48

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes] We open-sourced the code.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https: : /ttp///eerrpp.eeeee//ee cc ht.ppubiccc/tthicccsielelllaaaaa
didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, We discuss it in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
49

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We don't 't pre-train language models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, we do the proper citations.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
50

---

Answer: [Yes]
Justification:
Yes, we open-sourced the code.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing is used.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects are involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
51


Reviews:
[{"review_id": "Hgqn0zYYJx", "reviewer": "Reviewer_3nAE", "summary": "- This paper studies the question of how a system can be improved when the performance of a system has surpassed human performance on a task.\n- As a testbed, the paper uses problems from the MATH dataset which have been sorted into 5 levels by difficulty.\n- The authors first train process-supervised reward models on level 1-3 problems on the MATH dataset.\n- They then use the reward models learned from easy problems to supervise policies on hard (levels 4-5) problems on MATH.\n- They find that the reward models substantially improve the performance of the policy on hard tasks when used as either reward models in RL or as re-ranking models during inference, despite being only trained on easy problems.", "questions": "- I found nearly all the tables in the paper hard to read / extract information from.\n- It isn't clearly indicated (or at least I couldn't tell) how many times each model was trained, were there multiple runs, etc. I see what looks like error bars on some of the plots, but no explanation of these is given.", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "- The extensive comparison of different training methods (ReST, DPO, PPO) is useful even outside of the context of the research question.\n- The methodological decision to compare both re-ranking and RL on hard problems is very sound and makes me more confident in the conclusion of the paper.\n- Although there have been several recent papers on easy-to-hard generalization that establish that easy-to-hard generalization is possible, I think the experimental setup here takes a different angle by showing that the *reward models* learned on easy tasks transfer to harder tasks *in multiple ways* and specifically that evaluators generalize better than generators to hard tasks.", "weaknesses": "- The differences between the comparison categories in many cases are small, on the order of 1-2 percentage points. I also did not see any error bars or variance estimates (except maybe in Figure 4, though this is unclear). This makes assessment of the scientific validity of the results a bit more challenging.\n- The conclusions were demonstrated on only two tasks and both tasks were formal reasoning tasks. Would the conclusions transfer to natural language reasoning tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review and the positive feedback on our paper. We are pleased that you found our proposals interesting and our experiments thorough. It's particularly encouraging to hear that you recognize the novelty of our easy-to-hard generalization approach and our results useful even outside of the context of the research question. We address your questions below.\n\n**Weaknesses**\n\n**W1. The differences between the comparison categories in many cases are small, on the order of 1-2 percentage points. I also did not see any error bars or variance estimates (except maybe in Figure 4, though this is unclear).**\n\nWe are only reporting baseline results in Table 1. The main claim of the paper, that the easy-to-hard generalization of evaluators helps generators, is supported by results in Figure 3, Figure 4, Table 2, and Table 3. The accuracy improvements from the reward model are often significant (e.g., comparing weighted voting to majority voting or comparing RL models to SFT models).\nWe agree that observing the variance of the error is important. Figures 3 and 4 represent the curves for different combinations of random sampling trials, where the solid curves for the performance average and the shaded areas for the error ranges (performance variance). \n\n**W2. The conclusions were demonstrated on only two tasks and both tasks were formal reasoning tasks. Would the conclusions transfer to natural language reasoning tasks?**\n\nIn our easy-to-hard framework, we haven't specialized any assumptions in MATH or Code related to the problem (easy-to-hard generalization) we're studying, so our method should be transferable to other tasks in principle. We left the verification of our method in other domains as future work.\n\n**Questions**\n\n**Q1. I found nearly all the tables in the paper hard to read / extract information from.**\n\nWe have added an explanation about the performance variance for each curve and will include the revised explanation in the revised version of our paper. \n\n**Q2. It isn't clearly indicated (or at least I couldn't tell) how many times each model was trained, were there multiple runs, etc. I see what looks like error bars on some of the plots, but no explanation of these is given.**\n\nFor the training times, most of the training runs in the paper were only conducted once due to resource constraints, and also because we've observed that the performance was quite stable in our preliminary studies. For the plots, the error bars analysis is indeed included in each curve plot such as Figure 3, 4. We will add more descriptions to the plot in the paper.\n\nSpecifically, for all the problems, we sampled 2048 solutions. Taking N=32 on the x-axis as an example, we randomly select 32 solutions from the 2048 rollout samples, record the consensus score (majority voting, weighted voting, and BoN), and repeat this process 400 times. The solid curve represents the mean accuracy of these 400 sampled combinations of solutions, and the shaded margin of each curve represents the performance variance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "70GpY2jKFb", "reviewer": "Reviewer_dzZx", "summary": "In this paper, the authors propose *easy-to-hard* generalization, which is to train a reward model on simpler tasks and then use it to evaluate the solutions for more difficult tasks. They have conducted in-depth studies on MATH, and also demonstrated effectiveness on the coding benchmark APPS. This work serves as a nice proof-of-concept study of \"evaluation is easier than generation\", and suggests a new way toward scalable alignment without human supervision.", "questions": "The idea of \"evaluation is easier than generation\" is appealing, and it seems that the idea draws inspiration from the assumption that P < NP. However, for some tasks it seems that evaluation shares a similar level of difficulty with generation. \n\nFor example, let there be N integers: A_1, A_2, ..., A_N, and the task is to multiply them altogether: A_1 * A_2 * ... * A_N = ? . While doing multiplication is no easy task for LLMs, it seems that evaluating whether some potential answer is the result of A_1 * A_2 * ... * A_N is as difficult as generating the answer of the multiplication, since in either way, one should do it a serial manner (for i in 1,2..N). \n\nAny thoughts/evidence on this? \n\nAnd broadly speaking, are there cases when evaluation is as difficult as generation, or evaluation is even harder than generation? It would be great if the authors could shed light on this.", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The idea of *easy-to-hard* generalization is inspiring, and it's nice to see that the idea works out on the challenging MATH dataset.\n- The authors have conducted sufficient experiments and detailed analysis, which makes the paper quite worthy to be read and referred to.", "weaknesses": "- The definition of \"easy\" v.s. \"hard\" is not very clear, and it seems that the proof-of-concept experiments rely on the structure of the benchmarks, as the MATH dataset has 5 divisions of difficulty and APPS has 3. However, when there are not explicit tags of difficulty in a benchmark, what is the authors' definition for \"easy\" and \"hard\" in terms of each data sample?\n\n- Following the above weakness, a natural question for the authors is to demonstrate the practical value of this work when compared with recent work that scales synthetic data for MATH (e.g., [1]). Can the massive synthetic data be viewed as a mixture of easy and hard problems + the corresponding solutions? If we treat such mixed data as the easy part and train RMs on it, can we expect similar easy-to-hard generalization? Which kinds of ability would be \"unlocked\"?\n\n[1] Improve Mathematical Reasoning in Language Models by Automated Process Supervision. https://arxiv.org/abs/2406.06592\n\n- According to the figures shown in the paper, the \"Weighted Voting w/ RM\" method always  yields better performance as N increases. By comparison, \"Best-of-N w/ RM\" and \"Majority Voting\" can plateau or even become worse when N increases from 512 to 1024. Does the weighted voting with RM guarantee the increasing performance, or is it just by accident? Would be great if there are formal explanations to this.\n\n- While scaling the sampling times N has seen improvements, are there certain problems whose correct solutions are never sampled by the LLM when letting N be very large?\n\n- More case studies would be beneficial to provide a more intuitive understanding of how the RM on easy problems generalize to hard ones.\n\n- The authors have conducted in-depth analysis about the comparisons between PRM, ORM, and OPRM on MATH. While I appreciate the experiments, I wonder what the effect of PRM/ORM w.r.t. easy-to-hard generalization is. For example, is it true that we should always adopt PRM when it is possible to get process-based supervision (Lines 219~221)? If this is true, would the results on APPS be better when we could access process-based supervision for code (for example, treat the interleaved comments in a code snippet as that)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive feedback on our paper. We are glad that you appreciate the inspiring easy-to-hard generalization problem we’re working on, and thank you for acknowledging the thoroughness of our experiments. We address your questions below.\n\n**Weaknesses**\n\n**W1. The definition of \"easy\" vs \"hard\".** \n\nThe goal of this paper is not to provide a specific way to split data into easy and hard portions for any arbitrary domain but to show **how we can enable generalization on hard tasks by only supervising the model on easy tasks**.  Specifically, in the settings for scalable oversight (aligning superhuman AI), we can treat all the tasks that humans can annotate as easy, and **all the tasks that humans cannot supervise as hard**. This makes a clear definition of “easy” vs “hard” for real-world tasks.\nIt is only for research purposes that the experimental datasets used in the paper, as the reviewer noticed, have a clear division of the difficulty levels. This division helps verify the idea of easy-to-hard generalization (Lines 30-34), where the model is trained only on **easy data (simulating the tasks that humans can label)**, and then generalizing to **hard data (simulating the tasks that humans cannot handle)**.\n\n**W2. Can the massive synthetic data[1] be viewed as a mixture of easy and hard problems?** \n\nThe easy-to-hard generalization framework is also applicable to data generation methods such as [1]: we can treat all generated question-solution pairs that have been verified by ground-truth as easy problems, while those not verified by ground-truth (or open questions) are considered as hard problems. This is because the easy-to-hard generalization framework does not need to know the ground-truth solutions for the hard problems.\nWe leave combining our framework and other methods as future work.\n\n**W3. Does the weighted voting with RM guarantee the increasing performance?**\n\nIn our experiments with 7b-34b models, we found that weighted voting with RM is always better than BoN or majority voting. Here’re our insights:\n\n- why weighted voting is better than majority voting? Theorems 1 & 2 in [2] show the convergence of the accuracy with an increasing number of samples. Specifically, the limit is determined by the likelihood of generating the correct answers through all possible reasoning paths (and the likelihood should be viewed as a weighted sum for Weighted Majority Voting). As long as the reward model is “better than random (informally)”, i.e., assigning higher rewards to correct solutions on average, the accuracy limit of Weighted Majority Voting is higher than that of Majority Voting.\n\n- why weighted voting is better than best-of-n? [3] shows that the scaling curve of BoN is\n$R_{bon}(d) = d(\\alpha - \\beta \\cdot d)$, where $d$ is the square root of the KL divergence of the policy and $d = \\sqrt{log N - \\frac{N - 1}{N}}$.\nThis means the performance of BoN will ultimately become worse when reward over-optimization happens (i.e., $d > \\frac{\\alpha}{2\\beta}$).\n\n**W4. Are there certain problems whose correct solutions are never sampled?**\n\nWe conducted additional experiments on Pass@N and reported the results in the uploaded PDF. We found there are still some problems that cannot sample a correct answer. More specifically, **Pass@N is highly correlated with difficulty**. As illustrated in Figure 1 in the uploaded PDF, with a larger number of samples, the Pass@N for Level 1 problems is nearly saturated. However, for Level 5 problems, there are still many instances where a correct solution is not sampled.\n\n**W5. More case studies.**\n\nWe have included more case studies in Figures 3, 4 of the uploaded PDF. \nEvaluator can help generalize to harder ones in the following ways:\n- The evaluator can **help identify and reduce the confidence of hallucinations caused by misleading information in problems**. As demonstrated in Case Study 1, the solution selected by majority voting with an answer of 36 is misled by the different units of measurement in the problem (2.5 hours and 90 seconds), resulting in an incorrect solution. Then, the ORPM model successfully gives this solution a low score.\n- The evaluator can **assist in reducing the confidence of solutions that misuse mathematical theorems**. In Case Study 2, the majority solution incorrectly applies the theorem \"the sum of the exterior angles of a polygon is 360°\", leading to erroneous reasoning, and low confidence by the ORPM model.\n\n\n**W6. what the effect of PRM/ORM w.r.t. easy-to-hard generalization is. and would the results on APPS be better with accessing process supervision?**\n\nWe compared PRM, ORM, and OPRM in Appendix G, where we found PRMs and ORMs perform similarly, with PRMs slightly outperforming ORMs on hard tasks. However the OPRMs that are trained on the mixed data of PRMs and ORMs significantly outperformed both of them. Hence, we believe we should adopt PRMs and ORMs together for the OPRM, which can complement ORM and PRM.\n\nWe believe the results on APPS would be better if we could obtain a human-annotated (or synthetic) PRM dataset for code. However, that’s out of the scope of our paper.\n\n**Questions**\n\n**Q1. Are there cases when evaluation is even harder than generation?**\n\nNot all evaluations are easier than generation. As shown in [4], LLMs might generate content that exceeds their own understanding based on the given context. An LLM might create a **highly coherent and contextually linked story**, but when questioned about the **logical connections** within the story, it may fail to make accurate judgments.\n\n[1] Improve Mathematical Reasoning in Language Models by Automated Process Supervision, arXiv:2406.06592.\n\n[2] An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models, arXiv:2408.00724.\n\n[3] Scaling laws for reward model overoptimization, ICML 2023.\n\n[4] The Generative AI Paradox: “What It Can Create, It May Not Understand”, ICLR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bNkXps3X3r", "reviewer": "Reviewer_VNqz", "summary": "This paper addresses the issue that humans cannot always provide helpful demonstrations or supervision on tasks beyond their expertise. Based on the observation that evaluation is easier than generation, the authors propose \"easy-to-hard generalization,\" training a verifier on easy tasks and leveraging its generalization ability to supervise the generator on hard tasks. Experimenting mainly in math reasoning tasks, they demonstrate that easy-to-hard generalization from evaluators can enable easy-to-hard generalization from generators.", "questions": "1. What is the difference between \"easy-to-hard\" and \"weak-to-strong\"?\n2. In Table 1, the Full ICL setting performs worse than the Easy-To-Hard ICL setting. How do you explain this?\n3. In Table 1, the Easy-To-Hard SFT setting performs slightly worse than the Full SFT setting, which is expected. Do you think the evaluation of the Easy-To-Hard setting is influenced more by the format of problems or by the true grasp of the principles of solving hard tasks? Are there any better evaluation methods? This is an open question, and I would appreciate your thoughts on it.", "limitations": "The paper may lack innovation regarding the concept of \"easy-to-hard generalization\" and the method to achieve it. Many experimental conclusions appear to merely reproduce or support existing works and lack depth.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper introduces the Outcome & Process Reward Model (OPRM), which harnesses the complementary strengths of ORMs and PRMs. Experiments show that OPRM is more efficient. It also conducts a systematic experimental setup, testing various generators and evaluators, as well as optimization algorithms (BoN, Weighted Voting, and RL). It provides numerous experimental analyses in scalable alignment.", "weaknesses": "While the paper is well-written and presents a solid analysis, several weaknesses need addressing:\n\n1. What is the difference between \"easy-to-hard\" and \"weak-to-strong\"? You state that human supervision is available but not reliable in the weak-to-strong setting, but in the [OpenAI paper](https://cdn.openai.com/papers/weak-to-strong-generalization.pdf) it says, \"We find that when we naively finetune strong pretrained models on labels generated by a weak model, they consistently perform better than their weak supervisors, a phenomenon we call weak-to-strong generalization.\" In that study, they finetune GPT-4 with a GPT-2-level supervisor. Can a GPT-2-level supervisor be seen as a verifier on easy tasks? The novelty should be considered.\n2. In Table 1, the Full ICL setting performs worse than the Easy-To-Hard ICL setting. How do you explain this? The intuition is that sampling from both easy and hard exemplars may help solve hard problems more effectively than just demonstrating easy exemplars. Although the quality of PRM800K is lower than MetaMATH, your explanation for why the Full ICL setting is worse than the Easy-To-Hard ICL setting is insufficient (line 198).\n3. Can you report the average accuracy of verifying each step in a solution when you evaluate the evaluators? Section 3.5 does not explain why this is not reported.\n4. When citing images and sections, consider adding automatic jumps for easier navigation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review and constructive feedback. We appreciate your recognition of the paper’s clarity, the proposed OPRM methods, and the solid experiments analysis. We address your concerns and questions below.\n\n**Weaknesses & Questions**\n\n**W1&Q1. What is the difference between \"easy-to-hard\" and \"weak-to-strong\"? & Can a GPT-2-level supervisor be seen as a verifier on easy tasks?**\n\n“Easy-to-Hard” (E2H) studies how the model generalizes when trained by **clean** labels on **easy** tasks, while “Weak-to-Strong” (W2S) studies how the model generalizes when trained by **noisy** labels on **all available** tasks.\n\nWe listed a few differences between E2H and W2S as clarification, which we’ll add to the revised version of the paper:\n\n- W2S uses weak teacher’s prediction as the supervision, where no human annotation is used for the strong model. E2H uses human annotations, but they are limited to easy problems.\n\n- W2S studies classification or short-answer prediction problems, while E2H studies generative (or long-CoT reasoning) problems.\n\n- The two models used in W2S are the weak teacher and the strong student, which are models of different sizes but trained on the same task. The two models used in E2H are the generator and the evaluator, which can be of the same size, but trained on different tasks (as policy model or as reward model).\n\n- The research question in the W2S analogy is: Can we have the student model outperforming the teacher model? The research question in the E2H analogy is the following: **Can we produce a system (LLM+evaluator) trained on human annotations on easier tasks only but can perform well on harder tasks for which we do not have any human annotations?**\n\nFinally, both E2H and W2S are analogies of scalable oversight, which studies how we can align superhuman AI models.\n\n> “Can a GPT-2-level supervisor be seen as a verifier on easy tasks?”\n\nWe believe this is not a well-defined question. We would like to clarify that:1) W2S only studies classification problems, 2) E2H studies generalization of verifiers on hard tasks, not easy tasks.\n\n>  \"You state that human supervision is available but not reliable in the weak-to-strong setting, but in the OpenAI paper it says, ..., they finetune GPT-4 with a GPT-2-level supervisor.\"\n\nThe GPT-2-level supervisor in the W2S paper is used to simulate the noisy human supervision on tasks that are too difficult for humans to reliably evaluate.\n\n**W2&Q2. Explanation for why the Full ICL setting is worse than the Easy-To-Hard ICL setting is insufficient (line 198).**\n\nOne of our hypotheses is that ICL is mainly performing format learning, so **exemplars of easy problems might be simpler for the model to understand and follow**, whereas the format of difficult problems may be more challenging for the model to grasp. \nAnother hypothesis is that **the level of noise in hard data** is likely higher than in easy data. This is like how humans are more prone to making mistakes when annotating difficult questions (inconsistencies in reasoning solutions can also be considered a form of noise), making it difficult for the model to effectively extract knowledge from hard ICL data.\nThere is also research [1] suggesting that knowledge is stored in data in a hardness-invariant way. Therefore, selecting hard data for ICL does not necessarily lead to performance improvement.\n\n**W3. Can you report the average accuracy of verifying each step in a solution when you evaluate the evaluators?**\n\nWe conducted additional experiments using the PRM800K-test data, which includes correctness annotations for each step, to test our model's ability to distinguish correct reasoning steps. We randomly selected a portion of PRM800K-test data to balance positive and negative samples. The accuracy of the reasoning steps for the three models is as follows:\n| Reward Model         | Step ACC (%) | Outcome ACC (%) |\n|----------------------|--------------|-------------|\n| ORM-PRM800K-7B       | 64.3         | 71.7      |\n| PRM-PRM800K-7B       | **80.4**         | 63.5      |\n| OPRM-PRM800K-7B      | 79.8        |**74.4**     |\n\nThis table demonstrates the effectiveness of our trained PRM, showing that PRM has a significantly greater ability to distinguish steps compared to ORM. Additionally, in Figure 2 (Left) of the uploaded PDF, we present the Step ROC curves of three models, where PRM and OPRM exhibit better step discrimination abilities compared to ORM.\nHowever, it is important to note that a stronger ability to distinguish steps does not necessarily indicate that the evaluator is more helpful for generation. We then also present the Outcome ROC curves of three models on discriminating the final outcome. We collect data generated on MATH500 test set from our 7B policy model. According to the final outcome and groundtruth, we label each data and select a positive-negative balanced set to plot the Outcome ROC curves, where OPRM exhibits better outcome discrimination abilities compared to ORM and PRM. The above table also shows the effectiveness of OPRM on Outcome discrimination ability.\n\n**W4. Citing images and sections**\n\nThank you very much for your recommendation. We will correct this issue in the revised version.\n\n**Additional Questions**\n\n**Q3. Do you think the evaluation of the Easy-To-Hard setting is influenced more by the format? Are there any better evaluation methods?**\n\nIn this paper, we have **controlled all data to have a consistent format**. Therefore, the format will not influence the evaluation results and conclusions presented in the article. Our format is the same for all levels. We released all the data, which are ready for the reviewer to check. \nWe also believe accuracy with (greedy, majority voting, best-of-n, and weighted voting) are comprehensive enough to provide an evaluation of the mathematical reasoning tasks. \n\n\n[1] The Unreasonable Effectiveness of Easy Training Data for Hard Tasks, arXiv:2401.0675.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EvDfdPV0Er", "reviewer": "Reviewer_bhN7", "summary": "This paper attempts to address a forward-looking and challenging question: \"Can we limit human supervision to easier tasks, yet enable the model to excel in harder tasks?\" (referred to as Easy-to-Hard Generalization). Based on the observation that \"evaluation is easier than generation\", the authors propose to train a verifier on easy tasks, then make use of its generalization ability to supervise the generator on hard tasks. To harness the complementary strengths of outcome reward models (ORMs) and process reward models (PRMs), the authors introduce the Outcome & Process Reward Model (OPRM), so as to better utilize  Easy-to-Hard evaluators. Through extensive experiments, the authors verify that easier-level evaluators maintain their effectiveness on harder tasks. Further experiments explore the use of the easy-to-hard evaluator as a reward model in reinforcement learning and underscore the potential of using easy-to-hard evaluation to improve easy-to-hard generators.", "questions": "See weaknesses. I will consider raising my score if the authors can address my concerns.", "limitations": "I think the authors have addressed their limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The problem this paper aims to tackle is promising and challenging.\n* The proposed approach is intuitive and has strong motivation.\n* This paper is well-written and presents clear ideas.\n* Through extensive experiment,  the authors validate that the proposed method of \"training a verifier on simple tasks, then leveraging its generalization capability to guide the generator on complex tasks\" is workable.", "weaknesses": "* **The definition of difficult problems could be further refined.** The scenario considered in this paper is how to enhance the model's ability to perform difficult reasoning tasks when humans cannot provide effective supervisory signals. Given that the model's capabilities vary across the seven subsets in the MATH dataset on which the experiments are based, the definition of difficult problems may be biased. For instance, the LLM performs significantly better on the Algebra subset than on Geometry and Number Theory. Therefore, the improvement at levels 4-5 may primarily result from the performance enhancement in Algebra as it inherently has some evaluation ability for Algebra level 4-5 problems). Thus, it would be better to display the performance of the proposed method in different subsets of MATH, as well as the performance at levels 4 and 5 under different subsets, to prove that it can help the model solve \"truly difficult\" problems (such as Number Theory level 5).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review and positive feedback. We're glad to hear that you appreciate the easy-to-hard generalization problem we’re going on. Your recognition of our proposal and motivations is encouraging. We address your questions below.\n\n\n**Weaknesses**\n\n**W1 (a). The definition of difficult problems could be further refined**\n\nThe goal of this paper is not to provide a specific way to split data into easy and hard portions for any arbitrary domain **but to show how we can enable generalization on hard tasks by only supervising the model on easy tasks**. Specifically, in the settings for scalable oversight (aligning superhuman AI), we can treat **all the tasks that humans can annotate as easy, and all the tasks that humans cannot supervise as hard**. This makes a clear definition of “easy” vs “hard” for real-world tasks. It is only for research purposes that the experimental datasets used in the paper, as the reviewer noticed, have a clear division of the difficulty levels. This division helps verify the idea of easy-to-hard generalization (Lines 30-34), where the model is trained only on **easy data (simulating the tasks that humans can label)**, and then generalizing to **hard data (simulating the tasks that humans cannot handle)**.\n\n**W1 (b). For instance, the LLM performs significantly better on the Algebra subset than on Geometry and Number Theory. Therefore, the improvement at levels 4-5 may primarily result from the performance enhancement in Algebra as it inherently has some evaluation ability for Algebra level 4-5 problems). Thus, it would be better to display the performance of the proposed method in different subsets of MATH, as well as the performance at levels 4 and 5 under different subsets, to prove that it can help the model solve \"truly difficult\" problems (such as Number Theory level 5).**\n\nThanks for the suggestions. We indeed have conducted the fine-grained analysis of OPRMs’ re-ranking improvements divided by Level and Math Category in Figure 19 and Figure 20 in Appendix N, where Number Theory and Geometry are somewhat more difficult than Algebra. However, we can see in Figure 20 that **for almost all categories, OPRMs’ re-ranking can bring more than a 4% improvement**.\n\nBesides, we have added Figure 2 (Right) in the uploaded PDF to verify the effect under the level 4-5 subset in number theory and geometry. We can see that in this hard subset, **OPRM can bring nearly a 10% improvement**, further demonstrating the feasibility of the easy-to-hard approach and the superiority of OPRM. We will include more experiments on the hard parts of different category subsets in the revision paper. Many thanks for your suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers and AC,

Thank you all for your time and effort in reviewing our paper. We are grateful to 3nAE, dzZx, and bhN7 for recognizing **the adequacy and novelty of our experiments and motivations** and acknowledging **the importance of the problem we are exploring, easy-to-hard generalization**. We also thank VNqz and bhN7 for recognizing the intuition behind our proposed OPRM method. 

Our contributions are well-recognized and can be summarized as:
- We show the potential of easy-to-hard generalization, where models can be guided to **solve complex problems without direct human supervision on these harder tasks**.

- We demonstrate that the easy-to-hard generalization in **evaluator models can effectively guide the generalization of the policy model on challenging tasks**. This underscores the effectiveness of re-ranking strategies and reinforcement learning in leveraging evaluators to achieve performance gains on challenging tasks.

We have added several figures in the uploaded PDF to aid readers in understanding our paper. These figures will also be included in our revised paper:

- Figure 1: The Pass@N curve shows its high correlation with difficulty.
- Figure 2 (Left): The Step ROC Curve and Outcome ROC Curve.
- Figure 2 (Right): The performance of ORPM on Geometry Level 4-5 Problems and Number Theory Level 4-5 Problems.
- Figures 3 & 4: Case studies demonstrating how the evaluator can assist in solving hard mathematical questions.

We sincerely appreciate all the efforts from the reviewers and ACs put into improving our paper. We have responded to every raised concern and hope our response can address them. 

Thanks again for all the effort and time. 

Best, 

Authors

Meta-review: Summary: 
The paper addresses an interesting problem of whether, when training an LLM model, we can limit the human supervision to “easier” tasks yet enable the model to generalize well to “harder” tasks where human supervision is unavailable. This problem setting is motivated by scenarios where humans cannot always provide helpful and additionally informative supervision on tasks beyond their capabilities. Based on the observation that, in general, evaluation is easier than generalization, the authors propose to train a verifier on easy tasks and leverage its generalization ability to supervise the generator on more complex tasks. Exploiting the complementarity of ORMs and PRMs, the authors propose a novel framework called OPRMs for the problem. Extensive experimental validation on two sets of data (math and coding) backs the author’s claims on the efficacy of their proposed approach.  

Strengths: 
* The impact of the problem addressed in the paper is quite significant. 
* The techniques proposed in the paper are quite intuitive and are very well-motivated
* The experimental evaluation is quite thorough, further building confidence in the techniques proposed 
* The paper is very well written, well structured, and easy to understand 

Weakness: 
While the reviewers raised some minor weaknesses associated with the paper, the authors' rebuttal was sufficient to address them, prompting multiple reviewers to raise their scores. Some of the weaknesses raised were: 
* Imprecise definition of what is an easy problem and what’s a hard problem 
* The efficacy of the ideas proposed is validated on only two datasets. Will these ideas generalize to other tasks?  

Overall Recommendation: 
Overall, this is a high-quality paper addressing an exciting and impactful problem using a novel technique that is very well-motivated. The paper is well-written and easy to understand. While the reviewers had a few objections to the initial version of the paper, most (if not all) of those objections were appropriately addressed in the rebuttal. The unanimous vote is to accept this paper at NeurIPS 2024.
Decision: Accept (poster)