Please summarize the following article: 
The reasoning abilities of large language models (LLMs) are rapidly improving as measured by their performance
on numerous math, science and code benchmarks (Cobbe et al., 2021; Hendrycks et al., 2021b; Sawada et al.,
2023; Liang et al., 2022; Srivastava et al., 2022; Rein et al., 2023; Mialon et al., 2023; Chollet, 2019; Mishra
et al., 2022; Hendrycks et al., 2021a; Austin et al., 2021; Patel et al., 2021; Gao et al., 2021). Simultaneously,
Reinforcement Learning from Human Feedback (RLHF) (Bai et al., 2022; Ziegler et al., 2019; Ouyang et al.,
2022) and instruction fine-tuning (Wei et al., 2021; Mishra et al., 2021) have made significant progress in
aligning LLMs with human preferences. Improvements in model instructability have further increased apparent
model capability by making complex behaviors more accessible via instruction prompting. This has led to a
number of increasingly sophisticated prompting strategies augmenting LLM reasoning capabilities such as
Chain-of-Thought (Wei et al., 2022) or Tree-of-Thoughts (Yao et al., 2023).
Previous work in reinforcement learning (RL) such as AlphaGo (Silver et al., 2017), AlphaStar (Vinyals et al.,
2019), and OpenAI Dota 2 (Berner et al., 2019) demonstrate that RL techniques can be used to train neural
networks capable of sophisticated planning and reasoning in game environments. Cicero (Bakhtin et al., 2022)
in particular succeeds in combining an RL trained planning agent with a dialogue fine-tuned LLM to achieve
nearly super-human performance in the board game Diplomacy. Given these previous successes and the
inherent interactive nature of problem solving, applying RL to LLM reasoning seems a natural next step. In
this paper, we study how ideas from RL can be used to improve the reasoning capabilities of LLMs across a
variety of reward schemes and model initializations.
We begin by comparing the performance of different RL algorithms on reasoning tasks τ defined as a
distribution of question answer tuples (Q, A). The task τ can be extended to define a Markov Decision Process
(MDP) 4-tuple (S, A, Pa, Ra) where tokens serve as both actions and accumulated state with deterministic
1
arXiv:2403.04642v1 [cs.LG] 7 Mar 2024
dynamics. By default we use a sparse reward of +1 if the final answer is correct but also experiment with
dense rewards matching intermediate steps in a reference solution and rewards synthetically generated using a
reward model. We evaluate models with 7B and 13B parameters both starting from supervised fine-tuned
(SFT) checkpoints and pre-trained checkpoints. We report four metrics assessing model performance on a
task specific test set: 1) maj@1 score computed by greedily sampling once per question, 2) maj@96 score
computed by sampling K = 96 times per question and uniformly voting on the final answer, 3) rerank@96
score computed by sampling K = 96 times and choosing the final answer using an Outcome-Based Reward
Model (ORM), and 4) pass@96 score computed by sampling the model K = 96 times and taking the best
result according to the ground truth answer.
We find that overall the simplest method, Expert Iteration (EI) (Anthony et al., 2017), performs best across
all metrics for most reward setups and model initializations. Surprisingly, EI is nearly as sample efficient as
more sophisticated algorithms like Proximal Policy Optimization (PPO), both requiring only a few thousand
samples to converge even when initialized from a pretrained checkpoint. We also observe the gap between
pretrained model performance and SFT model performance significantly shrinks (< 10% gap on GSM8K)
after RL fine-tuning, with larger models having a smaller gap. Additionally, previous work identified a tradeoff
between test time maj@1 performance and pass@96 performance during supervised fine-tuning (Cobbe et al.,
2021), with continued training increasing maj@1 score at the expense of pass@96 score. We identify the
limited diversity of the dataset as a core reason for this. We show that RL fine-tuning can improve both
metrics simultaneously due to the fact that RL generates its own data during training, resulting in a more
diverse set of examples to learn from.
We then discuss why EI and return conditioned RL are competitive with PPO, suggesting two principal
factors. Firstly, the reasoning tasks we consider have entirely deterministic dynamics: a setting in which
direct behavior cloning and return conditioned RL is known to do well (Brandfonbrener et al., 2022). In
contrast, PPO often succeeds in environments with a high degree of stochasticity (Bhargava et al., 2023).
Second, we identify a lack of sophisticated exploration carried out by models during RL fine-tuning. This
limitation significantly impacts any performance or sample complexity advantages PPO may have when
fine-tuning the pretrained model. We come to this conclusion from a number of observations, noting in
particular quickly saturating pass@96 scores early in RL training. We conclude with a discussion of the
impacts of our observations on RLHF and the future of LLM fine-tuning via RL.
In summary we make the following contributions:
• A comprehensive study of PPO fine-tuning of LLMs on reasoning tasks using different types of rewards,
model sizes and initializations.
• A comparison to expert iteration and return-conditioned RL from which we find expert iteration reliably
attains the best performance and competitive sample complexity across the board.
• A discussion of the implications of our findings for RLHF and the future of RL fine-tuning for LLMs,
identifying exploration as a major limiting factor.
2 Related Work
LLM Reasoning: State-of-the-art large language models (OpenAI, 2023; Touvron et al., 2023; Bai et al., 2022;
Chowdhery et al., 2022) demonstrate increasingly impressive abilties on hard reasoning tasks as studied by
a wide range of math, science, and code benchmarks (Cobbe et al., 2021; Hendrycks et al., 2021b; Sawada
et al., 2023; Liang et al., 2022; Srivastava et al., 2022; Rein et al., 2023; Mialon et al., 2023; Chollet, 2019;
Mishra et al., 2022; Hendrycks et al., 2021a; Austin et al., 2021; Patel et al., 2021; Gao et al., 2021). Chain
of thought (CoT) (Wei et al., 2022) and related techniques (Chen et al., 2022; Yao et al., 2023; Besta et al.,
2023) have emerged as dominant methods siginficantly boosting LLM performance on these types of tasks.
CoT methods allow LLMs to defer giving their final answer by first generating a ”chain of thought” involving
intermediate computations needed to correctly solve the problem.
Another line of work combines base LLM reasoning capabilities with planning and search algorithms to further
boost performance on a wide range of tasks (Yao et al., 2023; Besta et al., 2023; Ye et al., 2022; Yao et al.,
2
2022; Dohan et al., 2022). Tree of thought (Yao et al., 2023) for example combines LLMs with a breadth first
search algorithm, relying on the LLM to both propose actions and evaluate state. Other works combine LLMs
with tools (Schick et al., 2023; Qin et al., 2023; Zhou et al., 2023a) further boosting reasoning capability.
Combining GPT-4 with a python code interpreter for generation and self-verification achieves an impressive
84% on the hard MATH benchmark (Hendrycks et al., 2021a; Zhou et al., 2023a).
Other works focus on LLMs for mathematical reasoning in natural language (Cobbe et al., 2021; Lewkowycz
et al., 2022; Azerbayev et al., 2023; Lightman et al., 2023; Patel et al., 2021; Zhu et al., 2023; Rafailov
et al., 2023). Particularly relevant to our study is Cobbe et al. (2021) which fine-tunes GPT-3 on supervised
math word problem (MWP) reasoning traces. In addition they train solution verifiers called Outcome Based
Reward Models (ORMs) which predict the probability of correctly solving a question Q giving a prefix of
intermediate steps Pi = (S1, ..., Si) i.e. p(is correct(A)|Q, Pi) where A is a solution with prefix Pi
. Process
based reward models (PRMs) (Uesato et al., 2022; Lightman et al., 2023) can also be trained to instead look
at the step-level accuracy of solutions. More recent work (Luo et al., 2023) utlizies a PRM distilled from
GPT-4 feedback as a reward signal during PPO.
RL for LLM fine-tuning: Reinforcement Learning from Human Feedback (RLHF) is perhaps the most
well-known application of RL techniques for fine-tuning LLMs. RLHF (Christiano et al., 2017; Ziegler et al.,
2019; Stiennon et al., 2020; Ouyang et al., 2022; Bai et al., 2022; Glaese et al., 2022; Peng et al., 2021;
Ramamurthy et al., 2022) most often works by training a reward model to capture human preferences over
a task τ . The reward model is then used to score LLM responses to prompts from the task after which
policy improvement is performed. PPO is most often used (Ouyang et al., 2022; Bai et al., 2022) but several
recent works including ReST (Gulcehre et al., 2023), Reward-Ranked Fine-tuning (Dong et al., 2023), and
AlpacaFarm (Dubois et al., 2023) all demonstrate simply fine-tuning on high return responses with the
standard cross-entropy loss can attain comparable performance. We broadly refer to this class of algorithms
as Expert Iteration.
A large body of work studying RL for LLM fine-tuning also exists outside of the RLHF sphere. Work on text
games (Yao et al., 2020; Ammanabrolu and Riedl, 2019) and other interactive textual environments (Zhou
et al., 2023b; Carta et al., 2023) seek to ground LLMs via interaction and RL. RL has also been applied to
improving model performance on controllable generation and question answering tasks (Lu et al., 2022; Liu
et al., 2022). Various forms of expert iteration have also been applied to improve LLM reasoning capabilities
(Huang et al., 2022; Yuan et al., 2023; Zelikman et al., 2022; Uesato et al., 2022). For example “Scaling
Relationship on Learning Mathematical Reasoning with Large Language Models” (Yuan et al., 2023) applies
a single round of expert iteration across multiple model sizes on GSM8K. They observe sizeable gains in all
metrics for smaller models, with gains diminishing for larger models. A related body of work studies RL
for code generation (Le et al., 2022; Shen et al., 2023; Rozi`ere et al., 2023). Shen et al. (2023) in particular
reports a huge increase in StarCoder’s (Li et al., 2023) maj@1 performance after a single round of expert
iteration, jumping from ∼30% to ∼60%.
Despite all the above work, it remains unclear exactly what factors account for the biggest impact during RL
fine-tuning due to wide variance in tasks, pretraining data, supervised fine-tuning data, RL algorithm used,
and the reward source. Our work conducts a thorough analysis of all these factors to understand exactly how
different algorithms compare when applied to improving LLM reasoning capability. As a result we are able to
identify key bottlenecks to further LLM improvement via RL and provide a discussion on promising future
directions.
3 Methods
Reasoning as an RL problem
We study the performance and sample complexity requirements for various RL algorithms when fine-tuning
LLMs on reasoning tasks. We consider Expert Iteration (EI) (Anthony et al., 2017), Proximal Policy
Optimization (PPO) (Schulman et al., 2017), and Return-Conditioned RL (RCRL) (Brandfonbrener et al.,
2022) as representative algorithms from the RL literature. In general, the goal of all RL algorithms is to
maximize the expected future return EA∼π(Q),(Q,·)∈τR(A) of a student policy π on task τ . We call the highest
return policy the optimal policy π
∗
. Each of our chosen algorithms goes about finding π
∗
in a different way.
3
PPO is an example of an online RL algorithm. Online algorithms engage in both an exploration phase and a
policy improvement phase which updates πθ using data generated during the exploration phase. PPO is also
an on-policy algorithm which samples model rollouts during exploration from the student policy πθ being
trained. During policy improvement, the student πθ updates its parameters via gradient descent by directly
maximizing for reward with the objective
J(θ) = Et

min(
π(at|st)
πold(at|st)
Aˆ
t, clip(1 − ϵ, 1 + ϵ,
π(at|st)
πold(at|st)
)Aˆ
t)

where Aˆ
t estimates the advantage i.e. difference between Q(s, a) (the expected return after taking action a at
state s) and value V (s) (the expected return at state s).
In practice, for PPO we sample 1024 rollouts at a time with a temperature of 0.7 and N = 4 rollouts per
question. Training is then run on these samples for K = 4 PPO epochs with a batch size of 256. Additionally,
we train using LoRA (Hu et al., 2021) with r = 128. Training is run for 4000 gradient steps. The best
checkpoint is then selected via performance on a validation set.
Expert iteration is also online but more off-policy than PPO. An initial expert policy approximation πˆ
∗
0
is sampled on the entire train set K times per question before any policy improvement. The πˆ
∗
0
is often
constructed using repeated sampling from an initial policy π0. For example, AlphaZero (Silver et al., 2017)
and subsequent work (Schick et al., 2023) combine π0 with Monte Carlo Tree Search. Sampling πˆ
∗
0
constructs
an initial set of rollouts D1 which are then distilled back into a policy π1 via a standard cross-entropy
loss: P
τ∈D
PH
t=1 −log(πθ(at|st)). This process can be repeated to construct policy πi fine-tuned on dataset
Di = Ri ∪ Di−1 where Ri corresponds to exploration done by πi−1.
In our setting we construct an approximation to the optimal policy πˆ
∗ by rejection sampling our student
policy πθ. After generating K samples S1, ..., SK on a question Q we construct D1 by filtering all (Q, Si)
pairs with return below a threshold T. De-duplication is then performed on the remaining samples.
In practice, during the expert iteration exploration phase we sample each question in the train set K = 96
times with temperature T = 1.0. To construct the training set we filter out incorrect solutions and duplicates.
Importantly, fine-tuning is then done from the pretrained base model with the same hyperparameters as SFT.
This is repeated until performance on a validation set saturates.
Return Conditioned RL Return conditioned RL algorithms seek to train policies conditioned on both the
current state s and desired return R when sampling an action. This is motivated by a desire to learn return
conditionable policies which can change depending on the desired return. Best performance can then be
sampled by conditioning on the highest possible return.
We consider an offline version of this class of algorithms similar to a decision transformer (Chen et al.,
2021). A training dataset D is constructed by generating state, action, return τ = ((st, at, gt))H
t=1 trajectories.
Training is done by predicting the action given state and return: P
τ∈D
PH
t=1 −log(πθ(at|st, gt)). In practice
we construct D by sampling solutions S = (S1, ..., SL), where each Si
is an intermediate step, from our best
EI trained policy πEI given a question Q. We generate return labels for each step Si by sampling πEI K many
times from Pi = (S1, ..., Si). This results in binary labels l1, .., lK evaluating the correctness of the generated
final answers. Si
is then labeled as “[GOOD]” if the average return 1
K
PK
k=1 lk ≥ T and otherwise is labeled
as “[BAD]”. Typically we set T = 0.5. We then filter the dataset to ensure a balanced number of correct and
incorrect solutions. See Section F in the appendix for more details about the step-label generating process.
Outcome Based Reward Modeling Multiple works (Cobbe et al., 2021; Uesato et al., 2022) train Outcome
Based Reward models ORMs as verifiers of candidate solutions to word problems. The ORM can then be
used to rerank multiple candidate solutions generated by a student model, significantly boosting performance.
Training data for the ORM is generated using a student policy π by sampling K solutions per question Q in
the task dataset. The ORM is trained as a classifier by predicting the probability of reaching the correct final
answer p(is correct(A)|Q, Pi) from an intermediate sequence of steps Pi = (S1, ..., Si), Pi ⊆ A = (S1, ..., SL).
4
maj@1 maj@96 rerank@96† pass@96
7B 13B 7B 13B 7B 13B 7B 13B
SFT 0.41 0.48 0.47 0.53 0.54 0.68 0.72 0.84
EIn 0.48 0.53 0.55 0.59 0.64 0.71 0.8 0.88
ORM EIn 0.48 0.53 0.54 0.58 0.65 0.71 0.81 0.87
ORM RCRL 0.45 0.51 0.5 0.56 0.54 0.69 0.73 0.83
Sparse PPO 0.44 0.51 0.49 0.55 0.58 0.67 0.77 0.85
Dense PPO 0.43 0.50 0.47 0.54 0.53 0.65 0.71 0.81
Sparse ORM PPO 0.46 0.51 0.51 0.55 0.59 0.67 0.79 0.83
Dense ORM PPO 0.46 0.51 0.52 0.55 0.59 0.67 0.76 0.83
Llema∗ 0.40 0.62 0.54 0.69 N/A N/A
RFT 0.47 0.54 0.58 0.65 N/A N/A
WizardMath 0.55 0.64 N/A N/A N/A
GPT-3∗∗ 0.2 0.31 N/A 0.39 0.55 0.71 NA
GPT-4∗∗∗ 0.91 N/A N/A N/A
Table 1 Results when initializing from SFT. EIn denotes n rounds of expert iteration until convergence with n = 2 for
7B and n = 2 for 13B. †Note all reranking is done using an ORM trained with samples from EIn. Results from other
works are included on the bottom for reference. N/A stands for not available. ∗Llema results reported for 7B/34B
sizes without fine-tuning. ∗∗GPT-3 results reported for 7B/175B sizes. ∗∗∗GPT-4 size unknown.
4 Experiments
We conduct our evaluations on GSM8K and SVAMP (Patel et al., 2021): two math word problem benchmarks.
In addition on GSM8K we consider two data regimes: first with SFT data and then without SFT data. We
evaluate all models using greedy sampling (maj@1) accuracy as well majority vote at 96 samples (maj@96),
ORM based reranking at 96 samples (rerank@96), and best of 96 sample (pass@96) accuracy. Unless otherwise
specified, test-time sampling is done greedily for maj@1 and with a temperature of 0.7 otherwise. We sample
the RCRL models one step/line at a time, conditioning on the “[GOOD]” token. We note while the notion of
a “step” is not clearly defined in general, in our case we can simply regard each step as ending with a sentence
or newline. All experiments are done using instruction-tuned Llama-2 7B and Llama-2 13B models.
4.1 Results with SFT Initialization
When given access to SFT data, we first supervise fine-tune Llama-2 models for 4 epochs with a global batch
size of 128 and an initial lr of 2e-5 decayed to 2e-7 with a cosine warmup schedule. We call the resulting
models SFT. When fine-tuning with PPO we initialize using this checkpoint. In contrast, for both EI and
RCRL we generate data with the SFT checkpoint but reset training to start from the pretrained base model.
Similarly to Zelikman et al. (2022), we find this model resetting is crucial for achieving best performance.
Results for both 7B and 13B models are reported in Table 1.
Expert iteration achieves the best performance with competitive sample complexity
Surprisingly, we find EI achieves the best performance with a maj@1 accuracy of 0.485 and 0.53 on 7B and 13B
models respectively. For both model sizes the best greedy accuracy is achieved after n = 2 expert iterations
(see Fig. 2), after which performance plateaus. In total, EI gives a sizable improvement of around 7% over
the SFT baseline. Similar gains can be seen in maj@96, rerank@96, and pass@96 scores with.
PPO models underperform EI, with ORM guided PPO giving the biggest improvement of around 5% over the
SFT baseline. Again, maj@96, rerank@96, and pass@96 accuracies show similar improvements. Interestingly,
despite further training on top of the SFT initialization, PPO models retain competitive rerank@96 and
pass@96 scores when compared to regression we see after further supervised fine-tuning. We believe this is
due to the relatively more diverse nature of the exploration dataset used to update the model.
Finally, RCRL models under-perform EI models despite training on EI generated data with an even balance
5
Figure 1 Sample complexities of SFT initialized models
on GSM8K. EI achieves better performance than PPO
with the same order of magnitude of samples.
Figure 2 Accuracy of EI models on GSM8K test vs.
number of iterations. Performance seems plateaus for SFT
initialized models after two iterations. The pretrained
checkpoints converge after four iterations.
between ‘[GOOD]’ and ‘[BAD]’ step labels. This matches similar results from Du et al. (2023) which use only
sparse labels for the entire rollout. Further, when sampling the RCRL model unconditionally the model often
generates the perfectly valid steps following a ‘[BAD]’ label resulting in a correct final answer. These results
suggest RCRL models are not correctly learning what constitutes a ‘[GOOD]’ versus ‘[BAD]’. This suggests
RCRL models are unable to usefully incorporate information from partially correct solutions at train time.
An ablation (See sec. A of the appendix) on the ratio of positive to negative labels finds a balanced ratio
yields the worst performance, with increasing the amount of positive data leading to better results.
In Figure 1 we plot the number of model rollouts against model performance in log-scale. PPO models achieve
their best accuracies after around 60,000 rollouts while EI models train with an order of magnitude more.
However, the resulting train time in both cases is about a day. This is largely due to memory requirements
from PPO, resulting in lower rollout throughput and smaller mini-batch sizes at train time. Additionally, in
the SFT case we did not experiment with reducing the number of samples from K = 96 per question for EI.
However, we expect this number can be significantly reduced without impacting performance. For a more
thorough investigation of sample complexity requirements, see Figure 5.
Extra guidance from ORMs or dense rewards provides little benefit Overall, the ORM slightly improves
PPO performance and negligibly impacts EI performance. For both algorithms it provides an improvement in
terms of sample complexity. However, this does not change final performance. See Figures 3 and 4 which plot
the performance against number of model rollouts for differnt reward regimes.
Giving dense rewards at best provides no extra benefit to performance when given either heuristically or
via the ORM. Giving a heuristic dense reward even slightly harms model performance relative to the sparse
setting. Recall we give intermediate reward by comparing intermediate model generated steps to the reference
solution. This likely encourages more overfit to exact solutions in the train set, limiting solution diversity.
RL improves maj@1 accuracy without impacting pass@96 performance Looking at the pass@96 accuracies
more closely, we see most similarly sized models are within 3% of the best result. This demonstrates with
enough sampling, most models are able to solve a very similar range of problems. Further, while the pass@96
accuracy of our best EI model initially seems much higher than the SFT checkpoint, this is only because the
SFT checkpoint has undergone much more training on a less diverse dataset. Simply supervised fine-tuning
for half as many steps results in a checkpoint with maj@1 = 0.36 but pass@96 = 0.76. This further suggests
RL training mostly impacts maj@1 accuracy without significantly improving on a pass@n accuracy which can
be achieved with a light amount of supervised fine-tuning.
The proximity of pass@96 accuracies among most models is in sharp contrast to the rerank@96 performance.
Here we find EI models enjoy around a 5% lead over other models. At first glance this seems contradictory
6
Figure 3 maj@1 scores of EI and ORM aided EI models
over the course of training. The ORM improves sample
efficiency but not performance.
Figure 4 maj@1 scores of PPO and ORM guided PPO
models over the course of training. As with EI models, the
ORM improves sample efficiency but not performance.
maj@1 maj@n rerank@n† pass@n
7B 13B 7B 13B 7B 13B 7B 13B
Prompted 0.05 0.03 0.14 0.18 0.17 0.24 0.22 0.27
EIn 0.31 0.4 0.35 0.47 0.39 0.63 0.45 0.83
ORM EI 0.28 0.37 0.33 0.43 0.37 0.59 0.42 0.76
Sparse PPO 0.32 0.41 0.37 0.48 0.41 0.65 0.5 0.83
Sparse ORM PPO 0.29 0.38 0.34 0.44 0.4 0.62 0.49 0.81
Dense ORM PPO 0.29 0.39 0.35 0.45 0.41 0.64 0.5 0.82
Table 2 Results for 7B/13B models when not using SFT initialization on GSM8K. Sparse PPO performs slightly
better than EIin this setting. ∗Note all reranking is done using an ORM trained with samples from EIn model.
with relatively similar pass@96 performance. However, we believe a non-trivial percentage of this gap is due
to overfit of the ORM to the EI model which was used to generate its training data.
4.2 Results with no SFT Initialization
We now consider the case when no SFT data is available for training. For questions in both SVAMP and
GSM8K we give pretrained models access to a two-shot prompt with samples drawn from the GSM8K
validation set. For EI models, we remove these prompts after the first round of exploration, instead relying on
the generated SFT data. As in the case with SFT data, we run both algorithms until performance saturates.
For PPO this happens after 250 steps on SVAMP and 1000 steps on GSM8K. For EI, this happens after n = 5
rounds of exploration and distillation. Results on both datasets are reported in Tables 2 and 3.
EI achieves the best performance overall Even without SFT data, EI achieves the best performance on
SVAMP, improving 7B/13B pretrained greedy model accuracies over 50% from 0.06/0.05 to 0.58/0.69%,
respectively. PPO performs slightly better than EI on GSM8K, improving from 0.05/0.03 to 0.31/0.4. Both
algorithms achieve comparable pass@96 scores across modes sizes, further supporting our observations from
the SFT regime that EI mostly improves maj@1 scores relative to PPO. The prompted 13B model on GSM8K
even attains 0.83 pass@96 accuracy which is close to the 0.84 pass@96 score achieved by the SFT model,
despite having no access to SFT data itself.
EI has the same sample complexity as PPO As before we plot the reward versus number of model rollouts
for PPO and EI in Figures 5 and 6. On GSM8K PPO models attain their best maj@1 accuracies after only
30,000 rollouts and on SVAMP even less. Surprisingly, EI models have the same sample complexity as PPO on
7
maj@1 maj@n rerank@n† pass@n
7B 13B 7B 13B 7B 13B 7B 13B
Prompted 0.06 0.05 0.2 0.25 0.24 0.29 0.3 0.36
EIn 0.58 0.69 0.6 0.75 0.62 0.78 0.70 0.93
Sparse PPO 0.44 0.51 0.55 0.66 0.58 0.73 0.72 0.89
Sparse ORM PPO 0.43 0.51 0.52 0.64 0.54 0.71 0.65 0.85
Dense ORM PPO 0.44 0.52 0.51 0.63 0.55 0.73 0.67 0.85
Table 3 Results for 7B/13B models when not using SFT initialization on SVAMP. EIn denotes the best EI model
after n iterations. EI outperforms PPO.
Figure 5 Sample complexities on GSM8K from pretrained
initialization.
Figure 6 Sample complexities on SVAMP. Surprisingly,
EI appears nearly as sample efficient as PPO.
SVAMP, requiring more samples to converge but also converging to a much higher accuracy. EI still appears
to have higher sample complexity on GSM8K, however as noted before this may be due to oversampling each
prompt during the exploration phase. To test this, we reduce the number of samples per prompt each round
of EI from K = 96 to K = 4. The resulting EI models require more iterations to converge but require far
less total samples, also converging in accuracy only a few percentage points lower than K = 96 samples per
prompt. With K = 4 rollouts per prompt EI has the same sample complexity as PPO on GSM8K.
This is a particularly surprising finding when compared to the performance of EI and PPO on more classical
RL problems training a neural network from scratch. Often PPO enjoys far better sample complexity in
these settings. One major difference here is the initialization of our student from a pretrained model which
imparts a very strong bias on the kind of behaviors and exploration encountered during RL training. Both
the extremely small sample complexity and the comparability of EI and PPO in this setting provide more
evidence that models are not truly engaging in complex exploration, but instead primarily drawing on what
they already know from the pre-training phase.
4.3 Implementation Details
It is well known RL training can be quite sensitive to architectural and hyperparameter choices. This is
even more so the case for LLM fine-tuning. In this section we ablate and discuss the factors we found most
important in our tasks.
PPO model architecture and training parameters To save memory we use a joint architecture for the PPO
policy and value heads. We found it important to use a relatively large value branch (L=4 transformer
layers) and detach the gradients coming from the value branch to the policy trunk. Without detachment we
found value gradients interfere with policy gradients, as similarly observed in Stiennon et al. (2020), causing
8
instability with a big update to either branch. See Figure 7 which compares maj@1 score of a student with a
large value branch and detached value gradients versus the default.
Figure 7 maj@1 performance of PPO fine-tuned models
against architectural changes. Note, we initialize training
from a 7B SFT model with maj@1 = 0.29.
Figure 8 Best K of N sampling parameters versus maj@1
score during training. K=4, N=4 yields a fast runtime
and best performance.
Low rank adaptation (LoRA) (Hu et al., 2021) with rank r = 128 helped significantly to further stabilize a full
layer fine-tuning while still maintaining performance (Sun et al., 2023). A large enough batch size (BS = 256)
and a small lr = 1e-6 also helped with stabilization. We additionally experimented with a partial fine-tune of
only the top M layers. This saved memory but at the cost of a few percentage points of performance.
We also found a non-trivial KL penalty of 0.05 to be critical for preventing model collapse after more than a
hundred gradient updates. This is in contrast to Bai et al. (2022) who do not see a significant need for the
KL constraint. We attribute its importance here to the somewhat unnatural distribution of text found in the
the reasoning tasks which consist of broken natural language and computations enclosed in <<x+y=z>> tags.
For tasks with distributions closer to pure natural language dialogue, such as those considered in Bai et al.
(2022), the KL constraint seems less necessary.
Sampling parameters affect exploration We found the best temperature to use for good exploration during
PPO training heavily depends on the initialization. When starting from an SFT checkpoint we choose T
= 0.7. However, sampling on a high temperature when starting from the pretrained prompted model often
results in collapse. In these cases we choose a low temperature (T = 0.2). Potentially better results for PPO
could likely be achieved by annealing the exploration temperature over the course of training. We similarly
experimented with the sampling temperature used during exploration in EI, ultimately deciding on T = 1.0
to maximize solution diversity without sampling too many degenerate solutions.
We also experimented with best K of N (KoN) sampling during PPO training to promote more solution
diversity. In this setup the K highest reward samples of N rollouts from a single prompt are kept for training
and the rest are discarded. Choosing parameters K ≪ N prioritize high reward samples and discard low
reward ones, resulting in a training distribution more similar to the curated EI dataset.
However, one important consideration is the impact of the K/N ratio on training time and sample complexity,
with smaller ratios taking proportionally longer. For example, K=1,N=8 takes 8 times as long as the default
K=1,N=1. Further, we ultimately found little benefit to small K/N ratios with most configurations yielding
decreased performance over K=1,N=1. In practice we found setting K=4, N=4 worked best. See Figure 8
which compares the performance of various choices of K and N.
Model size and initialization affect exploration We found both the quality of the student initialization and
the size of the student significantly affected the type of exploration engaged in during training. In particular
larger models engaged in more diverse exploration while models with worse generalization engaged in less
diverse exploration (See Appendix Section B). This in turn directly impacts model performance when trained
9
maj@1 maj@96 Rerank@96 pass@96
SFT2 0.36 0.45 0.53 0.76
SFT4 0.41 0.47 0.54 0.72
PPO2 0.43 0.48 0.59 0.8
PPO4 0.44 0.49 0.58 0.77
Table 4 Results for full supervised fine-tune (SFT4
), half supervised fine-tune (SFT2
) and their PPO fine-tunes.
Fine-tuning for only two epochs gets pass@96 = 0.76. This decreases to 0.72 with two additional epochs of fine-tuning.
on exploration data, with models engaging in more diverse exploration improving more from RL training.
To further examine the observations about overfitting, we supervise fine-tune a Llama-2-7B model for half as
many steps than the SFT model reported in Table 1. We call the model trained for four epochs SFT4 and the
model trained for two epochs SFT2
. Despite half the training, SFT2 has similar Rerank@96 and superior
pass@96 scores to SFT4 with the main difference being the maj@1 accuracies. When sampled K = 96 times on
each train prompt, SFT2 produces on average 3.7 unique correct solutions compared to SFT4 which produces
2.9 unique correct solutions. We also find SFT2 benefits significantly more from RL fine-tuning than SFT4
,
jumping from maj@1=0.36 to maj@1=0.43. It’s important to note some of this improvement also happens
with continued SFT training, however at the cost to model output diversity and pass@96 performance.
We believe RL fine-tuning is less prone to overfitting when compared to static SFT fine-tuning precisely
because of the exploration process which generates its own training data. This results in in more diverse
solution paths than the SFT training set, ameliorating overfit. This is also in line with recent work that found
RLHF to result in better (out-of-distribution) generalization than SFT on summarization and instruction
following tasks (Kirk et al., 2023). This benefit can be found both PPO and EI which have almost 10%
pass@96 improvement over continued SFT (yet a much smaller pass@96 improvement over a light SFT). To
support this hypothesis we plot the solution accuracies and diversities of EI models over each iteration in
Figures 10 and 12, respectively. Figure 12 also shows larger models generate more diverse solutions.
5 Discussion and Conclusions
Our study resulted in the following findings:
1. All the tested RL algorithms perform similarly on reasoning tasks, with Expert Iteration performing
best in most cases.
2. Both EI and PPO converge relatively quickly even without supervised fine-tuning, requiring only ∼60,000
model rollouts.
3. Neither algorithm benefits significantly from ORM guidance or a denser reward.
4. EI and PPO fine-tuning simultaneously improves maj@1 score and pass@n score in contrast with SFT.
The improvement of both maj@1 and pass@n performance noted above is due to the ability of online RL
algorithms to dynamically grow diverse sets of training examples via synthetic data generation. This allows
for longer training/more gradient updates on the same model without adversely impacting output diversity
and pass@n scores. In contrast, SFT training occurs on a static dataset. This limits how much training can
occur before maj@1 overfit occurs and output diversity suffers. However, RL training does not significantly
improve pass@n score beyond what can be achieved with light supervised fine-tuning. This suggests even
with RL training our best models are not discovering solutions beyond what can be discovered with (light)
supervised fine-tuning given the same rollout budget.
This observation, taken together with the fast convergence of both online algorithms and the low-impact of
ORM guidance and dense rewards, suggests models are not engaging in a significant amount of exploration
beyond pretraining/SFT data. Regardless of the type of algorithm used or the quality of the reward, all
student models engage in similar exploration, resulting in similar performance.
10
Crucial in our setting is the usage of a pretrained model imparting a strong exploration prior. Without such a
prior, exploration in a high-dimensional textual action space would be impossible. However, this prior also
appears to constrain the exploration engaged in at the beginning of training, with additional SFT training
only making things worse. We view the discovery of new techniques encouraging complex, rich exploration of
reasoning problems as fundamental to progress in LLM reasoning capability. More sophisticted prompting
strategies such as Tree of Thought (Yao et al., 2023) and combining LLM generative abilities with evolutionary
algorithms (Lehman et al., 2022) have already begun to make progress in this direction.
In addition to the limited exploration noted above, we also note reasoning environments are entirely deterministic. This is a setting in which EI and RCRL algorithms are already known to work well theoretically
(Brandfonbrener et al., 2022). PPO enjoys more advantage in environemnts with a high degree of stochasticity.
We also note prior work in RLHF finds PPO outperforms EI type approaches in human preference satisfaction
and instruction following (Gulcehre et al., 2023; Dubois et al., 2023; Kirk et al., 2023). Importantly, in our
setting we always have a reliable ground truth reward to optimize. However, in RLHF, models must optimize
against an unreliable reward model, often resulting in over-optimization (Gao et al., 2022). The relatively
superior performance of PPO over EI on RLHF tasks versus reasoning tasks suggests PPO better mitigates
such over-optimization. This is not too surprising since PPO training penalizes student models diverging
from the initial policy via both its clipped objective and additional KL-constraint. In contrast, EI training
has no such protection built in.