Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models
Authors: Avi Singh
John D Co-Reyes
Rishabh Agarwal
Ankesh Anand
Piyush Patil
Xavier Garcia
Peter J Liu
James Harrison
Jaehoon Lee
Kelvin Xu
Aaron T Parisi
Abhishek Kumar
Alexander A Alemi
Alex Rizkowsky
Azade Nova
Ben Adlam
Bernd Bohnet
Gamaleldin Fathy Elsayed
Hanie Sedghi
Igor Mordatch
Isabelle Simpson
Izzeddin Gur
Jasper Snoek
Jeffrey Pennington
Jiri Hron
Kathleen Kenealy
Kevin Swersky
Kshiteej Mahajan
Laura A Culp
Lechao Xiao
Maxwell Bileschi
Noah Constant
Roman Novak
Rosanne Liu
Tris Warkentin
Yamini Bansal
Ethan Dyer
Behnam Neyshabur
Jascha Sohl-Dickstein
Noah Fiedel
Venue: Accepted by TMLR ()

Abstract: Fine-tuning language models~(LMs) on human-generated data remains a prevalent practice. However, the performance of such models is often limited by the quantity and diversity of high-quality human data. In this paper, we explore whether we can go beyond human data on tasks where we have access to scalar feedback, for example, on math problems where one can verify correctness. To do so, we investigate a simple self-training method based on expectation-maximization, which we call \method, where we (1) generate samples from the model and filter them using binary feedback, (2) fine-tune the model on these samples, and (3) repeat this process a few times. Testing on advanced MATH reasoning and APPS coding benchmarks using PaLM-2 models, we find that \method{} scales favorably with model size and significantly surpasses fine-tuning only on human data. Overall, our findings suggest self-training with feedback can reduce dependence on human-generated data.

Published in Transactions on Machine Learning Research (04/2024)
Beyond Human Data: Scaling Self- Training for Problem- Solving with Language Models
Avi Singh*, John D Co-Reyes*, Rishabh Agarwal*,
Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, Abhishek Kumar, Alex Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Elsayed, Hanic Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snock, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yundi Qian, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, Noah Fiedel
*Equal Contribution, All authors are with Google DeepMind. Correspondence {singhami, jeoreges, rishabhagarwal) Sigoogle.com.
Reviewed on OpenReview: htths///oommeeeeeeee/////////u/d
Abstract
Fine-tuning language models (LMs) on human--eneratte data remains a prevalent practice. However, the performance of such models is often limited by the quantity and diversity of high-quality human data. In this paper, we explore whether we can go beyond human data on tasks where we have access to scalar feedback, for example, one can verily correctness. To do so, we investigate simple self-training method based on cenectationmmaxiiizaaii
math problems where
which we call ReSTEM where we (1) generate samples from the model and filter them using binary feedback, (2) fine-tune the model on these samples, and (3) repeat this process a few times. Testing on advanced MATH reasoming and APPS coding benchmarks using PaLM-2 models, we find that ReSTEM scales favorably with model size and significantly surpasses fine-tuning only on human data. Overall, our findings suggest self-training with feedback can reduce dependence on humaan-generated data.
1 Introduction
Large Language Models (LLMs) are revolutionizing the landscape of deep learning, showcasing remarkable capabilities in generating human-quality text and tackling diverse language tasks (Google et al., 2023; Ope- nAI, 2023). While supervised fine-tuning (SFT) on human-collected data further boosts their performance tasks of interest, acquiring high-quality human data poses significant bottleneck. This is particularly demanding for complex problem-solving tasks, requiring significant resources and expert knowledge. To address this hurdle, model-generated synthetic data emerges as promising alternative, offering scalability and cost-effectiveness, provided its quality can be ensured. While LLMs hold the potential to self-evaluate generated data, this paper explores a simpler sctting where an external, scalar feedback signal serves as a quality indicator for each generated sample.
To investigate training on model-generated data, we consider a simple yet powerful self-training approach for language models that requires only two capabilities: 1) generating samples from the model and 2) evaluating these samples with scoring mechanism. This approach shares similarities with Reinforced Self- Training (ReST) proposed by Gulechre et al. (2023). We make some modifications to ReST (detailed in Section 3), and call our approach ReSTEM, We show that ReSTEM can be viewed as applying expectation- maximization for reinforcement learning (Dayan & Hinton, 1997; Peters & Schaal, 2007), which we present formally in Section 3. Specifically, R&STEM alternates between the expectation and maximization steps:

---

Published in Transactions on Machine Learning Research (04/2024)
Reasoning: MATH
Code Generation: HumanEval GPT-
GPT-
PaLM 2-L (ReSTeM
4-shot Test Accuracy (%6
PaLM 2-1
0-shot Accuracy (%)
60
Mize rd Coder 158
2-1 (ReST
Minerva 540B
Coda Liama Python 34B
Minerva 628
MetaMath 708
50
GPT:3.5 (ChatGPT)
Eode LLaMA 348
FaLM 2-5" (Re51
WizardMath 708
Uemma 34B
PaLM 2-L
PaLM (ReST
40
PaLM 2-5*
Grak-o (33B)
PaLM 2-5
Inflection-1
Grok-0 (338)
Uemma
Inflection-1
LaMA-2 708
LaMA-2
Mistral
Mistral (maig4))
30
Figure 1: Self-training with RoSTEM substantially improves test performance of PaLM 2 models two challenging benchmarks: MATH and HumanEval. Results for other models are shown for general progress these tasks and are typically not comparable due to difference in model scales. GPT-4 results are taken from Bubeck et al. (2023). The x-axis approximately denotes release time (not to scale).
1. Generate (E-step): The language model generates multiple output samples for each input context. Then, we filter these samples using a binary reward to collect the training dataset.
2. Improve (M-step): The original language model is supervised fine-tuned the training dataset from the previous Generate step. The fine-tuned model is then used in the next Generate step.
ReSTEM, with its various adaptations (Section 4), has demonstrated success in enhancing language models across diverse domains, including machine translation (Norouzi et al., 2016; Gulcehre et al., 2023), semantic parsing (Agarwal et al., 2019), preference aligmment (Dong al., 2023), and elementary reasoning (Zelikman et al., 2022; Yuan et al., 2023). However, prior works primarily applied training with self-generated data to relatively small language models (up to 7B parameters), with limited scalability observed for larger models (Yuan et al., 2023). Complementing these efforts, our work aims to investigate the effectiveness and scalability of model-generated synthetic data compared to human- generated data in two challenging, less explored domains: competition-level mathematical problem-solving (MATH) (Hendrycks et al., 2021b) and code generation (APPS) (Hendrycks et al., 2021a).
Our empirical findings reveal significant advancements both mathematical reasoning and code generation capabilities when applying RcSTEM to PaLM 2 models of varying scales (Figure 1). Notably, models fine- tuned on model-generated synthetic data exhibit remarkably larger performance gains compared to those trained on human-written data (Figure 2, 3). Interestingly, exceeding a couple of iterations of ReSTEM leads to diminishing improvement, indicating potential overfitting on small amount of training problems (Figure 4). Additionally, models finc-tuned using RCSTEM improve pass@k as well as majority voting performance. Fur- thermore, these fine-tuned models demonst enhanced performance on related but held-out benchmarks, including math problems (GSMSK and Hungarian HS finals), coding (HumanEval), and Big- Big-Bench Hard tasks. We also perform ablation studies to investigate the effect of number of model-generated solutions, training problems, and iterations for RMSTEM fine-tuning. Overall, findings suggest self-training with feedback as promising approach to reduce dependence on human data.
The key contributions of this work are:
We introduce RMSTEM that enables learning from self-generated data for LLMs, employing prin- cipled expectation-maxxiizzttoo approach within reinforcement learning framework.
We demonstrate that training on self-generated solutions surpasses training on human-generated solutions in problem-solving domains, such as mathematics and code generation.
Through comprehensive ablation studies, we pinpoint the crucial elements necessary for attaining optimal performance.
2

---

Published in Transactions on Machine Learning Research (04/2024)
LLMs fine-tuned with ReSTEM exhibit robust transfer capabilities across various held-out tasks.
2 Preliminaries
An autoregressive language model produces an output sequence (v1, JU1..22.....77 given context (or source input) æ = (21. where the tokens æ1, Yt belong to a fixed vocabulary. Auto-regressive generation involves predicting tokens one at a time, based on the previously generated tokens. Assuming that the model is parameterized by 8, the conditional probability distribution of generating a sequence given x
Po(y x)) = II po y=,, 1=1
with the convention yı.o 0 and F1=-1 (2,, ...............................- For ease of notation, we define p(yala) == p(yeleer, x). The probability of predicting 4t token Me, p(y (x), determined using softmax with temper- ature p(y a)) = E exp(>./7))
U2.
exp(2x/7) where 2 is the logit score for the token yt- Higher values of temperature 7
introduces more randomness, while a lower value makes the output more deterministic by favoring the most probable words.
Given dataset D of inputs x and human-generated outputs y, supervised fine-tuning (SFT) trains the policy by minimizing the negative log likelihood loss:
(0) (a
Σ
log po (yr 11=-
(1)
We also assume access to a deterministic sequence-level (or terminal) reward r(æe, y). Then, the reinforcement learning (RL) objective corresponds to:
(R)( EE-D (ey [r(x,))]]
Optimizing CRI loss directly using online RL methods, such as policy gradients, requires updating and sampling from the policy numerous times during training. However, the computational cost of fine-tuning continual flow of new samples becomes limitation of online methods, especially when the sizes of the policy network grow to tens hundreds of billion parameters. We discuss alternative such online RL approaches in the next section.
3 Expectation--Maxiiizttoo for Reinforced Self-nraining
Exppettaionn-axximaaaiin (EM) for RL We first describe the EM-based framework for RL with language models, building upon the prior work by Dayan & Hinton (1997). Let's define binary optimality variable O, such that p(O = 1 | y) (r(æ,, y)), for some non-decreasing non-negative function f R - R We want maximize the ligelikeldhood of observing O 1 (obtaining high reward):
log 1 1|x) log Σ (o(y)l))p( |
However, the sum over all possible sequences y is typically intractable. Instead of maximizing log 1; one can consider maximizing its ELBO L(pe.g) q) with respect to parameters 0 and variational distribution x(ylx). Specifically,
p(0 1 x. (3) x))
pO x)) log Em(
q(y x))
p(0 1 x. y ()po(z)aar)
By(()l))
log
(Jensen's inequality)
g'y x:)
[log p(
x. y)] KL (g(2) xx) ())ppp =)]
=: =(p(p)
(2)
3

---

Published in Transactions on Machine Learning Research (04/2024)
Algorithm 1: RcST (Expetatoon-axsiiiaatoon)) Given initial policy (o.g., pre-trained LM), ReSTEM iteratively applies Generate and Improve steps to update the policy.
Input: D: Training datasct, Deat: Validation datasct, L(x, y: f): loss, r(x, y): Non-nogative reward function, 1: number of iterations, N: number of samples per context.
for i 1 to do
// Generate (E-step)
Generate dataset D; by sampling: D; = { (a², y' s.t. x ~ D, y ~ - } Annotate D; with the reward r(c, y).
// Improve (M-step)
while reward improves on Deal do Optimise to maximize objective: ((0) = ((),)) (r(e,)) y) logpo(y/=)
end end
Output: Policy pa
The EM algorithm (Dempster et al., 1977) for Equation alternates between an E-step and M-step: at iteration t, denote the language model parameter be f and the variational distribution to be g".
arg E-step: = max, [ppr, .9). Since L(prr, 9) can be written as (x)
q (y
m) xx (" m) p(O 1|,, p) (y æ)). Thus, this step is equivalent weighting the output samples from conditional language model distribution based their likelihood of obtaining high rewards.
M-step: g++1 arg ming Ey +y x) Pog æ). As such, this step corresponds to maximizing weighted negative lieeliodd loss.
arg maxp [ppo (
arg ming kL
=)llpo(y x)
Alternating between above steps ensures a monotonic improvement in the ELBO: L(pa 1 > Mpe + (pprr, g").
EM with non-negative rewards. the rewards are non-negative and f is set to the identity function, then p(O | y) & r(æ, y) which implies q** x) x r(æ, y)Po (y x). In this scenario, the updated policy parameters gt+1 resulting from the M-step at iteration t are given by:
g++1 == arg max E EDWD Ey-og(ul)) [r( y) log x>)
(3)
B
Comparing the above equation with the typical RL objective (CRl) reveals the key distinction between standard RL and EM-based RL: how outpul. data sampled. Standard RL continuously updates the policy and uses this latest policy to collect data. In contrast, FM-based RL employs fixed sampling policy from the previous iteration, decoupling data collection from policy optimization. This decoupling in EM-based approaches enables casier scaling to large policy networks, such as LLMs.
RcSTEM Motivated by the EM framework, we now discuss a simplified version of Reinforcod Self- Training (ReST) approach by Gulcehre et al. (2023). This approach, which we call RMSTEM decouples data collection (E-step) and policy optimization (M-step) typical RL pipeline. Algorithm 1 outlines the RMSTEM algorithm with multiple iterations, where each iteration corresponds one Generate and Improve step. We describe these steps in detail below.
Generate (E-stop): In this step, we generate datasct D; by sampling many output sequences from the current. policy pe: D; { (w )' S.I. ad D, y' po poly)>)) }. Here, the inputs are resampled from the original datasct x D. The output sequences D, are then scored with a binary reward function r(z,g) g). In our experiments, we condition the language model using a few-shot. prompt with programs for code generation and step-by-step solutions for math problems.
4

---

Published in Transactions on Machine Learning Research (04/2024)
Improve (M-step): In the tt iteration, we use the new dataset Dj from Generate step to fine-tune the policy po- To mitigate task-specific over-fitting, minimize drift from the base model by always fine tuning the base pretrained language model. For fine-tuning, we minimize the sewardweiihted negative log-likelihood loss J(O) = -(,, [r(z, y) log (o)lz)] Once the policy is improved, a new dataset of better quality samples can be created once again.
Differences with ReST (Gulcehre et al., 2023). Unlike ReST, we refrain from augmenting Dj in Generate step with human-generated outputs as such data may not always be optimal for learning or might not be easily available. Furthermore, each Improve step fine-tunes the base model instead of the model obtained from the previous ReST iteration. This results in comparable task-specific performance but much better transfer performance held-out tasks (see Figure 7).
Remark. Our experiments focus on problem-solving settings with binary rewards (either 0 or 1), unlike the bounded real-valued rewards assumed by Gulcehre et al. (2023). Specifically, for each Generate step, Gulcehre et al. (2023) perform multiple Improve steps, where each Improve step can be viewed as an M-step with the function f((( y)) = r(2, y) > т, where R+ increases in successive M-steps. However, with binary rewards, any value of 10, corresponds to the identical Improve steps.
4 Related work
Several prior methods can be instantiated using the popectation-maximizzaioo framework presented in Sec- tion 3. We discuss methods and their relation to ReSTEM in this section.
Expert Iteration (ExiT) (Anthony et al., 2017) alternates between two steps: expert improvement and policy distillation. During the expert improvement step (E-step), we combine a base policy with a search procedure to generate samples from a better policy, called the expert policy. Then, in the policy distillation step (M-step), we use these expert samples train the base policy supervised way, effectively improving it to match the expert policy. While ExiT used monte-carlo tree-search, we simply use temperature sampling for collecting samples from the expert policy in RoST. That said, improving the E-step in ReST using the ExIT framework via search and planning procedures with language models would be interesting for future work. For example, Huang et al. (2022) implement a single iteration of RcSTEM on simple math reasoning problems. However, unlike our setup, they do not assume access to a correctness reward and instead employ majority-voting (Wang et al., 2023) as a search procedure within the E-step.
Self- Taught Reasoner (STaR) (Zelikman et al., 2022) employed greedy decoding instead of tem- perature sampling for the E-step in RESTEM which restricted to one moner-generated solution per problem during data collection. Additionally, STaR proposod rationalization as an alternative lemperature sampling, where the language model is provided with the correct answer part of the input to generate correct solutions for difficult problems. However, in preliminary experiments, rationalization leads to substantial increase in false positive solutions that result in correct answer but with incorrect reasoning.
Rejection Sampling Fine-tuning (RFT) (Yuan et al., 2023) improves reasoning performance GSM8K and corresponds to running a single generate (F-slep) and improve (M-step) of RcSTEM While RFT demonstrated limited performance improvements on GSM8K with increasing language model capacity, RMSTEM achioves larger gains on more challenging APPS and MATH benchmarks when scaling PaLM model capacity. Moreover, we observe that using multiple iterations of ReSTEM result in larger performance gains.
Iterative Maximum Likelihood (IML) optimizes a policy using a reward-weighe log-likelihood objective on self-collected data. IML has been shown to perform well with relatively small-scale language models for semantic parsing (Liang et al., 2016; Agarwal et al., 2019), machine transla- tion (Wu al., 2016) and simple math reasoning (Ni et al., 2022). Each E-step and M-step in IMT.
5

---

Table 1: Differences between ReSTEM and other closcly related approaches utilizing synthetic data for Published in Transactions on Machine Learning Research (04/2024)
ReSTEN
REST
STaR
RFT
Starts from fine-tuned model
X
X
Finetunes from base model in each iteration Uses rationalizations for unsolved questions
X
N/A
X
J
X
Temperature sampling for exploration Experiments with Large LMs Multiple iterations
V
X
X
X
/
X
Larger gains on bigger models
N/A
N/A
X
Evaluation on held out tasks
X
advancing language model capabilities.
performed over a mini-batch training examples instead of the entire training dataset, as done RcSTEM In IMI, the learned policy can significantly diverge from the initial pretrained model, which can manifest but loses its ability to generalize to other tasks or domains. Additionally, the tightly coupled nature of data collection and policy optimization in IML leads to high computational cost with large LMs, making it significantly more expensive than ReSTEM
lask-specifie overfitting, where the model performs well on the larget task
Reward weighted regression (RWR) (Peters & Schaal, 2007) corresponds to EM where we set p(O 1|x, exp (r(z, (() in Section 3. RWR has been previously applied to robotic control, as it can be casily applied to non-binary reward functions. Norouzi et al. (2016) build on RWR to propose a general variant of IML for machine translation.
Reward ranked fine-tuning (RAFT) (Dong et al., 2023) can be interpreted as alternating between E-step and M-step over mini-batches, where E-step uses the the output sample with maximum reward for each input context. For binary reward functions, RAFT is analogous to IML and as such, can be viewed as an instantiation of EMS
Other related works: TRICE (Phan et al., 2023) proposes EM-based approach to maximize the marginal log-likelihood (MML) of generating a correct answer for reasoning problem, where the chain-of- thought rationale is treated as latent variable. While E-step in ReSTEM simply corresponds to sampling from the model and filtering with binary reward, TRICE uses Markov-chain Monte Carlo with a control variate to approximate the MML gradient. Sordoni et al. (2023) propose a gradient- freee EM-based approach, similar to RAFT, for prompt-optimization for frozen LLMs.
5 Experiments and analysis
The goal of our experiments answer the following questions:
1. How effective is RM compared to fine-tuning human-generated data?
How many iterations are needed for optimal performance? How quickly does RMSTEM leads to overfitting on training set?
3. How does RM affect passãk and majority voting performance?
4. If fine-tune using model-generated data specific task, do positíve transfer to related tasks? Is there any performance degradation compared to the base model when evaluating fine-tuned models on broad suile of tasks?
How much input data do we need to get most of the performance gains from RMS one iteration of ReSTEM sufficient?

---

Published in Transactions on Machine Learning Research (04/2024)
Hendrycks MATH
Transfer to GSM8K
Pass Test Accuracy (%)
Pass@1 Test Accuracy (%)
40
80
35
30
70
25
60
20
0
1
2
3
0
1
2
Num iterations
Num iterations
Palm-2-S Palm-2-L
Palm-2-L-SFT
Palm-2-5-5FT
Figure 2: ReSTEM for math problem-solving. Test performance MATH and GSMSK (transfer) for PaLM 2-S* and PaLM 2-T. as function of ReSTEM iterations. We also report performance of models fine-tuned via SFT on human-generated data as a baseline. Iteration corresponds to pre-trained model performance. Following Google el al. (2023), we use greedy decoding for evaluation.
Training Datasets. We evaluate RMSTEM primarily on mathematical problem solving using the Hendrycks' MATH datasel (Hendrycks et al., 2021b) and code generation using the APPS (Introduclory) dataset (Hendrycks et al., 2021a). MATH and APPS (Introductory) contain 7500 and 2342 training problems rospectively. We select those tasks because the model outputs can be automatically evaluated as correct or in- correct, perfectly suited for RM Both these datasets offer binary rewards: MATH, model-generated answers can be easily verified for correctness using the ground-truth answer, while on APPS, test cases determine whether the generated code is correct.
Models. We use the PaLM models (Google et al., 2023) with public APIs on Google Cloud for experiments, including PaLM 2-$ (Bison), PaTM 2-$* (Codey), and PaLM 2-I, (Unicorn).
Evaluation. We report generalization performance using the test splits of the MATH and APPS (Introduc- tory) datasets. For measuring transfer performance, look GSM8K (Cobbe et al., 2021), Hungarian HS finals (Paster, 2023), and HumanEyal (Chen et al., 2021) datasets. We also evaluate our models using the Big-Bench Hard (Suzgun et al., 2022) benchmark to evaluate general capabilities. All evaluations follow the settings from Google et al. (2023), unless specified otherwise.
Implementation Details. During each iteration of ReSTEM we generated a fixed number of solutions per problem for the E-step: 32 for the MATH dataset and 64 for the APPS dataset. For generating solutions, we sample from the language model using top-K sampling with K=40 and temperature of 0.7. However, directly using all these model-generated solutions more correct solutions for the casier problems. To mitigate this, we introduced a cut-off threshold for the maximum number of solutions per problem, a design choice also used by Zelikman et al. (2022), included in the fine-tuning dataset: 10 for both MATH and APPS. This approach ensures diversity in the training data and safeguards against overfitting on easier problems. For fine-tuning, we use the few-shot prompt (and the question) as input to the model, and use the model-generated solutions as targets. We only apply the next token prediction loss (Fquation 1) on the targets. Due to the cost of our experiments (thousands of TPC hours for every finc-tuning run), each experiment performed once.
lead to imbalanced dataset,
will have lot
5.1 ReST FM on MATH and APPS
Figures and 3 show the performance of RcSTEM when trained on the MATH and APPS datascts, re- spectively. We see that MATH benefits from performing multiple iterations of RCSTEM both in terms of performance the MATH test set, as well transfer to GSM8K. On the other hand, that most of
7

---

Published in Transactions on Machine Learning Research (04/2024)
APPS (Introductory)
Transfer to HumanEval
Pass@1 Test Accuracy (%)
26
Pass@1 Test Accuracy (%)
55
24
22
50
20
45
18
40
0
2
0
2
Num iterations
Num iterations
Palm-2-5*
Palm-2-L
Palm-2-L-SFT
Palm-2-S--SFT
Figure 3:
RM for code-generation. Test performance APPS (introductory) and Hu-
FanEval (transfer) for PaLM 2-S* and PaLM 2-T. as a function of RRSTEM iterations.
Hendrycks MATH
APPS (Introductory)
Pass@1 Performance 1%6
60
Pass@1 Performance (%)
60
55
50
50
40
45
40
35
20
2
0
2
Num iterations
Num iterations
Palm LTTran))
Palm 2 LiTest!
2L (Trii))
Polm (Test)
Figure 4: Train-test performance gap (left) MATH with PaLM-2-L, and (right) APPS with PaLM- 2-S* as function of RMSTEM iterations.
the gains for APPS come from the first ileration, and the performing more iterations leads to a regression in performance on both APPS and HumanFval.
Interestingly, Figures 2 and demonstrate that fino-tuning model-generated solutions substantially out- performs using human-written solutions, especially for the PaLM 2-L model. This aligns with findings of Yuan et al. (2023) and recent work on distilling LLMs using model-generated data (Agarwal et al., 2023; Gu et al., 2023). However, unlike Yuan et al. (2023), who observed diminishing returns from model-generated data on GSMSK when scaling model capacity, our results suggest an opposite trend: RMST leads to larger performance gains as model capacity increases. On the MATH dataset, the test accuracy improvement with RcSTEM 5.94% for PaLM 2-S compared 6.34% for the larger PaLM 2-L model. Similarly, on the APPS dataset, improvements the fact that the larger models start with much stronger initial performance, and improvements on these benchmarks generally get harder as the bascline performance goes up.
5.6% for PaLM 2-S* compared 6.4% for PaLM 2-L. This in addition to
Train-test performance gap. Figure 4 shows that while training performance increases linearly with the number of EM iterations, test set performance does not. For MATH, test performance improvements are small after the first iteration, and for APPS, we observe a regression in performance the 2nd iteration. We suspect that the regression in performance is likely due overfitting on the small of training problems. Since the APPS dataset is about third of the size of the MATH dataset, it suffers more from this problem.
8

---

Published in Transactions on Machine Learning Research (04/2024)
HumanEval
APPS (introductory)
Hendrycks MATH
Pass @ Tess Accuracy (96
Pass @ Test Accuracy 196)
Accuracy (%6)
80%
80%
40%
(ReST)
70%
60%
30%
60%
50%
40%
30%
Pay M--2--
Rwim-24
2a M-2- (Re5T)
10%
20%
Pwm--44 (ReSD)
20
10
40 Num samples (K)
Num samples (K)
Num samples (K)
Figure 5: Pass@K results for PaLM-2-L pretrained model as well as model fine-tuned with ReSTEM For a fixed number of samples K, fine-tuning with ReSTEM substantially improves Pass@K performance. We set temperature to 1.0 and use nucleus sampling with p = 0.95.
5.2 Impact on Pass@K and Majority-Voting Performance
To investigate the impact of fine-tuning with ReSTEM on the diversity of the final model's generated outputs, evaluate pass@k (Chen et al., 2021) and majority voting (Wang et al., 2023) performance of the finc-tuned PaLM 2-L model relative to the base model.
Pass@K measures the probability that at least one of the K generated solutions for a problem is correct, that is, outputs the correct answer for math problems or passes all the unit tests for code generation. Figure 5 shows the performance Palm-2-L on the pass@ metric. We see that model obtained after ReSTEM fine-tuning is stronger for all values of K, with the performance gap typically being the highest for K=1.
Majority voting first samples diverse set of reasoning paths instead of only taking the greedy one, and then selects the most consistent answer by marginalizing out the sampled reasoning paths. For Hendrycks MATH, it is possible to use majority voting to maximize Pass@1 performance, and we find that when using 64 samples per question, the PaLM 2-L fine-tuned with ReSTEM obtains a test accuracy of 48.82, while the base model gets 44.02.
5.3 Ablation Studies
Impact of multiple iterations Our results show that multiple iterations sometimes lead to over- fitting on the train set (Figure 4). This raises the question of whether multiple iterations are really necessary. Is it better to collect a larger dataset and perform just a single iteration of ReSTEM? To investigate this, we collect a dataset with the base PaLM-2-L model on Hendrycks MATH that is 3x as many solutions per problem as used in a single iteration of ReSTEM for the E-step. Fine-tuning with this dataset results in pass@1 performance of 40.3%, which is lower than the 41% in second and 41.9% in third iteration, as shown in Figure 2. These results indicate that performing multiple iterations of R&STEM leads to higher performance compared single iteration with 3x the data.
Comparing model-generated data with human data A key strength of ReSTEM is its ability to gen- crate multiple correct solutions for cach problem. This provides valuable additional training data compared to human-generated data, which lypically offers only a single solution per problem. While this makes com- parison in Figures 2 and not entirely fair, it also highlights the potential of RESTEM to boost performance with diverse and correct solutions.
In order to enable an apples-to-aplle comparison, we conduct the following study: we select all Hendrycks MATH questions for which we have at least one correct model-generated solution, resulting in about 5K questions. For these 5K questions, we run two fino-tuning experiments: SFT(5K) where we finc-tune on human--writen solutions (one per question), and ReST*(5K) where we fine-tune on model-generated solutions (also one per question, selected at random).
9

---

Published in Transactions on Machine Learning Research (04/2024)
Hendrycks MATH (Test)
PaLM 2-S on Hendrycks MATH (Test)
42
Pass@1 Performance (%)
Pass@1 Performance (%)
30.0
27.5
40
25.0
38
22.5
20.0
36
17.5
34
15.0
SFT (7K) SFT (5K) ReST' (5K) ReSTEM (5K)
SFT (Human) Distill* (2-L) ReST (2-5) Distill (2-L)
Method (Num questions)
Method (Data Source)
Figure 6: Left. Comparing ReSTRM with SFT on MATH. SFT refers to fine-tuning on human data, while ReST* refers to a version of ReSTEM with one iteration that uses only one correct sample per problem. Here, ReST denotes RcSTEM with 3 iterations. For each method, we denote the number questions in Data for Distillation.
parenthesis. Right. Impact
The results Figure (right), show that ReSTEM outperforms fin-ttning on human data even in this much more restricted sctting. Furthermore, the efficacy of RcST(5K) over ReST highlights the additional gain in performance that we can obtain by spending more compute sampling large number of solutions and performing multiple iterations of ReSTEM
Distillation with ReSTEM ReSTeM-generated data The above results indicate that self- generated data can be better than human data for fine-tuning language models. We hypothesize this may be because model- generated solutions are more in-distribution compared to human-written solutions. This raises the question of whether RcST*M-generatedd data can benefit different models than the one generating the data.
To answer this question, consider distillation setup MATH where fine-tune PaLM 2-S using data generated by PaLM 2-L, resulting in solutions for about 5K questions. Specifically, we two distillation experiments: Distill (2-L) where we fine-tune leacher-generated solutions (one per question), similar to ReST (5K), and Distill (2-L), which includes multiple solutions per problem, generated during the final iteration of EM with PaLM 2-L.
Our results, shown in Figure (right), reveal that Distill" surpasses the performance achieved by fine-tuning on Human-written solutions, despite having smaller number of training questions. Additionally, fine-tuning PaLM 2-S with multiple solutions from PaLM 2-L, namely Distill (2-L), is superior than using self-generated solutions via RM This improvement is likely due to the larger number of training questions with solutions in PaLM 2-L generated data compared to 2-S. Overall, these results indicate that model-generated data can be more effective for fine-tuning smaller models than relying on human-generated data.
ReST us EM One of the main differences between ReST and ReSTEM that ReSTEM always fine-tunes the base model for each iteration while ReST continues finetune the the model from the last itcration. We options using PaLM 2-S* in Figure 7 and observe that while ReST and RMSTEM have similar perfor- mance on APPS, the transfer performance to Hu- manEval is substantially better with BM
22.0
ductrrry
Poss@1 Best Accurary 69
48
Pass@1 let Accurary
21.8
46
21.6
21..4
42
ablation comparing those
21.2
40
ResTeR
Figure RM us ReST using PaLM 2-$*
Impact of dataset size Since one of the main ingredients needed for ReSTEM is dataset of input contexts (e.g., questions for MATH), we interested in evaluating the effect of number of input problems.
10

---

Published in Transactions on Machine Learning Research (04/2024)
Average Success Rate by Difficulty Level
Hendrycks MATH (Test)
100
Base
Pass Performance (%)
Average Success Rate (%)
ReS
60
36
20
Very Hard
Medium Problem Difficulty Level
Hard
Easy
Number Questions
Figure Left. Performance for a single ileration of ReSTEM as a function dataset size (number of questions) on MATH. Right. Improvement from ReSTEM based on the difficulty level of the question.
The results from our dataset ablations using the PaLM-2-L model Hendrycks MATH, Figure 8 (left), show that utilizing just 1000 MATH questions results in significant gains, implying that the method is very efficient in the number of prompts needed. However, we noted a slight decrease in performance when using 4,000 questions compared to 2,000, indicating potential variance in the fine-tuning process. Ideally, conducting this experiment multiple times would help quantify this variance, but this prohibitively resource-intensive Overall, we find that ReSTEM is quite sample efficient and performance gains from ReSTEM improve as we increase the dataset size.
Which Questions Benefit Most from ROSTEM We evaluate the performance enhancement of ReSTEM across different question difficultics in the Hendrycks MATH dataset. Questions are classified based on success rates from the base model at temperature setting of T-1.0 into four categories: "easy" (answered correctly 75%-100% of the time), "medium" (50%-75%) "hard" (25%-50%), and "very hard" (below 25%). Figure 8 (right) presents the average success rates for these categories, comparing the basc model to the -fnetuued model. The results demonstrate that R&STEM consistently improves performance across all difficulties, with the highest gains coming for questions categorized as medium and hard.
5.4 Impact on Reasoning capabilities
General capabilities. BIG-Bench provides suite of over 200 tasks that can be used to probe LLMs performance across range of fields and capabilities. BIG-Bench Hard (BBH) (Suzgun al., 2022) is subset of 23 BIG-Bench tasks where the previous generation of LLMs, such as Codex and PaLM 540B, performed below the average human rater. We follow the protocol of Google et al. (2023) and evaluate on BBH using both few-shot and chain-of-thought prompting. Figure 9 shows the performance of RMSTEM finetuned models, and compares them against the base PaLM-2 model. We see no major degradation on any of the BBH tasks. Furthermore, the model fino-tuned on Hendrycks MATH outperforms the base model on this suite when using chain-of -ftthhgthtthh prompting, and the model fine-tuned on APPS also shows slight performance gains. When using direct. prompting, all three models perform similarly.
Problem-solving. To stress test the math problem-solving capabilities a held-out "real-world' evaluation set, we evaluate our model on the 2023 Hungarian high school finals exam in mathematics, akin to Grok. We follow the evaluation protocol from Paster (2023). Specifically, we evaluate the PaLM 2-L, model, fine- tuned with EM on Hendrycks MATH, using the 1-shot prompt from Grok, sample solutions using temperature 0.1, and manually grade the outputs using the rubric provided by the examiners. The results from evaluation are shown in Figure 10. We find that PaLM-2-L fine-tuned with RMSTEM performs well on this exam, surpassing the performance of all existing models except GPT-4.
11

---

Published in Transactions on Machine Learning Research (04/2024)
PALM 2-L
PALM 2-L (MATH)
PALM 2-L (APPS)
Few-shot Performance with CoT
100
90
80
70
PALS 24L
60
PaLM (APP))
PWLM 2-L, (MATH)
50
40
Aueraae 8B4 Derformance
30
75
Causal Judgement
Disamblon wwg.guuaaaatt Sobiguaatooo poaiionnnn QA
Dyck Lang yage es
Failacie Geometri Shas Shapes Hyperby baton (woo
Movie Reco commendation dation Sult--tep Armmen
Object Counting taple
abogu in a Objects
Auin Detect hoo
Sempe para Sequent es of Lies
Unde Unddrstainn ssaandd nn
Tracking Shuf Shuffled ffld Objects (avg)
Logical Deduc cediuctionn ciio (avg (avg)
Snarks
Word Sorting
70
Reasoning about Colores
Boolean
Salient Translation Error
Big-Bench Hard (BBH) Task
CAT
Direct
Prompt Type
Figure 9: Comparing the RMSTEM models to the base model on the ations were conducted across multiple checkpoints, and the vertical black lines denote standard deviation.
Hard suite of tasks. Evalu-
Exam Score vs GSMBK Performance of Various Models
90
Claude 2
PaLM 2-L (ReSTruOM
80
MetaMath Mistral 78
OpenChat 3.5
GSMBK Score (%)
70
MetaMath 7B
Grok-1
60
Grok-O (335).
Turbo
Owen 78
Llmma 348
50
МAmMOTHH 78
40
Mistral 7B
30
Code
20
30
40
50
60
70
Hungarian HS Finals Exam Score (%)
Figure 10: Transfer results on Hungarian HS Finals Exam. Results for models other than PaLM-2-L finetuned with FM are taken from Paster (2023). Several models specialized for mathematics perform well on the widely-used GSM8K benchmark but perform poorly on the Hungarian exam. In contrast, PaLM 2-L model fine-tuned with ReSTEM performs well on both these benchmarks.
6 Discussion
In this paper, propose training on model-generated data combined with reward function, via RM for improving the performance of LLMs on problem-solving tasks. Furthermore, demonstrate that ReSTEM is theoretically grounded in the application of expectation-maxiizzatin ReSTEM on mathematical problem solving and code generation, and show that ReSTEM offers signifi- cant performance gains at a relatively low computational cost, especially when compared to the cost of pro-training. Our experiments also show that RM does not lead regression on other tasks. We conduct number of ablations better understand the strengths and weaknesses of this method, and find that it is data-efficient, but also requires vigilance to avoid over-fitting.
to RL. We evaluate
12

---

Published in Transactions on Machine Learning Research (04/2024)
There are number of limitations associated with ReSTEM First, this method requires moderately-sized training set of problems or prompis, which would need to be collected (from humans) for any new task of interest. Second, ReSTEM also requires access to mamually-designed or learned reward function, ideally one that can be computed automatically. Finally, while RcSTEM allows significant performance improvements in pass&1 performance,
may not quite close the gap to pass& performance for the same task (with a sufficiently large K). Future research in self-improvement in language models should focus on automating manual parts of the pipeline (likely through language models as well), and explore algorithmic improvements that reduce the gap to pass0K performance.
Acknowledgements
We would like to thank Tom Le Painc for providing feedback to an carly draft. We also acknowledge Benjamin Anderson, Sridhar Thiagarajan, Feryal Behbahani, Aleksandra Faust, Doina Precup, Olivier Bachem, and Slav Petrov for helpful discussions.
Author Contributions
Avi, Rishabh, and JD jointly led the project. Avi was responsible for training and evaluation infrastructure, ablations and experiments on MATH, JD led the experiments on APPS, Rishabh was responsible for the paper writing, evaluations, and distillation ablations.
Ankesh, Piyush, Ethan, and Behnam observed preliminary findings about efficacy of model-generated data on MATH for Minerva models and motivated this research. Piyush also helped Avi in setting up infrastructure, Xavier, Peter, Jamos, Jachcoon, Kelvin and Yamini took part in project discussions. Jascha and Noah sponsored and advised the project. All other authors provided feedback on this work.
References
Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi. Learning generalize from sparse and underspecified rewards. In International conference on machine learning, pp. 130 140. PMLR, 2019.
Rishabh Agarwal, Nino Vieillard, Piotr Stanezyk, Sabela Ramos, Matthicu Geist, and Olivier Bachem. Gkd: Generalized knowledge distillation for auto-regressive sequence models. ar preprint orXi::2306.13449 2023.
Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree scarch. Advances in neural information processing systems, 30, 2017.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, and Yi Zhang, Sparks of artificial general intelligence: Early experiments with GPT-4. CoRR, 0ts//2303.1227 2023. doi: 10.48550/ AAR.0//ARRRV..... RER.II.230...1..... 12220.112722 URL https://doi: org/10. 48550/ arrvv..Xivvvv 2303. 12712.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Har- rison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Fe- lipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katic Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. Xix preprint ar arXiv:2107.03377 2021.
Achiam,
13

---

Published in Transactions on Machine Learning Research (04/2024)
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXt::210.11166,, 2021.
Peter Dayan and Geoffrey E Hinton. Using expeciation-maxiinaatioo for reinforcement learning. Neural Compulation, 9(2):271 278, 1997.
Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal stotistical sociely: series B (melhodolooical)) 39(1):1-22 22, 1977.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang. Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint.
arXiv:2304, 06766, 2023.
Google, Rohan Anil, Andrew Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint ar trXte:2305.10/4 2023.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang, Knowledge distillation of large language models. arXiv preprint Xrx:2:2306. 08543, 2023.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. Xiv preprind ar arXiv:2308. 00:99388008988 08998, 2023.
Dan Hendryeks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. arXiv preprint (rXi::2105. 09938, 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. ar preprint arXiv:2103. N.X0:2203.00877,, 2021b.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang. Hongkun Yu, and Jiawei Han. Large language models can self-improve. CoRR, htss/2210.11611 2022. doi: 10. 10.48550/ ARXIV 2210.11611. URL https://doi.. org/ 10///11.000 48550/ hw1550//rriv.... 2210 11610.
Chen Liang, Jonathan Berant, Quoe Le, Kenneth D Forbus, and Ni Lao. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. arXiv preprint arXiv:1611. 1611.00000, 2016.
Ansong Ni, Jeevana Priya Inala, Chenglong Wang, Alex Polozov, Christopher Meek, Dragomir Radev, and Jianfeng Gao. Learning math reasoning from self-sampled correct and partially-corred solutions. In The Eleventh International Conference on Learning Representations, 2022.
Mohammad Norouzi, Samy Bengio, Navdeep Jaitly, Mike Schuster, Yonghui Wu, Dale Schuurmans, et al. Reward augmented maximum likelihood for neural structured prediction. Advances In Newral Information Processing Systems, 29, 2016.
OpenAI. Gpl-4 technical report, 2023.
Keiran Paster.
Testing language models on a held-out high school national finals exam. panaasssaaaaaaaaaama.a. 2023.
https:
Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pp. 745-750, 2007.
Du Phan, Matthew D Hoffman, David Dohan, Sholto Douglas, Tuan Anh Le, Aaron Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, and Rif Saurous. Training shain-of-thougee via latent-variable inference. arXiv preprint Xic:2311 02..0011 19, 2023.
14

---

Published in Transactions on Machine Learning Research (04/2024)
Alessandro Sordoni, Xingdi Yuan, Maro-Alexandre Côté, Matheus Percira, Adam Trischler, Ziang Xiao, Arian Hosseini, Friederike Niedtner, and Nicolas Le Roux. Joint prompt optimization of stacked 1lms using variational inference. In Thirth-seventh Conference on Neural Information Processing Systems, 2023.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Scbastian Gchrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether bhain-of-thougee can solve them. arXiv preprint arxiv:2200.00 09261, 2022.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine translation system: Bridging the gap between human and machine translation. arv preprint orXie:1609.08144, 2016.
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. Scaling relation- ship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.0183 018825 2023.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances Neural Information Processing Systems, 17115466-15488 2022.
15


Reviews:
[{"review_id": "7TX5dottOI", "reviewer": "Reviewer_sFSn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I am not an expert in this field and haven’t had any prior experience working in this field, so my evaluations are an educated guess. \n\nStrengths:\n\nQuality and significance: The authors conducted extensive experiments to show the effectiveness and efficiency of the proposed method, and indeed the performance is significantly improved with relatively low computational cost. \n \nOriginality: Assuming similar phenomena hasn’t been systematically investigated in prior works, the empirical results in this paper are novel. However, I find the idea to be a bit obvious, but I think this is made up by the extensive experimental results and ablation studies. \n\n\nWeakness:\n-\tThere’s no confidence intervals in all experimental results. For most experiments, the difference is big enough that it’s probably statistically significant. However for results like Figure 8, there are a lot of close results, and error bars there would provide more insights on what’s significant and what’s not.\n-\tThe idea of finetuning on model-generated data is a bit obvious.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work showed the effectiveness and efficiency of training on model-generated data combined with a reward function for improving the performance of LLMs on problem-solving tasks such as math and coding. The authors also show the connection between Reinforced Self Training and expectation-maximization.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think adding error bars to as much as possible results would make the paper stronger, especially to Figure 8, and improve my willingness to recommend for acceptance.", "broader_impact_concerns": "I think for reasoning tasks it’s relatively safe to self train on model generated data. However for tasks that involve human opinions and preferences, it is unclear whether/how the proposed method would reinforce existing bias in the model. But I’m not sure whether that’s within the scope of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "l8M8LnH6Hx", "reviewer": "Reviewer_Bd9R", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The examined self-training method is quite simple and improves language models of different sizes compared to relying only on human feedback. \n- Introduces two simplifications to the self-training method by Gulcere et al. (2023)  by removing the reliance on human-generated outputs during the E-step and always finetuning the base model to avoid overfitting.\n- Self-training leads to consistent improvements over models that have been trained only based on human feedback. The results hold for models of different sizes and models finetuned with this method lead to improvements on held-out benchmarks.\n\nWeaknesses\n- Even though there is reduced reliance on human feedback, the proposed method still requires numerous human prompts and a manual function to verify correctness. The main implication of this is that for new tasks there is significant effort involved and the process may not be easy to scale.\n- Improvements from self-training saturate with a few iterations and sometimes lead to overfitting; this can potentially hinder its adoption by the community.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper explores going beyond human data/feedback to improve language models on tasks such as math or coding problems where the correctness of the answers can be easily verified. Specifically, the authors investigate and propose a simplified version of an iterative self-training method by Gulcere et al. (2023) that involves generating samples filtering them based on binary feedback, and finetuning the model on them.  The main contribution is comparing PaLM 2 models trained with this method or with only human feedback in two domains, competitional-level mathematical problem-solving and code generation, as well as providing positive empirical findings for different model sizes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "A few suggestions for strengthening the work\n- The way this paper is positioned compared to ReST is somewhat confusing. In the introduction, it is implied that ReST is used as is but the model section introduces a few modifications with unexplored impact/consequences compared to the original version. It'd be good to provide a better motivation for the proposed modifications and surface them in the introduction. \n- I'd suggest quantifying the human effort involved in writing the prompts with the effort required to provide human feedback to make the benefit more clear to the reader.\n- Given that the ReST method is very closely related, it'd be useful to show in an ablation how the modifications made impact the performance by comparing to it.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "H7gzhJ5Ml5", "reviewer": "Reviewer_Un2a", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The method is simple, yet can be more effective than standard fine-tuning.\n\n- The experiments are pretty comprehensive (although see weaknesses and requested changes for some suggestions). They generally satisfactorily answer the questions posed by the authors at the beginning of the experimental section.\n\n- The approach is justified theoretically [although I still need to take a closer look at some details].\n\n**Weaknesses**\n\n- Reinforced Self-Training (Gulcehre et al., 2023) should likely be described in more detail. If a reader is not familiar with it, it can be difficult to understand exactly what it does, and how $ReST^{EM}$ relates to it.\n\n- Given that many practitioners already fine-tune models on human-generated data, they would possibly also be interested in the results if you apply $ReST^{EM}$ on an already fine-tuned model, instead of just the pre-trained one.\n\n- I found section 3 (Expectation-Maximization for Reinforced Self-Training) slightly difficult to follow.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors describe a framework, $ReST^{EM}$, to iteratively improve language models on tasks for which there is a binary correctness reward. At each round, (1) samples are generated from the model, and (2) the model is fine-tuned on the correct samples. The authors present how the approach is related to expectation-maximization. Experiments are run primarily on math and coding tasks, using PaLM models. In addition to the final performance, the paper also examines generalization to other datasets, distillation, etc.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical**\n\n- What is the \"x\" axis in figure 1? Please update the figure.\n\n- List your contributions more clearly in the introduction. Before getting to the related work section, it was difficult to understand what is proposed here and what already exists in the literature. The paragraph starting with \"$ReST^{EM}$, with its various adaptations (Section 4)\" should likely be reworked.\n\n- In figure 3 (left side), why is Palm-2-S*-SFT better than Palm-2-L-SFT? Is that an error or is it correct?\n\n**Would strengthen the work**\n\nI would be interested in seeing how the model improves based on the initial difficulty of a question. You could bucket questions based on the initial success rate (out of the 40 samples), then show the performance within each bucket at different iterations (or maybe just the final results). Are there some questions with 0% initial success rate that can be answered later on?\n\n**Minor**\n\nIn Eq. 1, it looks like T is a constant, but it depends on each target y. You could add a subscript to make that clearer.\n\n**Other**\n\n- Why is the approach named based on ReST instead of rejection-sampling fine-tuning or other similar methods?\n\n- On page 3, for the E-step, should there be a negative sign in front of $KL[q(y|x)||q^*(y|x)]$? Also, there seems to be an additional vertical bar in that expression that shouldn't be there.\n\n- Could you briefly expand the derivation of the equation in the M-step? [First in the response, then possibly in the paper]", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a simple method called ReST𝐸𝑀, which generates samples from a language model and fine-tunes it using binary feedback. The paper evaluates ReST𝐸𝑀 on advanced MATH reasoning and APPS coding tasks using PaLM-2 models, and shows that it outperforms fine-tuning only on human data. The paper demonstrates that ReST𝐸𝑀 scales favorably with model size and improves generalization to other math and coding datasets.

All reviewers have given positive ratings. I recommend acceptance overall.
Decision: Accept (poster)