REFT: Reasoning with REinforced Fine-Tuning (2024)

Abstract: One way to enhance the reasoning capability of Large Language Models (LLMs) is to conduct Supervised Fine-Tuning (SFT) using Chain-ofThought (CoT) annotations. This approach does not show sufficiently strong generalization ability, however, because the training only relies on the given CoT data. In math problemsolving, for example, there is usually only one annotated reasoning path for each question in the training data. Intuitively, it would be better for the algorithm to learn from multiple annotated reasoning paths given a question. To address this issue, we propose a simple yet effective approach called Reinforced Fine-Tuning (ReFT) to enhance the generalizability of learning LLMs for reasoning, with math problemsolving as an example. ReFT first warmups the model with SFT, and then employs on-line reinforcement learning, specifically the PPO algorithm in this paper, to further fine-tune the model, where an abundance of reasoning paths are automatically sampled given the question and the rewards are naturally derived from the ground-truth answers. Extensive experiments on GSM8K, MathQA, and SVAMP datasets show that ReFT significantly outperforms SFT, and the performance can be potentially further boosted by combining inference-time strategies such as majority voting and re-ranking. Note that ReFT obtains the improvement by learning from the same training questions as SFT, without relying on extra or augmented training questions. This indicates a superior generalization ability for ReFT 1.

heading: 1 Introduction
text: The state-of-the-art approaches to solving math problems (Luo et al., 2023; Wang et al., 2023a) employ Supervised Fine-Tuning (SFT) to train the models using Chain-of-Thought (CoT) annotations (Wei et al., 2022). As shown in Figure 1, a
* indicates equal contribution, † indicates corresponding author
1Code: https://github.com/lqtrung1998/mwp_ReFT
Question (x): Weng earns $12 an hour for babysitting. Yesterday, she just did 50 minutes of babysitting. How much did she earn?
Chain-of-Thought (e): We need to calculate her hourly rate and then multiply it by the amount of time she worked. First, we need to convert 50 minutes to hours. There are 60 minutes in an hour, so 50 minutes is equal to 50/60 = 5/6 hours. Next, we can calculate Weng's earnings by multiplying her hourly rate by the amount of time she worked: $12/hour x 5/6 hour = $10. Therefore, Weng earned $10 for 50 minutes of babysitting. The answer is 10.
CoT annotation outlines the intermediate reasoning steps toward solving a math problem.
Usually there is one CoT annotation for each question in the training data, i.e., one correct reasoning path, which is utilized in SFT. We observe that this may result in relatively weak generalization abilities of the SFT models. It is often the case that multiple valid CoT annotations exist for the same question (Cobbe et al., 2021a; Zhang et al., 2023), underscoring the need for a more powerful fine-tuning approach. To address this problem, we propose a simple yet effective approach called Reinforced Fine-Tuning (ReFT) (Figure 1 bottom).
ReFT commences with a warm-up stage involving Supervised Fine-Tuning (SFT) in one or two epochs (Figure 1, shaded box). This initial stage equips the model with the ability to generate correct responses to mathematical problems to some extent, as demonstrated in prior work (Cobbe et al., 2021a). Next, ReFT proceeds to further refine the
7601
model through the utilization of an online Reinforcement Learning (RL) algorithm (Sutton and Barto, 2018), specifically Proximal Policy Optimization (PPO) (Schulman et al., 2017) in this paper. In this way, ReFT is able to sample multiple correct reasoning paths or CoT annotations and learn from them (Figure 2, right).
Since the training data include ground-truth answers, the golden rewards can be naturally derived from them when training PPO. Consequently, there is no requirement for a separately trained reward model. In contrast, RLHF (Ouyang et al., 2022) has to utilize a reward model that is learned from human-labeled data.
During the warm-up stage, ReFT acquires a certain level of accuracy by supervised learning. In the RL stage, ReFT further enhances its ability by reinforcement learning through sampling various CoT reasoning paths. In this way, ReFT gets much richer supervision signals than SFT. This approach enables ReFT to greatly improve generalization in math problem-solving (Gao et al., 2018; Brown et al., 2020). Note that ReFT outperforms SFT by using the same training questions, without relying on extra or augmented training questions. In fact, ReFT does not conflict with such data engineering and can be seamlessly combined with it.
Our contributions are as follows:
• We introduce a novel fine-tuning approach, reinforced fine-tuning (ReFT), which utilizes reinforcement learning to solve math problems. ReFT exhibits enhanced generalization capabilities compared to conventional supervised fine-tuning when trained on the same dataset.
• We conduct extensive experiments using two foundational models, CodeLLAMA (Roziere et al., 2023) and Galactica (Taylor et al., 2022), on three standard datasets: GSM8K (Cobbe et al., 2021a), MathQA (Amini et al., 2019), and SVAMP (Patel et al., 2021). Our experiments cover both natural language and
program-based CoTs, demonstrating the significantly improved performance and generalization ability of ReFT.
• Additionally, we demonstrate that ReFT benefits from both majority voting (Wang et al., 2023b) and reward model reranking (Uesato et al., 2022) at inference-time, further improving its performance.

heading: 2 Related Work
text: Math Problem Solving Recent research efforts focus on CoT prompt design and data engineering. Most of them attempted to make CoT comprehensive and fine-grained to present the stepby-step reasoning solutions (Nye et al., 2021; Fu et al., 2023; Zhou et al., 2023b; Khot et al., 2023; Zelikman et al., 2022; Imani et al., 2023; Miao et al., 2023). Gao et al. (2023) further proposed to use the Python program as CoT prompt, demonstrating more accurate reasoning steps and significant improvements over the natural language CoT (Wei et al., 2022). Zhou et al. (2023a) introduced a prompting method that generates code to verify the intermediate reasoning step with GPT4 (OpenAI, 2023), thus achieving state-of-the-art performance on GSM8K (Cobbe et al., 2021a) and MATH (Hendrycks et al., 2021). Another line of work focuses on improving the quality of CoT (Wang et al., 2023a; Liu et al., 2023; Yu et al., 2023) and increasing the amount of CoT data (Luo et al., 2023; Yue et al., 2023) from OpenAI’s ChatGPT (gpt-3.5-turbo) or GPT-42.
Reinforcement Learning Our work is mostly related to the recent work that applies PPO (Schulman et al., 2017) to natural language process for aligning human preferences (Ouyang et al., 2022). Since then, several training algorithms have been proposed to efficiently improve the alignment, including direct preference optimization (DPO) (Rafailov et al., 2023), identity preference optimization (IPO) (Azar et al., 2023), and Kahneman-Tversky optimization (KTO) (Ethayarajh et al., 2023). Other than the purpose of alignment, we aim to adopt reinforcement learning as a fine-tuning paradigm to improve performance over conventional supervised fine-tuning.
Specifically for solving math problems, Uesato et al. (2022) and Lightman et al. (2023) trained an outcome-based or process-based reward model to
2https://chat.openai.com/
perform reranking (Cobbe et al., 2021a) to achieve much better performance over SFT and majority voting (Wang et al., 2023b). While our approach aims to improve the performance of the policy itself, these reward model reranking approaches can be easily integrated into the resulting policy model.

heading: 3 Method
text: In this work, we focus on natural language CoT (N-CoT) (Wei et al., 2022) (Figure 1) and programbased CoT (Gao et al., 2023) (P-CoT) using Python. Gao et al. (2023) proposed the programbased CoT for math problem solving. We can simply execute the program to obtain the answer. To ensure clarity and avoid ambiguity, we use the terms N-CoT and P-CoT to represent natural language and program-based CoTs, respectively.

heading: 3.1 Reinforced Fine-Tuning
text: The proposed Reinforced Fine-Tuning (ReFT) process consists of two stages: the warm-up stage and the reinforcement learning stage. The overall algorithm is shown in Algorithm 1.
Warm-up In this stage, the policy is fine-tuned for a few epochs on a dataset comprising of the “(question, CoT)” tuples: (x, e). It enables the model to have basic problem-solving skills to generate a proper response3. Formally, the CoT generation process can be decomposed into a sequence of next token prediction actions. The last action token, <eos>, signals the generation process to terminate. The CoT e is written as:
e = [a1, a2, ..., aL−1, aL=<eos>]
where L represents the maximum length. At timestep t, the action at is sampled from a policy πθ(·|st) where at can be any token in the vocabulary and the state st comprises of all tokens in the question and all tokens generated so far. After each action, the resulting state st+1 is the concatenation of the current state st and the action at:
st+1 =
{ x, t = 0
[st, at], 1 ≤ t ≤ L .
As the produced action is the <eos> token, the resulting state sL+1 is the terminal state and the
3The underlying concept is similar to the verifier training (Cobbe et al., 2021a) to generate multiple solutions.
generation process is finished. With this notation, the loss function for a sample can be written as:
LSFT (θ) = −Ee∼D [ L∑
t=1
log (πθ(at|st)) ] (1)
Reinforcement Learning In this stage, the policy improves its performance via a form of online self-learning using a dataset comprising of (question, answer) tuples: (x,y). Specifically, the policy model learns by repeatedly sampling responses (Figure 2), evaluating the response’s answer correctness, and updating its parameters in an online fashion (line 7-14 in Algorithm 1). We employ PPO (Schulman et al., 2017) with a clipped objective algorithm for training. Following Ziegler et al. (2019), the value model Vϕ is constructed by appending a linear value head on top of the last hidden states of the policy model πθ, which is the model after the warm-up stage. The reward of 0 is given for all action resulting in non-terminal state. At the terminal state, we use a reward function that directly compares the answer extracted from the state’s CoT and the ground-truth answer y . Here, the reward function returns 1 if the answer is deemed correct, otherwise 0 is returned. On dataset whose answers are all numeric, partial reward (Zhong et al., 2017; Le et al., 2022) of 0.1 can be applied when the answer can be extracted and it is of numeric type. For 1 ≤ t ≤ L, we write
r(st, at, st+1)=    1, EXTRACT(st+1) = y
0.1, EXTRACT(st+1) ̸= null, ̸= y 0, EXTRACT(st+1) = null
Such a partial reward can help reduce the effect of learning from sparse reward (Riedmiller et al., 2018; Trott et al., 2019). In addition, following Zheng et al. (2023), our total reward is the sum of the reward function score and the KullbackLeibler (KL) divergence (Kullback and Leibler, 1951) between the learned RL policy and initial policy scaled by a coefficient factor β.
rtotal(st,at, st+1) = r(st, at, st+1)
− βKL ( πθ(·|st),π(0)θ (·|st) )
The generalized advantage estimate (Schulman et al., 2018) is used for advantage calculation:
Ât =
L−t∑
l=0
(γλ)lδt+l,
Algorithm 1: Reinforced Fine-Tuning Input: Dtrain = {(x, e,y)}: Tuples of (question, CoT, answer), W : number of warm-up steps, T :
number of RL steps, U : number of updates per RL step, π(0)θ : Initial policy. Output: πθ: Final policy
1 πθ = π (0) θ 2 // Warm-up stage 3 for i← 1 to W do 4 x, e,y ∼ Dtrain // Sample mini-batch from Dtrain 5 θ = OPTIMIZATION_STEP(LSFT (θ)) // Equation 1 6 // Reinforcement learning stage 7 for i← 1 to T do 8 x, _,y ∼ Dtrain // Sample mini-batch without CoT 9 ê ∼ πθ(x) // On-policy CoT sampling 10 ŷ ← EXTRACT(ê) // Extract the answer from CoT 11 πθold ← πθ, Vϕold ← Vϕ 12 Compute δt, Ât, R̂t using πθold , Vϕold ,x, ê, ŷ and y 13 for j ← 1 to U do 14 θ,ϕ = OPTIMIZATION_STEP(LRL(θ,ϕ)) // Equation 2
15 return πθ
where the Temporal Difference (TD) is defined as
δt′ = −Vϕ(st′)+rtotal(st′ , at′ , st′+1)+γVϕ(st′+1) with the terminal state value Vϕ(sL+1) := 0, λ ∈ (0, 1] is the discount factor for rewards, and γ ∈ [0, 1] is the discount factor for TD. For the estimate of return, we leverages the λ-return R̂t, which can be written as the sum of the generalized advantage estimate and the value estimate:
R̂t = Ât + Vϕ(st)
Lastly, the policy and value objectives can be written as in two equations below
Lpolicy(θ) = −Ee∼πθold
[ min ( πθ(at|st) πθold(at|st) Ât,
clip (
πθ(at|st) πθold(at|st)
, 1− ϵ, 1 + ϵ ) Ât
)]
Lvalue(ϕ) = 1
2 Ee∼πθold
[ max (∥∥∥Vϕ(st)− R̂t ∥∥∥ 2 ,
∥∥∥clip ( R̂t − Vϕ(st), Ât − ϵ, Ât + ϵ )∥∥∥ 2 )]
where πθold , Vϕold are used for sampling CoT and computing Ât, R̂t. The unified loss function is the weighted sum of the above objectives.
LRL(θ,ϕ) = Lpolicy + αLvalue (2)
where α is the coefficient for the value objective.

heading: 4 Experiments

heading: 4.1 Datasets
text: We conduct experiments on three math problem datasets: GSM8K (Cobbe et al., 2021a), SVAMP (Patel et al., 2021) and MathQA (Amini et al., 2019). For both GSM8K and SVAMP, the format of answers is a numeric value. In MathQA, the format is instead a list of multiple choices (i.e., ABCD). Table 1 presents the statistics of all datasets. We perform few-shot prompting (Wei et al., 2022; Gao et al., 2023) using GPT-3.5-turbo to obtain both the N-CoT and P-CoT annotations4. The NCoT and P-CoT annotations are obtained following Jie et al. (2023). We also conducted an additional experiment on a numeric version of MathQA (Jie and Lu, 2023) where the format is also a numeric value. Such experiments are used to demonstrate our assumptions of potential reward hacking phenomenon (Skalse et al., 2022) on MathQA (§4.4).

heading: 4.2 Baseline
text: We compare ReFT with SFT and self-training (Xie et al., 2020; Amini et al., 2022) baselines. SFT simply fine-tunes the language model on the train-
4Examples of N-CoT and P-CoT representations can be found in Appendix A.
ing data. Experiments with self-training methods ensure a relatively fair comparison because these methods share the mechanism that the samples generated from the model are used for training.
We implemented Offline Self-Training (OfflineST) (He et al., 2020), and Online (Hoi et al., 2021) Self-Training (Online-ST). The Offline-ST method is similar to expert iteration (Anthony et al., 2017; Uesato et al., 2022; Zelikman et al., 2022). We first use the SFT checkpoint from the early checkpoint to sample the CoTs and verify them against the ground truth. We only retain those expert samples that have a correct answer. We perform SFT on the combination of original training data and the expert samples.
The Online-ST method is made to be closely comparable to ReFT. Following ReFT, Online-ST has the same warm-up process. After that, we perform continual training with the samples generated on the fly. At each training step, the model first samples CoTs for a batch and only retains those with correct answers. The resulting batch consists of both sampled and ground-truth CoTs. We then update the model parameters on this batch with the supervised fine-tuning objective LSFT . Compared with ReFT, Online-ST neither makes use of negative responses (with an incorrect answer) nor has a dedicated mechanism to prevent the model from significantly diverging from the initial model, which can manifest as task-specific overfitting and training instability.

heading: 4.3 Experimental Setup
text: We conduct experiments with two foundation models: Galactica-6.7B5 (Taylor et al., 2022) and CodeLLAMA-7B67 (Roziere et al., 2023). Both models are reported to have strong performance in math solving and are commonly adopted in recent literature on reasoning tasks (Yue et al., 2023; Luo
5huggingface.co/facebook/galactica-6.7b 6huggingface.co/codellama/CodeLlama-7b-hf 7Additional preliminary experiments were conducted using Gemma (GemmaTeam, 2024). However, these results are not included in the current version of this paper due to unresolved implementation issues that align with known challenges reported within the open-source community (https: //huggingface.co/google/gemma-7b/discussions).
et al., 2023). In addition to the comparison with baselines, we also apply common techniques, majority voting (Wang et al., 2023b) and reward model reranking (Lightman et al., 2023) on GSM8K.
Hyper-parameters In all experiments, the training is done with 8 A100-80GB GPUs using DeepSpeed (Rajbhandari et al., 2020; Rasley et al., 2020) Zero stage 2 and HuggingFace Accelerate (Gugger et al., 2022). During the warm-up stage of ReFT, we use AdamW (Loshchilov and Hutter, 2017) optimizer with 10% warm-up ratio. The batch size is 48 and learning rate is 1e-5. The maximum length is set to 1024. The number of epochs in the warmup stage is 2 in all settings except on MathQAMCQ and MathQAnumeric where we use up to 5 and 10 respectively. The model is trained for 300 epochs with a learning rate of 3e-7. Following Ziegler et al. (2019), the λ, γ, α, ϵ and U in PPO are set to 1, 0.95, 5, 0.2, and 2, respectively. The KL coefficient β is set to 0.01 for P-CoT and is set to 0.05 for NCoT experiments. Further hyperprameter settings about ReFT can be found in Appendix B.
For SFT baseline, we train the model for 40 epochs and choose the checkpoint with best performance. This number of epochs has been chosen to be sufficiently large to ensure SFT converges. For Offline-ST baseline, we sample the CoTs by using the checkpoint from the ReFT warm-up stage. Using the generation temperature of 1.0 and max length of 1024, we sample 100 CoTs for each question and only keep those with a correct answer. Following Singh et al. (2023), we then subsample the CoTs to 10 random unique CoTs per question to balance difficulties of questions. The number of fine-tune epoch is set to 20, which is sufficiently large to ensure the training to converge. As mentioned in §4.2, the Online-ST baseline tries to mimic the same setting as in ReFT. We have the same warm-up process and the hyperparameter setting is roughly the same as ReFT.
Reward Model Reranking Following (Cobbe et al., 2021a; Uesato et al., 2022), we train a reward model (RM) to determine the correctness of the CoT. To construct the RM training data, we use the model from the warm-up stage and perform sampling to obtain 100 CoTs for each question in the training set. The CoTs are deduplicated and the binary labels can be obtained by comparing the extracted answer against the ground truth.
As a common practice, the reward model is a
language model that is initialized from the best SFT checkpoint (Cobbe et al., 2021a; Ouyang et al., 2022). Similar to the outcome-based reward model (ORM) (Uesato et al., 2022), the reward model is trained to predict a binary label that indicates the “correct” or “incorrect” solution. Once the input passes through the reward model, classification is conducted with a linear classifier on the hidden state of the last token. Finally, the solution with the highest “correct” score among the candidates is selected as the final answer. We train the RM model for 3 epochs using a batch size of 24, the maximum length of 700 and a linear learning rate schedule with 10% warm-up period and the max learning rate of 1e−6.
Evaluation We report value accuracy for both N-CoT and P-CoT on all datasets. For majority voting and reranking (Table 4), we sample 100 CoTs for evaluation. In voting, the valid answer with majority counts is chosen as the final answer for computing accuracy. In reranking, we choose the CoT with the highest score and extract the answer.

heading: 4.4 Results
text: ReFT Outperforms SFT Table 2 compares the performance among the baselines and proposed ReFT on GSM8K, SVAMP, and MathQA datasets. We can observe that ReFT consistently achieves much better performance over the SFT except on MathQAMCQ N-CoT. Specifically, we have closed to 10-point and 12-point improvement over SFT with CodeLLAMA on GSM8K N-CoT and P-CoT, respectively. On average, we achieve 6.7-point and 7.4-point improvements with CodeLLAMA on all datasets in N-CoT and P-CoT, respectively. Notably, no additional annotations or reward models are used in ReFT. Such strong results demonstrate robust generalization of ReFT (see Analysis §5) and huge potential for further exploring the training
data with reinforcement learning (Lu et al., 2023). Offline self-training includes the sampling data from the initial policy for fine-tuning. We can see this simple baseline can improve the performance compared with SFT (He et al., 2020; Gulcehre et al., 2023) but the improvements are far behind the one made by ReFT. Such comparisons indicate that “exploring” is essential in ReFT to have good performance. Though online self-training achieves some more improvements with Galactica, it is still far behind ReFT on average. This result indicates that incorrect instances are also very essential to guide the model for better exploration. Comparisons with self-training also suggest the proposed approach with on-policy sampling and reinforcement learning is better than standard data augmentation approaches.
Reward Hacking for MathQA Our investigation of the negative results on MathQAMCQ in-
dicates that ReFT suffers from the reward hacking (Skalse et al., 2022) on the multi-choice question during training. Figure 3 shows how the sampled solutions produce “inaccurate rewards”, which makes the RL training suffer. As we can see, the sampled CoT obtains an incorrect answer “172” which is not half of the product of “18” and “22”. However, the final reasoning step still predicts the option “C” as the final answer as the model would always predict one of the options from {A, B, C, D, E} regardless of the correctness of intermediate CoT8. Thus, such a misleading CoT will receive a positive reward “1” and misguide the model to treat this as a correct CoT. The underlying reward hacking phenomenon severely tampers the model training (Everitt et al., 2021). This is also the reason that we chose the checkpoint with longer warm-up steps for MathQA N-CoT to reduce the reward hacking effect.
To further demonstrate the negative effect of MCQ questions, we experiment on the MathQA variant by Jie and Lu (2023), MathQAnumeric (Table 1), which removes the options in the question, and directly predicts the numeric answer. Table 3 presents the comparison against the baselines. We can observe that ReFT consistently outperforms the baselines using both Galactica and CodeLLAMA. Ideally, we could reduce the reward hacking effect on MathQAMCQ if we can obtain a more finegrained reward (e.g., process-based reward (Lightman et al., 2023)) for the intermediate reasoning
8We found that program-based CoTs are less likely to suffer as it is more rigorous than natural language.
steps. However, the development of a reliable process-based reward model is expensive, and requires extensive manual annotations of reasoning steps. Recognizing these challenges, we consider controlling reward hacking and its analysis as an important problem to be addressed in future work.
Majority Voting and Reranking Benefit ReFT Following Wang et al. (2023b); Uesato et al. (2022); Lightman et al. (2023), we also perform majority voting and reward model reranking to show that ReFT can benefit from these common techniques. Specifically, we perform sampling from both SFT and ReFT policies. We sample 100 CoT solutions for each question and employ the reward model described in §4.3 to perform reranking. Results in Table 4 demonstrate that ReFT consistently achieves the best performance on GSM8K by reward model reranking. ReFT + Voting significantly outperforms SFT + Voting by 8.6 points on average across all settings. ReFT with reranking outperforms SFT with reranking by more than 3 points.
Compared with existing open-source approaches (Luo et al., 2023; Wang et al., 2023a; Yue et al., 2023) (Table 4 bottom9), our best P-CoT variant achieves the best performance with accuracy 81.2 on GSM8K. In addition, these approaches mainly include extra data generated from ChatGPT and perform distillation during fine-tuning. In contrast, we improve the policy itself by exploiting the potential of existing training data and pushing the limit of the policy performance. Our best result reported in Table 4, i.e., the CodeLLAMA + ReFT + Reranking with P-CoT setting, even surpasses GPT-3.5-turbo. However, we obtain the result with a model that is only in the size of 7B.
Experiments with Small Model Intuitively, exploration could lead to imperfect demonstration
9Numbers are taken from original papers. The N-CoT and P-CoT results for MAmmoTH-Coder are reported in their appendix.
with a small language model. We conduct an experiment on P-CoT data using Galactica-125M10, Codeparrot-small11 and Codegen-350M12. Table 5 shows the performance comparison between SFT and ReFT. Surprisingly, ReFT still outperforms SFT on three datasets. Such improvements demonstrate the robustness of ReFT during the exploration of reasonable programs.
Ablation Study We perform the ablation study using CodeLLAMA on GSM8K P-CoT (Table 6). Without the partial reward, ReFT obtains a lower accuracy 74.4 but it is still much better than SFT. As mentioned in §3.1, such a partial reward can help reduce the effect of sparse reward (Trott et al., 2019) during training. In addition, the policy distribution will easily collapse to produce unexpected results (i.e., 0 accuracy) if we set the KL coefficient β to 0. It is certainly critical to impose constraints on the space that the policy explores (Ouyang et al., 2022). The initial warm-up step essentially makes such constraints and allows the policy to further explore within the range that is governed by β. We also experiment with a separate value model (Andrychowicz et al., 2021; Cobbe et al., 2021b), where the torso parameters are initialized the same as the policy model. We found that such a setting allows the policy to converge faster in early RL training, but eventually reaches an on par performance. Compared to the original setting of a shared value model, it is, however, twice the com-
10huggingface.co/facebook/galactica-125m 11huggingface.co/codeparrot/codeparrot-small 12huggingface.co/Salesforce/codegen-350M-mono
putation overhead due to one extra forward-pass, as well as twice the memory cost due to the storage of the separate value net. Finally, in Appendix C we give a case study to show how the generated P-CoT evolve for SFT and ReFT.

heading: 5 Analysis
text: Generalization Figure 4 shows the mean reward, evaluation accuracy, and the KL divergence during training of ReFT13 on GSM8K P-CoT using CodeLLAMA as foundation model. SFT converges and becomes overfiting when approaching 40th epoch. However, we can see the mean reward is around 80% to 90% for the ReFT policy at 40th epoch, and the value accuracy is also increasing. In addition, we can see that the KL divergence (Figure 4 (c)) is very large in the beginning and then maintains a reasonable value between 0 and 10. The stable KL divergence indicates our policy performs exploration within a space that contains appropriate programs. The underlying reinforcement learning mechanism greatly improves the generalization ability of ReFT (Brown et al., 2020).
Qualitative Evaluation We perform a human evaluation to qualitatively assess the output from the SFT model, Warmup checkpoint, and ReFT model. The evaluation uses 50 questions and samples the solutions in GSM8K test set that can be solved correctly by all three models. We ask four different annotators to score the reasoning path according to the following criteria, each scored on a scale from 0 to 1.
• Logic: evaluates if the logic leading to the answer is correct.
• Naming: evaluates if the variable conveys appropriate and reasonable semantics
• Compactness: evaluates if the reasoning paths contain redundant information.
13For illustration purpose, we only shows the mean reward and KL for 60 epochs.
A perfect score of 3 indicates good performance across these three dimensions. To ensure the evaluation is impartial and faithful, we strictly follow the setting: (1) The origin of each reasoning path (from SFT, Warmup, or ReFT) is anonymized to prevent annotator bias. (2) Four different annotators are responsible for different portions of the samples.
As seen in table 7, though the overall scores are quite close, ReFT performs slightly better than SFT, and outperforms the Warmup variant. Note that SFT is inherently trained to learn from the ground truth, thus, it is likely to have a high score. This comparative analysis underscores the robustness of ReFT in generating accurate and semantically coherent reasoning paths.
When ReFT surpasses SFT? To further investigate the relationship between ReFT and SFT, we perform ReFT training with different number of warm-up steps from SFT. Figure 5 shows the value accuracy of different ReFT variants against SFT14. Specifically, if the warmup step is 3, that means the policy initialize from the 3rd-epoch SFT checkpoint. We can see that the performance of all ReFT policies decreases right after the warm-up in the beginning, until the training epoch reaches around 8. Because the linear layer in the shared value model is randomly initialized, and it could take a few epochs to adjust the distribution. Starting from the 30th epoch, SFT converges and all ReFT variants are still improving. We can also see that all variants outperform SFT by a significant margin and there is no obvious advantage of any specific ReFT variant.

heading: 6 Conclusion
text: We have introduced reinforced fine-tuning (ReFT) as a new method for fine-tuning models to solve math problems. In contrast to SFT, ReFT optimizes a non-differentiable objective by exploring multiple CoT annotations in the search for the correct answer, rather than relying on a single annotation.
14We only show 60 epochs for illustration purposes. The performance for the later epoch is shown in Figure 4 (b).
Through extensive experimentation on three datasets using two foundation models, we have demonstrated that ReFT outperforms SFT in terms of performance and generalization ability. Moreover, we have showcased the compatibility of models trained with ReFT with techniques such as majority voting (Wang et al., 2023b) and reward model reranking (Cobbe et al., 2021a; Uesato et al., 2022).
Furthermore, ReFT has exhibited superior performance compared to several publicly available open-source models of comparable sizes in math problem-solving. This demonstrates the effectiveness and practical value of the ReFT approach.

heading: 7 Future Work
text: We have made the first attempt of applying reinforcement learning, specifically the PPO algorithm (Schulman et al., 2017), to fine-tune of LLMs for math problem-solving. Our future work includes utilization of offline reinforcement learning techniques (Levine et al., 2020; Gulcehre et al., 2023), development of a warm-up free method to enhance training efficiency and performance, thereby reducing the gap with the reranking method. Additionally, Lightman et al. (2023) suggests that a well-trained process-based reward model (PRM) can significantly enhance performance. Hence, it would be worthwhile to explore the implementation of process-based rewards in reinforcement learning training. Lastly, as ReFT is a versatile approach, we intend to apply it to more general reasoning tasks where the inference can be formalized with CoT.
Limitations
Training Efficiency As depicted in Figure 4 (b), it is evident that ReFT necessitates a greater number of epochs to reach convergence compared to SFT. This is primarily due to the fact that ReFT optimizes a non-differentiable objective and requires exploration of the generation space to attain correct answers. While a larger learning rate may expedite convergence, it also makes the policy more susceptible to instability and potential collapse. Alternatively, using a larger batch size is a viable option; however, it comes at the expense of increased computational costs.
Reward Hacking Our reward function relies solely on the final answer to determine the reward. However, as demonstrated in the experiments conducted on the MathQAMCQ N-CoT dataset, the policy can be easily manipulated if the possible space of final answers is limited, such as A,B,C,D. To mitigate the issue of reward hacking, it may be necessary to employ a more detailed or process-based reward function that takes into account a broader range of factors.

heading: A Examples of N-CoT and P-CoT Representations
text: We present examples of natural language CoT and program-based CoT from GSM8K dataset in Fig-
ure 6. We follow Jie et al. (2023) to perform fewshot prompting and obtain the CoT representations. The natural language CoT is generally the same as the one presented in Wei et al. (2022). The format program-based CoT is similar to the one in PAL (Gao et al., 2023), where we use a function to solve the problems. The function starts with a Python docstring that repeats the question and then a list of statements as reasoning steps.

heading: B Detailed Hyperparameter Setting
text: Supervised Fine-Tuning We train the model for 40 epochs with the batch size of 48 and the maximum length of 1024.. For small models, we increase the learning rate to 2e-5, and the number of epoch for training MathQAMCQ to 100 epochs.
ReFT Warm-up We perform warm-up for 2 epochs on GSM8K, SVAMP for both N-CoT and P-CoT. For MathQAMCQ, we perform warmup for 5 epochs on MathQAMCQ N-CoT and 2 epochs on MathQAMCQ P-CoT. Specifically for MathQAnumeric, we perform warm-up for 10 epochs because this dataset is much harder and the number of reasoning chains is longer than other datasets. For Galactica-125m and Codegen-350M, the warmup period is 10 epochs for GSM8K and SVAMP and is 40 epochs for MathQAMCQ. For Code-parrot, we increases the warm-up period to 40 epochs on all datasets to obtain reasonable warm-up performance.
ReFT RL The maximum length for question is set to 300, and the maximum length during sampling is set to 700. The batch size is 32, which is smaller than SFT due to extra memory consumption of the value model. The number of updates per RL step (i.e., ppo epoch) is set to 2 (Ziegler et al., 2019). We do not employ any weight decay and dropout following Ziegler et al. (2019). For small models, we train for 700 epochs with the learning rate of 3e-6, the global batch size of 256 and the α of 5, 1 and 0.1 for Galactica-125m, Codeparrotsmall and Codegen-350M model respectively.

heading: C Case Study
text: We show how SFT and ReFT evolve by investigating the generated P-CoT for a specific question. Figure 7 reports the responses of SFT and ReFT at checkpoint epoch 1, 3 and 5. At epoch 1, ReFT is in warmup stage so that its generated P-CoT looks similar to that of SFT. However, later at epoch 3 and 5, the P-CoTs differ. ReFT responses becomes shorter and correct while SFT reponses remains incorrect.

heading: D Attempts with DPO and IPO
text: In addition to the PPO algorithm in this work, we initially experimented with DPO (Rafailov et al., 2023) and IPO (Azar et al., 2023) in our early attempts on the GSM8K dataset. We obtained the preference data by sampling from the warmup checkpoint and adapted the implementation of these algorithms from the Transformer Reinforcement Learning (TRL) framework (von Werra et al., 2020). Preliminary results indicate that their performance is on par with the "Offline Self-Learning" baseline on GSM8K. This could be explained by the following reasons. Being offline methods, DPO and IPO cannot explore more CoT paths by themselves, thus, their performances are limited by the quality of data sampled from an sub-optimal policy. In addition, their policies are set to maximize the reward given by their implicitly trained reward models. Our preference data, which is sampled from the train set, might not be ideal for reward modeling as there may be neither incorrect responses for easy questions nor correct responses for hard questions. On the other hand, PPO not only is an online method that explores CoT solution paths automatically, but it also does not require reward model training as the reward function can be directly used. Thus, PPO is chosen in our later experiments.

heading: Future Work
text: Our future work includes utilization of offline reinforcement learning techniques (Levine et al. , 2020; Gulcehre et al. , 2023), development of a warm-up free method to enhance training efficiency and performance, thereby reducing the gap with the reranking method. Additionally, Lightman et al. (2023) suggests that a well-trained process-based reward model (PRM) can significantly enhance performance. Hence, it would be worthwhile to explore the implementation of process-based rewards in reinforcement learning training. Lastly, as ReFT is a versatile approach, we intend to apply it to more general reasoning tasks where the inference can be formalized with CoT. Limitations
Training Efficiency As depicted in Figure 4 (b), it is evident that ReFT necessitates a greater number of epochs to reach convergence compared to SFT. This is primarily due to the fact that ReFT optimizes a non-differentiable objective and requires exploration of the generation space to attain correct answers. While a larger learning rate may expedite convergence, it also makes the policy more susceptible to instability and potential collapse. Alternatively, using a larger batch size is a viable option; however, it comes at the expense of increased computational costs. Reward Hacking Our reward function relies solely on the final answer to determine the reward. However, as demonstrated in the experiments conducted on the MathQAMCQ N-CoT dataset, the policy can be easily manipulated if the possible space of final answers is limited, such as A,B,C,D. To mitigate the issue of reward hacking, it may be necessary to employ a more detailed or process-based reward function that takes into account a broader range of factors