Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Only
Authors: Qingru Zhang
Liang Qiu
Ilgee Hong
Zhenghao Xu
Tianyi Liu
Shiyang Li
Rongzhi Zhang
Zheng Li
Lihong Li
Bing Yin
Chao Zhang
Jianshu Chen
Haoming Jiang
Tuo Zhao
Venue: COLM 2025 (2025)
Keywords: Alignment with Demonstration
Self-Rewarding PPO
Large Language Models
TL;DR: We propose Self-Rewarding PPO, a novel fine-tuning method that combines the strengths of SFT and proximal policy optimization (PPO) to achieve more effective alignment from demonstration data.

Abstract: Supervised fine-tuning (SFT) has emerged as a crucial method for aligning large language models (LLMs) with human-annotated demonstrations. However, SFT, being an off-policy approach similar to behavior cloning, often struggles with overfitting and poor out-of-domain generalization, especially in limited-data scenarios. To address these limitations, we propose Self-Rewarding PPO, a novel fine-tuning method that leverages on-policy techniques to enhance generalization performance. Our approach combines the strengths of SFT and proximal policy optimization (PPO) to achieve more effective alignment from demonstration data. At its core is a reward function designed as the log policy ratio between the SFT model and the pretrained base model. This function serves as an implicit reward signal, using the pretrained policy as a baseline and the SFT policy as a target. By doing so, it enables on-policy fine-tuning without relying on human preference annotations. The integration of this self-rewarding mechanism with PPO addresses key limitations of SFT, improving generalization, data efficiency, and robustness. Our empirical evaluation across a range of natural language processing tasks demonstrates that Self-Rewarding PPO consistently outperforms traditional SFT methods. The results highlight the effectiveness of our approach in aligning LLMs using demonstration data, particularly in scenarios where high-quality annotated data is scarce.

Published as a conference paper at COLM 2025
Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Only
Qingru Zhang"; Liang Qiu,", llgee Hong", Zhenghao Xu', Tianyi Liu", Shiyang Li", Rongzhi Zhang", Zheng Li Lihong Li Bing Yin", Chao Zhang", Jianshu Chen', Haoming Jiang", Tuo Zhao
*Georgia Institute of Technology Amazon {qingru. zhang, tourzhao (earzhaoo@@@gteeaeo @egatch.. edu
Abstract
Supervised fine-tuning (SFT) has emerged as crucial method for align- ing large language models (LLMs) with human-annotated demonstrations. However, SFT, being an off-policy approach similar behavior cloning, often struggles with overfitting and poor out-of-domain generalization, es- pecially in limited-data scenarios. To address these limitations, we propose Self-Rewarding PPO, novel fine-tuning method that leverages on-policy techniques to enhance generalization performance. Our approach combines the strengths of SFT and proximal policy optimization (PPO) to achieve more effective alignment from demonstration data. At its core is a reward function designed the log policy ratio between the SFT model and the pretrained base model. This function serves as an implicit reward signal, using the pretrained policy By doing so, it enables on-policy fine-tuning without relying on human preference annotations. The integration of this self-rewarding mechanism with PPO addresses key limitations of SFT, improving generalization, data efficiency, and robustness. Our empirical evaluation across range of natural language processing tasks demonstrates that belf-Rewar PPO consistently outperforms traditional SFT methods. The results highlight the effectiveness of our approach in aligning LLMs using demonstration data, particularly in scenarios where high-quality annotated data is scarce.
baseline and the SFT policy as target.
1 Introduction
Large language models (LLMs) exhibit remarkable performance in various tasks ranging from text generation to complex reasoning (e.g., Brown et al., 2020; Touvron et al., 2023; OpenAT, 2023). Their ability to generate coherent and contextually relevant text has enabled breakthroughs in areas such as creative writing, code generation, and conversational Al (Chen et al., 2021; Thoppilan et al., 2022; Bubeck et al., 2023; Anil et al., 2023). However, aligning these models to ensure both safety and utility remains critical challenge.
Alignment refers to shaping model behavior to adhere to human values while avoiding harmful, biased, or unhelpful outputs (Bai et al., 2022a; Ganguli et al., 2022). The alignment process typically consists of two stages: (i) supervised fine-tuning on demonstration data, where models are fine-tuned on pairs of prompts and responses generated by experts (human or Al) to mimic desired behaviors (Wei et al., 2021; Chung al., 2022; Zhou et al., 2023a; Tunstall et al., 2023); and (ii) preference learning, where preference data is used to learn reward model, which is in turn used by reinforcement learning (RL) step to fine-tune the model (Christiano et al., 2017; Ouyang et al., 2022; Stiennon et al., 2020; Bai et al., 2022b). In this work, focus the first stage: aligning language models from demonstration data without relying on preference annotations.
Supervised fine-tuning (SFT) has become the de-facto approach for learning desired behav- iors from human-annotated demonstrations. This objective aligns closely with imitation
Work completed during Qingru Zhang's internship Amazon.
1

---

Published as a conference paper at COLM 2025
learning (Hussein et al., 2017; Osa et al., 2018). By maximizing the likelihood of expert's behaviors shown in the demonstration data, SFT is equivalent to performing behavior cloning, where the model aims to mimic the demonstrated actions (Bratko et al., 1995; Torabi et al., 2018; Florence et al., 2022). However, as an off off-policy approach akin to behavior cloning, SFT typically relies substantial volumes of high-quality data to achieve robust performance (Dubey et al., 2024). In the scenarios of limited data, SFT often suffers from overfitting to the training distribution, particularly with prolonged training, which hurts the model generalization on unseen examples and domains (Zhang et al., 2024; Chen et al., 2024).
Stage SFT
Darive
Param (y/m))
Cohurent Reward
log Perro (y(z)
Policy
Demonstration
Reward Biiluate
Input
Stage 2: PPO
Align
PPO Policy
Optimization
Nlgned
On-policy
Sampling
Prompt
Self-Rewarding PPO
Figure 1: Illustration of Self-Rewarding Proximal Policy Optimization (SRPPO). SRPPO defines a reward based on the log-density ratio between the supervised fine-tuned (SFT) and pretrained policies. It then applies the reward to further fine-tune the SFT policy by proximal policy optimization. SRPPO leverages on-policy data to improve the out-of-distribution generalization of the learned policy.
Notably, on-policy training methods, such as proximal policy optimization (PPO, Schulman et al. (2017)), have been widely adopted in preference learning stage (Ouyang et al., 2022; Bai et al., 2022b). Unlike SFT, PPO generates diverse on-policy samples for fine-tuning. It enhances data diversity and makes training process adapt to model's evolving behaviors, improving generalization performance. Their success on preference learning motivates us to explore if on-policy training techniques can be leveraged to benefit SFT. However, key challenges lies in deriving a meaningful reward signal from the data to enable on on-policy training. A few recent studies attempt to address this challenge from two perspectives.
On one hand, inspired by advances in imitation learning, ei al. (2024) and Sun & van der Schaar (2024) employ inverse reinforcement learning (IRL. (Ziebart et al., 2008; Ho & Ermon, 2016; Ghasemipour et al., 2020), to explicitly learn a reward model. While effective, these methods require training both the policy and reward models using a bi-level optimiza- tion framework, introducing significant complexity in terms of convergence and training stability.
On the other hand, SPIN (Chen et al., 2024) bypasses training an explicit reward model by assuming that the responses in demonstrations are always preferred over the model's on-policy samples. It applies DPO (Rafailov et al., 2024b) to fine-tune the model. As it does not involve an explicit reward, SPIN cannot accommodate additional prompts beyond those in the demonstrations during the training. Meanwhile, the preference assumption does not always hold, potentially leading to performance degradation. Despite these efforts, the design of meaningful reward signals from demonstrations remains largely unexplored and challenging.
In this study, propose Self-Rewarding PPO (SRPPO), novel fine-tuning method that bridges the gap between supervised fine-tuning and reinforcement learning fine-tuning, achieving more effective and robust alignment from demonstration through on-policy
2

---

Published as a conference paper at COLM 2025
training. At the core of our method, we propose a reward function named coherent reward that is designed as the log policy ratio between the supervised fine-tuning model (SFT policy) and the pretrained base model (pretrained policy). Specifically, two steps. First, we perform SFT to fine-tune the pretrianed base model using high-quality demonstrations pairs of prompts and desired responses. Next, we derive the coherent reward from the SFT and pretrained policies. Guided by the coherent reward, we apply PPO to continuously fine-tune the model using set of prompts. Figure illustrates the two stages of our method.
approach consists of
Unlike existing IRL methods Li et al. (2024), SRPPO eliminates the need to train reward model. Instead, the coherent reward is derived directly from the SFT policy, which is the same model to be trained, offering simple but effective self-rewarding mechanism. This reward design is inspired by coherent soft imitation learning (Watson et al., 2024). Coherent reward leverages the pretarined policy as a baseline and the SFT policy as a target. It establishes a training direction that pushes the model's behavior from the pretrained baseline to the mid-aligned SFT policy. The PPO stage then refines the model further along this alignment direction with on-policy sampling. Moreover, compared to SPIN, SRPPO allows the use of additional prompts beyond those in the demonstration data during the PPO step. This flexibility is particularly advantageous in scenarios where high-quality responses are scarce, but obtaining abundant similar prompts is relatively easy. such cases, a small amount of high-quality demonstration data can be used during SFT to establish an alignment direction that is captured by the coherent reward. Subsequently, during the PPO fine-tuning step, additional prompts can be utilized to sample more on-policy responses, which are then evaluated by the coherent reward, further refining the model along the established direction. Empirically, we observe that the coherent reward generalizes effectively from small set of representative demonstrations to broader range of prompts (Section 4). Therefore, Self-Rewarding PPO not only augments training data with on-policy samples, but also allows the use of additional prompts to enhance alignment.
We conduct experiments to demonstrate the effectiveness of Self-Rewar PPO using LLAMA3-SB and Mistral-7B our base model. Empirical results show that SRPPO signifi- cantly enhances thee-tnnigg performance compared to SFT and other alternative methods across various evaluation benchmarks, demonstrating its effectiveness in improving model alignment and generalization.
2 Background
Consider a language model parameterized by 0 and denote its output probability (or policy) by pe(y)x), where [x is the sequence of input prompts and y [y,,..., , is the sequence of output responses. LLMs are typically auto-regressive models: they generate tokens one-by-one, and predict the output probability of y given tokens in and yzj Ly.....,yjT yi-1 (y+1 is null):
po(y)x) = N Po(y,)
This process constitutes Markov decision process (MDP), where the state transitions are deterministic and the model generates tokens sequentially at every given position, leveraging only the sequence of previous tokens. In the following, we discuss two common procedures for fine-tuning 0: (i) supervised fine-tuning (SFT) over a demonstration dataset, and (ii) reinforcement learning with human feedback (RLHF) over a preference dataset.
SFT. Supervised fine-tuning (SFT) aligns or adapts pre-trained LLM to specific tasks, (e-g., instruction following, code generation, math reasoning). This process relies on demonstration dataset D {xx y)) that comprises prompts x sampled from the task distribution p, and their responses annotated by experts (expert( SFT uses a maximum- likelihood objective:
max (SF = (x [log Po ())xx))
(1)
Clearly, the above problem shares the same optimal solution as ming Exx<ppDKe (Pexpert (-1xx) p- (0) attains its optimum when the model
3

---

Published as a conference paper at COLM 2025
Po aligns perfectly with the expert behavior. So the fine-tuned model is expected to generate responses that resemble those of the expert. Therefore, SFT is closely related to imitation learning Osa et al. (2018), whose goal is to mimic the policy of an expert.
RLHE RL fine-tuning over a preference dataset is the second stage of aligning LLMs, after the SFT stage. Suppose we have deterministic reward model r(x,y) y) that evaluates given prompt-response pair (x, RLHF fine-tunes the model by solving the following RL problem:
max LRL (0) -Ex-pxy-po(txx (x)
(2)
-AExpp (po(((o(-(xx))) Prer(-1(x)) Prer- (-|[x)]],
where Pref a reference model. Due to the intractability of computing the KL regularization over all possible outputs y, (2) is typically solved by policy optimization techniques such as REINFORCE (Williams, 1992; Ahmadian al., 2024) and PPO (Schulman et al., 2017).
To obtain reward model r(x,y) RLHF often assumes preference dataset M = {x, w Vi where each data contains a pair of output (y yj) for prompt x. Here, yı is preferred over y/ by human annotator, denoted as V y, (Christiano et al., 2017; Ouyang et al., 2022). The Seaaley-Terry model (Bradley & Terry, 1952) is used to model the probability of choosing y/ over y/-
Pyy > y11x) = o(r(x, yo.) - r(x, y;)),
where o(-) is the sigmoid function. The reward model is trained with the following objective:
max RM = IB [(x.,...))-- log (ar (r(x, (rxx,,))(xxxx,)))))
((>-)
It is widely shown that models trained by episodically learning the policy (2) and learning the reward often outperforms those that are only trained using SFT (Ouyang et al., 2022). The reward model guides the performance of the LLM and allows a better generalization ability due to incorporating additional preference data from human annotator.
Discussion. In this study, we focus on aligning LLMs using demonstration data. As shown in (1), SFT is an off-policy approach akin to behavior cloning, where the model is fine-tuned to mimic expert behavior solely based on the provided data. Consequently, it often suffers from overfitting to training distribution, leading to subpar out-of-domain generalization. In contrast, RL fine-tuning in (2) is an on-policy method that samples responses directly from the model's 's current policy and optimizes it to maximize the reward of subsequent samples. This ada odabtaiiyyty allows the training process to align with the model's 's evolving behavior, resulting in improved generalization and robustness. Motivated by this, the paper studies the following question:
Can we leverage on-policy training techniques to bridge the gap between SFT and RL fine-tinun, thereby enhancing alignment from demonstrations only?
In next section, we delve into this prospect and address this question with our solution.
3 Method
Our proposed method, Self-Rewarding PPO (SRPPO), combines the strengths of both SFT and RL fine-tuning. At the core of SRPPO, we introduce novel reward function, Coherent Reward, which establishes an alignment direction coherent with supervised fine-tuning stage, thereby enabling continuous refinement through RL fine-tuning.
3.1 Coherent Reward
To enable on-policy training using demonstration data, we propose Coherent Reward, novel reward function derived as the log policy ratio between the initial pretrained model (pretrained policy PORD)) and the model fine-tuned on demonstrations (SFT policy POSET) Specifically, for any pair (x, y), the coherent reward is defined
F F(x,y) = llog PalSFT; (y|x)
log I1 P SEFT x,
(3)
POITT) (y/xx
1 - P PT (1, yzj
4

---

Published as a conference paper at COLM 2025
Our coherent reward is inspired by the coherent soft imitation learning method Watson et al. (2024). Intuitively, it leverages the pretrained policy as a baseline and the SFT policy as target. For a pair of prompt and response, it quantifies the divergence between two policy on this pair, thereby establishing training direction that transitions the model's behavior from the pretrained baseline to the mid-aligned SFT policy. For a given prompt-response pair, the reward quantifies the divergence between these two policies on the pair, thereby establishing a training direction that transitions the model's behavior from the pretrained baseline to the mid-aligned SFT policy. We then leverage this reward for subsequent RL fine-tuning, ensuring that the RL fine-tuning effectively builds upon the improvements of SFT stage, and further refine the model along this alignment trajectory.
3.2 Self-Rewarding PPO
As illustrated in Figure 1, our Self-Rewarding PPO method consists of two sequential training stages:
1. Supervised fine-tuning: Given demonstration dataset D { (x, y) fine- tune a pretrained base model POPTT on D by optimizing (1), and obtain the SFT policy PatSFT) from which we derive the coherent reward F as in (3).
2. RL fine-tuning: Given a prompt set P (y we further perform the on-policy RL fine-tuning to continuously refine the SFT policy by optimizing the objective (2), where the reward value our coherent reward.
For the RL fine-tuning stage, we use PPO as the policy optimization algorithm. Please see Appendix B for the details of PPO. Notably, other algorithms such as REINFORCE (Williams, 1992), GRPO (Shao et al., 2024), or RLOO (Ahmadian et al., 2024) can also be applied as alternatives.
When employing PPO to fine-tune the model, we treat states containing an [EOS] token as absorbing states. We assign the coherent reward the process level as defined in (4). Alternatively, we can revise the process-level coherent reward to token-wise reward
r(y)1 x, yer)) log PSSF, (y)lxye))) and assign it token level, which we discuss further in Appendix E.
Pa xyy
r (uy|x,ye,) - f(x,y) yf [OOS] or jrj=,,
(4)
otherwise.
Our coherent reward is straightforward yet meaningful. It offers a simple and effective self-rewarding mechanism for on-policy alignment training from demonstrations, without requiring rewarding learning or inverse reinforcement learning. Its advantages mainly stems from two key perspectives regarding on-policy responses y ~ Po( xx) and input prompts x ~ p. First, the coherent reward is derived from the same model being fine-tuned (i.e. SFT policy). Notably, during RL fine-tuning, on-policy responses y are sampled from the same model. Consequently, the coherent reward becomes inherently sensitive to variations in the model's own responses y Po(- [x). Compared to independent reward models, the coherent reward can capture subtle changes in y, providing more accurate and adaptive reward evaluations. Second, our method enables the inclusion of additional prompts beyond those in the demonstration dataset. This is substantial advantage compared to methods like SPIN (Chen al., 2024) that rely on DPO and are limited to demonstration prompts. This flexibility can be particularly beneficial when high-quality, human-annotated responses are scarce but similar prompts can easily obtained. In such cases, SFT is prone to overfitting, In contrast, SRPPO allows us to first fine-tune the model with small amount of demonstration data to establish an alignment direction. Subsequently, we can sample more prompts from task distribution continue refining the model along the alignment direction given by the coherent reward. In Section 4, empirically show how the coherent reward generalizes effectively from a small amount of demonstration data to a broader range of prompts. We summarize Self-Rewarding PPO in Algorithm 1.
p, and utilize them and their on-policy samples to
5

---

Published as a conference paper at COLM 2025
Table 1: Fine-tuning results of Mistral-7B under the minimum overlap setup. SFT con- ducted with Tulu-v2 only. The best results are shown in bold. The average score is calculated by first averaging two scores each tasks, and taking averaging of them across four tasks.
IFeval L.Acc / S.Ace
GSM8K
GPQA CoT EM Non-CoT EM LC win rate Win rate Ave.
AlpacaEval
All
Method
EM
Pretrain Baseline
30.58 / 29.38
37.3
12.50 / 27.23
0.07
0.12
21.81
SFT
42.45 / 40.53
46.47
23.88 / 26.34
8.95
4.60
29.96
SFT (Extended)
39.21 / 35,49
29.04
16.74 / 29.46
9.75 / 4.97
24.22
SPIN
45.08 / 38.73
42.99
19.87 / 26.56
5.81 4.29
28.29
SRPPO
47.60
41.37
46.93
24.33 / 26.56
12.47 13.23
32.43
Table 2: Fine-tuning results of LLAMA3-8B under the minimum overlap setup. SFT is conducted with Tulu-v2 only. The best results are shown in bold.
IFvval
GSM8k
GPQA CoT EM / Non-CoT EM LCC WR / WR
AlpacaEval
All
Method
L.Ace / S.Acc
EM
Ave.
Pretrain Baseline
28.30 / 26.85
50.11
12.95 / 27.23
0.23 / 0.25
24.50
SFT SFT (Extended)
28.42 / 28.30
47.69
25.67 29.69
9.48 / 4.97
27.74
34.77 / 32.13
45.19
16.74 / 32.14
10.41 5.34
27.74
PPO w/ preference RM
31.06 / 30.22
48.90
SRPPO
41.49
37.41
51.10
18.30
30.13
11.86
7.95
31.17
4 Experiments
We evaluate the effectiveness of Self- PPO by fine-tuning the pretrained Mistral- 7B (Jiang et al., 2023) and LLAMA3-8B (Dubey et al., 2024) models. The evaluation covers diverse set of benchmarks, including instruction following (IFEval, Zhou et al. (2023b)), math reasoning (GSMSK, Cobbe et al. (2021)), graduate-level question answering (GPQA, Rein et al. (2023)), and conversational ability (AlpacaEval, Dubois et al. (2024)). Our experiments highlight the following advantages of SRPPO:
Improved performance without additional human annotations: Without introducing new numan-annotated data, SRPPO significantly enhances model performance across wide range of evaluation benchmarks compared to SFT and other alternatives.
Effective generalization to additional prompts: SRPPO enables the inclusion of additional prompts for RL fine-tuning, thus further improves model performance. This demonstrates that the coherent reward can effectively generalize from human-annotated demonstration data to a broader range of prompts, enhancing alignment without new annotations.
4.1 Experimental Setup
Models and Datasets. We adopt the pretrained Mistral-7B (Jiang et al., 2023) and LLAMA3- 8B (Dubey et al., 2024) as our base models, and then conduct the supervised and RL fine-tuning to evaluate our method. For training, we leverage two datasets: TULU-v2-mix (Ivison al., 2023) and UltraFeedback (Cui et al., 2024). TULU-v2-mix is mixed collection of high-quality instruction datasets, comprising 326k examples from 11 diverse sources. UltraFeedback is large-scale, fine-grained preference dataset containing 64k examples that are realted to aspects of instruction-folloiing truthfulness, honesty, and helpfulness. Since TULU-v2-mix is high-quality dataset known for consistently improving model capabilities across various tasks, we primarily use it for supervised fine-tuning, ensuring an initial alignment of the models. To evaluate the generalization of our coherent reward beyond the demonstration data, use prompts from UltraFeedback during the PPO fine- tnnettunngg stage. This setup allows us to examine how well our reward mechanism transfers new prompts without additional human annotations.

---

Published as a conference paper at COLM 2025
Since the coherent reward is derived from the SFT policy, the choice of SFT training data is crucial to its effectiveness. Empirically, we find that overlapping the SFT demonstration data with the PPO prompt data helps derive more robust coherent reward. To systematically evaluate the generalization of our approach, we consider the following experimental setups for selecting SFT training data in SRPPO:
1. Minimum overlap: We conduct SFT only on TULU-v2-mix, ensuring minimal overlap between the SFT training pairs and PPO training prompts. This setup designed to assess the generalization capability of SRPPO when the PPO stage encounters prompts not seen during SFT. Tables and 2 presents results in this setting.
2. Medium overlap: To introduce a controlled degree of overlap, we sample 9k prompts from Ultral'eedback and annotate them with high-quality responses generated by GPT-4. The models are first fine-tuned using TULU-v2-mix, followed by additional fine-tuning on this small subset of UltraFeedback demonstrations. Table 3 shows the results.
3. Diminished overlap: We first fine-tune the models using TULU-v2-mix to establish an initial alignment. We then conduct additional supervised fine-tuning using both the small UltraFeedback demonstration subset and an additional 40k examples from TULU-v2-mix to further refine the models. Table 4 presents the results of this setup.
These setups allow us to analyze how different levels of training data overlap influence the effectiveness of the coherent reward.
Evaluation. We evaluate model performance across different perspectives, including instruction following (IFEval, Zhou al. (2023b)), math reasoning (GSM8k, Cobbe et al. (2021)), graduate-level question answering (GPQA, Rein et al. (2023)), and conversational ability (AlpacaEval, Dubois et al. (2024)). For IFEval, we report both instruct-level loose (L.Accc) and strict (S.Acc) accuracies. For GSM8k, we evaluate the 5-shot performance and report the exact match (EM). For GPQA, we assess both few-shot and few-shot Chain-of- Thought (CoT) performance (Wei et al., 2022b). These evaluations are conducted using the lim-evaluation-harnee framework (Gao et al., 2024) under its default settings. For AlpacaEval, we report length-controlie win-rate and overall win-rate.
Implementation Details. We use PyTorch (Paszke et al., 2019) to implement all the algo- rithms. Our implementation is based on the publicly available Huggingface Transformers (Wolf et al., 2019) and OpenRLHF (Hu et al., 2024) code-base. All the experiments are conducted on NVIDIA A100 GPUs
Regarding the hyperparameters of SFT, we set the batch size as 128 and trainig epochs as 2, choose the learning rates from (1 x 10-5,5 10-6, 1 10-6,5 5, x 10-?), and learning rate for both SRPPO and baseline methods. For the hyperparameters of PPO, we set the rollout buffer size as 1024, the training batch size as 128, the KL coefficient as 0.2 or 0.2. We initialize the critic model from the SFT policy, set its learning rate as 9 x 10 and warmup the critic fine-tuning for 35 rollout buffers. We then fine-tune the actor model for 2 episodes and select the actor learning rates from {5 x 10-8, 2 x 10 10-8, x 10 10-83.
and pick the optimal
0.5, and the clipping coefficient
Baselines. We compare Self-Rewarding PPO with the following methods:
SFT: It is the standard approach for aligning LLMs with demonstration data that optimizes (1). SFT is also the model from which SRPPO continues fine-tune. We the training epochs to 2, as this typically yields the best performance. Given different training data, we compare SRPPO against its SFT-only stage to showcase the effectiveness of PPO fine-tuning with our coherent reward.
SFT (Extended): This baseline extends the fine-tuning of the SFT policy by running ad- ditional SFT epochs. Specifically, it starts from the SFT policy and undergoes further fine-tuning, e.g., for a total of epochs, to examine how prolonged SFT affects model performance.
Itttps://githi htttta///giiuuuu..///ggggggfeeee http://wwwwwgggggggae//tesnsrrea
7

---

Published as a conference paper at COLM 2025
Table Fine-tuning results of Mistral-7B and LLAMA3-8B under the medium overlap setup. SFT is conducted with Tulu-v2 and small number of demonstrations of Ultrafeedback. The best results are shown in bold.
IFeval
GSMSK EM
GPQA
AlpacaEval
All
Model
Method
L.Acc / S.Ace
CoT EM / Direct EM LC WR / WR
Ave.
Pretrain Baseline
30.58 / 29.38
37.3
12.50
27.23
0.07
0.12
21.81
Mistral-7B
SFT w/ subset
46.40 / 43.05
35.18
18.75 / 25.45
22.63
16.21
30.36
SRPPO
49.40 / 44.00
41.39
20.31 / 27.23
21.73
21.18
33.33
Pretrained Baseline
28.30 / 26.85
50.11
12.95 / 27.23
0.23 0.25
24.50
LLAMA3-8B
SFT w/ subset
31.53 / 30.58
47.31
20.31 / 32.59
19.72
14.29
30.46
SRPPO
45.44 / 40.17
53.22
20.09 / 31.47
19.20
23.48
35.79
Table 4: Fine-tuning results of Mistral-7B under the diminished overlap setup. We first fine- tune the models using TULU-v2-mix, and then conduct additional SFT using both the small UltraFeedback demonstration subset and additional 40k examples from TULU-v2-mix to refine the model.
IFeval
GSM8k
GPQA CoT EM / Direct EM LC win rate Win rate
AlpacaEval
All
Method
L.Acc / S.Acc
EM
Ave.
Pretrain Baseline
30.58 / 29.38
37.30
12.50 / 27.23
0.07 0.12
21.81
SFT
47.36 / 44.36
27.52
15.85 / 27.23
13.57 / 7.52
26.37
SRPPO
48.56
44.72
30.55
19.87 / 27.23
15.59 / 9.01
28.26
PPO with an independent preference reward model: Instead of using our coherent reward, this baseline employs publicly available preference reward model [Pstirx-LLAMMA-RMM Dong et al., 2024) for PPO fine-tuning after the SFT stage. This baseline showcases the comparison between our coherent reward that does not rely on additional preference data, and reward models trained from preference data.
SPIN (Chen et al., 2024): This baseline is self-play-based fine-tuning method where the same target LLM generates synthetic data and critiques its own outputs.
4.2
Main Results
We present our main results in Tables 1 and 2. This corresponds to the first setup of minimum overlapping for selecting SFT data, where we perform SFT only on TULU-v2-mix and then apply PPO using UltraFeedback prompts. As illustrated, SRPPO consistently outperforms baseline methods IFEval, GSM8k and AlpacaEval, achieving the best overall average scores for both Mistral-7B and LLAMA3-8B While SFT (Extended) achieves the best Direct EM GPQA, SRPPO still shows competitive performance. Particularly, extending SFT training to more epochs improves performance some domains, such as question answering and instruction following. However, prolonged SFT also leads to overfitting to the training distribution, negatively impacting out-of-domain generalization. As observed in Table 2, extending SFT TULU-v2-mix to 6 epochs (compared to 2 epochs) improves accuracy on IFEval and GPQA, likely due to the fact that most TULU-v2-mix examples belong to similar domains. However, this extended training reduces performance on math reasoning (Tables 1 and 2). This result validates our argument in Section that prolonged SFT tends to cause overfitting, thereby hurting generalization to out-of-domain tasks. In contrast, SRPPO enhances model generalization, yielding performance gains across all four tasks. As shown in Table 2, SRPPO significantly improves performance on both instruction following and math reasoning, it effectively enables on-policy training additional prompts. These results indicate that SRPPO can effectively leverage additional prompts to enhance model capabilities without preference annotations.
Furthermore, as seen in Table 2, PPO with an independent preference reward model provides only marginal improvements over SFT, whereas SRPPO consistently outperforms SFT across
8

---

Published as a conference paper at COLM 2025
all benchmarks. This suggests the effectiveness of our self-rewarding mechanism. Since the coherent reward is derived directly from the SFT policy, it is inherently sensitive to variations in the model's own responses y ~ Pø during RL fine-tuning. Compared to independent reward models (Table 2), we hypothesize that the coherent reward can capture subtle changes in y, enabling more accurate and adaptive reward evaluations. Additionally, we compare SRPPO with SPIN in Table 1. Similar to SRPPO, SPIN is initialized from the SFT policy. However, unlike SRPPO, SPIN conducts its first-iteration DPO using the full 350k examples from TULU-v2-mix, dataset significantly larger than the one used by SRPPO. Despite this data advantage, while SPIN improves upon SFT, SRPPO consistently outperforms SPIN, demonstrating its effectiveness as fine-tuning method for alignment with demonstrations.
Tables 3 and 4 present the results for the setups of medium overlap and diminished overlap (c.f., Section 4. 4.1), respectively. In Table 3, during the SFT stage, we first fine-tune the models on large-scale TULU-v2-mi to establish their basic capabilities, followed by additional refinement using a 9k subset of high-quality demonstrations, where prompts are sampled from UltraFeedback and responses are annotated by GPT-4. This results in a medium overlap between the SFT prompts and the PPO prompts. We observe that SFT with this 9k subset significantly improves instruction-followii degrades math reasoning due to the scarcity of math-related data in this subset, inducing overfitting to training domains. In contrast, SRPPO effectively mitigates this issue, recov- ering math reasoning performance and yielding a substantial 4.06% EM improvement for Mistral-7B, which is consistent with observations in Tables 1 and 2. More importantly, SRPPO further enhances IFEval accuracy, as the prompt overlap leads to coherent reward that exhibits better generalization on PPO prompts. In the diminished-overlay setup, we begin with the SFT policy from the medium-overlap setup and further fine-tune it using 40k additional samples from TULU-v2-mix This step reduces the overlapping effects introduced by the 9k UltraFeedback demonstration subset. Even under this setup, SRPPO consistently outperforms SFT, demonstrating that the coherent reward effectively generalizes to addí- tional prompts. These results confirm that SRPPO enhances model performance through on-policy training with additional prompts, reinforcing its ability to improve generalization without relying on preference annotations.
and conversational ability but severely
Discussions
Our coherent reward approach can be applied beyond PPO, and to other on-policy training algorithms, including REINFORCE (Williams, 1992), RLOO (Ahmadian et al., 2024), GRPÒ (Shao et al., 2024), and VinePPO (Kazemnejad et al., 2024). Additionally, it has the potential to serve as an effective evaluator, facilitating tasks such as SFT data filtering by providing principled method for assessing response quality, thereby reducing the need for human involvement in the loop.
Empirically, we observe that the effectiveness of the coherent reward can be influenced by both the SFT training quality and the generalization ability of the pretrained base model. if the pretrained model has limited generalization to unseen domains and out-of-distribution prompts, such as a small-size model like Phi-2 (Javaheripi et al., 2023), its coherent reward faces challenges to generalize on a board range of prompts. Additionally, the quality of the SFT data and SFT training process play a crucial role in determining the effectiveness of the coherent reward. If the demonstration data is of low quality or if the SFT stage suffers from overfitting, the derived coherent reward may exhibit reduced generalization capability unseen prompts.
Moreover, we also explore modifying the coherent reward (3) into a token-wise reward. However, this approach introduces challenges related to length degeneration, complicating the training process. We provide a detailed discussion in Appendix E and leave this for future exploration. Due to space limit, we dicuss the related work in Appendix C.
6 Conclusions
In this study, we introduced Self-Rewarding PPO (SRPPO), a novel fine-tuning framework that bridges the gap between supervised fine-tuning (SFT) and reinforcement learning

---

Published as a conference paper at COLM 2025
(RL) fine-tuning by leveraging coherent reward derived directly from the SFT policy. SRPPO enables on-policy fine-tuning using demonstration data alone, making it a scalable and effective approach for LLM alignment. Experimental results demonstrate that SRPPO achieves significant improvements over SFT methods and other alternatives across multiple benchmarks, showcasing its effectiveness.
References
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXio:2402.14770, 2024.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm technical report. Xiv preprint ar XrX:::3305.1044 104003 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. ar preprint aXXiv:2204.05862, 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitu- tional ai: Harmlessness from feedback. arviv preprint arXiv:2212.0807.. 2022b.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3 / 40:/4443324345, 1952.
Ivan Bratko, Tanja Urbančič, and Claude Sammut. Behavioural cloning; phenomena, results and problems. IFAC Proceedings Volumes, 28(21):143-149 1995.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.
Alex J Chan, Hao Sun, Samuel Holt, and Mihaela van der Schaar. Dense reward for free in reinforcement learning from human feedback. arXiv preprint arXio:2402.00782, 2024.
Jonathan Daniel Chang, Masatoshi Uehara, Dhruv Sreenivas, Rahul Kidambi, and Wen Sun. Mitigating covariate shift in imitation learning via offline data with partial coverage. In Advances in Neural Information Processing Systems, 2021.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained code. ar Xiv preprint ar arXiv:2107.00 033774, 2021.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine- tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024.
Hyunsoo Cho. Unveiling imitation learning: Exploring the impact of data falsity to large language model. In Findings of the Association for Computational Linguistics: ACL 2024, 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
10

---

Published as a conference paper at COLM 2025
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling essructioonffietuune language models. arv preprint ar (r1::22210..1411 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, al. Training verifiers to solve math word problems. ar preprint ar Xiv:21100. 14168, 2021.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ultrafeedback: Boosting language models with scaled ai feedback. In Forty-first International Conference on Machine Learning, 2024.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and long Zhang. Rlhf workflow: From reward modeling to online rlhf. Xiv preprint ar Xrio:2245.07863. 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. XrXiv preprint or Xrv::24007 21t.22447..2783,, 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori Hashimoto. Length-controoled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint ar axXi::2404.04475, 2024.
Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conference on Robot Learning, pp. 158-168. PMLR, 2022.
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint ar #Xiv:2209.07855 2022.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, 2023.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL https://zenodo. org//reoodd////oee.e/ 12688602.
Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence mini- persseccive on imitation learning methods. In Conference robot learning, PP- 1259-1277. PMLR, 2020.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. ar preprint Xiw::2501.12944 2025.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arx preprint arXio:2402.04799 2024.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016
Jian Hu, Xibin Wu, Weixun Wang, Xianyu, Dehao Zhang, and Yu Cao. Openrlhf: An easy- to-use, scalable and high-peeformance rlhf framework. ar Xiv preprint ar arXiv:2405.11143 2024.
11

---

Published as a conference paper at COLM 2025
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah Smith, Iz Beltagy, et al. Camels in changing climate: Enhancing Im adaptation with tulu 2. arviv preprint ar arXio:2311.10702, 2023.
Hamish Tvison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lam- bert, Noah A. Smith, Yejin Choi, and Hannaneh Hajishirzi. Unpacking DPO and PPO: Disentangling best practices for learning from preference feedback. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 1()):
3, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. ar preprint ar 2rXin:2310.06855, 2023.
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking Il potential for Ilm reasoning through refined credit assignment. arXiv preprint arXiv:2410.01679, 2024.
Jiaxiang Li, Siliang Zeng, Hoi-To Wai, Chenliang Li, Alfredo Garcia, and Mingyi Hong. Getting more juice out of the sft data: Reward learning from human demonstration improves sft for llm alignment. arXiv preprint ar #rXio22405.17888 2024.
Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. Forty-first International Conference on Machine Learning, 2023.
Andrew Y Ng. Stuart Russell, et al. Algorithms for inverse reinforcement learning. In Icml, volume 1, pp. 2, 2000.
OpenAl. Gpt-4 technical report, 2023.
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, et al. An algorithmic perspective on imitation learning. Foundations and Trends@ in Robotics, 7(1-2):1-179, 2018.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-2777 27744,, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, An- dreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-pprformance deep learning library. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' AlAch--Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 8024-8035, 2019.
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous naviga- tion. Neural computation, 3(1):88-9) 98, 1991.
12

---

Published as a conference paper at COLM 2025
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024a.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024b.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google- proof qa benchmark, 2023. URL https://arxiv. http/bb/2250 aa///////111111/////// 12022.
Stephane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Stafistics, 2011.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithmss. In Xrv preprint ar 1rXie:1707.06347, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint ar arXio:2402.03300 2024.
Joar Max Viktor Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking. Advances in Neural Information Processing Systemss, 2022.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Hao Sun and Mihaela van der Schaar. Inverse-rligmment Inverse reinforcement learning from demonstrations for Ilm alignment. arviv preprint arXi::2405.15624, 2024.
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of Ilms should leverage suboptimal, on-policy data. ar Xrv preprint ar arXiv:2404.7 14367, 2024.
Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, et al. Under- standing the performance gap between online and offline alignment algorithms, arXiv preprint ar arXxi:2245508444 2024.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. Xiv preprint ar Rtte:2201002333 2022.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. Xiv 2018.
preprint
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arxiv preprint Xix::2230 009880, 2023.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint ar XrXiv:2310.16944, 2023.
Joe Watson, Sandy Huang, and Nicolas Heess. Coherent soft imitation learning. Advances in Neural Information Processing Systems, 36, 2024.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. ar Xiv preprint ar XtX:t:21090016552 2021.
13

---

Published as a conference paper at COLM 2025
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Adounces in Neural Information Processing Systems, 2022b. URL net/forum?id= VjQIMeSB J.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist rein- forcement learning. Machine learning, 8:229-253 256,, 1992.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. ar preprint arXiv:1910.03771, 2019.
Markus Wulfmeier, Michael Bloesch, Nino Vieillard, Arun Ahuja, Jorg Bornschein, Sandy Huang, Artem Sokolov, Matt Barnes, Guillaume Desjardins, Alex Bewley, Sarah Maria Elis- abeth Bechtle, Jost Tobias Springenberg, Nikola Momchev, Olivier Bachem, Matthieu Geist, and Martin Riedmiller. Imitating language via scalable inverse reinforcement learning. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is DPO superior PPO for LLM alignment? comprehensive study. In Forty-first International Conference on Machine Learning, 2024.
Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets LLM finetuning: The effect of data, model and finetuning method. In The Twelfth International Conference on Learning Representations, 2024. URL httts:://ppnrreete wet..e..oooooummdd- 5HCnKDeTws.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less is more for alignment. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a. URL https: http://ppenrvvieeeteeeeeeeee KBMOKmX2he.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. instruction-folloiing evaluation for large language models. arXiv preprint ar Xrx::2331.00999
2023b.
Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi, Sanqiang Zhao, Kaiqiang Song, Silei Xu, and Chenguang Zhu. Wpo: Enhancing rlhf with weighted preference optimization. arXiv preprint 20o.24066.1877,, 2024.
Brian D Ziebart, Andrew L Maas, Andrew Bagnell, Anind K Dey, al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, IL, USA, 2008
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human prefer- ences. ar Xiv preprint ar 000...9999,,,,,,,, 2019.
14

---

Published as a conference paper at COLM 2025
A Detailed Algorithm of SRPPO
Algorithm 1 SRPPO
1: Input: Demonstartion dataset 'D; Prompt set P, Pretrained LLM PoirT).
2: Supervised fine-tune Pairr; using DD and obtain Pg/SFT)
3: Derive using Poirt; and Po/SFT)-
4: PPO fine-tune poISFT) using P and the reward F.
5: Output: the aligned model.
B PPO Algorithm
This section describes the PPO algorithm.
Algorithm 2 Proximal Policy Optimization (PPO)
Input:
Initial actor model Tm critic model Vg, reward function r, task prompts D, clip range e, batch size B.
Initialize πιg Tinit-
2: for iteration = 1,2,... do
3: Sample batch prompts {x:],=] D.
4: Generate responses { where y; 7 (-12;).
1B
5: Initialize batch data B
6: for index = 1,2,..., do
7:
Break down the prompt-response pair (x; y:) into state-action (prefix-next token)
pairs {(s),
8:
Query reward function to get r; = r(s; at) for E [T].
9:
Compute advantages A using GAE (5) for E [T].
10:
Collect batch data B + B U {(s), a:, A
end for
12:
Update critic parameters & by minimizing Laritie defined in (6).
13: Update actor parameters 9 by minimizing Cactor defined (7).
14: end for
Output: Trained policy model πe.
In Algorithm 2, the GAE with hyperparameters y and À is defined
A ((aA/or4 where 5 = (VG($++1) (s()))
(5)
1=0
The critic loss Ceritic ( is defined as:
Leritic
LB (Vp(st) RI) where
R₁ = Σ Tr
(6)
k=0
The actor loss Cactor (0) is defined as:
Lactor (0)
EB
min
TT (a)(s)) Tezya (ar|st)
A, clip
TT(ai)(s) Tie (atls:)'
1-,,,1,,, e,1+e) Ar)
(7)
where Thaa the policy before the update, and the clipping parameter. In our experi- ments, we set 0.2, y and = 0.95.
C Related Work
Previous works have explored framing LLM alignment as imitation learning (IL) problem Sun van der Schaar (2024); Wulfmeier et al. (2024); Li et al. (2024); Cho (2024). This approach views the maximum likelihood estimation in standard SFT as behavior cloning
15

---

Published as a conference paper at COLM 2025
(BC) (Pomerleau, 1991), which directly maps states to expert demonstrations. Although straightforward, BC is often unreliable due to challenges with out-of-domain generalization and compounding errors Ross et al. (2011); Chang et al. (2021). To address these limitations, pioneering studies have introduced Inverse Reinforcement Learning (IRL) (Ng et al., 2000; Ziebart et al., 2008) and Adversarial Imitation Learning (AIL) (Ho & Ermon, 2016), which aim to infer reward function that captures underlying objectives of the expert, leading to more robust policies that generalize better new scenarios. These methods have become the foundation for contemporary IRL and AlL-based techniques tailored for LLM fine-tuning Chen al. (2024); Sun & van der Schaar (2024); Wulfmeier et al. (2024); Li al. (2024). Inspired by recent advancements in Coherent Soft Imitation Learning (CSIL) (Watson et al., 2024), we introduce a combination of BC and IRL approach using novel coherent reward that measures the divergence between the fine-tuned policy and the pre-trained policy, removing the need to explicitly or implicitly train a separate reward function.
The current state-of-the-art method for aligning large language models (LLMs) begins with SFT using behavior cloning (BC) Ouyang et al. (2022); Wei et al. (2022a) or inverse reinforcement learning (IRL) Sun & van der Schaar (2024); Li et al. (2024); Wulfmeier et al. (2024) on pairs of instructions and demonstrations. Next, RLHF is applied, relying on additional preference-annotated data Christiano et al. (2017); Ziegler al. (2019); Stiennon et al. (2020); Bai et al. (2022b); Li et al. (2023); Chan et al. (2024). RLHF uses a separate reward model trained on these preferences and optimizes the SFT model using on-policy RL techniques such as PPO (Schulman et al., 2017). Despite the success of these methods, collecting preference annotations costly, Additionally, issues like overfitting make reward modeling susceptible to overoptimization or reward hacking Skalse et al. (2022); Gao et al. (2023); Guo et al. (2025), which can lead to undesirable behaviors in the target policy. In contrast, our work removes the need for expensive preference annotations and sensitive reward modeling.
The significaace of on-policy learning in LLM alignment has been widely discussed Guo et al. (2024); Dong et al. (2024); Tang et al. (2024); Tajwar et al. (2024). Several studies have shown that on-policy methods, such as PPO, outperform off-policy methods like direct preference optimization (DPO) Rafailov et al. (2024a) in terms of out-of-distribution generalization, reasoning capabilities, and generation diversity Xu et al. (2024); Ivison et al. (2024); Tang et al. (2024). Addtionally, on-policy methods avoid the distributional gap between data collection and the target policy, which Tajwar et al. (2024); Zhou et al. (2024). Our method combines a coherent reward with PPO to iteratively align large language models. The on-policy nature of PPO ensures that the policy is continuously updated based on its current behavior and allows for the exploration of a diverse response space, improving the model generalization to new scenarios.
lead to suboptimal optimization in off-policy approaches
D Hyperparameter Setup
In this section, we present the detailed hyperparameters for experimental results in Tables to 4.
Table Hyper-parameter setup for SFT with TULU-v2-mix and a 9k subset of Ultrafeedback demonstrations.
SFT Training Data
Model
Method learning rate or actor learning rate Epochs or Steps
SFT
x 10-6
10 epochs
Mistral-7B
TUT.U-v2-mix and 9k Ultra-
SRPPO
x 10
3 episodes
feedback demonstration
SFT
x 10-6
10 epochs
LLAMA3-8B
SRPPO
x 10-8
3 episodes
16

---

Published as a conference paper at COLM 2025
Table 6: Hyper-parameter setup.
SFT Training Data Model
Method
learning rate Epochs or Steps
SFT
5 x 10
eepochs
SFT (Extended)
5 10
epochs
Mistral-7B
SPIN (the first iteration)
5 10-?
2500 steps
TULU-v2-mix
SRPPO
5 10
episodess
SFT
1 x 10-5
epochs
SFT (Extended)
1 10-5
epochs
LLAMA3-SB
PPO w/ preference RM
5 x 10
2 episodes
SRPPO
5 x 10-8
3 episodes
Table 7: Hyper-parameter setup for SFT with TULU-v2-mix and a 9k subset of Ultrafeedback demonstrations, and 40k subset of TULU-v2-mix.
SFT Training Data
Model
Method learning rate or actor learning rate Epochs or Steps
TULU-v2-mix and 9k Ult
SFT
x 10
8 spoccs episodes
Mistral-7B
feedback demonstration
SRPPO
x 10
E Token-wise Coherent Reward
We assign the coherent reward the process level as defined in (4). Alternatively, we can revise the process-level coherent reward to a token-wise reward and assign it at token level:
POISET) xej))
r(y;lx, log
(8)
POrTT (y Vzjj)
However, during training, we observed a significant issue: the model consistently biased toward generating increasingly long sequences. This behavior lead to deteriorated perfor- mance, as the model produced unnaturally verbose outputs. Upon further analysis, we identified that the uncontrolled length growth stemmed from the token-wise reward design:
Almost every generated token was assigned a non-negative reward, regardless of its contribution to the sequence's 's overall quality.
The [EOS] (end-of-sequence) token, which signals termination of the generation, did not receive any distinguishable reward signal.
reward
total_ength tag Leingth
tag: train/reward
1.2e+3
2e+3
1e+3
1.6e13
1.2++
800
800
600
400
10
15
20
25
30
85
40
45
50
55
5
10
15
30
40
45
50
85
60
Figure 2: PPO training curve using the token-wise coherent reward (8).
As a result, the model learned to extend sequences unnecessarily, maximizing cumulative rewards without considering the intended stopping point. To address this issue, structured the reward formulation to focus on the entire sequence rather than individual tokens:
17

---

Published as a conference paper at COLM 2025
We calculated the log policy ratio for the likelihood of the entire sequence.
This sequence-level reward was then assigned exclusively to the [EOS] token, effectively treating as summary evaluation of the entire generation.
By tying the reward to the [EOS] token, we resolved the uncontrolled length growth issue, as the model was no longer incentivized to generate excessively long sequences. Instead, the reward mechanism now encourages the model to generate sequences of appropriate length that align well with the desired policy behavior. This reward redesign successfully stabilized PPO training and mitigated the length bias problem. It ensured that the model balanced the trade-off between sequence quality and length, resulting in outputs that were both coherent and concise. Our solution highlights the importance of carefully designing reward mechanisms in reinforcement learning settings, particularly for autoregressive generation tasks where sequence length plays a critical role.
18


Reviews:
[{"review_id": "M3gx6Fu3qw", "reviewer": "Reviewer_mRvM", "summary": "SFT is an off policy approach similar to behavior cloning and suffers from overfitting or poor out of domain generalization.  SRPPO is claimed to be a novel fine tuning method that leverages on policy techniques to enhance generalization.  The reward function is a log likelihood ratio between the SFT model and the base model in this paper (e.g., self-rewarding).  This enables no preference annotations.  The empirical evaluation is claimed to demonstrate outperformance of SFT methods.\n\nThe intro goes into SFT and preference learning for alignment.  RLHF suffers from challenges in deriving a reward signal.  Alternative approaches include IRL. SPIN bypasses this by applying DPO.  The authors call this a “coherent reward.”  The authors apply PPO to continuously fine-tune the model using a set of prompts.  The paper was inspired by coherent soft imitation learning.  Compared to SPIN, SRPPO allows the use of additional prompts.  Section 2 gives the background, including SFT and RLHF.  Section 3 gives the approach, including the coherent reward definition and self-rewarding PPO.  Section 4 gives results.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Growing a weak LLM into strong one without human feedback is a promising and interesting research direction.  The authors present an approach toward that goal.", "reasons_to_reject": "There are a number of apparent issues with the paper that put the results into question:\n* SPIN (a reasonable baseline) is not evaluated across all scenarios.\n* There is no apparent baseline with human feedback.\n* For some decisions (e.g., reward function) and experiments (the three experimental setups on page 7), it was not clear why these were made or what intuitions we should derive from them.  See questions below.\n* The code does not appear to be released, which will limit reproducibility.\n* The differences from SPIN appear to be minimal and it is not clear the experiments are as complete as in that paper, despite substantial overlap in approach.", "questions_to_authors": "How much SFT should be done is never discussed in the paper.  The whole aim here is to only do a bit of SFT, I would assume and use that directionally for downstream performance.  \n\nIt’s also not clear if there was ever a full comparison to SFT on downstream tasks.\n\n* It was mentioned multiple places in the paper that compared to SPIN, you get additional prompts.  What does this buy you? Why was SPIN not evaluated in Table 2, Table 3 or Table 4?  SPIN / DPO should generally be evaluated.\n* Why don't you compare against approaches that have human feedback?  It's not clear how, with these approaches, you get a resulting model that is resonably aligned with human preferences without human feedback?\n* I wanted there to be a figure like Figure 2 in the SPIN paper, but with SRPPO.  Were the same metrics evaluated as in that paper?\n* The reasons for the three setups on page 7 was never intuitively explained.  And diminished and minimum on the surface have similar meanings.  What was the reason for each experiment?  Why were multiple SFT subsets not evaluated?\n* It would be much more impactful to release the code.  Is that possible\n\n\nSmall things:\nP2, pretrianed -> pretrained  \nP3, pretarined -> pretrained", "extra_scores": {}}, {"review_id": "h6BtmtXBiK", "reviewer": "Reviewer_fYjb", "summary": "This paper introduces an RLHF approach that does not rely on preference data but only high-quality demonstration data. Specifically, the authors compute the logarithm of the relative ratio of the SFT model and the base model response probabilities to derive a reward that reflects how well the response aligns with the demonstration data. Then the authors apply PPO to further optimize the reward. The proposed method is evaluated with Mistral-7B and LLaMA3-8B on four math&chat benchmarks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper introduces an RLHF method that does not rely on preference data but only high-quality demonstration data.\n2. The authors ablate several choices on the SFT data, demonstrating the necessity to adopt a minimum overlap setup for the SFT and PPO stages.\n3. The authors draw a connection between the proposed method and Coherent Soft Imitation Learning.", "reasons_to_reject": "1. Lack of analysis on why the proposed method could be better than PPO with a preference reward model. It is expected that the authors provide a theoretical analysis on how to frame SRPPO within the framework of inverse RL or Coherent Soft Imitation Learning.\n2. Lack of an ablation study on the quality of the demonstration data. Though being free of the preference dataset makes the method promising, the quality of demonstration data should be an important influencing factor for SRPPO.\n3. There are limited baselines. It would be better to incorporate more baselines such as DPO and model merging.", "questions_to_authors": "1. Why could the proposed method be better than PPO with a preference reward model?\n2. How would the quality of demonstration data influence the performance of SRPPO?", "extra_scores": {}}, {"review_id": "W2aDFXTd1a", "reviewer": "Reviewer_zG4C", "summary": "This paper introduces self-rewarding PPO (SRPPO), a fine-tuning method that combines supervised fine-tuning (SFT) and proximal policy optimization (PPO), without the need for extra human preference labels or reward models. Specifically, SRPPO begins with an SFT-ed policy, which is then optimized using PPO with a coherent reward, inspired by coherent soft imitation learning. The coherent reward is defined as the log ratio between the SFT-ed policy and the pre-trained base model. Experiments on Mistral-7B and Llama3-8B demonstrate improvements over SFT-only and other baselines.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The motivation and the method sections are well written and easy to follow.\n- SRPPO eliminates the need for human preferences, thus it is more cost effective than other alignment methods.\n- Experimental results show promising gains of two open-source models across a variety of tasks.", "reasons_to_reject": "The paper can be greatly improved with (1) additional stronger and more controlled baselines, and (2) additional analysis on how the quality of SFT-ed policy impacts SRPPO.\n\nIf I read it correctly, the extended-SFT baseline does not leverage the exactly same prompt distribution used by SRPPO stage. A stronger SFT baseline would be continue the SFT model (initial policy for SRPPO RL finetuning) with demonstrations where the prompt is used by SRPPO stage for all three data overlap settings. With current results, it is unclear if the improvement is by data or by better algorithmic design. Similarly for the SPIN baseline, making sure the prompt distribution is the same can help strengthen the results. \n\nThe reward model baseline experiment can be further improved. The PPO baseline was using an external reward model without reasons for why that particular RM was chosen. There are also missing values for the PPO baseline row in the table. Trying a few more reward models and showing consistent gains over those PPO baselines can help increase the confidence on the performance of SRPPO. \n\nThe proposed coherent reward (log p_SFT / p_PT) seems to be only meaningful when the SFT-ed policy is noticeably better than the base model. More (theoretical or empirical) analysis on how the quality of the SFT policy influences the SRPPO tuning can help better understand the benefits and the limitations of the proposed method.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o238Rj06Iv", "reviewer": "Reviewer_m2aH", "summary": "The authors propose finetuning/align LLMs in a staggered manner: First use supervised finetuning (SFT) on a demnstration dataset, then do reinforcement learning on the target domain data using the ratio between the finetuned model and the baseline model as reward. This way, there's no need for a critic model.\nIn the experiments shown this approach seems to work well.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Applying RLHF using only one model with good results.", "reasons_to_reject": "- I'm a bit doubtful about the baselines, specifically there's no other true baseline than a DPO-based approach. A comparison to \"Self-Rewarding Language Models\" would be helpful.\n- Hard to follow at times, lots of RL lingo even in the abstract.", "questions_to_authors": "- Line 3: Can you rewrite this sentence such that it's clearer what \"off-policy\" means here?\n- Line 122: It would be helpful to not just state \"clearly\" here.\n- Eq. 2: Can you note here what $p_{ref}$ is in your case?\n- Alg. 1: Typo \"Demonstartion\"", "extra_scores": {}}, {"review_id": "hO1PBtEd0h", "reviewer": "Reviewer_vgGq", "summary": "The paper presents a simple yet effective strategy called Self-Rewarding PPO (SRPPO) that enables fine-tuning a model with only demonstrations (positive examples), doing without preference data to perform reinforcement learning by using a coherent reward based on a log policy ratio between the SFT model and the pretrained model. SRPPO enables on-policy fine-tuning without preference data by addressing the limitations of overfitting of SFT and improves generalization by being able to continue training with unseen prompts. The authors show that SRPPO outperforms or performs on par for various instruction-following and reasoning benchmarks compared to various baselines for Mistral 7B and LLAMA3-8B.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- SRPPO is a simple yet effective method that outperforms or performs on par for various benchmarks. The authors also provide results with varying degrees of overlap between SFT and PPO data to demonstrate the effectiveness of SRPPO in generalizing beyond the data that it seen during SFT.\n- The paper is well-written, making it easy to follow and understand the main research question presented in lines 153-154. The essence of the paper's contributions is clearly situated in relation to prior related work.", "reasons_to_reject": "- Improvements for general instruction-following tasks seem clear while those for reasoning and math-related tasks are less evident compared to baselines. (GSM8k and GPQA in table 1 and 2).\n- Experiments are limited to relatively small models and therefore it is unclear whether SRPPO is effective for larger models as well.\n- The benefit of SRPPO is limited to a specific scenario where high-quality responses are scarce but similar prompts are abundant (lines 91-92) and thus window of opportunity for this method to be practical may be narrow. The authors do not experiment with a systematic variation in the amount of SFT data to establish the data requirements for SRPPO.", "questions_to_authors": "- line 77: pretrianed -> pretrained\n- line 88: what is meant by mid-aligned? why is a SFT model mid-aligned as opposed to fully aligned? \n- Missing related work: https://arxiv.org/abs/2406.00888\n- line 170-173 is redundant\n- line 196: rewarding learning -> reward learning\n- line 236: realted -> related\n- line 239-242: could the shift from TULU-v2-mix to UltraFeedback be quantified concretely in order to back up the validity of this setup for measuring generalizability? I know this is somewhat measured by the evaluations that vary the degree of overlap between SFT and PPO data but intuitively I would expect coherent reward to only provide correct reward signals when the SFT data and the prompts seen in PPO are from a similar domain for SRPPO to be effective.\n- PPO is generally challenging to work with to find hyperparameters with good performance. Can you share details on this search process? Do you observe large fluctuations in performance depending on the chosen set of hyperparameters?\n- Another interesting experiment would be to see how using different epochs of SFT and size of SFT data affects the performance of SRPPO to balance the effect of overfitting to the training distribution while having the SFT Policy functioning as an effective coherent reward. Being able to balance this tradeoff effectively would further add to the usefulness of SRPPO.\n- Can you further elaborate on coherent soft imitation learning and how it motivates SRPPO?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes SRPPO, a method for instruction-tuning from demonstrations without human preference data. The main idea is to use a _coherent reward_, which is the log-ratio of the sequence likelihood under the SFT'd model to the likelihood under the base model. Basically, you get more reward if your sequence is relatively more likely under the SFT'd model compared to the base model. This reward can be computed on arbitrary sequences, which allows the authors to train on additional prompts, without demonstrations. The authors compare to baselines including SFT, SPIN, RLHF with a simple reward model, and show promising results.

The proposed method is simple and intuitive, and the results are promising. The reviewers raised concerns with the quality of baselines. The method does allow for training on extra data (new prompts), which the baselines can not do. Also, the method does not rely on human preferences, so comparing to RLHF is challenging. As a result, coming up with a fair evaluation setting is somewhat challenging. The authors suggested that they will include additional results for the SPIN method in the updated manuscript.

The authors provide an ablation on the overlap between the demonstrations and additional prompt distribution. Some reviewers suggested that more ablations should be done on the amount and quality of SFT data. 

Overall, the paper proposes a method that enables on-policy RL without human preference data, which is a unique property, and could be useful in practice. Concurrent work [1, 2] works in a similar setting, although with a focus on reasoning.

My main concern with the paper is that it is not clear why the proposed coherent reward is a good reward function, and what are the limits of its applicability. With enough optimization, PPO should learn "reward hacks", i.e. idiosyncrasies of the SFT / Base model which allow the policy to get high reward in a way that is not aligned with the true goal. It would be good to see more analysis on the quality of the reward signal, and possibly comparisons to alternative choices, e.g. just the likelihood under the SFT model. 

[1] Learning to Reason without External Rewards
[2] Absolute Zero: Reinforced Self-play Reasoning with Zero Data

[Automatically added comment] At least one review was discounted during the decision process due to quality]
Decision: Accept