SimPO: Simple Preference Optimization with a Reference-Free Reward
Authors: Yu Meng
Mengzhou Xia
Danqi Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Language Models
Preference Optimization
Reinforcement Learning from Human Feedback
Primary area: natural_language_processing
TL;DR: We propose SimPO, a simple and effective preference optimization algorithm.

Abstract: Direct Preference Optimization (DPO) is a widely used offline preference optimization algorithm that reparameterizes reward functions in reinforcement learning from human feedback (RLHF) to enhance simplicity and training stability. In this work, we propose SimPO, a simpler yet more effective approach. The effectiveness of SimPO is attributed to a key design: using the _average_ log probability of a sequence as the implicit reward. This reward formulation better aligns with model generation and eliminates the need for a reference model, making it more compute and memory efficient. Additionally, we introduce a target reward margin to the Bradley-Terry objective to encourage a larger margin between the winning and losing responses, further improving the algorithm's performance. We compare SimPO to DPO and its latest variants across various state-of-the-art training setups, including both base and instruction-tuned models such as Mistral, Llama 3, and Gemma 2. We evaluate on extensive chat-based evaluation benchmarks, including AlpacaEval 2, MT-Bench, and Arena-Hard. Our results demonstrate that SimPO consistently and significantly outperforms existing approaches without substantially increasing response length. Specifically, SimPO outperforms DPO by up to 6.4 points on AlpacaEval 2 and by up to 7.5 points on Arena-Hard. Our top-performing model, built on Gemma-2-9B-it, achieves a 72.4\% length-controlled win rate on AlpacaEval 2, a 59.1\% win rate on Arena-Hard, and ranks 1st on Chatbot Arena among $<$10B models with real user votes.

SimPO: Simple Preference Optimization with a Reference-Free Reward
Yu Meng' Mengzhou Xia² Danqi Chen
1Computer Science Department, University of Virginia
Princeton Language and Intelligence (PLI), Princeton University
aininniiiiinniiaaaaainiiiiniiiia edu yumeng5 @evi rginia
{ mmengzhou, danqic) @cs. pprncctonn.ed
Abstract
Direct Preference Optimization (DPO) a widely used offline preference opti- mization algorithm that reparameterizes reward functions in reinforcement learning from human feedback (RLHF) to enhance simplicity and training stability. In this work, propose SimPO, simpler yet more effecttive approach. The effectiveness SimPO attributed to key design: using the average log probability of a seeueeetii
sequence the implicit reward. This reward formulation better aligns with model generation and eliminates the need for a reference model, making it more compute and memory efficient. Additionally, Bradley-Terry objective to encourage a larger margin between the winning and losing responses, further improving the algorithm's 's performance. We compare SimPO to DPO and its recent variants across various state-of tfiie-offthh-arttt training setups, including both base and instruction-tus tuned models such as Mistral, Llama 3, and Gemma 2. We evaluate on extensive chat-based evaluation benchmarks, including AlpacaEval 2, MT-Bench, and Arena-Hard Our results demonstrate that SimPO consistently and significantly outperforms existing approaches without substantially increasing response length. Specifically, SimPO outperforms DPO by up 6.4 points on AlpacaEval and by up to 7.5 points on Arena-Hard Our top-performing achieves a 72.4% length-controlled win rate on
introduce target reward margin the
model, built on Gemma-2-9B- AlpacaEval 2, a 59.1%% % win rate on Arena-Hard, and ranks st on Chatbot Arena among <10B models with real user votes.
Introduction
Learning from human feedback is crucial in aligning large language models (LLMs) with human values and intentions [47], ensuring they helpful, honest, and harmless [5]. Reinforcement learning from human feedback (RLHF) [18, 58, 68] a popular method for fine-tuning language models to achieve effective alignment. While the classical RLHF approach [58, 65] has shown impressive results, it presents optimization challenges due to its multi-stage procedure, which involves training a reward model and then optimizing policy model maximize that reward [13].
Recently, researchers have been exploring simpler offline algorithms. Direct Preference Optimization (DPO) [61] is one such approach. DPO reparameterizes the reward function in RLHF to directly learn policy model from preference data, eliminating the need for an explicit reward model. has gained widespread practical adoption due to its simplicity and stability. In DPO, the implicit reward is formulated using the log ratio of the likelihood of response between the current policy model and the supervised fine-tuned (SFT) model. However, this reward formulation is not directly aligned with
"Equal Contribution.
Code and models can found at httppp///ggthbb..../////ooooooop
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

CDpO (Ree)
AlpacaEval LC Win Rate (%)
Arena Arene-Hard Win Rate (4)
E loger
Flog
molpe [2)
Flog
Fv(au (x)
DPQ SimPO
DPO
SimPo
+6.
18.2
Csump (Tp)
=
2 lo
log Mo
[x)
log mo(sh
Mistral
Mistrel
Ulema3
Uamms
Motral
Mistral
UJama
Uarred
Instructds
Baaw
Instruct
Base
Figure I: SimPO and DPO mainly differ in their reward formulation, indicated in the shaded box. SimPO outperforms DPO significantly across a range settings on AlpacaEval and Arena-Hard.
the metric used to guide generation, which is approximately the average log likelihood of response generated by the policy model. We hypothesize that this discrepancy between training and inference may lead to suboptimal performance.
this work, we propose SimPO, simple yet effective offline preference optimiza- tion algorithm (Figure 1). The core of algorithm aligns the reward function in the preference optimization objective with the generation metric. SimPO consists
Table I: Length-controln (LC) and raw win rate (WR), and generation lengths of top models the AlpacaEval Leaderboard. Bold are the models we trained.
Model
LC (%)
WR ( (%)
Len.
Gemma-2-9B-it-SmmP0
72.4
65.9
1833
of two major components: (1) a length- normalized reward, calculated the aver- age log probability of all tokens a re- sponse using the policy model, and (2) target reward margin to ensure the reward difference between winning and losing re-
GPT-4 Turbo (04/09)
55.0
46.1
1802
Gemma-2-9B-it
51.1
38.1
1571
44.7
40.5
1825
Claude Opus
40.5
29.1
1388
Mamma-3-8B-Instrettaa
40.3
37.9
1837
Llama-3- Seaa-----7--nnnttue
34.4
33.2
1919
sponses exceeds this margin. In summary, SimPO has the following properties:
Mamma---B-Instruue
26.0
25.3
1899
Simplicity: SimPO does not require a reference model, making it more lightweight and easier to implement compared DPO and other reference-basee based methods.
Significant performance advantage: Despite its simplicity, SimPO significantly outperforms
DPO and its latest variants (e.g., recent reference- frre objective ORPO [38]). The performance advantage is consistent across various training setups and extensive chat-based evaluations, includ- ing AlpacaFvall 2 [51, 28] and the challenging Arena-Hard [50] benchmark. It achieves up 6.4 point improvement on AlpacaBval and a 7.5 point improvement on Arena-Hard compared to DPO (Figure
Minimal length exploitation: SimPO does not significantly increase response length compared to the SFT DPO models (Table 1), indicating minimal length exploitation [28, 66, 80].
Extensive analysis shows that SimPO utilizes preference data more effectively, leading to a more accurate likelihood ranking of winning and losing responses held-out validation set, which in turn translates to a better policy model. As shown in Table 1, our Gemma-2-99 2-9-i--------- model achieves state-of-the-aat performance, with 72.4% length-controlled win rate AlpacaEval and a 59.1% win rate on Arena-Hard, establishing it as the strongest open-source model under 10B parameters. Most notably, when evaluated Chatbot Arena [17] with real user votes, our model significantly improved upon the initial Gemma-2-9B-in model, advancing from 36th to 25th place and ranking first among all <10B models on the leaderboard,2
2 SimPO: Simple Preference Optimization
In this section, we first introduce the background of DPO ($2.1). Then we identify the discrepancy between DPO's reward and the likelihood metric used for generation, and propose an alternative reference-free reward formulation that mitigates this issue ($2.2). Finally, derive the SimPO objective by incorporating a target reward margin term into the Bradley-Terry model ($2.3).
As of September 16th, 2024.

---

2.1 Background: Direct Preference Optimization (DPO)
DPO [61] is one of the most popular preference optimization methods. Instead of learning an explicit reward model [58], DPO reparameterizes the reward function using closed-form expression with the optimal policy:
rxz v) B log Toly x) x)
+ B Blog Z(z),
where my is the policy model, Tie is the reference policy, typically the supervised fine-tuned (SFT) model, and (xx) is the partition function. By incorporating this reward formulation into the Bradley- Terry (BT) ranking objective [11], p(ye > y x) a (r(z, yw) r(z, 91),, DPO expresses the
probability of preference data with the policy model rather than the reward model, yielding the following objective:
To (yu 2z) where (ar, Mwr M)) are preference pairs consisting of the prompt, the winning response, and the losing response from the preference dataset D.
To x) ( Tiet) -.( log (BLog
B log
(2)
Tier x))
(er i x))
2.2 A Simple Reference-Free Reward Aligned with Generation
Discrepancy between reward and generation for DPO. Using Eq. (1) as the implicit reward has the following drawbacks: (1) it requires reference model Tet during training, which incurs additional memory and computational costs; and (2) it creates a mismatch between the reward optimized in training and the log-like ligllike optimized during inference, where no reference model is involved. This means that in DPO, for any triple (xc, tho, W1), satisfying the reward ranking rarr, you) > r(x,, v1) does not necessarily mean that the likelihood ranking (Yw yo xx) > po (31 x)) met (here po the average log- loooo in Eq. (3)). In our experiments, we observed that only ~ 50% of the triples from the training set satisfy this condition when trained with DPO (Figure 4b). This observation aligns with concurrent work [14], which finds that existing models trained with DPO exhibit random ranking accuracy in terms of average log-likelihood, even after extensive preference optimization.
Length-normalized reward formulation. One solution is to use the summed token log probability as the reward, but this suffers from length bias-longer sequences tend to have lower log probabilities. Consequently, when yhr is longer than y/s optimizing the summed log probability as reward forees the model to artificially inflate probabilities for longer sequences to ensure Yo receives higher reward than yt- This overcompensation increases the risk of degeneration. To address this issue, we consider using the average log- likelikeehhod as the implicit reward:
1v
(y
x) log To (y ax)
1
Σ
log (1 xr, Fet))
M
This metric is commonly used for ranking options in beam search [33, 49] and multiple-choice tasks within language models [12, 37, 58]. Naturally, we consider replacing the reward formulation in DPO with pa in Eq. (3), so that it aligns with the likelihood metric that guides generation. This results in mnrtai-normalize reward:
B
TSimPO (z, y) = log Toly x))
log To (yr xr Bes))
(4)
3
where 3 is constant that controls the scaling of the reward difference. We find that normalizing the reward with response lengths is crucial; removing the length normalization term from the reward formulation results in bias toward generating longer but lower-quality sequences (see Section 4.4 for more details). Consequently, this reward formulation eliminates the need for reference model, enhancing memory and computational efficiency compared to reference-dependen algorithms.
2.3
The SimPO Objective
Target reward margin. Additionally, we introduce a target reward margin term, y >> O, to the Bradley-Terry objective to ensure that the reward for the winning response, r(z, yw), exceeds the
3

---

reward for the losing response, r(x, 31), by at least
ply > y1 x) (r(z, yw) r(z, w) -7).
(5)
The margin between two classes is known to influence the generalization capabilities of classifiers [1, 10, 22, 30]. In standard training settings with random model initialization, increasing the target margin typically improves generalization. In preference optimization, the two classes are the winning and losing responses for single input. In practice, we observe that generation quality initially improves with an increasing target margin but degrades when the margin becomes too large ($4.3). One of DPO's variants, IPO [6], also formulates target reward margin similar SimPO. However, its full objective is not as effective as SimPO ($4.1).
Objective. Finally, we obtain the SimPO objective by plugging Eq. (4) into Eq. (5):
log To (x) ->)
(siPoo(TT) = -E -E(r.y... log
log To (V
(6)
Mvl
In summary, SimPO employs implicit reward formulation that directly aligns with the generation metric, eliminating the need for a reference model. Additionally, it introduces a target reward margin y help separating the winning and losing responses. In Appendix F, we provide gradient analysis of SimPO and DPO to further understand the differences between the two methods.
Preventing catastrophic forgetting without KL regularization. Although SimPO does not impose KL regularization, we find that a combination of practical factors ensures effective learning from preference data while maintaining generalization, leading to empirically low KL divergence from the reference model. These factors are: (1) a small learning rate, (2) a preference dataset that covers diverse domains and tasks, and (3) the intrinsic robustness of LLMS to learn from new data without forgetting prior knowledge. We present KL divergence experiments Section 4.4.
Experimental Setup
Models and training settings. We perform preference optimization with two families of models, Llama-3-8B [2] and Mistral-7B [40], under two setups: Base and Instruct. In this section, our goal to understand the performance SimPO vs. other preference optimization methods in different experimental setups. Our strongest model is based on Gemma-2-9B (Instruct setup) with stronger reward model, these results in Appendix J.
[79] (Table 1). We will present and discuss
For the Base setup, we follow the training pipeline of Zephyr [75]. First, we train a base model (i.e., misttulai/Mistraa--BBOOII.
meta- Maaama/Mtt-LLama--3 obtain an SFT model. Then, we perform preference optimization on the UltraFeedback dataset [23] using the SFT model as the starting point. This setup provides high level of transparency, the SFT models are trained on open- open-source data.
the UltraChat-2001 dataset [25] to
For the Instruct setup, we use an off-the-shelf Instruction-tuned model (i.e., meta- Peta-llama/Metta Lama-3-8B-IInttucc, or mistralai/Mistrall-T7 -Bn-----tttttut--..) as the SFT models.* These models have undergone extensive instruction-tunin SFT models in the Base setup. However, they are also more opuque because their RLHF procedure not publicly disclosed. To mitigate the distribution shift between SFT models and the preference optimization process, we generate the preference dataset using the SFT models following [74]. on-policy setting. Specifically,
processes, making them more powerful and robust than the
This makes our Instruct setup closer the UltraFeedback dataset and regenerate the chosen and rejected response pairs (Yw- y1) with the SFT models. For each prompt æ, generate 5 responses using the SFT model with sampling temperature of 0.8. We then use Him-beender/PiiRMM [41] to score the responses, selecting the
use prompts from
"This margin termed home advantage Bradley-Teryy models [1. 30).
'It is unclear whether the released instruct checkpoints have undergone supervised fine-tuning (SFT) or the complete RLHF pipeline. For simplicity, we refer to these checkpoints as SFT models.

---

Table 2: Evaluation details for AlpacaEval 2 [51], Arena-Hard [50], and MT-Bench [94]. The baseline model refers to the model compared against. GPT-4 Turbo corresponds to GPT-4-Previeew- 106..
# Exs.
Baseline Model
Judge Model
Scoring Type
Metric
AlpacaEval
805
Turbo GPT-4-0314
GPT-4 Turbo
Pairwise comparison
LC & raw win rate Win rate
Arena-Hard
500
GPT-4 Turbo
Pairwise comparison
MT--Bench 80
GPT-4/GPT-- Turbo
Single-answer grading
Rating of 1-10
highest-scoring one as you and the lowest-scoring one as y/. We only generated data in a single pass instead of iteratively as in [74].
In summary, have four setups: Llama-3-Base, (llam-3--Instruee Mistral- and Mistral-Instruct. We believe these configurations represent the state-of-the-art, placing our models among the top performers on various leaderboards. We encourage future research adopt these settings for better and fairer comparisons of different algorithms. Additionally, we find that tuning hyperparameters is crucial for achieving optimal performance with all the offline preference optimization algorithms. including DPO and SimPO. Generally, for SimPO, setting 3 between 2.0 and 2.5 and y between 0.5 and 1.5 leads to good performance across all setups. For more details, please refer to Appendix B.
Evaluation benchmarks. We primarily assess our models using three of the most popular open- ended instruction-followee benchmarks: MT-Bench [94], AlpacaEval 2 [51], and Arena- Hard v0.1 [50]. These benchmarks evaluate the models' versatile conversational abilities across a diverse set of queries and have been widely adopted by the community (details in Table 2). AlpacaEval 2 consists of 805 questions from 5 datasets, and MT--Bnchh covers 8 categories with 80 questions. The most recently released Arena-Hard is enhanced version of an MT-Bench, incorporating 500 well- defined technical problem-sol solving queries. We report scores following each benchmark's 's evaluation protocol. For AlpacaEval 2, we report both the raw win rate (WR) and the length-controlled win rate (I.C) [28]. The LCC metric is specifically designed to be robust against model verbosity. For Arena-Hard, report the win rate (WR) against the baseline model. For MT--Bench, we report the average MT-Bench score with GPT-4 and GPT-4-Preview- 106 as the judge model. For decoding details. please refer to Appendix B. We also evaluate on downstream tasks from the Huggingface Open Leaderboard benchmarks [9], with additional details in in Appendix C.
Baselines. We compare SimPO with other offline preference optimization in Table 3.7
Table 3: Various preference optimization objectives given pref- erence data D (I. Pm M1,, where x is an input, and yor and y are the winning and losing responses.
methods listed RRHF [86] and SLiC-HF [91] are ranking losses. RRHF uses length-normalized log-likelihood, similar to SimPO's reward function, while SLiC- HF uses log- likelihoodd directly and includes an SFT objective. IPO [6] is theoretically grounded approach method that avoids DPO's assumption that pairwise preferences can be replaced with pointwise rewards. CPO [83] uses sequence likelihood as a reward and trains alongside an SFT objective. KTO [29] learns from non-paired preference data.
Method
Objective
RRHF [86]
max (0.
Taul log my (oo(Bll))))) [51 log (o((())))) A log To( (x)
SLiC-HF [91]
max (0, log mo Ro(Mula) log Ro (ar) A log (o ix)
DPO [61]
logg (3 log saly--in)
(9 lug mataris) mastoris)
IPO [6]
[x] (log, welyx [3]
log Telmiis)
meilor 1n)
CPO [83]
log (5 log mo(ya]la) x- 9 log mo(xe)z)) log mo(
KTO [29]
->.0 log ro(u-11) =ut) Ave
log sxtmle))
where Fier E( [SKLL (5o( (arlo(x))
ORPO [38]
log po(uulll)))) log (leg, peloule) log 1-poinni)) )
where po(ylue) exp ( log no(ula))
R R-DPO [60]
log
(3 log warls
Talas
Teimis) m)) 1 log (artorls)
+ (nlyal - alsil))
SimPO
log
log To (yy)aa))))) Bon log no (x))=))) [o4
*We also experimented with using a stronger reward model, RLHFloo/ArmoRM- 1AA-M---laa33-B--- [79], to rank generated data, which yields significantly improved performance (see Appendix H and Appendix J). This is the reward model we used in our Gemma 2 experiments.
'GPT--Preview 106 produces more accurate reference answers and judgments compared to GPT-4. 'Many recent studies [85, 69] have extensively compared DPO and PPO [65]. We will leave the comparison of PPO and SimPO to future work.
5

---

Table 4: AlpacaEval 2 [51], Arena-Hard [50], and MT-Bench [94] results under the four settings.
LC and WR denote length-controle and raw win rate, respectively. We train SFT models for Base settings the UltraChat dataset. For Instruct settings, off-the-shee models as the SFT model.
Mistral-Bas (7B)
Instrul- (7B)
Method
AlpacaEval Arena-Hard
MT-Bench
AlpacaEval 2
Arena-Hard
MT-Bencc
(%)
(R)))
Turbo GPT4 (%)
(%)
GPT.4
SFT
8.4
6.2
1.3
4.8
6.3
17.1
14.7
12.6
6.2
7.5
RRHF 186]
11.6
10.2
5.8
5.4
6.7
25.3
24.8
18.1
6.5
7.6
SLiC-HF [91]
10.9
8.9
7.3
5.8
7.4
24.1
24.6
18.9
6.5
7.8
DPO [61]
15.1
12.5
10.4
5.9
7.3
26.8
24.9
16.3
6.3
7.6
IPO [6]
11.8
9.4
7.5
5.5
7.2
20.3
20.3
16.2
6.4
7.8
CPO [83]
9.8
8.9
6.9
5.4
6.8
23.8
28.8
22.6
6.3
7.5
KTO [29]
5.6
5.4
7.0
24.5
23.6
17.9
7.7
ORPO [38]
14.7
12.2
7.0
7.3
24.5
24.9
20.8
7.7
R-DPO [60]
17.4
12.8
8.0
5.9
7.4
27.3
24.5
16.1
6.2
7.5
SimPO
20.8
16.6
6.0
7.3
32.1
34.8
21.0
7.6
Llama-3-Ban (8B)
(8B)
Method
AlpacaEval Arena-Has Hard
MT-Bench
AlpacaEval 2 Arena-Hard
MT-Bench
(%) (%)
(R))%
GPT-4 Turbo GPT-4
LC (%)
WR (%)
WR(%)
GPT- Turbo GPT-
SFT
3.3
5.2
26.0
25.3
8.1
RRHF [86]
12.1
10.1
6.3
5.8
7.0
31.3
28.4
26.5
6.7
7.9
SLiC-HF [91]
12.3
13.7
6.0
6.3
7.6
26.9
27.5
26.2
6.8
8.1
DPO 1611
15.5
15.9
6.5
7.7
40.3
37.9
8.0
IPO [6]
14.4
17.8
35.6
35.6
8.3
CPO [83]
10.8
8.1
5.8
6.0
7.4
28.9
32.2
28.8
7.0
8.0
KTO [29]
14.2
12.4
12.5
6.3
7.8
33.1
31.8
26.4
6.9
8.2
ORPO [38]
12.2
10.6
10.8
6.1
7.6
28.5
27.4
25.8
6.8
8.0
R-DPO [60]
17.6
14.4
17.2
6.6
7.5
41.1
37.8
33.1
7.0
8.0
SimPO
22.0
20.3
23.4
6.6
7.7
44.7
40.5
33.8
7.0
8.0
ORPO [38] introduces reference-moddllffce odd ratio term to directly contrast winning and losing responses with the policy model and jointly trains with the SFT objective. R-DPO [60] is modified version of DPO that includes additional regularization term prevent exploitation of length. We thoroughly tune the hyperparameters for each baseline and report the best performance. We find that many variants of DPO do not empirically present advantage standard DPO. Further details can be found in Appendix B.
Experimental Results
In this section, we present main results of our experiments, highlighting the superior performance of SimPO on various benchmarks and ablation studies (84.1). We provide an in-depth understanding of the following components: (1) length normalization ($4.2), (2) the margin term y ($4.3), and (3) why SimPO outperforms DPO ($4.4). Unless otherwise specified, the ablation studies are conducted using the Mistral-Bas Base setting.
4.1 Main Results and Ablations
SimPO consistently and significantly outperforms existing preference optimization methods. shown Table while all preference optimization algorithms enhance performance over the SFT model, SimPO, despite its simplicity, achieves the best overall performance across all benchmarks and settings. These consistent and significant improvements highlight the robustness and effectiveness of SimPO. Notably, SimPO outperforms the best baseline by 3.6 to 4.8 points on the AlpacaEval LC win rate across various settings. On Arena-Hard SimPO consistently achieves superior performance,
'ORPO can directly train on preference data without the SFT stage. For fair comparisons, we start ORPO from the same SFT checkpoints other baselines, which yields better results than starting from base checkpoints.

---

Table 5: Ablation studies under Mistral-Base and Mistral- settings. We ablate each key design of SimPO: (1) removing length normalization in Eq. (4) (i.e., w/o LN); (2) setting target reward margin y to be 0 in Eq. (6) (i.e, 7 = 0).
Mistral-Bas Base (7B) Setting
Mistral-Instruce (7B) Setting
Method
AlpacaEval 2 Arena-Hard
MT-Bench
AlpacEEval Arena-Hard
MT-Bench
(C)
(%)
(R())
GPT-4 Turbo
GPT-4
(C)
WR
WR())
GPT- Turbo GPT-4
DPO
15.1
12.5
10.4
5.9
7.3
26.8
24.9
16.3
6.3
7.6
SimPO
21.5
20.8
16.6
6.0
7.3
32.1
34.8
21.0
6.6
7.6
w/o LN
11.9
13.2
9.4
5.5
7.3
19.1
19.7
16.3
6.4
7.6
= 0
16.8
14.3
11.7
5.6
6.9
30.9
34.2
20.5
6.6
7.7
Avgg liog posk, Noll
walks
10
Srrae Mag Say
0-
-00
0.34
= 0.82
2.00
1000
(U0)
Length G....
55
60]
Resporee bength lai
Reeporee longth Iai
(a) Reward optimization.
SimPO.
(c) SimPO without LN
Figure 2: Effect of length normalization (LN). (a) Relationship between reward margin and length difference between winning and losing responses. (b) Spearman correlation between average log probability and response length for SimPO. (c) Spearman correlation for SimPO without LNN
though it is occasionally surpassed by CPO [83]. We find that CPO generates responses that are, on average, 50% longer than those generated by SimPO (See Table 10). Arena-Hard might favor longer generations due to the absence of a length penalty in its evaluation.
Benchmark quality varies. Although all three benchmarks are widely adopted, we find that MT- Bench exhibits poor separability across different methods. Minor differences between methods on MT-Bench may be attributed to randomness, likely due to the limited scale of its evaluation data and its aingle-instanee scoring protocol. This finding aligns with observations reported in [50]. In contrast, AlpacaEval 2 and Arena-Han Hard provide more meaningful distinctions between different methods. We observe that the win rate on Arena-Hos Hard is significantly lower than on AlpacaEval 2, indicating that Keea-Hard is more challenging benchmark.'
The Instruct setting introduces significant performance gains. Across all benchmarks, we observe that the Instruct setting consistently outperforms the Base setting. This improvement is likely due to the higher quality of SFT models used for initialization and the generation of more high-quality preference data by these models.
Both key designs in SimPO are crucial. In Table 5, we demonstrate results from ablating each key design of SimPO: (1) removing length normalization in Eq. (4) (i.e., w/o LN); (2) setting the target reward margin be in Eq. (6) (i.e. 7 0). Removing the length normalization has the most negative impact on the results. Our examination reveals that this leads to the generation of long and repetitive patterns, substantially degrading the overall quality of the output (See Appendix E). Setting to 0 yields also leads to a performance degradation compared to SimPO, indicating that it is not the optimal target reward margin. In the following subsections, we conduct in-depth analyses to better understand both design choices.
4.2 Length Normalization (LN) Prevents Length Exploitation
LN leads to an increase in the reward difference for all preference pairs, regardless of their length. The Bradley-Terry objective in Eq. (5) essentially aims to optimize the reward difference
Although our models excel on benchmarks, these evaluations have limitations, including restricted query space and potential biases from model-based evaluations. Efforts like WildBench [88] aim to expand these spaces, where SimPO models demonstrate competitive performance.
7

---

Alpes LO (10)
Deward Aucutaay
00
201
Deardty
0.3
08
Dunite
0.8
0.8
0.2
16
16
27
22
7 24
0.1
24
14
24
0.0
0.0
Targor. Rewwan] Margin ?
Reward GE r(z, sc) r(x.a))
Avg. log prob, on wizning poly-
(a) Performance w/ different
(b) Reward diff. distribution.
(c) Log prob. distribution.
Figure 3: Study of the margin 7. (a) Reward accuracy and AlpacaEval2 LC win rate under different Y values. (b) Reward difference distribution under different 7y values. (c) Log likelihood distribution on chosen responses under different values.
A r(z, yw:) (x, 3t) to exceed the target margin 7, We investigate the relationship between the learned reward differences and the length difference Al |//l - |y between the winning and losing responses from the training set of UltraFeedback. We measure the difference of reward (rsimpo: Eq. (4)) using the SFT model, the SimPO model, and model trained with SimPO but without length normalization. We present the results in Figure 2a and observe that SimPO with LN consistently achieves a positive reward margin for all response pairs, regardless of their length difference, and consistently improves the margin over the SFT model. In contrast, SimPO without LN results in negative reward difference for preference pairs when the winning response is shorter than the losing response, indicating that the model learns poorly for these instances.
Removing LN results strong positive correlation between the reward and response length, leading to length exploitation. Figures 2b and 2c illustrate the average log likelihood (pe in Eq. (3)) versus response length on held-out set for models trained with SimPO and SimPO without LN. The model trained without LN exhibits much stronger positive Spearman correlation between likelihood and response length compared to SimPO, indicating a tendency to exploit length bias and generate longer sequences (see Appendix E). In contrast, SimPO results Spearman correlation coefficient similar to the SFT model (see Figure 6a).
4.3 The Impact of Target Reward Margin in SimPO
Influence of y reward accuracy and win rate. We investigate how the target reward margin in SimPO affects the reward accuracy on a held-out set and win rate on AlpacaEval 2, presenting the results in Figure 3a. Reward accuracy measured as the percentage of preference pairs where the winning response ends up having a higher reward for the winning response than the losing response (i.e., r(v, Mw) r(z: m))). We observe that reward accuracy inereases with y both benchmarks, indicating that enforcing larger target reward margin effectively improves reward accuracy. However, the win rate on AlpacaEval first increases and then decreases with 7, suggesting that generation quality is not solely determined by the reward margin.
Impact of y on the reward distribution. We visualize the distribution of the learned reward margin r(z, y - rxx y1) and the reward of winning responses r(r, yn:) under varying values in Figure 2b and Figure 2c, Notably, increasing " tends to flatten both distributions and reduce the average log likelihood of winning sequences. This initially improves performance but can eventually lead to model degeneration. We hypothesize that there is trade-off between accurately approximating the true reward distribution and maintaining well-calibrated likelihood when setting the value. Further exploration this balance is deferred to future work.
4.4 In-Depth Analysis of DPO vs. SimPO
In this section, we compare SimPO to DPO in terms of (1) likelihood-length correlation, (2) re- ward formulation, (3) reward accuracy, and (4) algorithm efficiency. We demonstrate that SimPO outperforms DPO in terms of reward accuracy and efficiency.
DPO reward implicitly facilitates length normalization. Although the DPO reward expression (e(g))) (with the partition function excluded) lacks an explicit term for length
r(r,8) = 3 log (=)())))) normalization, the logarithmic ratio between the policy model and the reference model can serve to

---

Noy. (vlol))
DPO
SimPO
Teining mntele
>
23.0k
21.2k
Bread Arcunity
X0H00
-10
8.0k
8.3k
0.30
Tx
10
2 20
Ps. <0.0
Mistral-Bae
Mistral-Insirut
Response inuth lol
Geueration cuetric
Length correlation (DPO).
Contingency table (DPO).
(c) Reward Accuracy.
Figure 4: Comparison between SimPO and DPO, measured on UltraFeedback. (a) Spearman
correlation between average log probability and response length for DPO. (b) Contingency table of rankings based on DPO rewards and the average log likelihood (measured on the training set). (c)
Reward accuracy of DPO and SimPO.
0.6
DPO
StaPO
AlpacaEoll2 LC (R)
100
RL. diergenee
0.4
Percestage (1)
75-
73 min
GB 69 CI
DP0 2.00)
50
60 min
0.0g
100
200
300
8
8 301 DPO
8-16 8 StnPO
Rus time
Poak GPU memory
Training stop
(a) KL divergence w/ different B.
(b) Performance w/ diffcrent B.
(c) Efficiency of DPO vs. SimPO.
Figure Comparison between SimPO and DPO (continued). (a) With different in DPO and SimPO, KL divergence from the policy model to the reference model on Yu- (b) AlpacaEval2 LC win rate of DPO and SimPO with different B. (c) Runtime and memory usage for DPO and SimPO.
implicitly counteract length bias. As shown in Table and Figure 4a, employing DPO reduces the Spearman correlation coefficient between average log likelihood and response length compared to the approach without any length normalization (referred to as "SimPO w/o LN"). However, it still exhibits a stronger positive correlation when compared to SimPO."
DPO reward mismatches generation likelihood. There divergence between DPO's reward formulation, ro (z, v) B log
Table Spearman correlation p be- tween average log likelihood of differ- ent models and response length on held-out set.
<e(ulm) and the average log likelihood "wrly/z)
metric, Po x)) M log To(y x)), which directly impacts generation.
shown in Figure 4b, among the instances on the UltraFeedback training set where re (2 ywo > ra( 31) almost half of the pairs have po(y1 æ) Pe(n r). In contrast, SimPO directly employs the average log likelihood (scaled by () as the reward expression, thereby eliminating the discrepancy completely, demonstrated in Figure 6b.
SimPO w/o LN
DPO
SimPO
P
0.82
0.59
0.34
DPO lags behind SimPO in terms of reward accuracy. In Figure 4c, compare the reward accuracy of SimPO and DPO, assessing how well their final learned rewards align with preference labels held-out set. SimPO consistently achieves higher reward accuracy than DPO, suggesting that our reward design facilitates better generalization and leads to higher quality generations.
KL divergence SimPO and DPO. In Figure 5a, we present the KL divergence between the policy model trained with DPO and SimPO and the reference model with different 3, measured on the winning responses from held-out set during training. Figure 5b shows the corresponding AlpacaEval 2 LC win rate. Although SimPO does not apply any form of regularization against the reference model, the KL divergence SimPO is reasonably small. Increasing reduces the KL divergence for both DPO and SimPO, with DPO exhibiting more pronounced reduction at higher B values. In this particular setting (Mistral-base). Figure 5b demonstrates that smaller / can
10 Note that this correlation does not fully reflect the generation length. Despite DPO showing a stronger correlation, the length of its generated responses is comparable to or even slightly shorter than those of the SimPO models. Please find more details in Appendix E.

---

improve AlpacaEval 2 performance, despite the higher KL divergence. "| We hypothesize that when the reference model is weak, strictly constraining the policy model the reference model may not be beneficial. As a caveat, while we did not observe any training collapse or degeneration with proper tuning, principle, SimPO could potentially lead to reward hacking without explicit regularization against the reference model. In such scenario, the model might achieve low loss but degenerate.
SimPO is more memory and compute-efficient than DPO. Another benefit of SimPO is its efficiency as it does not use a reference model. Figure 5c illustrates the overall run time and per-GPU peak memory usage of SimPO and DPO in the Llama-3-Bass setting using 8xH100 GPUs. Compared to vanilla DPO implementation, 12 SimPO cuts run time by roughly 20% and reduces GPU memory usage by about 10%, thanks eliminating forward passes with the reference model.
5 Related Work
Reinforcement learning from human feedback. RLHF is technique that aligns large language models with human preferences and values [18, 97, 58, 7]. The classical RLHF pipeline typically comprises three phases: supervised fine-tuning [96, 71, 32, 21, 44, 25, 77, 15, 81]. reward model training [31, 56, 16, 52, 35, 46], and policy optimization [65, 4]. Proximal Policy Optimization (PPO) [65] is widely used algorithm in the third stage of RLHF, The RLHF framework is also widely applied to various applications, such as mitigating toxicity [3, 45, 92], ensuring safety [24], enhancing helpfulness [73, 78), searching and navigating the web [57], and improving model reason- ing abilities [34]. Recently, [13] has highlighted challenges across the whole RLHF pipeline from preference data collection to model training. Further research has also demonstrated that RLHF can lead to biased outcomes, such as verbose outputs from the model [28, 66, 80).
Offline vs. iterative preference optimization. Given that online preference optimization algo- rithms are complex and difficult to optimize [95, 64], researchers have been exploring more efficient and simpler alternative offline algorithms. Direct Preference Optimization (DPO) [61] is notable example. However, the absence of an explicit reward model in DPO limits its ability to sample preference pairs from the optimal policy. To address this, researchers have explored augmenting preference data using a trained SFT policy [91] or a refined SFT policy with rejection sampling [55], enabling the policy to learn from data generated by the optimal policy. Further studies have extended this approach to an iterative training setup, by continuously updating the reference model with the most recent policy model or generating new preference pairs each iteration [27, 42, 62, 82, 87). In this work, we focus exclusively on offline settings, avoiding any iterative training processes.
Preference optimization objectives. A variety of preference optimization objectives have been proposed besides DPO. Ranking objectives allow for comparisons among more than two instances [26, 54, 67, 86]. Another line of work explores simpler preference optimization objectives that do not rely on a reference model [38, 84). similar to SimPO. [8] proposes a method to jointly optimize instructions and responses, finding it effectively improves DPO. [93] focuses post-training extrapolation between the SFT and the aligned model to further enhance model performance. In this work, we compare SimPO to series of offline algorithmss. including RRHF [86]. SLiC-HF [91], 1], DPO [61], IPO [6]. CPO [83], KTO [29], ORPO [38], and R-DPO [60], and find that SimPO can outperform them in both efficiency and performance. Recently, [70] proposed generalized preference optimization framework unifying different offline algorithms, and SimPO can be seen as a special case.
Conclusion
In this work, we propose SimPO, simple and effective preference optimization algorithm that consistently outperforms existing approaches across various training setups. By aligning the reward function with the generation likelihood and introducing target reward margin, SimPO eliminates the need for a reference model and achieves strong performance without exploiting the length bias. Extensive analysis demonstrates that the key designs in SimPO are crucial and validates the efficiency and effectiveness of SimPO. A detailed discussion of the limitations can be found in Appendix A.
We observe that in some settings (e.g.. Llama-3-Instruwe Instrruct, a large 3 (e.g.. 3 = 10) leads to better performance. 12 DPO can be as memory efficient as SimPO if it were implemented to separate the forward passes of the reference model from the actual preference optimization. However, this implementation is not standard practice.
10

---

Acknowledgments
The authors would like to thank Li Dong, Tianyu Gao, Tanya Goyal, Di Jin, Yuchen Lin, Kaifeng Lyu, Sadhika Malladi, Eric Mitchell, Lewis Tunstall, Haoxiang Wang, Wei Xiong. Zhen Xu, Libing Yang, Zhiyu Zhao, and members of the Princeton NLP group for their valuable feedback and discussions. We thank Niklas Muennighoff for his advice on training and reproducing training KTO models. We thank Haoran Xu for helping verify our CPO runs. Mengzhou Xia is supported by an Apple Scholars in AIML Fellowship. This research is also funded by the National Science Foundation (IS-2221179)) and a Sloan Research Fellowship.
References
[1] Alan Agresti. Categorical data analysis, volume 792. John Wiley & Sons, 2012.
[2] Al@Meta. Llama 3 model card. 2024.
[3] Afra Amini, Tim Vieira, and Ryan Cotterell. Direct preference optimization with an offset. arXiv preprint 200.1:24022.057,,,, 2024.
[4] Thomas Anthony. Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017.
[5] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Natfield-Doddee Danny Hernandez, John Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, and Jared Kaplan. A general language assistant laboratory for alignment. ArXiv, abs/21 12.00861, 2021.
[6] Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm understand learning from human preferences. ArXiv, abs/2310. 12036, 2023.
[7] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2xXiv:204.00500 2022.
[8] Hritik Bansal, Ashima Suvarna, Gantavya Bhatt, Nanyun Peng, Kai-Wei Chang, and Aditya Grover. Comparing bad apples to good oranges: Aligning large language models via joint preference optimization. arXiv preprint arXiv:2404.0055 2024.
[9] Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open LLM leaderboard. https: //hgggingfacee co spaces Huggingg
11m__m________._mmm__ lem___aaderrrrrr 2023,
[10] Bernhard E Boser, Isabelle M Guyon, and Vladimir N Vapnik. A training algorithm for optimal margin classifiers. In Proceedings of the fifth annual workshop on Computational learning theory, pages 152, 1992.
[11] Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39:324, 1952.
[12] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. In NeurIPS, 2020.
[13] Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217. 2023.

---

[14] Angelica Chen, Sadhika Malladi, Lily H Zhang, Xinyi Chen, Qiuyi Zhang, Rajesh Ranganath, and Kyunghyun Cho. Preference learning algorithms do not learn preference rankings. In NeurlPS, 2024.
[15] Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. AlpaGasus: Training better Alpaca with fewer data. ICI.R 2024.
[16] Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. ODIN: Disentangled reward mitigates hacking in RLHF. arXiv preprint arXiv:2402.07319 2024.
[17] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph H. Gonzalez, et al. Chathot arena: An open platform for evaluating LLMs by human preference. arXiv preprint #XXv:2403.04132 2024.
[18] Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[19] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the A12 reasoning challenge. ArXiv, abs/1803.05457 2018
[20] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
[21] Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tuned LLM, 2023.
[22] Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20:273-297. 1995.
[23] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. UltraFeedback: Boosting language models with high-quality feedback. In ICML, 2024.
[24] Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHE: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310. 12773, 2023.
[25] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng. Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In EMNLP. 2023.
[26) Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, SHUM KaShun, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023.
[27] Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF workflow: From reward modeling to online RLHF. arXiv preprint 2XXx:2405.0786.. 2024.
[28] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length- AlpacaEval: A simple way to debias automatic evaluators. ArXiv, abs/2404.04475. 2024.
[29] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO: Model alignment prospect theoretic optimization. ArXiv, abs/2402.01306, 2024.
12

---

[30] David Firth and Heather Turner. Bradley-terry models in R: the BradleyTerry2 package. Journal of Statistical Software, 48(9), 2012.
[31] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization.
In International Conference on Machine Learning, pages 10835-10866. PMLR, 2023.
[32] Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Picter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April, 1:6, 2023,
[33]
Alex Graves, Sequence transduction with recurrent neural networks. ArXiv, abs/1211.37111 2012.
[34]
Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. arXiv preprint ar Xiv:2240 048424 2024.
[35] Alex Havrilla, Sharath Raparthy, Christoforus Nalmpantis, Jane Dwivedi- Yu, Maksym Zhu- ravinskyi, Eric Hambro, and Roberta Railneau. GLoRe: When, where, and how improve LLM reasoning via global and local refinements. arXiv preprint arXiv:2402. 2024.
[36] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020.
[37] Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. Surface form competition: Why the highest probability answer isn't always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7038-7051, 2021.
[38] Jiwoo Hong, Noah Lee, and James Thome. ORPO: Monolithic preference optimization without reference model. ArXiv, abs/2403.0765 07691, 2024.
[39] Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beaver'Tail:: Towards improved safety alignment of LLM via human- -urfa--refereence dataset. ArXiv, abs/2307.04657, 2023.
[40] Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lu- cile Saulnier, L'elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mistral 7B. ArXiv, abs/2310 10.02255 2023.
[41] Dongfu Jiang. Xiang Ren, and Bill Yuchen Lin. LLM-Blender: Ensembling large language models with pairwise ranking and generative fusion. In ACL, 2023.
[42] Dahyun Kim, Yungi Kim, Wonho Song, Hyeonwoo Kim, Yunsu Kim, Sanghoon Kim, and Chanjun Park. sDPO: Don't use your data all at once. ArXiv, abs/2403. 19270, 2024.
[43] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6900 2014.
[44] Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Minh Nguyen, Oliver Stanley, Richárd Nagyfi, et al. Ope- nassistant conversations-des derrrattion-demmrratiiining large language model alignment. In Thirty-seventh Conference Neural Information Processing Systems Datasets and Benchmarky Track, 2023.
[45] Tomasz Korbak, Kejian Shi. Angelica Chen, Rasika Vinayak Bhalerao. Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. In International Conference on Machine Learning, pages 17506-17533 PMLR. 2023.
[46] Nathan Lambert, Valentina Pyatkin, Jacob Daniel Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Raghavi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hanna Hajishirzi. RewardBench: Evaluating reward models for language modeling. ArXiv, abw//a03..../ 13787, 2024.
13

---

[47] Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: research direction. arXiv preprint 181.07111 2018.
[48] Hector Levesque, Emesst Davis, and Leora Morgenstern. The Winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012.
[49] Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep reinforcement learning for dialogue generation. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods Natural Language Processing, pages 1192-120,, Austin, Texas, November 2016. Association for Computational Linguistics.
[50] Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The Arena-Hard pipeline, April 2024.
[51] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaEval: An automatic evaluator of instruction- following models. https:
com/ su- 1ab/ aataaballppa alpaca _eval, 2023.
[52) Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023.
[53] Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In ACL. pages 3212-3252, 2022.
[54] Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, al. LiPO: Listwise preference optimization through learning-to- toanng-tt-rrnkk rank. arXiv preprint arXiv:2402.01877 2024.
[55] Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. In The Twelfth International Conference on Learning Representations, 2024.
[56] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical rea- soning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308. 30.50333.. 2023.
[57] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vincet Kosaraju, William Saunders, et al. WebGPT, Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.093322, 2021.
[58] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. In NeurIPS, 2022.
[59] Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with pPO-ivee arXiv preprint 20Xiv:2402.13228. 2024.
[60] Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. ArXiv, abs/2403. 19159, 2024.
[61] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In NewrIPS, 2023.
14

---

[62] Corby Rosset, Ching-An An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models self-immrove with general preferences. ArXiv, abs/2404.037115 2024.
[63] Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747, 2016.
[64] Michael Santacroce, Yadong Lu, Han Yu, Yuanzhi Li, and Yelong Shen. Efficient RLHE: Reducing the memory usage of PPO. arXiv preprint arXiv:2309.00754. 2023.
[65] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (xXv:/707.06347,, 2017.
[66] Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations RLHF. arXiv preprint arXiv:2310.03716, 2023.
[67] Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. In AAAI, 2024.
[68] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
[69] Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, al. Understand- ing the performance gap between online and offline alignment algorithms. arXiv preprint arXiv:2405. 08448, 2024.
[70] Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach offline alignment. arXiv preprint arXiv:2402.057499 2024.
[71] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-followin Ilama model, 2023.
[72] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. 2024.
[73] Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. Fine- tuning language models for factuality. In The Twelfth International Conference on Learning Representations, 2024.
[74] Hoang Tran, Chris Glaze, and Braden Hancock. Iterative DPO alignment. Technical report, Snorkel AI, 2023.
[75] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of LM alignment. ArXiv, abs/2310.16944, 2023.
[76] Boxin Wang. Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zi-Han Lin, Yuk Kit Cheng, Sanmi Koyejo. Dawn Xiaodong Song, and Bo Li. DecodingTrust. A comprehensive assessment of trustworthiness in GPT models. In NeurIPS, 2023.
[77] Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu, OpenChat: Advancing open-source language models with mixed-quality data. In ICLR, 2024.
15

---

[78] Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of LLMs for diverse user preferences: Directional preference alignment with multi-objeetive rewards. In ACI., 2024.
[79] Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable pref- erences via multi-objective reward modeling and mixture-of-experts. Findings of EMNLP, 2024.
[80] Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. In Thirty-seventh Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[81] Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. LESS: Selecting influential data for targeted instruction tuning. In ICML, 2024.
[82] Wei Xiong, Hanze Dong, Chenlu Ye. Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for RLHF under KL-constraint. In Forty-first International Conference on Machine Learning, 2024.
[83] Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Ken- ton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of L.LM performance in machine translation. ArXiv, abs/2401 008417, 2024.
[84] Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.16682. 2023.
[85] Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is DPO superior to PPO for LLM alignment? a comprehensive study. arXiv preprint arXiv:2404. 10711 2024.
[86] Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang. Songfang Huang, and Fei Huang. RRHF: Rank responses align language models with human feedback. In NeurlPS, 2023.
[87] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar. Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10022 2024.
[88] Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. WildBench: Benchmarking LLMS with challenging tasks from real users in the wild. arXiv e-prints, pages arXiv-2406, 2024.
[89] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Mårquez, edi- tors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800, Florence, Italy, July 2019. Association for Computational Linguistics.
[90] Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yumtian Deng. WildChat: IM ChatGPT interaction logs in the wild. In The Twelfth International Conference on Learning Representations, 2024.
[91] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter Liu. SLiC-HF: Sequence likelihood calibration with human feedback. ArXiv, 10s/23 2023.
[92] Chujie Zheng, Pei Ke, Zheng Zhang, and Minlie Huang. Click: Controllable text generation with sequence likelihood contrastive learning. In Findings of ACL, 2023.
[93] Chujie Zheng, Ziqi Wang, Heng Ji, Minlie Huang, and Nanyun Peng. Weak-to-strong extrapola- tion expedites alignment. arXiv preprint arXiv:2404. 16792, 2024.
[94] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Fric Xing, et al. Judging auud-ee- with MT-Beecch and Chatbot Arena. In NeurIPS Datasets and Benchmarks Track, 2023.
16

---

[95] Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. Secrets of RLHF in large language models part PPO. arXiv preprini arXiv:2307.04464, 2023.
[96] Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. LIMA: Less is more for alignment. NeurIPS, 2023.
[97] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprinl 1909008999 2019.
17

---

A Limitations
More in-depth theoretical analysis. Despite the empirical success and intuitive motivation of SimPO, a more rigorous theoretical analysis necessary to fully understand the factors contributing to its effectiveness. Additionally, we introduce an additional hyperparameter, the target reward margin, which requires manual tuning. Future work could explore how to determine the optimal margin automatically and provide more theoretical understanding of SimPO.
Safety and honesty. SimPO is designed optimize the generation quality of language models by pushing the margin between the average log likelihood of the winning response and the losing response exceed target reward margin. However, it does not explicitly consider safety and honesty aspects, which are crucial for real-world applications. Future work should explore integrating safety and honesty constraints into SimPO ensure that the generated responses are not only high-quality but also safe and honest. The dataset used in this work, UltraFeedback [23], primarily focuses on helpfulness, and future research may consider a more comprehensive study utilizing larger-scale preference datasets [39, 90] and evaluation benchmarks [76] that place strong emphasis on safety aspects. Nonetheless, we observe that this method consistently achieves high TruthfulQA [53] performance compared to other objectives in Table 9, suggesting its potential for safety alignment.
Performance drop math. We observed that preference optimization algorithms generally decrease downstream task performance, particularly on reasoning-heay tasks like GSM8k, as shown in Table 9. SimPO occasionally results in performance comparable hypothesize that this may be related to the choice of training datasets, hyperparameters used for training, mismatch of chat templates used for downstream task evaluations. One explanation is that the preference optimization objective may not be effectively increasing the likelihood of preferred sequences despite increasing the reward margin. [59] first observed this phenomenon and point out that this can hinder learning from math preference pairs where changing one token can flip the label (e.g., changing 2 = to 2 2 5). They propose a simple regularization strategy to add back a reference- seference-model calibrated supervised fine-tuning loss to the preference optimization objective, and effectively mitigate this issue. Future work may consider integrating this regularization strategy into SimPO to improve performance on reasoning-heay tasks.
worse than DPO. We
B Implementation Details
We find that hyperparameter tuning is crucial for achieving optimal performance of preference optimization methods. However, the importance of careful hyperparameter tuning may have been underestimated in prior research, potentially leading to suboptimal baseline results. To ensure fair comparison, we conduct thorough hyperparameter tuning for all methods compared in our experiments.
General training hyperparameters. For the Base training setups, we train SFT models using the UltraChat-200k dataset [25] with the following hyperparameters: a learning rate of 2e-5, a batch size of 128, a max sequence length of 2048, and a cosine learning rate schedule with 10% warmup steps for epoch. All the models are trained with an Adam optimizer [43].
Table 8: The hyperparameter values in SimPO used for each training setting.
Setting
Y
Learning rate
Mistral-Base
2.0
1.6
3c-7
Histral-Instrun
2.5
0.3
Se-7
Llama-3 3-
2.0
1.0
6e-7
-lama---Insttee 2.5
1.4
le-6
For the preference optimization stage, conduct preliminary experiments to search for batch sizes in [32, 64, 128] and training epochs in [1, 2, 3]. We find that batch size of 128 and single training epoch generally yield the best results across all methods. Therefore, we fix these values for all preference optimization experiments. Additionally, we set the max sequence length be 2048 and apply cosine learning rate schedule with 10% warmup steps on the preference optimization dataset.
Nethod-speciffe training hyperparameters. We have noticed that the optimal learning rate varies for different preference optimization methods and greatly influences the benchmark performance. Therefore, individually search the learning rates in the range of [3e-7, 5c-7, 6c-7, 1c-6] for each
18

---

Table 7: Various preference optimization objectives and hyperparameter search range.
Method
Objective
Hyperparameter
RRHF (86)
max (0. Ivua log Ny (x) + sul log mo (x)) A log mo(lw
[0.1, 0.5, 10, 10.0)
A 0.5, 100, 10.0) 8 [0.1, 0.5, 1.0, 2.0)
SLIC-HF [91]
max (0, 6 log Tho (Umrlt))) log mo (ar)) Alg mo(ww()))
DPO [61]
log ( log TurlRulF) (R)
salowla)
5 log Me(wila)
murlor =)
8 e [0.01, 0.1]
IPO [6]
llog reltssle)
Tniwil)) log markinla
[0.01, 0.1, 1.0)
27
CPO [83]
log or [18 log No (Mu in) log mol mn) (oull))
A 1.0, (0.01,0.05,0.
salwwll)) Au log Turlin (F)
Fret Ayer Free 13 log mafsnla))
KTO [29]
where Andt B(a [ØKL (url))))) (ar(y)a))) mertuils))
8 (0.01.0.05. 0.1]
(e Alog log pollwle) Potsull) where po(y) 22) exp [9] log (o(())))
log PoP0011
ORPO [38]
A 0.51 1.0, 2.0]
R-DPO [60]
log log zelawlz) (arlbwll) Rarlor x)
log reluixz + (alyill (omul-alppt alw/1)
0.05, 0.1,0.5, 1.0) 8 (0.01, 0.05, 0.1]
SimPO
Ing
log (o(ww(
Del log (or -7)
3 [2.0, 2.5]
[0.3, 0.5, 1.0, 1.
Avg. log prob. polgla ax)
To 1!
40
Training metric
16. 7k
10000
20
-10
Fw: < n1
13.8k
0
0.33
15
200
400
600
800
1000
Po Pt
Pe: < Pt
Response length ly]
Generation metric
Mistral-SFT.
(b) Contingency table (SimPO).
Figure 6: (a) Likelihood-lengii correlation plot for Mistral-SFT fine-tuned on UltraChat-2000 200k. (a) Contingency table rankings based on SimPO rewards and the average log likelihood (measured on the training set).
method. Table shows the detailed information on method -seecfffc hyperparameters search ranges for baselines." 13 Table 8 shows SimPO's hyperparameters used under each setting.
Decoding hyperparameters. For AlpacaEval 2, we use sampling decoding strategy to generate responses, with temperature 0.7 for the Mistral-Base setting following zephyr- 7b- be temperature of 0.5 for the and temperature 0.9 for both Llama settings. For Frena-Hard, we use the default greedy decoding for all settings and methods. For MT-Bench, we follow the official decoding configuration which defines different sampling temperatures for different categories.
setting following Snorkel w-teeeeiaiaaalllaaaleaa PettrralPaaiin -atssrraa-aaaa--PeePPP PRM-DO,
Computation environment. All the training experiments in this paper conducted 8xH100 GPUs based on the alignment-handee handbook repo." 16
Phhere is a discrepancy between the KTO runs in their original paper, where the original runs use RMSProp optimizer [63]. We use Adam optimizer [43] for all the experiments. "nttps:// //ttttbb www.niigs/epphrr//////
eval/
yaml
15 grid search the temperature hyperparameter for Llama-3-Base setting with DPO over 0.1, 0.1, 0.5, 0.7, 0.9, and for all different methods.
httpps: / /gi thub. con/huggingtace wtaptface/aaaaggge http://aaaagaa/eeeee///////eeeee -iigmeent-haddooo
19

---

C Downstream Task Evaluation
To examine how preference optimization methods affect downstream task performance, evaluate models trained with different methods on various tasks listed on the Huggingface Open Leader- board [9]. These tasks include MMLU [36], ARC [19], HellaSwag [89], TruthfulQA [53], Wino- grad [48], and GSM8K [20]. We follow the established evaluation protocols and present the results for all models in Table 9, Generally, find that preference optimization 's effect varies across tasks.
Knowledge largely retained with a small loss. Compared to the SFT checkpoint, we find that all preference optimization methods generally maintain MMLU performance with minimal decline. In this aspect, SimPO is largely comparable to DPO.
Reading comprehension and commonsense reasoning improves. For ARC and HellaSwag, preference optimization methods generally improve performance compared the SFT checkpoint. One hypothesis is that the preference optimization dataset contains similar prompts to these tasks, which helps the model better understand the context and improve reading comprehension and commonsense reasoning abilities.
Truthfulness improves. Surprisingly, we find that preference optimization methods consistently improve TruthfulQA performance compared to the SFT checkpoint, and the improvement could be as high as over 10% in some cases. Similarly, we hypothesize that the preference dataset contains instances that emphasize truthfulness, which helps the model better understand the context and generate more truthful responses.
Math performance drops. GSM8K the benchmark that shows the most volatility across methods. Notably, except for ORPO, almost all approaches lead to consistent drops in one more settings. We hypothesize that ORPO retains performance largely due to its supervised fine-tuning loss for regulation. [59] adds reference-model calibrated supervised fine-tuning loss to the preference optimization objective, and find that it effectively solves the issue and maintains performance on math tasks well.
Overall, identifying a pattern downstream performance is challenging. Comprehensive analysis is difficult due to using different pretrained models, preference optimization datasets, and objectives. Recent works indicate that gradient- baaeed approaches could be effective in finding relevant data for downstream tasks [81], and could possibly extended to understand the effect of preference optimization. We believe a thorough study on how preference optimization affects downstream performance would be valuable and call for a rigorous and more comprehensive analysis in future work.
D Standard Deviation of AlpacaEval and Arena-Hard
We present the standard deviation of AlpacaEval and the 95% confidence interval of Arena-Hard in Table 10, All these metrics are reasonable and do not exhibit any significant outliers or instability.
E Generation Length Analysis
Length normalization decreases generation length and improves generation quality. Remov- ing length normalization from the SimPO objective results in an approach similar to Contrastive Preference Optimization (CPO) [83], which interpolates reward maximization with a supervised fine-tuning loss and has demonstrated strong performance in machine translation. However, without the supervised fine-tuning loss, the reward maximization objective without length normalization is suboptimal in preference optimization.
We analyze the generation length of models trained with or without length normalization on Al- pacaEval 2 and Arena-Hart As shown in Figure 6, length normalization significantly decrease the generation length by up to 25% compared to when it not used in most cases. However, even though the generation length is shorter, the models with length normalization consistently achieve much higher win rates on both benchmarks. This suggests that length normalization can effectively control the verbosity of the generated responses, and meanwhile improve the generation quality.
20

---

Table Downstream task evaluation results of tasks on the huggingface open leaderboard.
MMLU (5)
ARC (25)
HellaSwag (10)
TruthfulQA (0)
Winograd (5)
GSM8K (5)
Average
Mistral-Base
SFT
60.10
58.28
80.76
40.35
76.40
28.13
57.34
RRHF
57.41
52.13
80.
43.73
76.64
4.78
52.48
SLiC-HF
59.24
55.38
81.15
48.36
77.35
33.74
59.20
DPO
58.48
61.26
83,59
53.06
76,80
21.76
59.16
IPO
60.23
60.84
83,30
45.44
77.58
27.14
59.09
CPO
59.39
57.00
80.75
47.07
76.48
33.06
58.96
KTO
60.90
62.37
84.88
77.27
38.51
63.42
ORPO
63.20
61.01
84.09
78.61
R-DPO
59.58
61.35
84.29
46.12
76.56
18.12
57.67
SimPO
59.21
62.63
83.60
50.68
77.27
22.21
59.27
SFT
60.40
63.57
84.79
66.81
76.64
40.49
65.45
RRHF
59.75
64.42
85.54
67.98
76.64
37.76
65.35
SLIC-HF
60.59
59,90
84.05
65.30
76.32
39.65
64,30
DPO
60.53
65.36
85.86
66.71
76.80
40.33
65.93
IPO
60.20
63.31
84.88
67.36
75.85
39.42
65.17
CPO
60.36
63.23
84.47
67.38
76.80
38.74
65.16
KTO
60.52
65.78
85.49
68.45
75.93
38.82
65.83
ORPO
60.43
61.43
84.32
66.33
76.80
36.85
64.36
R-DPO
60.71
66,30
86.01
68.22
76.72
37.00
65.82
SimPO
60.53
66.89
85.95
68.40
76.32
35.25
63.56
Llama-3-Base
SFT
64.88
60.15
81.37
75.77
62.30
RRHF
64.71
62.12
82.03
77.51
64.27
SLiC-HF
64.36
61.43
81.88
54.95
77.27
48.82
64.79
DPO
64.31
64.42
83.87
53.48
76.32
38.67
63.51
IPO
64.40
62.88
80.46
54.20
72.22
22.67
59.47
CPO
64.98
61.69
82.03
54.29
76.16
46.93
64.35
KTO
64.42
63.14
83.55
55.76
76.09
38.97
63.65
ORPO
64.44
61.69
82.24
56.
77.51
50.04
65.34
R-DPO
64.19
64.59
83.90
53.41
75.93
39.27
63.55
SimPO
64.00
65.19
83.09
59.46
77.19
31.54
63.41
Llama-3- Instrrcct
SFT
67.06
61.01
78.57
51.66
74.35
68.69
66.89
RRHF
67.20
61.52
79.54
53.76
74.19
66.11
67.05
SLiC-HF
66.41
61.26
78.80
53.23
76.16
66.57
67.07
DPO
66.88
63.99
80.78
59.01
74.66
49.81
65.86
IPO
66.52
61.95
77.90
54.64
73.09
58.23
65.39
CPO
67.05
62.29
78.73
54.01
73.72
67.40
67.20
KTO
66.38
63.57
79.51
58.15
73.40
$7.01
66.34
ORPO
66.41
61.01
79.38
54.37
75.77
R-DPO
66.74
64.33
80.97
74.82
65.18
SimPO
65.63
62.80
78.33
60.70
73.32
50.72
65.25
Length is not reliable indicator generation quality. We further analyze the generation length of models trained with different methods on AlpacaEval 2 and Arena-Hard, as shown in Table 10. Generally, we find that no single method consistently generates longer or shorter responses across all settings. Additionally, even though some methods may generate longer responses, they do not necessarily achieve better win rates on the benchmarks. This indicates that the length the generated responses is not reliable indicator of generation quality.
SimPO demonstrates minimal exploitation of response length. We observe that SimPO has shorter generation length compared DPO the Llama-3-Instruct case but exhibits higher generation length in other settings, with up to 26% longer responses AlpacaEval 2. Conversely, SimPO only increases length by only around 5% on Arena-Hard compared DPO. fair to say that the generation length heavily depends on the evaluation benchmark. A stronger indicator is that SimPO consistently achieves a higher length-controll win rate on AlpacaEval compared the win rate, demonstrating minimal exploitation of response length.
21

---

100
100
78.6%
9.6%
A A<B
56.9%
19.5%
A = DPO
60
= DPO
00
40 %
%
4> 8
A> B
40
2.6%
9.2%
5.7%
17.9%
20
20
A B
A a>= B
A B
A== B
SimPO
SimPO
(a) Mistral-Bant
(b) Mistral-Instruct.
Figure 7: Win rate heatmap of Mistral-Base and Mistral-Instruct on AlpacaEval 2. B represents the baseline model (i.e, GPT-4-Previee- 1106)
20
20
18.5%
13.0%%
1.2%
5.9%
0.6%
13.4%
11.1%
2.5%
6.7%
0.6%
12..9%
16.95
2.00%
9.955
0.9%
10.3%
13.65
2.1%
8.85
1.0%
2.5%
10%
3.0%
10%
3.85%
3.8%
0.8%
2.456
0.4%
4.15%
6.1%
1.3%
6.156
0.65%
5
5
0.3%
0.5%
0.00%
0.3%
0.0%%
0.4%
0.496
0.1%%
0.7%
0.3%
A
4
433 8
439
SinPO
SinPO
(a) Mistral-Bace
(b) Mistral-Instruct.
Figure 8: Win rate heatmap of Mistral-Base and Mistral-Instruct Arena-Hard. B represents the baseline model (i.e., GPT-4-0314).
F Gradient Analysis
We examine the gradients SimPO and DPO to understand their different impact on the training process.
WDD ToLsimP((TO) -BE
(. (z.
Se
V log me (y (x)
V log Te (zr)
yuul
incretioe likelihood on thu
docresse Rikelihood
(7)
CoLoPo(TT) -BE wD de
Vo log IT' (yo likelilhood you
Va log no (
likelihooo on 9n
where
89 ( 3 [
log (x)la))
B Murl
log no (yel +
+7).
do (Blog
To (21 (x) Trer( (2)
log Te (Man (x) कात (Mer (=)
represent the gradient weight in SimPO and DPO, respectively. It can be seen that the differences are twofold: (1) comparing the gradient weights sy and dg, SimPO's gradient weight sy does not involve the reference model and has straightforward interpretation: the weights will be higher for samples where policy model incorrectly assigns higher likelihood to yı than yor (2) comparing the gradient updates, SimPO's gradients on yı and yo are This corresponds the empirical findings [60] that DPO may exploit length bias: longer sequences with more tokens will receive larger gradient updates in DPO, dominating the training process.
while DPO's are not.
22

---

G
Qualitative Analysis
We present the win rate heatmap of Mistral-Base and Mistral-Instruct on AlpacaEval and Arena- Hard in Figure 7 and Figure 8, respectively. Based on this analysis, we present qualitative examples of responses generated by SimPO model, DPO model and the baseline model GPT-44Preview-1100 on AlpacaEval 2.
Comparing SimPO with DPO. Figure and Figure 10, we present an example where Mistral- Base-SimPO generates a better-structure answer compared to Distral-Base-DDe Given the question, "How can you determine a person genuinely interested in conversation or simply being polite?", the DPO model generates response with long list of bullet points, making it difficult to understand the relationships between different points. contrast, the SimPO model produces well-structured answer with high-level categorization of different behaviors, followed by detailed suggestions for each category. This makes the answer more readable and easier to understand.
Comparing Instruct models with Base models when trained with SimPO. In Figure 11, present an example where -lama-3-IIntrrus generates a more detailed and well-formatted answer compared to the baseline model, and as well as the Alama-3-Base-Simee What language does Argentina people speak? 1VTT4-Preview-1000 1106 gives more detailed answer in explaining how the Argentina Spanish differs from standard Spanish. However, the answer is not well formatted and a bit hard to parse. Llama-3- Instruct-Simte SimPPO gives detailed and well fol-mtormtteed answer, which is easier to read and understand, and offers sufficient details.
model. Given the question: only gives a very brief answer.
23

---

Figure 9: An AlpacaEval 2 generation from the MistrallBae model after training with DPO.
Input
Q: How can you determine if a person is genuinely interested in a conversation or simply being polite?
Mistral-Ban DPO
AI, I dont have access to contextual information nonverbal cues, but here general signs that may indicate whether soneone is genuinely interested in conversation or being polite:
Active listening: genuinely interested person will actively listen to what you'e saying, ask questions, and provide feedback or comments. They may alsc maintain eye contact, nod, or use verbal cues like "uh-huh" theyfe engaged.
see" show
2. Follow-up questions: A genuine interest in the conversation often leads to follow-up questions. someone asking questions that show theyre interested in learning more about the topic, its a good sign theyfe engaged.
Body language: Nonverbal cues like leaning in, smiling, and mirroring your body language can indicate genuine interest. However, note that sone people may have different boay language styles, this may not always be reliable indicator.
4. Topic continuation. If the conversation flows naturally, and both parties are contributing equally, its sign of mutual interest. If one person cominates the conversation keeps steoring it back to their own interests, it may indicate lack of genuine interest.
Time spent: IF the conversation lasts longer than expected or planned, ité a sign that both parties are engaged and interested.
Follow-up actions: If someone shows genuine interest, they may follow up with you after the conversation, whether ité through a message, email, or phone call.
Remember, however, that sometimes people may polite or professional, even if theyře net genuinely interested. Itš ossential to consider the context and the relationship you have with the person to determine their level of interest accurately.
24

---

Table 10: Detailed results of AlpacaEval 2 and Arena-Hard. LCC. means length-controlled win rate, WR means raw win rate, and STD means standard deviation of win rate. Length is the average generation length. For Arena-Hard, we report the win rate and 95% confidence interval.
AlpacaEval 2
Arena-Hard
Models
LC (%)
WR (6)
STD (%)
Length
WR
high
95 CI low
Length
Mistral-Base
SFT
8.4
6.2
1.1
914
1.3
1.8
0.9
521
RRHF
11.6
10.2
0.9
1630
6.9
8.0
596
SLiC-HF
10.9
8.9
0.9
1525
7.3
8.5
6.2
683
DPO
15.1
12.5
1.0
1477
10.4
11.7
9.4
628
IPO
1380
8.5
674
CPO
9.8
8.9
0.9
1827
5.8
6.7
4.9
823
KTO
13.1
9.1
0.9
1144
5.6
6.6
4.7
475
ORPO
12.2
1475
764
R-DPO
17.4
12.8
1.0
1335
9.9
11.1
8.4
528
SimPO
21.4
20.8
1.2
1868
16.6
18.0
15.1
699
Mistral- Mistral-Instruct
SFT
17.1
14.7
1.1
1676
12.6
14.1
11.1
486
RRHF
25.3
24.8
1.3
1927
18.1
19.5
16.4
517
SLIC-HF
24.6
2088
18.9
20.6
578
DPO
26.8
24.9
1.3
1808
16.3
18.0
15.2
518
IPO
20.3
20.3
1.2
2024
16.2
17.9
14.4
740
CPO
23.8
28.8
1.3
3245
22.6
25.0
20.8
812
KTO
24.5
23.6
1.3
1901
17.9
20.3
16.1
496
ORPO
24.5
24.9
1.3
2022
20.8
22.5
19.1
527
R-DPO
27.3
24.5
1.3
1784
16.1
18.0
14.6
495
SimPO
32.1
34.8
1.4
2193
21.0
22.7
18.8
539
Llama-3-Bat
SFT
1082
4.0
437
RRHF
10.8
8.1
0.9
1186
6.6
7.5
5.7
536
SLiC-HF
12.1
10.1
0.9
1540
10.3
11.5
8.9
676
DPO
18.2
15.5
1585
15.9
18.1
563
IPO
14.4
14.2
1.I
1856
17.8
19.5
16.0
608
CPO
12.3
13.7
1.0
2495
11.6
13.2
10.4
800
KTO
12.4
1646
12.5
14.2
10.9
519
ORPO
12.2
10.6
0.9
1628
10.8
12.3
9.6
639
R-DPO
17.6
14.4
1.1
1529
17.2
18.5
15.7
527
SimPO
22.0
20.3
1.2
1795
23.4
25.4
21.6
704
Llama -lama-3-Instret
SFT
26.0
25.3
1.3
1920
22.3
23.9
20.3
596
RRHF
31.3
28.4
1.33
1805
26.5
28.4
24.6
502
SLiC-HF
26.9
27.5
1.3
1977
26.2
28.4
24.4
584
DPO
40.3
37.9
1,4
1883
32.6
34.8
30.3
528
IPO
35.6
35.6
1.4
1983
30.5
32.8
28.4
554
CPO
28.9
32.2
1.4
2166
28.8
30.6
26.6
624
KTO
33.1
31.8
1.4
1909
26.4
28.7
24.3
536
ORPO
28.5
27.4
1.3
1888
25.8
27.4
23.8
535
R-DPO
41.1
37.8
1.4
1854
33.1
35.3
30.9
522
SimPO
44.7
40.5
1.4
1825
33.8
35,9
32.0
504
25

---

Table 11: Average response lengths on AlpacaEval 2 and Arena-Hard trained with Mistral-Base Mistral-Instruct.
AlpacaEval
Arena-Hard
Model
Mistral-Base
Mistral-Instruct
Mistral-Base
Mistral-Instruct
LC (%)
Length
LC (%)
Length
WR (%)
Length
WR (%)
Length
SimPO
21.5
1868
32.1
2193
16.6
699
21.0
539
SimPO w/o LN
11.9
2345
19.1
2067
9.4
851
16.3
679
26

---

Figure 10: An AlpacaEval generation from the Mistral-Base model after training with SimPO. Compared to the output generated by the DPO model, as shown Figure 9, the generation by SimPO is better structured with hierarchical discussions, making the information more clearly presented and readable.
Input + Q: How can you determine if a person is genuinely interested in a conversation or simply being polite?
Mistral-Bar SimPO + Hore are some indicators to consider:
Engagement:
Active listening: Does the persen maintain eye contact, nod, ask follow-up questions, and respond fully individuals tend show genuine interest by demonstrating attentiveness and participation in the conversation.
what you say? Engaged
Initiative: Does the person take the lead in steering the conversation, expressing enthusiasm, sharing personal experiences, or asking questions beyond polite small talk? Someone who genuinely cares about the topic or connection may show initiative in keeping the discussion engaging.
Body language: Observe nonverbal cues such as leaning in, mirroring posture, facial expressions, and gestures that convey enthusiasm, curiosity, or openness. These signals suggest a deeper level of interest beyond mere politeness.
2. Content and depth:
Topic selection: Does the persen show genuine curiosity or passion about the subject matter, asking relevant questions, sharing insights, or expressing opinions? If they seem invested in exploring the topic's nuances, it indicates politeness.
genuine interest beyond
Level of detail: Does the persen provide specific examples, personal anecactes, or elaborate on their thoughts, incicating deeper level of engagement and investment in the conversation?
3. Pollox-up actions:
Continued communication: Does the person initiate further conversations, send follow-up messages, or suggest meeting up outside the initial interaction? Such actions suggest a genuino desire to maintain the connection and explore further topics, indicating genuine interest beyond
politeness.
Personalirations Does the person remember details discussed earlier, reference shared interests, or show interest in Learning more about your life experiences? Perscnalized interactions indicate a genuine interest building a relationship beyond mere politeness.
Overall, genuine interest in conversation involves active participation, enthusiasm, depth, and follow-up actions that demenstrate a genuine desire to connect and engage beyond more politeness.
27

---

Figure 11: A case study on AlpacaEval 2 demonstrates that Lama-3-Instrucce trained with SimPO, provides a better formatted and more detailed answer than both Llama-3- Base, also trained with SimPO, and the baseline model GPT-4- 106-Preview. This illustrates how the instruction setting typically outperforms the base setting.
Input
Q: what language does a argenting people speak
Ll --aaee SimPO
The official language spoken in Argentina is Spanish, specifically Argentine Spanish, which shares similarities with other varieties Spanish spoken in reighboring countries like Gruguay, Chile, and Paraguay.
Ll 1llam---- 1ama-3-Instruct SimPO
In Argentina, the official language is Spanish. Specifically, the dialcet spoken in Argentina is known as Rioplatense Spanish, which is a variation of Spanish with some unique characteristics and influences from Italian, German, and other immigrant languages.
Rioplatense Spanish distinguished by: Instead of using tú (you)
1. Voseo:
as the informal singular proncun, Argentines use and its associated verb conjugations.
Lunfardo: A slang vocabulary derived from Italian, Corsican, and other immigrant languages, which has been incerporated into everyday speech.
Accent and promunciation: Argentines tend to pronounce the "c" and "x" sounds with soft. "s" or "sh" sound, rather than hard "th" sound like in standard Spanish.
Overall, Argentine Spanish known for its distinctive flavor and is widely understood throughout the Spanish-speaking world.
However, it's worth noting that there may regional variations within Argentina, and some dialects may differ slightly from the standard Rioplatense variety.
gpt4-turbo-1166-rres
The primary language spoken in Argentina is Spanish. Specifically, the variety of Spanish spoken there is known as Rioplatense Spanish, which is characterized by its unique accent and the use of the pronoun "vos" instead of "tú" for the second person singular. Accitionally, there are various indigenous languages spoken in different regions the country, as well as immigrant languages such as Italian, German, and English, but these are much less widespread than Spanish.
28

---

H Llama-3-Instruct v0.2 (Jul 7, 2024)
Table 12: Results Llama-3-Instruc (8B) setting, utilizing preference labels annotated by stronger reward model (ArmoRM [79], we term version 0.2).
Llama-3- --nsrruct (8B)
Method
AlpacaEval
Arena-Hard
MT-Bench
(S)
Length
Length
GPT44
GPT-4
SFT
26.0
25.3
1920
22.3
596
6.9
8.1
SimPO v0.1
44.7
40.5
1825
33.8
504
7.0
8.0
RRHF [86]
37.9
31.6
1700
28.8
467
7.1
8.2
SLiC-HF [91]
33.9
32.5
1938
29.3
599
6.9
8.1
DPO [61]
48.2
47.5
2000
35.2
609
7.0
8.2
IPO [6]
46.8
42.4
1830
36.6
527
7.2
8.2
CPO [83]
34.1
2086
30.9
604
7.2
8.2
KTO [29]
34.1
32.1
1878
27.3
541
7.2
ORPO [38]
38.1
33.8
1803
28.2
520
7.2
8.3
R-DPO [60]
48.0
45.8
1933
35.1
608
7.0
8.2
SimPO v0.2
53.7
47.5
1777
36.5
530
7.0
8.0
Table 13: Downstream task evaluation results of tasks on the huggingface open leaderboard.
MMLU (5)
ARC (25)
HellaSwag (10)
TruthfulQA (0)
Winograd (5)
GSM8K (5)
Average
1lama-3-Instrun
SFT
67.06
61.01
78.57
51.66
74.35
68.69
66.89
RRHF
67.20
61.52
79.54
53.76
74.19
66.11
67.05
SLiC-HF
66.41
61.26
78.80
$3.23
76.16
66.57
67.07
DPO
63.99
80.78
59.01
74.66
49.81
65.86
IPO
66.52
77.90
$4.64
73.09
58.23
65.39
CPO
67.05
62.29
78.73
$4.01
73.72
67.40
67.20
KTO
66.38
63.57
79.51
$8.15
73.40
$7.01
66.34
ORPO
66.41
61.01
79.38
54.37
75.77
64.59
66.92
R-DPO
66.74
80.97
60.32
74.82
43,90
65.18
SimPO
65.63
62.80
78.33
60.70
73.32
50.72
65.25
Llama-3-Instruct
v0.2
SFT
67.06
61.01
78.57
$1.66
74.35
68.69
66.89
RRHF
66.60
63.74
80,98
59.40
76.32
58.68
67.62
SLiC-HF
66.91
61.77
79.17
56.36
76.40
68.23
68.14
DPO
67.33
64.08
80.08
56.33
75.61
54.51
66.32
IPO
67.32
63.23
78.71
58.12
74,51
56.33
66.37
CPO
66.86
62.80
79.10
55.62
73.88
67.78
67.67
KTO
67.25
63.57
79.66
$5.56
74.98
66.41
67.91
ORPO
66.78
63.40
80.09
$7.52
76.72
66.72
68.54
R-DPO
80,22
$6.44
75.61
$2.99
66.17
SimPO
66.51
66.64
78.97
63.86
74.74
55.65
67.73
SimPO w/ SFT
66.74
63.82
78.82
60.52
73.72
64.06
67.95
In this section, update the Llama-3-Instruet setting, primarily by utilizing stronger reward model to annotate our generated preference dataset.
Enhanced reward model yields significantly better results. our previous version, we use PairRM [41] as reward model to rank generated candidate responses. The results, presented in Table 12, show that switching the reward model from PairRM [41] to ArmoRM [79] for ranking the data markedly improves model performance. This underscores the importance of high-quality preference optimization dataset for enhancing performance. Notably, SimPO has achieved 53.7 LC win rate AlpacaEval 2 and 36.5 on Frena-Hard surpassing the previous version by 9.0 and 2.7 points, respectively.
We use the following hyperparameters for SimPO under the Llama-3 Instruct v0.2 setting: , = 10 and > = 3. The other hyperparameters (e.g., learning rate, batch size, max sequence lengths) are kept the same as the original Llama-3-88 (amma---BB-Imaa 8-8B-Instruct setting.

---

Strong SFT model and high-quality policy data diminish algorithm differences. With a strong SFT model like Mlama---BBIInstrue and the preference optimization data quality improves, the differences between algorithms become less pronounced. For instance, DPO achieved similar win rate as SimPO in terms of raw win rate, and DPO, IPO, and R-DPO all exhibited comparable raw win rates on Arena-Hard. However, SimPO maintains an advantage by producing shorter sequences, resulting in a significantly better LC win rate on AlpacaEval 2.
Stronger downstream task performance. The v0.2 ver- sion also shows improved performance in downstream tasks across various objectives. However, DPO, IPO, R-DPO, and SimPO continue experience decline in reasoning- intensive domains such as GSM8K. In contrast, objectives that include an SFT component maintain their performance in mathematical tasks.
Table 14: AlpacaEval performance of SimPO and SimPO with an SFT loss.
Method
LC (%)
WR (%)
SimPO v0.2
53.7
47.5
w/ SFT
41.4
36.5
Incorporating SFT regularization in SimPO. Several reference-free algorithms, including RRHF [86], SLiC-HF [91], CPO [83], and ORPO [38], employ SFT regularization in their ob- jectives. SFT regularization can be an effective method to prevent reward hacking, ensuring that the solution maintains low loss without resulting in degraded generations. We also experiment with the integration of an SFT loss in SimPO, yielding the following objective:
SsimPO w/ SET (To) BB...... log
log (o (x)
B
log No (bnl2)) (p)ll))>>) log To (Vwlz)))
[gvel
As shown in Table 14, the addition of the SFT regularization leads to a decrease in performance on AlpacaEval 2. However, we note that SFT regularization provides substantial benefits certain tasks such as GSM8K, as shown in Table 12. These contrasting results suggest that the impact of SFT in preference optimization may vary depending on the training setup and the nature of the task. Further comprehensive studies on this topic are left for future research.
I Applying Length Normalization and Target Reward Margin to DPO (Jul 7, 2024)
Since the release of the paper, we have had inquiries from researchers about whether the key design elements of SimPO Lingh-lengthh length normalization and target reward margin margin-could benefit DPO. By doing so, we will derive the following two objectives:
(
3 log
( x)
( x)) Trer(11
LDoo w/ LTO Teet) = - -E (w..v..r) log a
|vvl log
|Vvll
Teet Yyw
(yw
xx)
x))
No (Bw
x))
To 1u
x))
Cpeo w/ (TT; Teet) - - BB
log B3log
3 log
(
xx)
Weet yy
x
An intuitive understanding of how length normalization could benefit DPO is that, despite DPO's reward design being implicitly normalized by the reference model, the policy model might still exploit
length bias from the data, resulting in a disproportionately high probability for longer sequences. Applying length normalization could help mitigate this effect.
We train models with the objectives mentioned above and compare their performance to that of DPO and SimPO, as shown in Table 15.
The results indicate that, unlike SimPO, length normalization and target reward margin do not consistently benefit DPO. Specifically, length normalization significantly improves DPO performance only in the Mistral-Bart setting, where the preference optimization dataset shows strong length bias. However, it does not provide a benefit the Mistral-Instrnn setting, where the lengths of winning and losing responses are comparable. This is likely because DPO already includes implicit instance-wise target reward margin via the reference model, as shown in the derivation below.
To ( x))
To ( x))
PPO = log a
( log
,3 log
Trer
x))
Teer (1r xx)
= =log 0
(x) - B log To
(x) - (3 (3 log Tree (y x)) B log (er xx)
30

---

Table 15: Applying length normalization (LN) and target reward margin (7)
DPO.
Mistral-Base (7B) Setting
Mistral- (7B) Setting
Method
AlpacaEval 2 Arena-Hard
MT-Bench
AlpacaEval 2
Arena-Hard
MT-Beecch
(%)
WR (%)
WR (%)
GPT-4 Turbo
GPT-4
LC (%)
WR (%)
WR
GPT-4 Turbo
GPT-
SimPO
21.5
20.8
16.6
6.0
7.3
32.1
34.8
21.0
6.6
7.6
DPO
15.1
12.5
10.4
5.9
7.3
26.8
24.9
16.3
6.3
7.6
w/ LN
21.0
17.7
15.2
5.9
7.2
21.7
20.9
15.6
6.4
7.7
w/ y
15.2
12.1
10.3
5.7
7.3
23.0
24.6
14.7
6.3
7.6
Applying SimPO to Gemma 2 Models (Sept 16, 2024)
Performance degradation on other benchmarks for Llama-3- checkpoints. After releas- ing the Llama-3-SimPO checkpoints, we received extensive feedback about performance degradation on benchmarks measuring specific capabilities, such as MMLU and GSM8K. To investigate this issue, we continued training the Llama-3-8B-Instie in Table 16. We find that using higher learning rate results in stronger model in chat-oriented benchmarks, at the cost catastrophic forgetting on GSM8K and MMLU. With a smaller learning rate, the model's performance on chat benchmarks is slightly worse, but its performance on GSM8K and MMLU is better retained. This demonstrates a trade-off between chat-oriented benchmarks and other benchmarks when continuing training from strong instruction-tunes model.
model with different learning rates, as reported
Table 16: Results on AlpacaEval 2, ZeroEval GSM, and ZeroEval MMLU when continuing training from Llama-3-8B-Inssee Instruct with different learning rates. indicates the released checkpoint.
Model
AlpacaEval 2 LC (%)
ZeroEval GSM (%)
ZeroEval MMLU (%
Llama-3-8B-Inan
26.0
78.5
61.7
SimPO (lr=4e-7)
38.8
77.9
62.6
SimPO (lr=5e-7)
44.6
77.0
62.3
SimPO (Ir=1c-6)*
53.7
57.4
54.9
Applying SimPO to Gemma 2 models presents different trend. We evaluate SimPO using Google's recently released Gemma-2-9B-- model [72], which represents a strong open-source model. For training data, we generate up to 5 responses per prompt from the UltraFeedback dataset [23] and use the ArmoRM model [79] annotate preferences between responses. We compare our SimPO against DPO-trained variant, both fine-tuned from the Gemma-2-9B-1 base model. As shown in Appendix J, SimPO demonstrates superior performance on chat benchmarks like AlpacaEval 2 and Arena-Hard while maintaining the model's original zero-shot capabilities on tasks like GSM8K and MMLJ. Notably, we find that varying the learning rate during fine-tuning has minimal impact on the model's 's performance. These results suggest an underlying property difference between the Llama-3 checkpoints and the Gemma checkpoints, and might worth further investigation.
Gemma-2-9B 2-9----iiO- significantly improved the ranking of the Gemma-2-2 2-9mma---9Bit model on Chatbot Arena. During the development stage, we relied solely on automated metrics to evaluate the model's performance. To determine if these metrics aligned with real preferences, submitted our best- perffornro model, hosted by LMSYS [17]. We find that our model improved the original Gemma-2--9B. it ranking from 36th to 25th, making the SimPO variant the top-ranked <10B model on the Chatbot Arena leaderboard based on real user votes of September 16th, 2024.
to the Chatbot Arena leaderboard
We evaluate the zero-shot performance of the models on GSMSK and MMLU using the ZeroEval repository which adopts unified setup.
31

---

Table 17: Benchmark performance of Gemma-2-95 trained with DPO and SimPO UltraFeedback (responses regenerated with Gemma-2-9B following the same dataset construction process as Llama-3-Instrus (8B) described in Section 3). SimPO results in better instruction following perfor- mance than DPO without degrading math abilities (GSM) or general knowledge (MMLU) of the original model. * indicates the released checkpoint.
Instruction Following
Capabilities
Model
AlpacaEval 2 LC
Arena-Hard (%)
ZvroEval GSM (%)
ZeroEval
(%)
MMLU (%)
Gemma-2-9B-ii
51.1
40.8
87.4
72.7
Gemma-2-9B-DPPO
67.8
58.9
88.5
72.2
Gemma-2-9BB- 2-9-SSmPO (I==6-77)
58.3
72.2
Sommma-2-9B-Siine (Ir=8e-7)*
72.4
59.1
88.0
72.2
Demma-2-99B-o 2-9B-SimPO (I==le-6)
71.0
58.3
87.4
71.5
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims have been validated with extensive empirical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
33

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We released our code, and provided experimental details to reproduce our
results.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
34

---

Answer: [Yes]
Justification:
We released our code and datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https: cttss://niis.oom cc/
public/ gui
CodeSu'om / ss coooooooocccc cy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips: is.ps.. ccccc////o ccc/pblice i c/ggiiiii des Codeeuumm ss on icyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We presented experiment settings and details in Section and Appendix B.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide standard deviation for confidence intervals for the major evaluation benchmarks Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
35

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We mentioned our computation environment in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: ttttps://neuripee httpp//cll//ttttccccciieee.....a
Answer: [Yes]
Justification:
Our research was conducted in accordance with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is a methodological paper to improve preference optimization algorithms and does not have direct societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
36

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not involve releasing data models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credited the creators of the assets used in the paper, and provide URLs to all assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use package should be provided. For popular datasets, hapersw's thoode. com / Bataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
37

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have released our code, datasets, and trained models, and provided detailed documentation.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
38


Reviews:
[{"review_id": "YNZxHRJqWd", "reviewer": "Reviewer_a8Vq", "summary": "The paper introduces SimPO (Simple Preference Optimization), an extension of Direct Preference Optimization (DPO), by replacing the reference-policy-dependent implicit reward with a reference-free reward.\nSpecifically, SimPO utilizes the average log probability of a sequence as the implicit reward, aligning it with the generation process and eliminating the need for a reference model. This results in improved computational and memory efficiency. Additionally, a target reward margin is introduced into the Bradley-Terry objective to ensure a significant reward difference between winning and losing responses. Extensive experiments show that SimPO consistently outperforms DPO and its variants across various benchmarks, demonstrating its effectiveness in improving model performance.", "questions": "- Is it possible to add a careful discussion and conduct comparative experiments about DPO with a target reward margin (Amini et al., 2024)?\n- The paper suggests that reducing the discrepancy between reward and generation in DPO results in a more preferable reward for preference optimization. While theoretical analysis might be challenging, is it possible to experimentally validate this hypothesis? For instance, can you evaluate the relationship between the discrepancy's size and preference optimization's performance? This could include testing different decoding methods, such as a beam search with larger beam widths in the length-normalized version that will align more with the average log-likelihood reward.\n- Since SimPO omits KL divergence regularization from the initial policy, assessing how the KL divergence behaves would be helpful. Evaluating the model's performance in terms of KL divergence, as analyzed by Gao et al. (2023), could provide valuable insights.", "limitations": "The authors have adequately discussed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The focus on addressing the discrepancy between reward and generation in DPO is novel and valuable, providing a new perspective on extending DPO.\n- SimPO enhances computational and memory efficiency by eliminating the need for a reference model.\n- Extensive experiments across various benchmarks demonstrate that SimPO consistently outperforms DPO and its variants.", "weaknesses": "- Although SimPO simplifies the optimization process by removing the reference model calculation of response probabilities, it introduces a new hyperparameter, the target reward margin $\\gamma$, which requires tuning.\n- While the motivation to address the discrepancy between reward and generation in DPO is understandable, there is a lack of mathematical discussion on how the proposed implicit reward is effective for preference optimization. For instance, standard random sampling with temperature 1 could eliminate the discrepancy without length normalization. However, the experimental results suggest that length normalization is crucial, questioning whether the initial motivation to align optimization and decoding is definitively effective for preference optimization.\n- The target reward margin plays a critical role in SimPO, but there is no discussion or comparative experiments with \"DPO with an offset\" (Amini et al., 2024), which also introduces a similar margin into the DPO loss. This comparison is necessary to understand the relative advantages of SimPO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We’d like to thank the reviewer for acknowledging the novelty and simplicity of our proposed approach. We address your raised points as follows.\n\n**The target reward margin $\\gamma$ requires extra tuning**:\nWe acknowledge that the newly introduced target reward margin requires additional tuning. However, we’d like to emphasize that within a reasonable range of $\\gamma$ values, SimPO consistently outperforms the DPO baseline, as shown in the table below (also Figure 3a in the manuscript). Furthermore, we find that using $\\gamma$=1.5 generally yields sufficiently good results across all settings. While slight tuning can further improve performance, it is not mandatory.\n| Models           | AlpacaEval LC Win Rate  |\n|------------------|:-------------------------:|\n| SimPO ($\\gamma$=0)   |  16.8 |\n| SimPO ($\\gamma$=0.8) |  20.2 |\n| SimPO ($\\gamma$=1.6) |  22.0 |\n| SimPO ($\\gamma$=2.4) |  16.8 |\n| DPO  | 15.1 |\n\n**Lack of mathematical understanding**:\nWhile it will be challenging to derive a rigorous theoretical understanding yet, we offer two possible explanations:\n1. SimPO's reward, the average log-likelihood of a sequence, closely aligns with the objective used during decoding. Although it's challenging to prove rigorously, random sampling with a temperature of 1 is likely to match SimPO's reward metric better than either (1) DPO or (2) SimPO without length normalization.\n    * DPO involves a reference model in its formulation, which is not used during decoding. \n    * Analogous to beam search, which ranks candidate sequences of varying lengths using average log-likelihood, length normalization is crucial when comparing sequences of different lengths (i.e., winning and losing responses) for reward calculation in SimPO. The reward without length normalization is biased, as evidenced by the tendency to assign higher likelihoods to longer sequences (Figure 2). This can result in excessively long outputs, such as degeneration into repetitive tokens.\n2. Another perspective for understanding the method is that the SimPO reward decouples the reward from the sequence length. According to [1], sequence length positively correlates with model-based judgment. SimPO’s disentanglement allows us to train the policy without biasing it to generate longer sequences merely to achieve a higher reward. Instead, it encourages the model to learn and focus on the differences beyond sequence lengths.\n\nWe hope this helps clarify the effectiveness of our approach!\n\n[1] [A Long Way to Go: Investigating Length Correlations in RLHF](https://arxiv.org/abs/2310.03716)\n\n**DPO with an offset (Amini et al., 2024)**: We conducted additional experiments to determine if adding an offset (Amini et al., 2024) would further enhance DPO. After tuning $\\gamma$, we found that it does not lead to any further improvement. Specifically, we used $\\gamma$=0.1 for mistral-base and $\\gamma$=0.0 for mistral-instruct. Increasing gamma beyond these values resulted in worse results.\n|              |    |         Mistral-base 7B     |            |   |       Mistral-instruct 7B       |            |\n|--------------|:---------------:|:------------:|:----------:|:-------------------:|:------------:|:----------:|\n|              | AlpacaEval 2 LC Win Rate | AlpacaEval 2 Win Rate | Arena-Hard |   AlpacaEval 2 LC Win Rate   | AlpacaEval 2 Win Rate | Arena-Hard |\n| DPO          |       15.1      |     12.5     |    10.4    |         26.8        |     24.9     |    16.3    |\n| DPO w/ $\\gamma$ |       15.2      |     12.1     |    10.3    |       26.8        |     24.9     |    16.3   |\n\nThe objective of DPO inherently includes an instance-wise target reward margin $\\gamma_{\\text{ref}}$, as shown below: \n$$\n\\mathcal{L} = \\log \\sigma \\left( \\beta \\log \\frac{\\pi_\\theta(y_w \\mid x)}{\\pi_{\\text{ref}}(y_w \\mid x)} - \\beta \\log \\frac{\\pi_\\theta(y_l \\mid x)}{\\pi_{\\text{ref}}(y_l \\mid x)}\\right)\\\\\n= \\log \\sigma \\bigg( \\beta \\log \\pi_\\theta(y_w \\mid x) - \\beta \\log \\pi_\\theta(y_l \\mid x) - \\big(\\beta \\log \\pi_{\\text{ref}}(y_w \\mid x) - \\beta \\log \\pi_{\\text{ref}}(y_l \\mid x)\\big)\\bigg)\n$$\nwhere \n$\\gamma_{\\text{ref}} = \\beta \\log \\pi_{\\text{ref}}(y_w \\mid x) - \\beta \\log \\pi_{\\text{ref}}(y_l \\mid x)$. \nThis may explain why adding an extra margin to DPO will not be as effective as it is with SimPO. We will add these results and discussions to our revision!\n\n**Discuss about KL divergence**: \nIn the PDF attached to our general response, we illustrate the KL divergence of SimPO and DPO across different $\\beta$, measured on the winning responses from a held-out set during training. In summary:\n* We find that SimPO, even without KL regularization, yields a reasonably small KL divergence from the policy model to the SFT model, which is comparable to DPO under its optimal $\\beta$.\n* Within the range of $\\beta$ explored, increasing $\\beta$ reduces the KL divergence for both DPO and SimPO, but it can also overly constrain the model from effective learning of the preference dataset, resulting in lower generation quality. This suggests that KL divergence is not monotonically related to reward scores or model quality (Gao et al., 2023 present a similar phenomenon).\nWe hope this helps clarify the reviewer’s concern about KL divergence!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LOk6jOrU72", "reviewer": "Reviewer_oFKm", "summary": "Simpo introduces a simplistic new method of alignment leveraging the log-probability of the sequence eliminating the need to leverage the SFT policy making the implementation computationally less complex. By introducing a notion of margin in the loss function, SimPO outperforms several existing alignment algorithms including DPO and variants without increasing response length. Overall, it provides an interesting light-weight alignment procedure with strong empirical benefits.", "questions": "1. Can you mention if your original objective is strongly convex or how the particular objective is derived?\n2. Can you provide the KL regularization of your model to the SFT and show how much it deviates from the SFT in comparison to baselines?", "limitations": "Check above", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The approach eliminates the need for the reference model which can be computationally heavy or might not be available in practice\n2. The methods shows improved empirical performance over DPO and existing baselines through extensive experimentations.\n3. The method provides an interesting way of regularizing the length and shows improved performance without increasing the length of the response.", "weaknesses": "1. The primary definition of RLHF has a constraint to the SFT policy through the KL regularization which restricts the model to overoptimize the reward function. Also, it ensures the base performance of the LLM remains intact on several other domains and tasks. However, the proposed method doesn't use KL to the SFT and removes the SFT which makes it unclear what to attribute the improvement to. Specifically, one can violate the constraint and do better with the reward for that specific or similar task, but what about the performance in other tasks that the pre-trained/SFT model was good at?\n\n2. DPO has a closed-form unique solution due to the strong convexity of the KL regularized RLHF objective resulting in the particular objective. However, it's not clear if we don't use the KL regularization, what the original optimization problem we are optimizing for. Is it still strongly convex? If not, what are the insights that this objective will have some good convergence properties? \n\n3. Under all these questions, the motivation for the objective is heuristically driven and unclear from where it comes from", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We'd like to thank the reviewer for the thoughtful feedback. We address your questions as follows.\n\n**Without KL to the SFT, one can violate the constraint and do better with the reward for that specific or similar task, but what about the performance in other tasks that the pre-trained/SFT model was good at?** \nWhile reward hacking is theoretically possible without explicit regularization, several practical factors mitigate this risk: (1) the learning rate, (2) the preference dataset, and (3) the SFT model’s robustness to forgetting. \n1. The learning rate of preference optimization algorithms is typically very small (e.g., 5e-7), which naturally prevents significant divergence from the SFT model.\n2. The preference datasets are often constructed to cover a broad range of tasks and domains, helping retain the model's existing knowledge and task versatility.\n3. Large language models generally have a substantial capacity to learn from new data without catastrophic forgetting of previously learned tasks. This robustness can mitigate the need for explicit regularization, similar to how instruction tuning (SFT) retains the model’s pretrained knowledge without KL regularization.\n\nThe above factors combined might be sufficient to ensure that the model learns human preferences while retains its generalization capabilities. Empirically, with appropriate hyperparameters, SimPO can result in a similar KL divergence to DPO (see our response to your last question) and comparable academic benchmark performance (see Table 9 in the Appendix). Additionally, we demonstrate new results in the table below that fine-tuning gemma2-9b-it with SimPO yields significantly improved instruction following ability without degradation on academic benchmarks including math (GSM) and general knowledge (MMLU).\n\n|models | AlpacaEval 2 LC | GSM (0 shot) | MMLU (0 shot) | \n|-----|:------:|:------:|:----------:|\n| gemma-2-9b-it  |  51.1  |  87.4 |  72.7 |   \n| gemma-2-9b-it-DPO  |  67.8  |  88.5 |  72.2 |   \n| gemma-2-9b-it-SimPO |  72.4  |  88.0 |  72.2 | \n\n**DPO has a closed-form unique solution due to the strong convexity of the KL regularized RLHF objective resulting in the particular objective. Without KL, is the objective still strongly convex? If not, what are the insights that this objective will have some good convergence properties?** \nThe derivation of the DPO objective from RLHF relies on the specific condition of achieving the optimal policy $\\pi^*_{\\theta}$. However, this assumption is rarely met in practice due to the inherent challenges in optimizing deep neural networks. Therefore, there's no guarantee that DPO faithfully implements the original RLHF objective. Our work departs from this assumption: we directly address the potential misalignment between the DPO objective's reward metric and the decoding objective. This focus allows us to tackle the practical limitations of DPO without relying on idealized assumptions.\n\nFurthermore, comparing preference optimization (PO) objectives in terms of convexity or convergence is inherently difficult. All these objectives, including RLHF (PPO), DPO, and SimPO, are generally non-convex with respect to the model parameters $\\theta$. This is because the policy model $\\pi_{\\theta}$ is parametrized by multi-layer, non-linear Transformer architectures, making it a non-convex function with respect to $\\theta$. Consequently, any PO objective that depends on $\\pi_{\\theta}$ will also be inherently non-convex with respect to the model parameters.\n\n**The motivation for the objective is heuristically driven and unclear from where it comes from**: \nOur SimPO objective is systematically derived by aligning the reward in training with the generation likelihood in decoding:\n1. We use the sequence likelihood objective, optimized by decoding algorithms, as the reward metric for winning/losing responses during preference optimization.\n2. We then apply the Bradley-Terry objective over these reward metrics to formulate the SimPO objective.\n3. The target reward margin hyperparameter in SimPO is analogous to the margin loss in SVMs and represents the home advantage in Bradley-Terry models.\n\n**Can you provide the KL regularization of your model to the SFT and show how much it deviates from the SFT in comparison to baselines?** In the PDF attached to our general response, we illustrate the KL divergence of SimPO and DPO. Figure 1 shows that (a) increasing $\\beta$ in DPO can reduce the KL divergence, but (b) it can also overly constrain the model from effective learning of the preference dataset, resulting in lower generation quality. Therefore, there is a trade-off between learning preference data and staying close to the initial model. With appropriate hyperparameters, SimPO can achieve similar KL divergence to DPO.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ypz7lW0y29", "reviewer": "Reviewer_TXhi", "summary": "This paper proposes a new offline-RLHF algorithm SimPO, which significantly improves current DPO variants on a collection of benchmarks.", "questions": "I do not have any specific questions. As a final comment, if possible, I personally like to see some concrete examples of responses generated by SimPO and DPO (beyond the response lengths) to have a better understanding of the actual improvement.", "limitations": "Most limitations have been declared by the authors.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The SimPO algorithm is intuitive, simple to implement, and works well, with a good presentation.\n\nA few concrete strong points are list below.\n\n1. The algorithm is intuitive and the overall design is novel in the current literature. The paper proposes two simple techniques, including directly optimizing the generation probability and adding a discrepancy term. Both two techniques are intuitive. Overall, little efforts in offline RLHF methods have fully explored these two directions systematically. \n\n2. The experiments are solid. This paper conducts experiments on a diverse collection of chat benchmarks. The results are significant. \n\n3. The overall presentation is great and easy to follow.", "weaknesses": "The following comments are for potential improvement of this work and future research directions. These are not complaints.\n\n1. Some fine-grained case studies can be beneficial. The paper primarily focuses on quantitative evaluations. But it would be interesting to see concrete changes between DPO and SimPO, i.e., some representative responses generated by SimPO and DPO, to see what has changed. The LN experiments are great. But things can be better. \n\n2. With the current formulation, the SimPO objective reminds me of those online RLHF methods. Assuming a -1/1 reward function, many online RLHF methods optimize similar objectives, i.e., direction optimizing the responses with a 1 reward response (winning responses) while reducing the reward of bad ones (losing responses). It will be interesting to see some discussions between SimPO and online RLHF methods. Also, many recent practices have observed that online DPO works greatly. It would be interesting to see if the same phenomena can be observed for SimPO, which looks much closer to online RLHF variants. This is a potential direction to make this paper a much stronger one.\n\n3. Another possible direction to explore is about tasks other than chat, e.g., reasoning tasks like coding and maths. The evaluations in this paper are indeed sufficient as they are, but some discussions on a broad domain can be appreciated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging the novelty, simplicity, and significant empirical results of SimPO!\n\n**Fine-grained case studies**:\nThank you for the suggestion! We refer the reviewer to Figures 8 and 9 in our original submission, as well as Figure 2 in the PDF attached to our general response. These examples intuitively demonstrate why SimPO outperforms DPO:\n* SimPO can produce better-structured answers than DPO. (Figures 8 and 9 in our paper)\n* SimPO can generate more concise and clear responses than DPO. (Figure 2 in the rebuttal PDF)\n\nWe acknowledge that qualitative analysis of model outputs is becoming increasingly challenging, given the breadth and depth of the questions in these benchmarks. We will provide more qualitative cases in our next revisions. \n\n**Online methods**:\nThank you for the suggestions! We are actively working on an online version of SimPO. Specifically, we alternate between generating online preference optimization data and training the model with the generated data. Here are some initial results we have for a stronger version of the model — gemma-2-9B-it:\n\n|  model | LC Win rate | Raw Win Rate |\n|----------------|:-----------:|:------------:|\n| gemma-2-9b-it  |     51.1    |     38.1     |\n| SimPO (1 iter) |     74.6    |     66.7     |\n| SimPO (2 iter) |     76.5    |     76.2     |\n\nWe find that the online version of SimPO continues to improve performance with each additional iteration. \n\n**Tasks other than chat, e.g., reasoning tasks like coding and maths**: \nIn our initial manuscript (also shown in the table below), we did find that after further training on the Llama-3-Instruct-8B model, the chat ability of the models enhances at the sacrifice of the degradation on GSM8k (math) and MMLU (knowledge). However, after more exploration, we find that such a performance degradation is largely attributable to the lack of robustness of the base model (i.e., Llama-3) rather than the SimPO objective. \nWe’d like to present the following new results from the gemma-2-9b-it model. The last two rows in the table below demonstrate that SimPO largely retains general knowledge (MMLU) and even slightly improves the original model’s math ability (GSM). Additionally, it significantly enhances coding ability, as demonstrated by the Arena-Hard benchmark, which primarily consists of real-world coding questions.\n\n|models | AlpacaEval 2 LC | AlpacaEval 2 WR | Arena-Hard  | GSM (0 shot) | MMLU (0 shot) | \n|-----|:------:|:------:|:----------:|:----:|:---------:|\n| Llama-3-Instruct-8B  |  26.0  |  25.3  |   22.3 |  78.5 |  61.7 |   \n| Llama-3-Instruct-8B-SimPO |  44.7  |  40.5  |   33.8 |  71.3 |  58.5 |  \n| gemma-2-9b-it  |  51.1  |  38.1  |   40.8 |  87.4 |  72.7 |   \n| gemma-2-9b-it-SimPO |  72.4  |  65.9  |   59.1 |   88.0 |  72.2 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0gXP1VY9eM", "reviewer": "Reviewer_Wuuc", "summary": "This paper presents SimPO, an offline preference optimization method for LLM alignment. SimPO replaces the KL term in DPO with a length-regularized log-probability and adds a margin value for regularization. Extensive experiments in chat benchmarks show that SimPO significantly outperforms DPO and other preference optimization variants, despite its simplicity. The authors conclude that the success of SimPO can be attributed to the better alignment between training and decoding objectives, as well as reduced exploitation of generating lengthy responses.", "questions": "+ Could the authors explain why length-controlled WR is important? From my perspective, controlling response length is helpful during training because it encourages the model to generate short but useful responses. However, forcing the generation of short responses during evaluation is not necessary. Good responses are good regardless of length. Evaluating length-controlled WR seems to set a privileged metric for SimPO.\n\n+ Since SimPO removes the KL regularization, how does it perform with poor data quality, which is a common application scenario? Is it more fragile and sensitive to data quality?", "limitations": "The limitations have been addressed by the authors in Section 6 and in my weaknesses section.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ This paper is well-written and easy to follow. The motivation is clear, and the main approach is quite understandable.\n\n+ The experiments are extensive and well-designed. In addition to benchmark numbers, the authors also provide an in-depth analysis of generation lengths, log probabilities, and differences in rewards for DPO and SimPO models. This greatly helps readers understand the source of the performance gains.", "weaknesses": "I do not find any significant weaknesses beyond the limitations proposed by the authors. Since I generally agree with these limitations, I am reiterating them in my review:\n\n+ A theoretical understanding of SimPO is lacking. Although SimPO's design is not theoretically grounded, it performs well in practice, as validated by the authors' experiments.\n\n+ The experiments in the paper solely focus on evaluating helpfulness, disregarding safety, honesty, etc. I think this is the major weakness. SimPO removes the KL regularization from the reference model, so it might intuitively suffer from safety issues. This problem should be further studied in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging the clarity and comprehensive evaluation of our paper. We address your raised points as follows.\n\n**KL regularization and safety**:\nWe would like to present our most recent results based on the gemma-2-9b-it model. We measured the original gemma-2-9b-it model, its DPO variant, and the SimPO variant using Sorry-Bench (Xie et al., 2024 [1]), a benchmark for testing the refusal behavior of language models when faced with unsafe queries. Our findings indicate that further tuning the Gemma model on the UltraFeedback dataset with either DPO or SimPO increases the attack success rate, probably due to the lack of safety-related data in UltraFeedback. However, we also find that SimPO is safer than DPO. Therefore, we believe that SimPO does not raise more safety concerns than DPO. \n\n|  model  | Sorry-Bench attack success rate |\n|----------------------|:------------------------------:|\n| gemma-2-9b-it        |    10.89% |\n| w/ DPO               |  50.44% |\n| w/ SimPO (without safety preference data)   |  33.33% |\n\nWe further direct the reviewer to the PDF attached to our general response, where we demonstrate that even without explicit KL regularization, SimPO achieves a comparable KL divergence from the initial SFT model as KL-regularized objectives like DPO.\n\n[1] [SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal Behaviors](https://arxiv.org/abs/2406.14598)\n\n**Why is length-controlled WR important**:\nThe length-controlled win rate (LC-WR), introduced by Dubois et al. (2024) [2], effectively addresses the length bias issue prevalent in model-based evaluations such as AlpacaEval. This metric has demonstrated a better correlation with human judgments compared to raw win rates [2]. Hence, it has been adopted as the preferred metric for ranking models on the AlpacaEval leaderboard. Given its significance and widespread acceptance in the field, we have incorporated the length-controlled WR as one of our primary evaluation metrics in this study.\n\n[2] [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://arxiv.org/abs/2404.04475v1)\n\n**Is SimPO more fragile and sensitive to data quality?**\nOur extensive experiments across various settings do not indicate that SimPO is more susceptible to low-quality data than other methods. In fact, the UltraFeedback dataset used in our study inherently includes a variety of data quality levels, as the responses are generated by randomly-sampled models from a diverse pool, including relatively weak models (e.g., Alpaca-7B). Despite this variability in data quality, SimPO consistently demonstrates significant empirical advantages over other baselines. \n\nWe hypothesize that SimPO's robustness to data quality issues stems from two key factors:\n1. The use of a small learning rate, which is the common practice in preference optimization algorithms and naturally prevents significant divergence from the initial model.\n2. The inherent robustness of some large language models (e.g., gemma models), which allows them to maintain performance even when exposed to some degree of low-quality input.\n\nThese factors combined may provide sufficient protection against catastrophic forgetting, even without the need for explicit KL regularization.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their thoughtful feedback, and we'd like to share some additional analysis and results since the submission!

**Additional Analysis in the attached PDF**   
We include the following additional studies: 
* KL divergence plots of SimPO vs. DPO
* Qualitative studies of SimPO vs. DPO

**Exceptionally strong results from applying SimPO to gemma-2-9b-it**   
Furthermore, we’d like to present our new results from training gemma-2-9b-it with SimPO. The resulting model tops the AlpacaEval 2 leaderboard and the Arena-Hard benchmark among similar-sized models. Importantly, we find that training the gemma model with SimPO retains its knowledge (MMLU) and even slightly improves the original model’s math ability (GSM). This exciting new set of results demonstrates the effectiveness of SimPO across different model types without performance degradation on other benchmarks.

|models | AlpacaEval 2 LC | AlpacaEval 2 WR | Arena-Hard  | GSM (0 shot) | MMLU (0 shot) | 
|-----|:------:|:------:|:----------:|:----:|:---------:|
| gemma-2-9b-it  |  51.1  |  38.1  |   40.8 |  87.4 |  72.7 |   
| gemma-2-9b-it-SimPO |  72.4  |  65.9  |   59.1 |   88.0 |  72.2 |

Meta-review: This paper proposes SimPO for preference optimization without using a reference model. The two key designs are length-normalized reward and target reward margin. Experiments on AlpacaEval, MT-Bench and Arena-Hard verify the proposed method. Reviewers raised concerns about why and how without KL regularization SimPO is able to preserve close enough distance between trained and reference policies, as well as more experimental results, and the authors addressed most of the concerns. I suggest that the authors incorporate the discussions and additional experiments in subsequent versions of the paper.
Decision: Accept (poster)