SimPO: Simple Preference Optimization with a Reference-Free Reward

Authors: yu  meng
mengzhou  xia
danqi  chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 19:49:57

Abstract: Direct Preference Optimization (DPO) is a widely used offline preference optimization algorithm that reparameterizes reward functions in reinforcement learning from human feedback (RLHF) to enhance simplicity and training stability. In this work, we propose SimPO, a simpler yet more effective approach. The effectiveness of SimPO is attributed to a key design: using the _average_ log probability of a sequence as the implicit reward. This reward formulation better aligns with model generation and eliminates the need for a reference model, making it more compute and memory efficient. Additionally, we introduce a target reward margin to the Bradley-Terry objective to encourage a larger margin between the winning and losing responses, further improving the algorithm's performance. We compare SimPO to DPO and its latest variants across various state-of-the-art training setups, including both base and instruction-tuned models such as Mistral, Llama 3, and Gemma 2. We evaluate on extensive chat-based evaluation benchmarks, including AlpacaEval 2, MT-Bench, and Arena-Hard. Our results demonstrate that SimPO consistently and significantly outperforms existing approaches without substantially increasing response length. Specifically, SimPO outperforms DPO by up to 6.4 points on AlpacaEval 2 and by up to 7.5 points on Arena-Hard. Our top-performing model, built on Gemma-2-9B-it, achieves a 72.4\% length-controlled win rate on AlpacaEval 2, a 59.1\% win rate on Arena-Hard, and ranks 1st on Chatbot Arena among $<$10B models with real user votes.

Main paper: 
 introduction: 
 Learning from human feedback is crucial in aligning large language models (LLMs) with human values and intentions [47] , ensuring they are helpful, honest, and harmless [5] . Reinforcement learning from human feedback (RLHF) [18, 58, 68] is a popular method for fine-tuning language models to achieve effective alignment. While the classical RLHF approach [58, 65] has shown impressive results, it presents optimization challenges due to its multi-stage procedure, which involves training a reward model and then optimizing a policy model to maximize that reward [13] .Recently, researchers have been exploring simpler offline algorithms. Direct Preference Optimization (DPO) [61] is one such approach. DPO reparameterizes the reward function in RLHF to directly learn a policy model from preference data, eliminating the need for an explicit reward model. It has gained widespread practical adoption due to its simplicity and stability. In DPO, the implicit reward is formulated using the log ratio of the likelihood of a response between the current policy model and the supervised fine-tuned (SFT) model. However, this reward formulation is not directly aligned with the metric used to guide generation, which is approximately the average log likelihood of a response generated by the policy model. We hypothesize that this discrepancy between training and inference may lead to suboptimal performance.Table 1 : Length-controlled (LC) and raw win rate (WR), and generation lengths of top models on the AlpacaEval 2 Leaderboard. Bold are the models we trained.In this work, we propose SimPO, a simple yet effective offline preference optimization algorithm (Figure 1 ). The core of our algorithm aligns the reward function in the preference optimization objective with the generation metric. SimPO consists of two major components: (1) a lengthnormalized reward, calculated as the average log probability of all tokens in a response using the policy model, and (2) a target reward margin to ensure the reward difference between winning and losing responses exceeds this margin. In summary, SimPO has the following properties:• Simplicity: SimPO does not require a reference model, making it more lightweight and easier to implement compared to DPO and other reference-based methods.• Significant performance advantage: Despite its simplicity, SimPO significantly outperforms DPO and its latest variants (e.g., a recent reference-free objective ORPO [38] ). The performance advantage is consistent across various training setups and extensive chat-based evaluations, including AlpacaEval 2 [51, 28] and the challenging Arena-Hard [50] benchmark. It achieves up to a 6.4 point improvement on AlpacaEval 2 and a 7.5 point improvement on Arena-Hard compared to DPO (Figure 1 ).• Minimal length exploitation: SimPO does not significantly increase response length compared to the SFT or DPO models (Table 1 ), indicating minimal length exploitation [28, 66, 80] .Extensive analysis shows that SimPO utilizes preference data more effectively, leading to a more accurate likelihood ranking of winning and losing responses on a held-out validation set, which in turn translates to a better policy model. As shown in Table 1 , our Gemma-2-9B-it-SimPO model achieves state-of-the-art performance, with a 72.4% length-controlled win rate on AlpacaEval 2 and a 59.1% win rate on Arena-Hard, establishing it as the strongest open-source model under 10B parameters. Most notably, when evaluated on Chatbot Arena [17] with real user votes, our model significantly improved upon the initial Gemma-2-9B-it model, advancing from 36th to 25th place and ranking first among all <10B models on the leaderboard. 22 SimPO: Simple Preference OptimizationIn this section, we first introduce the background of DPO ( §2.1). Then we identify the discrepancy between DPO's reward and the likelihood metric used for generation, and propose an alternative reference-free reward formulation that mitigates this issue ( §2.2). Finally, we derive the SimPO objective by incorporating a target reward margin term into the Bradley-Terry model ( §2.3). 
background: direct preference optimization (dpo): 
 DPO [61] is one of the most popular preference optimization methods. Instead of learning an explicit reward model [58] , DPO reparameterizes the reward function r using a closed-form expression with the optimal policy:EQUATIONwhere π θ is the policy model, π ref is the reference policy, typically the supervised fine-tuned (SFT) model, and Z(x) is the partition function. By incorporating this reward formulation into the Bradley-Terry (BT) ranking objective [11] , p(y w ≻ y l | x) = σ (r(x, y w )r(x, y l )), DPO expresses the probability of preference data with the policy model rather than the reward model, yielding the following objective:EQUATIONwhere (x, y w , y l ) are preference pairs consisting of the prompt, the winning response, and the losing response from the preference dataset D.Discrepancy between reward and generation for DPO. Using Eq. ( 1) as the implicit reward has the following drawbacks: (1) it requires a reference model π ref during training, which incurs additional memory and computational costs; and (2) it creates a mismatch between the reward optimized in training and the log-likelihood optimized during inference, where no reference model is involved. This means that in DPO, for any triple (x, y w , y l ), satisfying the reward ranking r(x, y w ) > r(x, y l ) does not necessarily mean that the likelihood ranking p θ (y w | x) > p θ (y l | x) is met (here p θ is the average log-likelihood in Eq. ( 3)). In our experiments, we observed that only ∼ 50% of the triples from the training set satisfy this condition when trained with DPO (Figure 4b ). This observation aligns with a concurrent work [14] , which finds that existing models trained with DPO exhibit random ranking accuracy in terms of average log-likelihood, even after extensive preference optimization.Length-normalized reward formulation. One solution is to use the summed token log probability as the reward, but this suffers from length bias-longer sequences tend to have lower log probabilities. Consequently, when y w is longer than y l , optimizing the summed log probability as a reward forces the model to artificially inflate probabilities for longer sequences to ensure y w receives a higher reward than y l . This overcompensation increases the risk of degeneration. To address this issue, we consider using the average log-likelihood as the implicit reward:EQUATIONThis metric is commonly used for ranking options in beam search [33, 49] and multiple-choice tasks within language models [12, 37, 58] . Naturally, we consider replacing the reward formulation in DPO with p θ in Eq. ( 3), so that it aligns with the likelihood metric that guides generation. This results in a length-normalized reward:EQUATIONwhere β is a constant that controls the scaling of the reward difference. We find that normalizing the reward with response lengths is crucial; removing the length normalization term from the reward formulation results in a bias toward generating longer but lower-quality sequences (see Section 4.4 for more details). Consequently, this reward formulation eliminates the need for a reference model, enhancing memory and computational efficiency compared to reference-dependent algorithms.Target reward margin. Additionally, we introduce a target reward margin term, γ > 0, to the Bradley-Terry objective to ensure that the reward for the winning response, r(x, y w ), exceeds the reward for the losing response, r(x, y l ), by at least γ:EQUATIONThe margin between two classes is known to influence the generalization capabilities of classifiers [1, 10, 22, 30] . 3 In standard training settings with random model initialization, increasing the target margin typically improves generalization. In preference optimization, the two classes are the winning and losing responses for a single input. In practice, we observe that generation quality initially improves with an increasing target margin but degrades when the margin becomes too large ( §4.3). One of DPO's variants, IPO [6] , also formulates a target reward margin similar to SimPO. However, its full objective is not as effective as SimPO ( §4.1).Objective. Finally, we obtain the SimPO objective by plugging Eq. ( 4) into Eq. ( 5):EQUATIONIn summary, SimPO employs an implicit reward formulation that directly aligns with the generation metric, eliminating the need for a reference model. Additionally, it introduces a target reward margin γ to help separating the winning and losing responses. In Appendix F, we provide a gradient analysis of SimPO and DPO to further understand the differences between the two methods.Preventing catastrophic forgetting without KL regularization. Although SimPO does not impose KL regularization, we find that a combination of practical factors ensures effective learning from preference data while maintaining generalization, leading to an empirically low KL divergence from the reference model. These factors are: (1) a small learning rate, (2) a preference dataset that covers diverse domains and tasks, and (3) the intrinsic robustness of LLMs to learn from new data without forgetting prior knowledge. We present KL divergence experiments in Section 4.4.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss limitations in Section 6.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
experimental setup: 
 Models and training settings. We perform preference optimization with two families of models, Llama-3-8B [2] and Mistral-7B [40] , under two setups: Base and Instruct. In this section, our goal is to understand the performance of SimPO vs. other preference optimization methods in different experimental setups. Our strongest model is based on Gemma-2-9B (Instruct setup) with a stronger reward model, RLHFlow/ArmoRM-Llama3-8B-v0.1 [79] (Table 1 ). We will present and discuss these results in Appendix J.For the Base setup, we follow the training pipeline of Zephyr [75] . First, we train a base model (i.e., mistralai/Mistral-7B-v0.1, or meta-llama/Meta-Llama-3-8B) on the UltraChat-200k dataset [25] to obtain an SFT model. Then, we perform preference optimization on the UltraFeedback dataset [23] using the SFT model as the starting point. This setup provides a high level of transparency, as the SFT models are trained on open-source data.For the Instruct setup, we use an off-the-shelf instruction-tuned model (i.e., meta-llama/Meta-Llama-3-8B-Instruct, or mistralai/Mistral-7B-Instruct-v0.2) as the SFT models. 4 These models have undergone extensive instruction-tuning processes, making them more powerful and robust than the SFT models in the Base setup. However, they are also more opaque because their RLHF procedure is not publicly disclosed. To mitigate the distribution shift between SFT models and the preference optimization process, we generate the preference dataset using the SFT models following [74] . This makes our Instruct setup closer to an on-policy setting. Specifically, we use prompts from the UltraFeedback dataset and regenerate the chosen and rejected response pairs (y w , y l ) with the SFT models. For each prompt x, we generate 5 responses using the SFT model with a sampling temperature of 0.8. We then use llm-blender/PairRM [41] to score the 5 responses, selecting the highest-scoring one as y w and the lowest-scoring one as y l . We only generated data in a single pass instead of iteratively as in [74] . 5In summary, we have four setups: Llama-3-Base, Llama-3-Instruct, Mistral-Base, and Mistral-Instruct.We believe these configurations represent the state-of-the-art, placing our models among the top performers on various leaderboards. We encourage future research to adopt these settings for better and fairer comparisons of different algorithms. Additionally, we find that tuning hyperparameters is crucial for achieving optimal performance with all the offline preference optimization algorithms, including DPO and SimPO. Generally, for SimPO, setting β between 2.0 and 2.5 and γ between 0.5 and 1.5 leads to good performance across all setups. For more details, please refer to Appendix B.Evaluation benchmarks. We primarily assess our models using three of the most popular openended instruction-following benchmarks: MT-Bench [94] , AlpacaEval 2 [51] , and Arena-Hard v0.1 [50] . These benchmarks evaluate the models' versatile conversational abilities across a diverse set of queries and have been widely adopted by the community (details in Table 2 ). AlpacaEval 2 consists of 805 questions from 5 datasets, and MT-Bench covers 8 categories with 80 questions. The most recently released Arena-Hard is an enhanced version of an MT-Bench, incorporating 500 welldefined technical problem-solving queries. We report scores following each benchmark's evaluation protocol. For AlpacaEval 2, we report both the raw win rate (WR) and the length-controlled win rate (LC) [28] . The LC metric is specifically designed to be robust against model verbosity. For Arena-Hard, we report the win rate (WR) against the baseline model. For MT-Bench, we report the average MT-Bench score with GPT-4 and GPT-4-Preview-1106 as the judge model. 6 For decoding details, please refer to Appendix B. We also evaluate on downstream tasks from the Huggingface Open Leaderboard benchmarks [9] , with additional details in in Appendix C.Table 3 : Various preference optimization objectives given preference data D = (x, y w , y l ), where x is an input, and y w and y l are the winning and losing responses.RRHF [86] max 0, -1 |yw| log π θ (y w |x) + 1 |yl| log π θ (y l |x)λ log π θ (y w |x) SLiC-HF [91] max (0, δlog π θ (y w |x) + log π θ (y l |x))λ log π θ (y w |x) DPO [61] log σ β log πθ(yw|x) πref(yw|x)β log πθ(yl|x) πref(yl|x)IPO [6] log πθ(yw|x) πref(yw|x) -log πθ(yl|x) πref(yl|x) -1 2τ 2 CPO [83] -log σ (β log π θ (y w |x) -β log π θ (y l |x)) -λ log π θ (y w |x) KTO [29] -λ w σ β log πθ(yw|x) πref(yw|x) -z ref + λ l σ z ref -β log πθ(yl|x) πref(yl|x) , where z ref = E (x,y)∼D [βKL (π θ (y|x)||π ref (y|x))]ORPO [38] log p θ (y w |x)λ log σ log pθ(yw|x) 1-pθ(yw|x)log pθ(yl|x) 1-pθ(yl|x) , where p θ (y|x) = exp 1 |y| log π θ (y|x)R-DPO [60] -log σ β log πθ(yw|x) πref(yw|x) -β log πθ(yl|x) πref(yl|x) + (α|y w | -α|y l |) SimPO -log σ β |yw| log π θ (y w |x) -β |yl| log π θ (y l |x) -γBaselines. We compare SimPO with other offline preference optimization methods listed in Table 3 . 7 RRHF [86] and SLiC-HF [91] are ranking losses. RRHF uses length-normalized log-likelihood, similar to SimPO's reward function, while SLiC-HF uses log-likelihood directly and includes an SFT objective. IPO [6] is a theoretically grounded approach method that avoids DPO's assumption that pairwise preferences can be replaced with pointwise rewards. CPO [83] uses sequence likelihood as a reward and trains alongside an SFT objective. KTO [29] learns from non-paired preference data. ORPO [38] 8 introduces a reference-model-free odd ratio term to directly contrast winning and losing responses with the policy model and jointly trains with the SFT objective. R-DPO [60] is a modified version of DPO that includes an additional regularization term to prevent exploitation of length. We thoroughly tune the hyperparameters for each baseline and report the best performance. We find that many variants of DPO do not empirically present an advantage over standard DPO. Further details can be found in Appendix B.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
experimental results: 
 In this section, we present main results of our experiments, highlighting the superior performance of SimPO on various benchmarks and ablation studies ( §4.1). We provide an in-depth understanding of the following components: (1) length normalization ( §4.2), (2) the margin term γ ( §4.3), and (3) why SimPO outperforms DPO ( §4.4). Unless otherwise specified, the ablation studies are conducted using the Mistral-Base setting.SimPO consistently and significantly outperforms existing preference optimization methods.As shown in Table 4 , while all preference optimization algorithms enhance performance over the SFT model, SimPO, despite its simplicity, achieves the best overall performance across all benchmarks and settings. These consistent and significant improvements highlight the robustness and effectiveness of SimPO. Notably, SimPO outperforms the best baseline by 3.6 to 4.8 points on the AlpacaEval 2 LC win rate across various settings. On Arena-Hard, SimPO consistently achieves superior performance, though it is occasionally surpassed by CPO [83] . We find that CPO generates responses that are, on average, 50% longer than those generated by SimPO (See Table 10 ). Arena-Hard might favor longer generations due to the absence of a length penalty in its evaluation.Benchmark quality varies. Although all three benchmarks are widely adopted, we find that MT-Bench exhibits poor separability across different methods. Minor differences between methods on MT-Bench may be attributed to randomness, likely due to the limited scale of its evaluation data and its single-instance scoring protocol. This finding aligns with observations reported in [50] . In contrast, AlpacaEval 2 and Arena-Hard provide more meaningful distinctions between different methods. We observe that the win rate on Arena-Hard is significantly lower than on AlpacaEval 2, indicating that Arena-Hard is a more challenging benchmark. 9The Instruct setting introduces significant performance gains. Across all benchmarks, we observe that the Instruct setting consistently outperforms the Base setting. This improvement is likely due to the higher quality of SFT models used for initialization and the generation of more high-quality preference data by these models.Both key designs in SimPO are crucial. In Table 5 , we demonstrate results from ablating each key design of SimPO: (1) removing length normalization in Eq. ( 4) (i.e., w/o LN); (2) setting the target reward margin to be 0 in Eq. ( 6) (i.e., γ = 0). Removing the length normalization has the most negative impact on the results. Our examination reveals that this leads to the generation of long and repetitive patterns, substantially degrading the overall quality of the output (See Appendix E). Setting γ to 0 yields also leads to a performance degradation compared to SimPO, indicating that it is not the optimal target reward margin. In the following subsections, we conduct in-depth analyses to better understand both design choices.LN leads to an increase in the reward difference for all preference pairs, regardless of their length. The Bradley-Terry objective in Eq. ( 5) essentially aims to optimize the reward difference ∆r = r(x, y w )r(x, y l ) to exceed the target margin γ. We investigate the relationship between the learned reward differences and the length difference ∆l = |y w | -|y l | between the winning and losing responses from the training set of UltraFeedback. We measure the difference of reward (r SimPO ; Eq. ( 4)) using the SFT model, the SimPO model, and a model trained with SimPO but without length normalization. We present the results in Figure 2a and observe that SimPO with LN consistently achieves a positive reward margin for all response pairs, regardless of their length difference, and consistently improves the margin over the SFT model. In contrast, SimPO without LN results in a negative reward difference for preference pairs when the winning response is shorter than the losing response, indicating that the model learns poorly for these instances.Removing LN results in a strong positive correlation between the reward and response length, leading to length exploitation. Figures 2b and 2c illustrate the average log likelihood (p θ in Eq. ( 3)) versus response length on a held-out set for models trained with SimPO and SimPO without LN. The model trained without LN exhibits a much stronger positive Spearman correlation between likelihood and response length compared to SimPO, indicating a tendency to exploit length bias and generate longer sequences (see Appendix E). In contrast, SimPO results in a Spearman correlation coefficient similar to the SFT model (see Figure 6a ).Influence of γ on reward accuracy and win rate. We investigate how the target reward margin γ in SimPO affects the reward accuracy on a held-out set and win rate on AlpacaEval 2, presenting the results in Figure 3a . Reward accuracy is measured as the percentage of preference pairs where the winning response ends up having a higher reward for the winning response than the losing response (i.e., r(x, y w ) > r(x, y l )). We observe that reward accuracy increases with γ on both benchmarks, indicating that enforcing a larger target reward margin effectively improves reward accuracy. However, the win rate on AlpacaEval 2 first increases and then decreases with γ, suggesting that generation quality is not solely determined by the reward margin.Impact of γ on the reward distribution. We visualize the distribution of the learned reward margin r(x, y w )r(x, y l ) and the reward of winning responses r(x, y w ) under varying γ values in Figure 2b and Figure 2c . Notably, increasing γ tends to flatten both distributions and reduce the average log likelihood of winning sequences. This initially improves performance but can eventually lead to model degeneration. We hypothesize that there is a trade-off between accurately approximating the true reward distribution and maintaining a well-calibrated likelihood when setting the γ value. Further exploration of this balance is deferred to future work.In this section, we compare SimPO to DPO in terms of (1) likelihood-length correlation, (2) reward formulation, (3) reward accuracy, and (4) algorithm efficiency. We demonstrate that SimPO outperforms DPO in terms of reward accuracy and efficiency.DPO reward implicitly facilitates length normalization. Although the DPO reward expression r(x, y) = β log π θ (y|x) πref(y|x) (with the partition function excluded) lacks an explicit term for length normalization, the logarithmic ratio between the policy model and the reference model can serve to implicitly counteract length bias. As shown in Table 6 and Figure 4a , employing DPO reduces the Spearman correlation coefficient between average log likelihood and response length compared to the approach without any length normalization (referred to as "SimPO w/o LN"). However, it still exhibits a stronger positive correlation when compared to SimPO.10 There is a divergence between DPO's reward formulation, r θ (x, y) = β log π θ (y|x) πref(y|x) , and the average log likelihood metric, p θ (y | x) = 1 |y| log π θ (y | x), which directly impacts generation. As shown in Figure 4b , among the instances on the UltraFeedback training set where r θ (x, y w ) > r θ (x, y l ) , almost half of the pairs have p θ (y w | x) < p θ (y l | x). In contrast, SimPO directly employs the average log likelihood (scaled by β) as the reward expression, thereby eliminating the discrepancy completely, as demonstrated in Figure 6b .DPO lags behind SimPO in terms of reward accuracy. In Figure 4c , we compare the reward accuracy of SimPO and DPO, assessing how well their final learned rewards align with preference labels on a held-out set. SimPO consistently achieves higher reward accuracy than DPO, suggesting that our reward design facilitates better generalization and leads to higher quality generations. KL divergence of SimPO and DPO. In Figure 5a , we present the KL divergence between the policy model trained with DPO and SimPO and the reference model with different β, measured on the winning responses from a held-out set during training. Figure 5b shows the corresponding AlpacaEval 2 LC win rate. Although SimPO does not apply any form of regularization against the reference model, the KL divergence of SimPO is reasonably small. Increasing β reduces the KL divergence for both DPO and SimPO, with DPO exhibiting a more pronounced reduction at higher β values. In this particular setting (Mistral-base), Figure 5b demonstrates that a smaller β can improve AlpacaEval 2 performance, despite the higher KL divergence. 11 We hypothesize that when the reference model is weak, strictly constraining the policy model to the reference model may not be beneficial. As a caveat, while we did not observe any training collapse or degeneration with proper tuning, in principle, SimPO could potentially lead to reward hacking without explicit regularization against the reference model. In such a scenario, the model might achieve a low loss but degenerate.SimPO is more memory and compute-efficient than DPO. Another benefit of SimPO is its efficiency as it does not use a reference model. Figure 5c illustrates the overall run time and per-GPU peak memory usage of SimPO and DPO in the Llama-3-Base setting using 8×H100 GPUs. Compared to a vanilla DPO implementation, 12 SimPO cuts run time by roughly 20% and reduces GPU memory usage by about 10%, thanks to eliminating forward passes with the reference model. 
related work: 
 Reinforcement learning from human feedback. RLHF is a technique that aligns large language models with human preferences and values [18, 97, 58, 7] . The classical RLHF pipeline typically comprises three phases: supervised fine-tuning [96, 71, 32, 21, 44, 25, 77, 15, 81] , reward model training [31, 56, 16, 52, 35, 46] , and policy optimization [65, 4] . Proximal Policy Optimization (PPO) [65] is a widely used algorithm in the third stage of RLHF. The RLHF framework is also widely applied to various applications, such as mitigating toxicity [3, 45, 92] , ensuring safety [24] , enhancing helpfulness [73, 78] , searching and navigating the web [57] , and improving model reasoning abilities [34] . Recently, [13] has highlighted challenges across the whole RLHF pipeline from preference data collection to model training. Further research has also demonstrated that RLHF can lead to biased outcomes, such as verbose outputs from the model [28, 66, 80] .Offline vs. iterative preference optimization. Given that online preference optimization algorithms are complex and difficult to optimize [95, 64] , researchers have been exploring more efficient and simpler alternative offline algorithms. Direct Preference Optimization (DPO) [61] is a notable example. However, the absence of an explicit reward model in DPO limits its ability to sample preference pairs from the optimal policy. To address this, researchers have explored augmenting preference data using a trained SFT policy [91] or a refined SFT policy with rejection sampling [55] , enabling the policy to learn from data generated by the optimal policy. Further studies have extended this approach to an iterative training setup, by continuously updating the reference model with the most recent policy model or generating new preference pairs at each iteration [27, 42, 62, 82, 87] . In this work, we focus exclusively on offline settings, avoiding any iterative training processes.Preference optimization objectives. A variety of preference optimization objectives have been proposed besides DPO. Ranking objectives allow for comparisons among more than two instances [26, 54, 67, 86] . Another line of work explores simpler preference optimization objectives that do not rely on a reference model [38, 84] , similar to SimPO. [8] proposes a method to jointly optimize instructions and responses, finding it effectively improves DPO. [93] focuses on post-training extrapolation between the SFT and the aligned model to further enhance model performance. In this work, we compare SimPO to a series of offline algorithms, including RRHF [86] , SLiC-HF [91] , DPO [61] , IPO [6] , CPO [83] , KTO [29] , ORPO [38] , and R-DPO [60] , and find that SimPO can outperform them in both efficiency and performance. Recently, [70] proposed a generalized preference optimization framework unifying different offline algorithms, and SimPO can be seen as a special case. 
conclusion: 
 In this work, we propose SimPO, a simple and effective preference optimization algorithm that consistently outperforms existing approaches across various training setups. By aligning the reward function with the generation likelihood and introducing a target reward margin, SimPO eliminates the need for a reference model and achieves strong performance without exploiting the length bias. Extensive analysis demonstrates that the key designs in SimPO are crucial and validates the efficiency and effectiveness of SimPO. A detailed discussion of the limitations can be found in Appendix A. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Our work is a methodological paper to improve preference optimization algorithms and does not have direct societal impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our work does not involve releasing data or models that have a high risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We properly credited the creators of the assets used in the paper, and provide URLs to all assets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We have released our code, datasets, and trained models, and provided detailed documentation.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: More in-depth theoretical analysis. Despite the empirical success and intuitive motivation of SimPO, a more rigorous theoretical analysis is necessary to fully understand the factors contributing to its effectiveness. Additionally, we introduce an additional hyperparameter, the target reward margin, which requires manual tuning. Future work could explore how to determine the optimal margin automatically and provide a more theoretical understanding of SimPO.Safety and honesty. SimPO is designed to optimize the generation quality of language models by pushing the margin between the average log likelihood of the winning response and the losing response to exceed a target reward margin. However, it does not explicitly consider safety and honesty aspects, which are crucial for real-world applications. Future work should explore integrating safety and honesty constraints into SimPO to ensure that the generated responses are not only high-quality but also safe and honest. The dataset used in this work, UltraFeedback [23] , primarily focuses on helpfulness, and future research may consider a more comprehensive study utilizing larger-scale preference datasets [39, 90] and evaluation benchmarks [76] that place a strong emphasis on safety aspects. Nonetheless, we observe that this method consistently achieves high TruthfulQA [53] performance compared to other objectives in Table 9 , suggesting its potential for safety alignment.Performance drop on math. We observed that preference optimization algorithms generally decrease downstream task performance, particularly on reasoning-heavy tasks like GSM8k, as shown in Table 9 . SimPO occasionally results in performance comparable to or worse than DPO. We hypothesize that this may be related to the choice of training datasets, hyperparameters used for training, or a mismatch of chat templates used for downstream task evaluations. One explanation is that the preference optimization objective may not be effectively increasing the likelihood of preferred sequences despite increasing the reward margin. [59] first observed this phenomenon and point out that this can hinder learning from math preference pairs where changing one token can flip the label (e.g., changing 2 + 2 = 4 to 2 + 2 = 5). They propose a simple regularization strategy to add back a reference-model calibrated supervised fine-tuning loss to the preference optimization objective, and effectively mitigate this issue. Future work may consider integrating this regularization strategy into SimPO to improve performance on reasoning-heavy tasks.We find that hyperparameter tuning is crucial for achieving optimal performance of preference optimization methods. However, the importance of careful hyperparameter tuning may have been underestimated in prior research, potentially leading to suboptimal baseline results. To ensure a fair comparison, we conduct thorough hyperparameter tuning for all methods compared in our experiments. General training hyperparameters. For the Base training setups, we train SFT models using the UltraChat-200k dataset [25] with the following hyperparameters: a learning rate of 2e-5, a batch size of 128, a max sequence length of 2048, and a cosine learning rate schedule with 10% warmup steps for 1 epoch. All the models are trained with an Adam optimizer [43] .For the preference optimization stage, we conduct preliminary experiments to search for batch sizes in [32, 64, 128] and training epochs in [1, 2, 3] . We find that a batch size of 128 and a single training epoch generally yield the best results across all methods. Therefore, we fix these values for all preference optimization experiments. Additionally, we set the max sequence length to be 2048 and apply a cosine learning rate schedule with 10% warmup steps on the preference optimization dataset.Method-specific training hyperparameters. We have noticed that the optimal learning rate varies for different preference optimization methods and greatly influences the benchmark performance. Therefore, we individually search the learning rates in the range of [3e-7, 5e-7, 6e-7, 1e-6] for each method. Table 7 shows the detailed information on method-specific hyperparameters search ranges for baselines. 13 Table 8 shows SimPO's hyperparameters used under each setting.Decoding hyperparameters. For AlpacaEval 2, we use a sampling decoding strategy to generate responses, with a temperature of 0.7 for the Mistral-Base setting following zephyr-7b-beta,14 a temperature of 0.5 for the Mistral-Instruct setting following Snorkel-Mistral-PairRM-DPO, and a temperature of 0.9 for both Llama 3 settings. 15 For Arena-Hard, we use the default greedy decoding for all settings and methods. For MT-Bench, we follow the official decoding configuration which defines different sampling temperatures for different categories.Computation environment. All the training experiments in this paper were conducted on 8×H100 GPUs based on the alignment-handbook repo. 16To examine how preference optimization methods affect downstream task performance, we evaluate models trained with different methods on various tasks listed on the Huggingface Open Leaderboard [9] . These tasks include MMLU [36] , ARC [19] , HellaSwag [89] , TruthfulQA [53] , Winograd [48] , and GSM8K [20] . We follow the established evaluation protocols and present the results for all models in Table 9 . Generally, we find that preference optimization's effect varies across tasks.Knowledge is largely retained with a small loss. Compared to the SFT checkpoint, we find that all preference optimization methods generally maintain MMLU performance with minimal decline. In this aspect, SimPO is largely comparable to DPO.Reading comprehension and commonsense reasoning improves. For ARC and HellaSwag, preference optimization methods generally improve performance compared to the SFT checkpoint. One hypothesis is that the preference optimization dataset contains similar prompts to these tasks, which helps the model better understand the context and improve reading comprehension and commonsense reasoning abilities.Truthfulness improves. Surprisingly, we find that preference optimization methods consistently improve TruthfulQA performance compared to the SFT checkpoint, and the improvement could be as high as over 10% in some cases. Similarly, we hypothesize that the preference dataset contains instances that emphasize truthfulness, which helps the model better understand the context and generate more truthful responses.Math performance drops. GSM8K is the benchmark that shows the most volatility across methods. Notably, except for ORPO, almost all approaches lead to consistent drops in one or more settings. We hypothesize that ORPO retains performance largely due to its supervised fine-tuning loss for regulation. [59] adds a reference-model calibrated supervised fine-tuning loss to the preference optimization objective, and find that it effectively solves the issue and maintains performance on math tasks as well.Overall, identifying a pattern in downstream performance is challenging. Comprehensive analysis is difficult due to using different pretrained models, preference optimization datasets, and objectives.Recent works indicate that gradient-based approaches could be effective in finding relevant data for downstream tasks [81] , and could possibly extended to understand the effect of preference optimization. We believe a thorough study on how preference optimization affects downstream performance would be valuable and call for a rigorous and more comprehensive analysis in future work.We present the standard deviation of AlpacaEval 2 and the 95% confidence interval of Arena-Hard in Table 10 . All these metrics are reasonable and do not exhibit any significant outliers or instability.Length normalization decreases generation length and improves generation quality. Removing length normalization from the SimPO objective results in an approach similar to Contrastive Preference Optimization (CPO) [83] , which interpolates reward maximization with a supervised fine-tuning loss and has demonstrated strong performance in machine translation. However, without the supervised fine-tuning loss, the reward maximization objective without length normalization is suboptimal in preference optimization.We analyze the generation length of models trained with or without length normalization on Al-pacaEval 2 and Arena-Hard. As shown in Figure 6 , length normalization significantly decrease the generation length by up to 25% compared to when it is not used in most cases. However, even though the generation length is shorter, the models with length normalization consistently achieve much higher win rates on both benchmarks. This suggests that length normalization can effectively control the verbosity of the generated responses, and meanwhile improve the generation quality.Length is not a reliable indicator of generation quality. We further analyze the generation length of models trained with different methods on AlpacaEval 2 and Arena-Hard, as shown in Table 10 . Generally, we find that no single method consistently generates longer or shorter responses across all settings. Additionally, even though some methods may generate longer responses, they do not necessarily achieve better win rates on the benchmarks. This indicates that the length of the generated responses is not a reliable indicator of generation quality.SimPO demonstrates minimal exploitation of response length. We observe that SimPO has a shorter generation length compared to DPO in the Llama-3-Instruct case but exhibits a higher generation length in other settings, with up to 26% longer responses on AlpacaEval 2. Conversely, SimPO only increases length by only around 5% on Arena-Hard compared to DPO. It is fair to say that the generation length heavily depends on the evaluation benchmark. A stronger indicator is that SimPO consistently achieves a higher length-controlled win rate on AlpacaEval 2 compared to the raw win rate, demonstrating minimal exploitation of response length. (a) Mistral-Base. A << B A < B A = B A > B A >> B A = SimPO A >> B A > B A = B A < B A << B A = DPO 0.A << B A < B A = B A > B A >> B A = SimPO A >> B A > B A = B A < B A << B A = DPO 0.We examine the gradients of SimPO and DPO to understand their different impact on the training process.EQUATIONwheres θ = σ β |y l | log π θ (y l |x) - β |yw| log π θ (yw|x) + γ , d θ = σ β log π θ (y l |x) πref(y l |x) -β log π θ (yw|x) πref(yw|x)represent the gradient weight in SimPO and DPO, respectively. It can be seen that the differences are twofold: (1) comparing the gradient weights s θ and d θ , SimPO's gradient weight s θ does not involve the reference model and has a straightforward interpretation: the weights will be higher for samples where the policy model incorrectly assigns higher likelihood to y l than y w ; (2) comparing the gradient updates, SimPO's gradients on y l and y w are length-normalized, while DPO's are not. This corresponds to the empirical findings [60] that DPO may exploit length bias: longer sequences with more tokens will receive larger gradient updates in DPO, dominating the training process.We present the win rate heatmap of Mistral-Base and Mistral-Instruct on AlpacaEval 2 and Arena-Hard in Figure 7 and Figure 8 , respectively. Based on this analysis, we present qualitative examples of responses generated by a SimPO model, a DPO model and the baseline model GPT-4-Preview-1106 on AlpacaEval 2.Comparing SimPO with DPO. In Figure 9 and Figure 10 , we present an example where Mistral-Base-SimPO generates a better-structured answer compared to Mistral-Base-DPO. Given the question, "How can you determine if a person is genuinely interested in a conversation or simply being polite?", the DPO model generates a response with a long list of bullet points, making it difficult to understand the relationships between different points. In contrast, the SimPO model produces a well-structured answer with high-level categorization of different behaviors, followed by detailed suggestions for each category. This makes the answer more readable and easier to understand.Comparing Instruct models with Base models when trained with SimPO. In Figure 11 , we present an example where Llama-3-Instruct generates a more detailed and well-formatted answer compared to the baseline model, and as well as the Llama-3-Base-SimPO model. Given the question: What language does Argentina people speak? Llama-3-Base-SimPO only gives a very brief answer. GPT-4-Preview-1106 gives a more detailed answer in explaining how the Argentina Spanish differs from standard Spanish. However, the answer is not well formatted and a bit hard to parse. Llama-3-Instruct-SimPO gives a detailed and well-formatted answer, which is easier to read and understand, and offers sufficient details. Figure 11 : A case study on AlpacaEval 2 demonstrates that Llama-3-Instruct, trained with SimPO, provides a better formatted and more detailed answer than both Llama-3-Base, also trained with SimPO, and the baseline model GPT-4-1106-Preview. This illustrates how the instruction setting typically outperforms the base setting.Input → Q: what language does argentina people speak Llama-3-Base SimPO → The official language spoken in Argentina is Spanish, specifically Argentine Spanish, which shares similarities with other varieties of Spanish spoken in neighboring countries like Uruguay, Chile, and Paraguay.Llama-3-Instruct SimPO → In Argentina, the official language is Spanish. Specifically, the dialect spoken in Argentina is known as Rioplatense Spanish, which is a variation of Spanish with some unique characteristics and influences from Italian, German, and other immigrant languages.Rioplatense Spanish is distinguished by: 1. Voseo: Instead of using tú (you) as the informal singular pronoun, Argentines use vos and its associated verb conjugations.2. Lunfardo: A slang vocabulary derived from Italian, Corsican, and other immigrant languages, which has been incorporated into everyday speech.3. Accent and pronunciation: Argentines tend to pronounce the "c" and "z" sounds with a soft "s" or "sh" sound, rather than a hard "th" sound like in standard Spanish.Overall, Argentine Spanish is known for its distinctive flavor and is widely understood throughout the Spanish-speaking world. However, it's worth noting that there may be regional variations within Argentina, and some dialects may differ slightly from the standard Rioplatense variety.gpt4-turbo-1106-preview → The primary language spoken in Argentina is Spanish. Specifically, the variety of Spanish spoken there is known as Rioplatense Spanish, which is characterized by its unique accent and the use of the pronoun "vos" instead of "tú" for the second person singular. Additionally, there are various indigenous languages spoken in different regions of the country, as well as immigrant languages such as Italian, German, and English, but these are much less widespread than Spanish.Strong SFT model and high-quality policy data diminish algorithm differences. With a strong SFT model like Llama-3-8B-Instruct, and as the preference optimization data quality improves, the differences between algorithms become less pronounced. For instance, DPO achieved a similar win rate as SimPO in terms of raw win rate, and DPO, IPO, and R-DPO all exhibited comparable raw win rates on Arena-Hard. However, SimPO maintains an advantage by producing shorter sequences, resulting in a significantly better LC win rate on AlpacaEval 2. Stronger downstream task performance. The v0.2 version also shows improved performance in downstream tasks across various objectives. However, DPO, IPO, R-DPO, and SimPO continue to experience a decline in reasoningintensive domains such as GSM8K. In contrast, objectives that include an SFT component maintain their performance in mathematical tasks.Incorporating SFT regularization in SimPO. Several reference-free algorithms, including RRHF [86] , SLiC-HF [91] , CPO [83] , and ORPO [38] , employ SFT regularization in their objectives. SFT regularization can be an effective method to prevent reward hacking, ensuring that the solution maintains low loss without resulting in degraded generations. We also experiment with the integration of an SFT loss in SimPO, yielding the following objective:LSimPO w/ SFT(πθ ) = -E (x,yw ,y l )∼D log σ β |yw| log π θ (yw|x) - β |y l | log π θ (y l |x) -γ + λ log π θ (yw|x) .As shown in Table 14 , the addition of the SFT regularization leads to a decrease in performance on AlpacaEval 2. However, we note that SFT regularization provides substantial benefits to certain tasks such as GSM8K, as shown in Table 12 . These contrasting results suggest that the impact of SFT in preference optimization may vary depending on the training setup and the nature of the task. Further comprehensive studies on this topic are left for future research.I Applying Length Normalization and Target Reward Margin to DPO (Jul 7, 2024)Since the release of the paper, we have had inquiries from researchers about whether the key design elements of SimPO-length normalization and target reward margin-could benefit DPO. By doing so, we will derive the following two objectives:L DPO w/ LN (π θ ; π ref ) = -E (x,yw,y l )∼D log σ β |y w | log π θ (y w | x) π ref (y w | x) - β |y l | log π θ (y l | x) π ref (y l | x) . L DPO w/ γ (π θ ; π ref ) = -E (x,yw,y l )∼D log σ β log π θ (y w | x) π ref (y w | x) -β log π θ (y l | x) π ref (y l | x) -γ .An intuitive understanding of how length normalization could benefit DPO is that, despite DPO's reward design being implicitly normalized by the reference model, the policy model might still exploit length bias from the data, resulting in a disproportionately high probability for longer sequences. Applying length normalization could help mitigate this effect.We train models with the objectives mentioned above and compare their performance to that of DPO and SimPO, as shown in Table 15 .The results indicate that, unlike SimPO, length normalization and target reward margin do not consistently benefit DPO. Specifically, length normalization significantly improves DPO performance only in the Mistral-Base setting, where the preference optimization dataset shows a strong length bias. However, it does not provide a benefit in the Mistral-Instruct setting, where the lengths of winning and losing responses are comparable. This is likely because DPO already includes an implicit instance-wise target reward margin via the reference model, as shown in the derivation below. Performance degradation on other benchmarks for Llama-3-SimPO checkpoints. After releasing the Llama-3-SimPO checkpoints, we received extensive feedback about performance degradation on benchmarks measuring specific capabilities, such as MMLU and GSM8K. To investigate this issue, we continued training the Llama-3-8B-Instruct model with different learning rates, as reported in Table 16 . We find that using a higher learning rate results in a stronger model in chat-oriented benchmarks, at the cost of catastrophic forgetting on GSM8K and MMLU. 17 With a smaller learning rate, the model's performance on chat benchmarks is slightly worse, but its performance on GSM8K and MMLU is better retained. This demonstrates a trade-off between chat-oriented benchmarks and other benchmarks when continuing training from a strong instruction-tuned model. Applying SimPO to Gemma 2 models presents a different trend. We evaluate SimPO using Google's recently released Gemma-2-9B-it model [72] , which represents a strong open-source model.L DPO = log σ β log π θ (y w | x) π ref (y w | x) -β log π θ (y l | x) π ref (y l | x) = log σ β log π θ (y w | x) -β log π θ (y l | x) -β log π ref (y w | x) -β log π ref (y l | x) =γref .For training data, we generate up to 5 responses per prompt from the UltraFeedback dataset [23] and use the ArmoRM model [79] to annotate preferences between responses. We compare our SimPO against a DPO-trained variant, both fine-tuned from the Gemma-2-9B-it base model. As shown in Appendix J, SimPO demonstrates superior performance on chat benchmarks like AlpacaEval 2 and Arena-Hard while maintaining the model's original zero-shot capabilities on tasks like GSM8K and MMLU. Notably, we find that varying the learning rate during fine-tuning has minimal impact on the model's performance. These results suggest an underlying property difference between the Llama-3 checkpoints and the Gemma 2 checkpoints, and might be worth further investigation.Gemma-2-9B-it-SimPO significantly improved the ranking of the Gemma-2-9B-it model on Chatbot Arena. During the development stage, we relied solely on automated metrics to evaluate the model's performance. To determine if these metrics aligned with real user preferences, we submitted our best-performing model, Gemma-2-9B-it-SimPO, to the Chatbot Arena leaderboard hosted by LMSYS [17] . We find that our model improved the original Gemma-2-9B-it ranking from 36th to 25th, making the SimPO variant the top-ranked <10B model on the Chatbot Arena leaderboard based on real user votes as of September 16th, 2024. Justification: The claims have been validated with extensive empirical results.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.As of September 16th, 2024.This margin is termed home advantage in Bradley-Terry models[1,30].It is unclear whether the released instruct checkpoints have undergone supervised fine-tuning (SFT) or the complete RLHF pipeline. For simplicity, we refer to these checkpoints as SFT models.We also experimented with using a stronger reward model, RLHFlow/ArmoRM-Llama3-8B-v0.1[79], to rank generated data, which yields significantly improved performance (see Appendix H and Appendix J). This is the reward model we used in our Gemma 2 experiments.GPT-4-Preview-1106 produces more accurate reference answers and judgments compared to GPT-4.Many recent studies[85,69] have extensively compared DPO and PPO[65]. We will leave the comparison of PPO and SimPO to future work.ORPO can directly train on preference data without the SFT stage. For fair comparisons, we start ORPO from the same SFT checkpoints as other baselines, which yields better results than starting from base checkpoints.Although our models excel on benchmarks, these evaluations have limitations, including restricted query space and potential biases from model-based evaluations. Efforts like WildBench[88] aim to expand these spaces, where SimPO models demonstrate competitive performance.Note that this correlation does not fully reflect the generation length. Despite DPO showing a stronger correlation, the length of its generated responses is comparable to or even slightly shorter than those of the SimPO models. Please find more details in Appendix E.We observe that in some settings (e.g., Llama-3-Instruct), a large β (e.g., β = 10) leads to better performance.DPO can be as memory efficient as SimPO if it were implemented to separate the forward passes of the reference model from the actual preference optimization. However, this implementation is not standard practice.There is a discrepancy between the KTO runs in their original paper, where the original runs use a RMSProp optimizer[63]. We use an Adam optimizer[43] for all the experiments.https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/ models_configs/zephyr-7b-beta/configs.yamlWe grid search the temperature hyperparameter for the Llama-3-Base setting with DPO over 0.1, 0.3, 0.5, 0.7, 0.9, and fix it for all different methods.https://github.com/huggingface/alignment-handbookWe evaluate the zero-shot performance of the models on GSM8K and MMLU using the ZeroEval repository which adopts a unified setup. 


Decision: This paper proposes SimPO for preference optimization without using a reference model. The two key designs are length-normalized reward and target reward margin. Experiments on AlpacaEval, MT-Bench and Arena-Hard verify the proposed method. Reviewers raised concerns about why and how without KL regularization SimPO is able to preserve close enough distance between trained and reference policies, as well as more experimental results, and the authors addressed most of the concerns. I suggest that the authors incorporate the discussions and additional experiments in subsequent versions of the paper.

Reviews: review: paper_summary: paper_summary: The paper introduces SimPO (Simple Preference Optimization), an extension of Direct Preference Optimization (DPO), by replacing the reference-policy-dependent implicit reward with a reference-free reward.
Specifically, SimPO utilizes the average log probability of a sequence as the implicit reward, aligning it with the generation process and eliminating the need for a reference model. This results in improved computational and memory efficiency. Additionally, a target reward margin is introduced into the Bradley-Terry objective to ensure a significant reward difference between winning and losing responses. Extensive experiments show that SimPO consistently outperforms DPO and its variants across various benchmarks, demonstrating its effectiveness in improving model performance.
strength_weakness: strength_weakness: - Although SimPO simplifies the optimization process by removing the reference model calculation of response probabilities, it introduces a new hyperparameter, the target reward margin $\gamma$, which requires tuning.
- While the motivation to address the discrepancy between reward and generation in DPO is understandable, there is a lack of mathematical discussion on how the proposed implicit reward is effective for preference optimization. For instance, standard random sampling with temperature 1 could eliminate the discrepancy without length normalization. However, the experimental results suggest that length normalization is crucial, questioning whether the initial motivation to align optimization and decoding is definitively effective for preference optimization.
- The target reward margin plays a critical role in SimPO, but there is no discussion or comparative experiments with "DPO with an offset" (Amini et al., 2024), which also introduces a similar margin into the DPO loss. This comparison is necessary to understand the relative advantages of SimPO.
questions: questions: - Is it possible to add a careful discussion and conduct comparative experiments about DPO with a target reward margin (Amini et al., 2024)?
- The paper suggests that reducing the discrepancy between reward and generation in DPO results in a more preferable reward for preference optimization. While theoretical analysis might be challenging, is it possible to experimentally validate this hypothesis? For instance, can you evaluate the relationship between the discrepancy's size and preference optimization's performance? This could include testing different decoding methods, such as a beam search with larger beam widths in the length-normalized version that will align more with the average log-likelihood reward.
- Since SimPO omits KL divergence regularization from the initial policy, assessing how the KL divergence behaves would be helpful. Evaluating the model's performance in terms of KL divergence, as analyzed by Gao et al. (2023), could provide valuable insights.
limitations: limitations: The authors have adequately discussed the limitations.
paper_summary: paper_summary: Simpo introduces a simplistic new method of alignment leveraging the log-probability of the sequence eliminating the need to leverage the SFT policy making the implementation computationally less complex. By introducing a notion of margin in the loss function, SimPO outperforms several existing alignment algorithms including DPO and variants without increasing response length. Overall, it provides an interesting light-weight alignment procedure with strong empirical benefits.
strength_weakness: strength_weakness: 1. The primary definition of RLHF has a constraint to the SFT policy through the KL regularization which restricts the model to overoptimize the reward function. Also, it ensures the base performance of the LLM remains intact on several other domains and tasks. However, the proposed method doesn't use KL to the SFT and removes the SFT which makes it unclear what to attribute the improvement to. Specifically, one can violate the constraint and do better with the reward for that specific or similar task, but what about the performance in other tasks that the pre-trained/SFT model was good at?

2. DPO has a closed-form unique solution due to the strong convexity of the KL regularized RLHF objective resulting in the particular objective. However, it's not clear if we don't use the KL regularization, what the original optimization problem we are optimizing for. Is it still strongly convex? If not, what are the insights that this objective will have some good convergence properties? 

3. Under all these questions, the motivation for the objective is heuristically driven and unclear from where it comes from
questions: questions: 1. Can you mention if your original objective is strongly convex or how the particular objective is derived?
2. Can you provide the KL regularization of your model to the SFT and show how much it deviates from the SFT in comparison to baselines?
limitations: limitations: Check above
paper_summary: paper_summary: This paper proposes a new offline-RLHF algorithm SimPO, which significantly improves current DPO variants on a collection of benchmarks.
strength_weakness: strength_weakness: The following comments are for potential improvement of this work and future research directions. These are not complaints.

1. Some fine-grained case studies can be beneficial. The paper primarily focuses on quantitative evaluations. But it would be interesting to see concrete changes between DPO and SimPO, i.e., some representative responses generated by SimPO and DPO, to see what has changed. The LN experiments are great. But things can be better. 

2. With the current formulation, the SimPO objective reminds me of those online RLHF methods. Assuming a -1/1 reward function, many online RLHF methods optimize similar objectives, i.e., direction optimizing the responses with a 1 reward response (winning responses) while reducing the reward of bad ones (losing responses). It will be interesting to see some discussions between SimPO and online RLHF methods. Also, many recent practices have observed that online DPO works greatly. It would be interesting to see if the same phenomena can be observed for SimPO, which looks much closer to online RLHF variants. This is a potential direction to make this paper a much stronger one.

3. Another possible direction to explore is about tasks other than chat, e.g., reasoning tasks like coding and maths. The evaluations in this paper are indeed sufficient as they are, but some discussions on a broad domain can be appreciated.
questions: questions: I do not have any specific questions. As a final comment, if possible, I personally like to see some concrete examples of responses generated by SimPO and DPO (beyond the response lengths) to have a better understanding of the actual improvement.
limitations: limitations: Most limitations have been declared by the authors.
paper_summary: paper_summary: This paper presents SimPO, an offline preference optimization method for LLM alignment. SimPO replaces the KL term in DPO with a length-regularized log-probability and adds a margin value for regularization. Extensive experiments in chat benchmarks show that SimPO significantly outperforms DPO and other preference optimization variants, despite its simplicity. The authors conclude that the success of SimPO can be attributed to the better alignment between training and decoding objectives, as well as reduced exploitation of generating lengthy responses.
strength_weakness: strength_weakness: I do not find any significant weaknesses beyond the limitations proposed by the authors. Since I generally agree with these limitations, I am reiterating them in my review:

+ A theoretical understanding of SimPO is lacking. Although SimPO's design is not theoretically grounded, it performs well in practice, as validated by the authors' experiments.

+ The experiments in the paper solely focus on evaluating helpfulness, disregarding safety, honesty, etc. I think this is the major weakness. SimPO removes the KL regularization from the reference model, so it might intuitively suffer from safety issues. This problem should be further studied in the paper.
questions: questions: + Could the authors explain why length-controlled WR is important? From my perspective, controlling response length is helpful during training because it encourages the model to generate short but useful responses. However, forcing the generation of short responses during evaluation is not necessary. Good responses are good regardless of length. Evaluating length-controlled WR seems to set a privileged metric for SimPO.

+ Since SimPO removes the KL regularization, how does it perform with poor data quality, which is a common application scenario? Is it more fragile and sensitive to data quality?
limitations: limitations: The limitations have been addressed by the authors in Section 6 and in my weaknesses section.

Comments: title: Reponse to Reviewer a8Vq
Thank you for the support!
Thank you for the support!
Looking forward to your feedback!
Response to Reviewer oFKm
Thanks
Response to Rebuttal by Authors
Thank you for your response.
Explore beam search with a large beam
comment: Thanks for the reply. The explanations provided were convincing and significantly contributed to a deeper understanding of SimPO. I appreciate the detailed discussion and would encourage careful inclusion of these points in the revised paper. I have also reviewed the other reviewers' comments and the corresponding responses. I have no additional concerns, so I have raised the score by one level.
Thank you for the thoughtful comments; this is an excellent question! To clarify how length normalization (LN) impacts preference learning, we need to consider three factors:
- **Length bias of total log-likelihood**: As you mentioned, longer sequences typically accumulate a lower (more negative) summed log-likelihood, inherently disadvantaging longer sequences.
- **Length difference in preference pairs**: We found that, in many instances, the winning response ($y_w$) is longer than the losing response ($y_l$). 
- **Objective of preference learning**: Preference learning aims to assign higher rewards to winning responses ($y_w$) compared to losing ones ($y_l$).

Without LN, the reward metric $r_{\text{SimPO w/o LN}}(x,y) = \log \pi_\theta(y \mid x)$ is based on the total log-likelihood. Due to the length bias, longer winning responses ($y_w$) generally have a lower total log-likelihood compared to shorter losing responses ($y_l$). To correctly rank these $y_w$ higher, the model must overcome this bias by assigning disproportionately high probabilities to each token in $y_w$ to ensure the reward on $y_w$ exceeds that on $y_l$. This compensatory effect can cause the model to become miscalibrated, as illustrated in Figure 2(c), where it begins to excessively favor longer sequences. As a result, during decoding, such an ill-calibrated model tends to generate longer sequences. Essentially, LN acts as a calibration mechanism to prevent this overcompensation in the training stage.

To further illustrate this for a mathematical understanding, let's analyze the loss of SimPO with and without LN. For simplicity, we omit the expectation symbol in the following formulas:

- With LN: $ L_{\text{SimPO}} (\pi_\theta) = - \log \sigma  \left( \frac{\beta}{|y_w|} \log \pi_\theta(y_w|x) - \frac{\beta}{|y_l|} \log \pi_\theta(y_l|x) - \gamma \right)$
- Without LN: $ L_{\text{SimPO w/o LN}} (\pi_\theta) = - \log \sigma  \left( \log \pi_\theta(y_w|x) - \log \pi_\theta(y_l|x) - \gamma \right)$
 
It can be seen that without LN, the reward difference $\Delta r = \log \pi_\theta(y_w|x) - \log \pi_\theta(y_l|x)$ is based on **total log-likelihoods**, and is generally more negative when $y_w$ is longer than $y_l$ due to the length bias. Consequently, $\sigma(\Delta r - \gamma)$ approaches 0, and thus the final loss for such instances, $- \log \sigma(\Delta r - \gamma)$, will be very large. This heavily biases the model towards learning such preference pairs. As demonstrated in Figure 2(a), this can result in the failure of learning the opposite cases where $y_w$ is shorter than $y_l$ (the loss for these cases will be very small by applying a similar reasoning).

We hope this addresses the reviewer's question, and we’ll include these clarifications in the revision. If you find this explanation satisfactory, we kindly ask you to consider raising the score, thank you so much for your time! We are also happy to help further clarify any questions you have!
Dear reviewer oFKm,

As the discussion period is coming to an end, we kindly ask that you review our responses and let us know if you have any additional concerns. If possible, we would appreciate it if you could adjust your scores accordingly. Thank you!
Thanks to the author(s) for clarifying the concerns. The additional experiments are helpful in analyzing SimPO more deeply. However, I am confused about the mathematical understanding regarding the role of length normalization. In the context of beam search, length normalization is typically used to address the bias toward generating shorter sequences, which arises because the log probabilities (which are always negative since the max probability is 1) tend to favor shorter sequences. This adjustment prevents the model from preferring short sequences excessively (e.g., [A]).

However, the statement in the paper (“removing the length normalization term from the reward formulation results in a bias toward generating longer but lower-quality sequences,” in line 101) and response (“The reward without length normalization is biased, as evidenced by the tendency to assign higher likelihoods to longer sequences”) seem to contradict the conventional understanding of length normalization. Could the authors please clarify this apparent inconsistency?

On the second point ("Another perspective for ... "), I appreciate the explanation that SimPO’s reward decouples the sequence length from the reward, allowing the model to focus on meaningful differences beyond just the length.

[A] Kenton Murray, David Chiang: “Correcting Length Bias in Neural Machine Translation,” ACL 2018. (https://aclanthology.org/W18-6322/)
Dear Reviewer TXhi,

Thank you once again for your thoughtful review and valuable feedback! We appreciate your support!

Sincerely,  
Authors
Dear Reviewer Wuuc,

Thank you once again for your thoughtful review and valuable feedback! We appreciate your support!

Sincerely,  
Authors
Dear Reviewer a8Vq,

Thank you once again for your thoughtful review and valuable feedback! As the discussion period is ending tomorrow, we would greatly appreciate knowing whether our response has adequately addressed your questions.

If you have any additional comments or concerns, please feel free to share them with us.

Sincerely,  
Authors
Thank you for the reply! We address your questions as follows:

**How the optimization expression comes is not clear, once we remove the KL**: 

While DPO is derived from the KL-regularized RLHF objective, it's important to highlight that this equivalence only holds at the "optimal policy" (i.e., $\pi_{\theta} = \pi^*$). However, this optimal condition is rarely, if ever, achieved during the actual training process due to the complexities of optimizing deep neural networks, thereby breaking the correspondence between RLHF and DPO in practice. Therefore, we believe that a full correspondence between RLHF and *PO is not the prerequisite for *PO algorithms to perform well empirically.

The primary motivation behind SimPO is to **address the training-decoding discrepancy in DPO** which is a significant practical issue: In DPO’s objective, the implicit reward metric being optimized during training incorporates the reference model, which is absent during decoding. This mismatch can lead to a counterintuitive outcome: even if a winning response $y_w$ has a higher reward than a losing response $y_l$ during training (i.e., $r(x, y_w) > r(x, y_l)$), the model might still be more likely to generate $y_l$ during decoding, as illustrated in Figure 4 (b) in our manuscript.

**Also, can you explain why even after not having any KL constraint, SimPO can achieve similar KL divergence to DPO? Which hyperparameters will cause it to stay close and why?** 

For *PO algorithms, the reference model is the initial SFT model where preference optimization begins (that’s why the KL divergence of DPO/SimPO always starts from 0). Two key hyperparameters in SimPO directly influence the KL divergence by controlling how much the model deviates from its initial state:
* **Learning rate**: The learning rate of DPO/SimPO is typically very small (e.g., 5e-7), which naturally constrains model updates from the initial checkpoint, resulting in small KL divergence from the reference model. Increasing the learning rate can lead to greater KL divergence and performance degradation for both methods. 
* **The $\beta$ hyperparameter in the training objective**: The $\beta$ hyperparameter scales the reward difference in the SimPO objective: $\mathcal{L} = -\log \sigma  \left( \frac{\beta}{|y_w|} \log \pi_\theta(y_w|x) - \frac{\beta}{|y_l|} \log \pi_\theta(y_l|x) - \gamma \right)$. A large $\beta$ amplifies even small differences between the winning likelihood $\frac{\beta}{|y_w|} \log \pi_\theta(y_w|x)$ and the losing likelihood $\frac{\beta}{|y_l|} \log \pi_\theta(y_l|x)$. Consequently, even a slight advantage of the winning response will lead to a near-zero loss, minimizing model updates and keeping the KL divergence small.

In practice, we observe that for both DPO and SimPO, hyperparameters can significantly impact the results, further supporting the point that deriving from RLHF does not guarantee empirical robustness.
Thanks for the reply and I appreciate the authors' efforts.
Thanks for the points. I, in general, agree with the comment regarding the closed form of DPO and that it directly doesn't hold for a parametrized setting, which is ok. However, at least for the Tabular class of policies, it holds true and it indeed has the closed form. However, in your problem, how the optimization expression comes is not clear, once we remove the KL.
Also, can you explain why even after not having any KL constraint, SimPO can achieve similar KL divergence to DPO ? Which hyperparameters will cause it to stay close and why?
I'v read the authors' comments to all reviewers. I don't have any additional concerns and would like to remain my score, voting for acceptance.
Thanks for the insightful suggestion to test the discrepancy between the reward metric and decoding methods! We follow the reviewer's suggestion to test beam search with different beam sizes (e.g., 1, 5, 10) using the mistral-base-SimPO model. We find that increasing beam sizes does increase the generation quality measured by win rate, as shown in the following table. It's even better than the number reported in our paper! We think that this helps validating that aligning rewards and decoding metrics more explicitly is beneficial. However, a clear downside is that the runtime significantly increases as beam size increases, and we think this might be why people stick with greedy decoding/sampling instead of using beam search for LLM generation. 

| Decoding Method | LC Win Rate |  WR  |   runtime   |
|:---------:|:-----------:|:----:|:--------:|
|  sampling |     21.5    | 20.8 |   5min   |
|     greedy (beam=1)     |     22.6    | 22.3 |   5min   |
|     beam=5     |     22.2    | 21.6 | 1h 51min |
|     beam=10    |     24.5    | 23.5 | 3h 56min |