SimPER: A Minimalist Approach to Preference  Alignment without Hyperparameters
Authors: Teng Xiao
Yige Yuan
Zhengyu Chen
Mingxiao Li
Shangsong Liang
Zhaochun Ren
Vasant G Honavar
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Model
Alignment
RLHF
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Existing preference optimization objectives for language model alignment require additional hyperparameters that must be extensively tuned to achieve optimal performance, increasing both the complexity and time required for fine-tuning large language models. In this paper, we propose a simple yet effective hyperparameter-free preference optimization algorithm for alignment. We observe that promising performance can be achieved simply by optimizing inverse perplexity, which is calculated as the inverse of the exponentiated average log-likelihood of the chosen and rejected responses in the preference dataset. The resulting simple learning objective, SimPER, is easy to implement and eliminates the need for expensive hyperparameter tuning and a reference model, making it both computationally and memory efficient. Extensive experiments on widely used real-world benchmarks, including MT-Bench, AlpacaEval 2, and 10 key benchmarks of the Open LLM Leaderboard with 5 base models, demonstrate that SimPER consistently and significantly outperforms existing approaches—even without any hyperparameters or a reference model. For example, despite its simplicity, SimPER outperforms state-of-the-art methods by up to 5.7 points on AlpacaEval 2 and achieves the highest average ranking across 10 benchmarks on the Open LLM Leaderboard. The source code for SimPER is publicly available at: https://github.com/tengxiao1/SimPER.

Published as a conference paper at ICLR 2025
SIMPER: A MINIMALIST APPROACH TO PREFERENCE ALIGNMENT WITHOUT HYPERPARAMETERS
Teng Xiao'r, Yige Yuan Zhengyu
Li', Shangsong Liang', Zhaochun Ren', Vasant G Honavar' PPnnssylvania State University University of Chinese Academy of Sciences Meituan Inc Tencent Lab Sun Yat-Sen University LLiden University tengxiao@psu. edu, yuanyige923@gmail. com, vhonavar@psu. edu
ABSTRACT
Existing preference optimization objectives for language model alignment require additional hyperparameters that must be extensively tuned to achieve optimal performance, increasing both the complexity and time required for fine-tuning large language models. In this paper, we propose a simple yet effective hyperparameter- free preference optimization algorithm for alignment. We observe that promising performance can be achieved simply by optimizing inverse perplexity, which is calculated as the inverse of the exponentiated average log-likelihood of the chosen and rejected responses in the preference dataset. The resulting simple learning objective, SimPER (Simple alignment with Perplexity optimization), is easy to implement and climinates the need for expensive hyperparameter tuning and reference model, making it both computationally and memory efficient. Extensive experiments on widely used real-world benchmarks, including MT- Bench, AlpacaEval 2, and 10 key benchmarks of the Open LLM Leaderboard with 5 base models, demonstrate that SimPER consistently and significantly outperforms existing approaches even without any hyperparameters or a reference model, For example, despite its simplicity, SimPER outperforms sfate-of-the-at methods by up 5.7 points on AlpacaEval and achieves the highest average ranking across 10 benchmarks on the Open LLM Leaderboard. The source code for SimPER is publicly available at the Github: https: // /gi thub. com conntteggxial//Simeee
INTRODUCTION
Learning from preference data plays crucial role in fine-tuning large language models to ensure that pretrained LLMS are aligned with human or societal values and preferences (Bai et al., 2022; Ouyang et al., 2022; Stiennon et al., 2020). In recent years, reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022: Christiano et al., 2017) has been proposed for fine-tuning language models based on human preferences. In the RLHF pipeline (Ouyang et al., 2022), reward model first fit to a dataset of human preferences in the form of a classifier between chosen and rejected responses. Next, LLM policy is trained using RL algorithms such proximal policy optimization (PPO) (Schulman et al., 2017) to generate responses given the input prompts with high reward.
While RLHF produces models with impressive capabilities across diverse tasks, ranging from pro- gramming to creative writing, it introduces notable complexities into the training process (Engstrom al., 2020; Rafailov et al., 2024), involving inefficient and unstable optimization, well training on separate reward and policy models. This potentially worsens the sample complexity and compro- mises efficient convergence. To address these issues, offline preference finettuning (Tajwar al., 2024) methods, such as DPO (Rafailov et al., 2024), IPO (Azar et al., 2024), and KTO (Ethayarajh al., 2024a). have been proposed to replace RLHF with supervised learning on human preference data. More recently, SimPO (Meng et al., 2024) eliminates the need for reference model, making DPO more compute and memory efficient. These methods eliminate the need for explicit reward modeling by directly using the likelihood of language model policy define a implicit reward fitted to the preference data, while achieving notable competitive performance (Tajwar et al., 2024).
"qqual contribution.

---

Published as a conference paper at ICLR 2025
Mistra -7B-Base
Mistral-78-Instruct
Llama-3-80-Base
Uama--388--4 3-8--nnstut
7.8
Score
73
7.2
7.8
74
0.3
10
1.2
14
1.6
0.3
1.0
3.4
1.6
0.3
1.0
3.2
14
1.6
1.0
1.2
1.4
1.6
SImPER
SimPO
Figure 1: Evaluation on the MT-Bench Score (1-10) of SimPO and our SimPER across different large language models reveals the high sensitivity and instability of SimPO with respect to its hyperparame- ter y across models. contrast, our SimPER, which operates without any hyperparameters in the objective function, consistently and significantly outperforms SimPO across a wide range of models. Additional experimental evidence on other widely used benchmarks is provided in Section 4.
However, these methods require additional hyperparameters that must be carefully tuned as shown in Table 1, and the performance of current preference optimization methods, such as DPO, KTO, IPO, and others, is highly sensitive these hyperparameters across dif- ferent LLMs, as already shown by (Hugging- Face, 2023; Liu et al., 2024a; Meng et al., 2024; Liu et al., 2024c; Wu et al., 2024). In
Table 1: State-of-the-ie preference fine-tuning losses are compared terms of properties: hyperparameters, the number of hyperparameters, and the elimination of the reference model (see Appendix B.2 for details of these methods and hyperparameters).
Method Hyperparameters
#tpperparmmders
w/o Reference Model
DPO
d
IPO
KTO
Au Au
Figure 1, we also show that tuning hyperpa- rameters is also crucial to achieve optimal performance with the recent state-of-the. -frt algorithm SimPO, which eliminates the need for reference model. This challenge largely prevents us from aligning large language models in real-world applications, given that a single post-training process for alignment is usually very expensive and takes long time (Dubey al., 2024). To this end, we ask an important research question for large language model alignment: Can we design an efficient and effective hyperparameter-
SEAC
6,A
SimPO
7.3
2
SiXPER
0
preference optimization method for alignment?
In this paper, we answer this question aftirmatively. We propose SimPER (Simple alignment with Perplexity optimization), simple yet effective offline preference optimization objective that elimi- nates the need for reference model and any tunable hyperparameters. The key to SimPER is directly optimizing the reverse perplexity of chosen and rejected responses within the preference dataset. Perplexity (Jelinek et al., 1977) is well-known evaluation metric for language modeling, commonly used to assess a model's s ability to process long text. is calculated as the inverse of the exponentiated average log-like likelihood of the responses. SimPER achieves alignment by solving reverse perplexity optimization problem, minimizing perplexity over the chosen response while maximizing perplexity over the rejected response, enabling the model to better align with human preferences. Our simple SimPER validates that perplexity is also an effective optimization indicator for LLM alignment.
Moreover, our further analysis proves that, unlike optimizing Kullback- divergence (KLD) in SimPO, our algorithm effectively minimizes the Total Variation distance (TVD). From a gradient perspective, the robust nature of TVD balances gradients from positive and negative responses, which ensures that the contribution from negative samples does not overshadow those from its positive counterpart, thereby mitigating the issue of decreasing the likelihood of chosen response during preference optimization as noticed by recent works (Meng et al., 2024; Pal et al., 2024).
We empirically demonstrate that SimPER enjoys promising performance on extensive benchmarks such as the Open LLM Leaderboard (Beeching et al., 2023a), MT-Bench (Zheng et al., 2023), and AlpacaEval (Li et al., 2023b). Despite its simplicity, our results show that SimPER consistently and significantly outperforms existing approaches across various large language models, without the need for any hyperparameters and a reference model in the objective function for alignment.
RELATED WORK
Reinforcement Learning from Human Feedback (RLHF) an effective technique designed to align LLMs with human preferences (Christiano et al., 2017). The training process for RL.HF includes
2

---

Published as a conference paper at ICLR 2025
three stages: initial supervised fine-tuning (Zhou et al., 2024; Xia et al., 2024), training of the reward model from human preference data (Gao et al., 2023a; Xiao & Wang, 2021), and policy optimization using reinforeement learning, notably Proximal Policy Optimization (PPO) (Schulman et al., 2017). While RLHF provides significant benefits in various domains, such as instractionffolloiing (Ouyang al., 2022), safety alignment (Bai et al., 2022), and truthfulness enhancement (Tian et al., 2023), requires more complex training pipeline compared to traditional supervised learning methods.
Recent literature highlights the inherent complexity of online preference optimization algorithms, driving the exploration of more efficient offline alternatives. A notable advancement is Direct Preference Optimization (DPO) (Rafailov et al., 2024), which eliminate the need for explicit reward modeling by directly using the likelihood of policy to define an implicit reward fitted to the preference data. Inspired by DPO, various methods such as IPO (Azar et al., 2024). KTO (Ethayarajh et al., 2024b), and others (Yuan et al., 2024a; Xu et al., 2024; Hong et al., 2024; Xiao et al., 2024b;a) have been proposed. While these approaches are effective, they typically necessitate an extensive search for one more hyperparameters, 2024) removed the need for a reference model in DPO, yet introduced two additional hyperparameters: the reward scaling factor and the target reward margin, which require significant manual tuning. Tuning hyperparameters often entails an iterative ridl--nn-erroo process, resulting in substantial computational overhead, particularly for large-scale language models. In this paper, we introduce SimPER, a simple yet effective objective that eliminates the need for costly hyperparameter tuning and a reference model, thus enhancing both learning and memory efficiency in practice.
well as the use of reference model. Recently, SimPO (Meng et al.,
Also worth mentioning is a body of work on perplexity (Jelinek et al., 1977) in language modeling. Researchers use perplexity, an evaluation metric aligned with the causal language modeling objective of LLMs, to assess whether a test input falls within the LLM's expertise and how it relates to the LLM's pretraining data (Chen et al., 2024b; Marion et al., 2023; Gonen et al., 2023). lower per- plexity indicates that the model's predictions are generally more accurate, while a higher perplexity suggests that the model finds the content more unpredictable. Recent work (Ankner et al., 2024; Muennighoff et al., 2024) also uses perplexity to identify high-quality subsets of large-scale text datasets that improve performance. Perplexity has also been used to fuse knowledge from multiple models (Mavromatis et al., 2024). As perplexity provides a sequence-length normalized expression of the model's 's confidence, recent works have utilized the inverse perplexity score to detect hallucina- tions (Valentin et al., 2024) and for confidence estimation (Liu et al., 2024b). In contrast these works, we propose a simple yet effective alignment objective based on perplexity, demonstrating that perplexity is also a surprisingly effective indicator for achieving alignment on preference data.
3 THE PROPOSED METHOD
3.1 BACKGROUND
Notations. We consider the problem of preference fine-tuning: Let the text sequences x
[2), ... denote the prompt, and y = =[y1, [1,,,92,...] and y: [3), .2....] denote two responses, sampled from the reference policy T x). The response pairs are then presented to an oracle who express preferences for responses given the prompt, denoted as yw y: | where yw and yı denote chosen and rejected responses, respectively. Given dataset D, containing preference (x, yws Yt), the goal is to learn language model policy ( x) parameterized by 0 for aligning human preference.
DPO. DPO (Rafailov et al., 2024) is one of the most popular offline preference optimization methods. Instead of learning explicit reward model like RLHF, DPO uses the log-likell likelihood of the policy to implicitly represent the reward function via a closed-form expression with the optimal policy:
r(x, y) = 3 (log ( x) log Tref (y x)) + 3 log ((x).
(1)
DPO aims to optimize me based on the Bradley-Terry (BT) preference model (Bradley & Terry, 1952), p (y > y1 x) = a (r(x. - -x,,)))) and with the following maximum likelihood objective:
T Tolyw x)
T x)
Cppo 0 D) = (x)))))))>>>> log (8 log
[ye x) log
(2)
Frel (y: xx)
where 3 is tunable hyperparameter controlling the deviation from the reference model.
IPO. The Identity Preference Optimization (IPO) (Azar et al., 2024) also avoids a reward learning process and potentially unstable RL training. Specifically, IPO chooses to directly minimize the
3

---

Published as a conference paper at ICLR 2025
following squared loss regression problems by defining an alternative reward function:
( ( w
x (Xe
x)
27
LIPO(0;D) D) = E(x,y=..)))>)
(3)
(
x x))Teer (Yw w
x)
2/3
where 3 also a hyperparameter. A potential advantage of IPO over DPO is that these methods don't assume specific preference model, like BT,, and can work with general preference probabilities.
SimPO. Simple Preference Optimization (SimPO) (Meng et al., 2024) has recently been proposed to eliminate the need for reference model in DPO while achieving promising performance. SimPO probability of response pairs with a margin based on the BT model:
optimizes the
B Y
B y1
log To x) - [)] (4)
(SimP((; (0; (xx.y
log ( log me (y
x)
where y is an additional hyperparameter indicating a target reward margin. In practice, minimizing the above preference fine-tuning objective, or any other contrastive objectives such as those of KTTO (Ethayarajh et al., 2024b) and SLiC (Zhao et al., 2023) (see Appendix B.2), requires extensive hyperparameter tuning. These hyperparameters (e.g.. 3 and ?) play a critical role in alignment performance, shown by (Bai et al., 2022; Liu al., 2024a; Meng et al., 2024) (also see Figure 1), and need to be manually adjusted to achieve optimal performance, significantly increasing complexity and time cost. In this paper, address this limitation by proposing simple yet effective alignment objective, SimPER, which eliminates the need for a reference model and any tunable hyperparameters required by previous work, making the alignment process on large language model more efficient.
3.2 THE LEARNING OBJECTIVE OF SIMPER
In this section, we elaborate on SimPER. The key idea behind SimPER is to encourage the model to minimize the perplexity the chosen response while simultaneously maximizing the perplexity of the rejected response within the preference dataset. Specifically, we optimize the inverse perplexity, which is calculated as the inverse of the exponentiated average negative log-likelihood of the response. The average log- ligeliill of the response under the policy model To is defined follows:
lyl
(x, y) = log (y x)
log To xx) = yyl |yl
log no (y: Xx)))),
(5)
where (y x)
x) 1sT is the defined geometric mean over the sequence of token probabilities. The perplexity is defined as the exponentiated its average negative log- log-liielihood as:
T (y
Perplexity(y | = exp (rr (y x)) = exp (-My) log Togooly xx)
Ttg
(6)
and serves as metric closely tied the causal language modeling objective, allowing whether new input aligns with the model's knowledge and how it corresponds to the pretraining data (Jelinek et al., 1977; Marion et al., 2023: Gonen et al., 2023). Leveraging perplexity measurement of how well language model predicts the response given the prompt, we formulate the aligmment with preference data as an optimization problem that does not require any hyperparameters and reference model during training. Formally, our SimPER learning objective is given as follows:
CsimPER (0; D) = Perplexxity - "y x) Perplexity- "'ri
(7)
-xpp ( 1
Iywl log (y xx) + exp (
log To xx)
(8)
where we directly optimize the reverse perplexity of the chosen and rejected response as the reverse perplexity, i.c., the geometric mean over the sequence of token probabilities, effectively quantifies the model confidence as shown in (Valentin et al., 2024; Liu et al., 2024b). Intuitively, SimPER increases the likelihood of the chosen response and decreases the likelihood of rejected response by optimizing the reverse perplexity, effectively aligning the language model with the preference data.
In summary, SimPER employs simple yet effective formulation that directly aligns with the perplexity generation metric, climinating the need for a reference model. We empirically find that SimPER still achieves a strong performance without requiring any hyperparameters, unlike previous methods.
4

---

Published as a conference paper at ICLR 2025
Chosen
Rejected
Margins
2.0
-2.5
22-
-10
0.8
Steps
Steps
Steps
SImPER
SlmPO
SimPO
SimPO
SimPO
Figure 3: The training dynamics during training of SimPER and SimPO with different hyperparameters on the Mistral-7B (Results on Llama3-8B can be found in Section 4.2). We can observe that SimPER exhibits the least decline in chosen likelihoods, while still achieving the most significant increase in likelihood margins of rejected and chosen, compared to SimPO across various hyperparameters.
3.3 ANALYSIS AND DISCUSSION
In this section, we provide a gradient and divergence analysis further understand our SimPER,
Gradient Analysis. We examine the gradients of SimPER and the state-of-the-art method DPO (Rafailov al., 2024) and SimPO (Meng et al., 2024) and to glean some insight into the optimization process. Note that our analysis also holds for other methods such as IPO (Azar et al., 2024) and SLiC (Zhao et al., 2023). One advantage of the SimPER framework is that the gradients of both chosen and rejected responses are more balanced, thus, we can prevent the model from overfitting the rejected responses. We first analyze the following gradients of DPO and SimPD:
( x) VoTo(yi x)
VoLDro(O:D) = -3E We
(9)
To (y
x) To |x)
VoPo(X v.
xx) Vopolyi x)
VeLsimPo(O:D) = (x
dø
(10)
pe (y |x)
pe(y: x)
Ro x) where the weights we = ( log )x)(()) B (N x) and de = a 8
noly (x)
byil log To (y:lx)
Iy log (y + 7) represent the gradient weight in DPO and SimPO, respectively. Po is the geometric mean over the sequence of token probabilities defined in Equation (5). It can be seen that the gradient the model probability weighted by the reciprocal of the model probability on this rejected response. If the rejection likelihood ( x) + 0 Po(y: x) + O, the norm of the gradient on rejected response will be large, which leads to a huge step of parameter update towards decreasing the likelihood of rejected response compared to the increasing of the chosen response.
For instance, in DPO, the gradient ratio between the decrease in the probability of rejected responses and the increase in the probability of chosen responses is as follows: mo(yulx)) Vota(y))x))
mo(y;i)) Vooooyylxx)" which becomes infinite when (y: x) + A larger gradient ratio leads a faster reduction in the probability of a rejected response compared to that of a chosen response, resulting in a more pronounced decrease for rejected responses, which explains why DPO and SimPO tend to push the model to decrease the likelihood of both chosen and rejected responses during training, as shown in Figure 3. This occurs because rejected and chosen responses often share some tokens, leading to decline in performance on reasoning- heavy tasks, such math and coding, as demonstrated in several recent studies (Meng et al., 2024; Pal et al., 2024; Pang et al., 2024; Chen et al., 2024a). For comparison, we calculate the gradient of our SimPER with respect to 9 using Equation (8):
VeLsmpERR (0: D) (0))==E<xxx.p.> -xx......))
(epe
x) Vepo(y: x)]
(11)
Where the gradient ratio between rejected and chosen responses constant, it has a smaller norm than SimPO (Equation 10). This means that the unlearning of rejected responses is more conservative, and SimPER reduces the gradient imbalance issue between chosen and rejected responses. As shown in Figure SimPER effectively prevents the likelihood of chosen responses from decreasing significantly compared to SimPO, while still achieving substantial margins between the likelihood of chosen and rejected responses. Our experiments also show that SimPER significantly outperforms SimPO.
Divergence Analysis. We next present a theoretical analysis of SimPER, demonstrating its key properties that advantageous for fine-tuning L.MSs with preferences. Recent works (Tajwar et al.,
5

---

Published as a conference paper at ICLR 2025
2024; Xiao al., 2024b; Ji et al., 2024) identify mode-seeking behavior as crucial property for preference alignment, as it reduces the likelihood of rejected responses. In what follows, we show that SimPER also theoretically promotes mode-seeking behavior by optimizing the Total Variation distance (TVD) between the model distribution To and the distribution of chosen response. For simplicity, remove the length averaging from SimPER for the analysis.
Theorem 3.1. Minimizing SFT with respect to f is approximately minimizing the KLD between Tī and the distribution of the chosen response the preference dataset, while minimizing our SimPER approximately minimizing the TVD.
min LSFT 7 min [L(Fehosen (y Ty x)) = E yey #ehosen (y x) log TT x)
Tchowen (y x)
(12)
0
0
min SIMPER + min (F (y x x)) = 2 Fchosen (y x) To x)]
(13)
The proof is provided in Appendix A. This theorem demonstrates that SimPER asymptotically optimizes the TVD (Van Handel, 2014; Ji et al., 2023) between the cho- sen data distributions and the model distribution. In theory, while SimPER and SFT aim discover identical optimal poli- cies, achieving this in practice would require full data coverage and infinite computation. These requirements are not in practice, and hence, the choice of divergences and the opti- mization procedure affects performance. TVD measures the average absolute difference between Tchosen and in all pos- sible text sequences, and SimPER learns to properly allocate its probability mass to best represent the main portion of the distribution of the chosen response, while ignoring outliers. This promotes mode-seeking behavior, which concentrates the probability mass on certain high-reward regions. In contrast, the KLD in SFT encourages assigning equal probability to all responses in the dataset, leading an overestimation of the long tail of the target distribution, as illustrated in Figure 2.
Data
KLD
TVD
Density
Figure 2: Illustration of the charac- teristics of KL.D) and TVD. While SFT exhibits mass-covering behav- ior by minimizing forward KL, SimPER exhibits mode-seeking be- havior, similar RLHF (Tajwar et al., 2024), by minimizing TVD.
In summary, forward KLD encourages all chosen responses in datasets have equal probability, leading to an overestimation of the long tail of the target distribution, whereas reverse TVD sharpens the probability mass on certain high-reward regions of chosen response. Thus, alignment commits to generating a certain subset of high-reward responses, which is more effectively realized by promotes sede-s behavior as shown in recent works (Tajwar al., 2024; Xiao al., 2024b).
EXPERIMENTS
Models. Following (Meng al., 2024), we perform alignment with several families of open- source models, Llama3-8B (Base and Instruct.) (Dubey et al., 2024) and Mistral-7B (Base and Instruct.) (Jiang al., 2023a). We also Pythia-2.8B (Biderman et al., 2023; Rafailov et al., 2024).
Datasets. For the Llama3-8B-Ban and Mistral- BBerall---Basee setups, we follow the same training pipeline as Zephyr (Tunstall al., 2023) and evaluate SimPER on the widely used benchmark dataset for preference fine-tuning: the UltraFeedback Binarized dataset (Cui et al., 2023; Tunstall et al., 2023). For the Llama3-8B-Insrruct and Mistral-7B-Instruct setups, we evaluate using the on-policy datasets generated by SimPO (Meng et al., 2024). Specifically, we use prompts from UltraFeedback and regenerates the chosen and rejected response pairs with the SFT models. For each prompt. it generates five responses using the SFT model with sampling. We then use IIm-blender/PaiRRN (Jiang et al., 2023b) to score the five responses, selecting the highest-scoring one as the chosen response and the lowest-scoring one the rejected response. For Pythia-2.8B Anthropic-HH dataset (Bai et al., 2022) is used for dialogue generation to produce helpful and harmless responses (Rafailov et al., 2024).
Evaluation benchmarks. Following previous work (Rafailov et al., 2024; Tunstall et al., 2023), we evaluate methods fine-tuned on the benchmarks on HuggingFace Open LLM Leaderboard v1 benchmarks (AlpacaEval2, MT-Bench). For
and v2 (Gao et al., 2023b) and instruction-oollwwiii evaluation on Anthropic-HEl we use GPT-4 for zero-shot pair-wise evaluation, consistent with human judgments (see prompts in Appendix B. The task and evaluation details are given in Appendix B.1.

---

Published as a conference paper at ICLR 2025
Table 2: AlpacaEval 2 (Li et al., 2023a) and MT-BBech (Zheng et al., 2023) results under the four settings. LC and WR denote length-controlled and raw win rate, respectively. Our SimPER can achieve surprisingly good performance without any hyperparameters across various settings.
Mistral-7B-Ban BBse
Mistral-7B-Insruuct
Llama3-8B-Base
Llama3-$B-Instre
Method AlpacaEval MT-Bench AlpacaEval 2 MTT-ench AlpacaEval MT-Bench AlpacaEval MT-Bench
(C(%) (%)
GPT4
LC(%) WR WR(%)
GPT4
LC (%)
(R(%)
GPT4
(C(%) WR (%)
GPT4
SFT
8.4
6.2
6.3
17.1
14.7
7.5
6.2
4.6
6.6
26.0
25.3
8.1
DPO
15.1
12.5
26.8
24.9
7.6
18.2
7.7
40.3
SLiC
10.9
8.9
24.1
24.6
7.8
12.3
7.6
27.5
8.1
IPO
11.8
9.4
7.2
20.3
20.3
7.8
14.4
14.2
7.4
35.6
35.6
8.3
KTO
13.1
9.1
7.0
24.5
23.6
7.7
14.2
12.4
7.8
33.1
31.8
8.2
CPO
9.8
8.9
6.8
23.8
28.8
7.5
10.8
8.1
7.4
28.9
32.2
8.0
SimPO
21.5
20.8
7.3
32.1
34.8
7.6
22.0
20.3
7.7
44.7
40.5
8.0
SimPER
22.4
21.3
7,5
37.8
39.5
7.8
25.2
22.9
7.7
48.5
45.7
8.2
Table Evaluation results on various tasks from the Huggingface Open Leaderboard show that our simple yet effective SimPER achieves superior comparable performance to other, more complex preference fine-tuning methods, despite eliminating both hyperparameters and reference model. Although SimPO (Meng et al., 2024) also eliminates the reference model, SimPER demonstrates significant improvements over it across various settings without relying on any hyperparameters.
Method MMLU-PRO
IFEval
GPQA
MUSR
GSMBK
TrethfulQA
Winograd
Arg. Rank
DPO
26.73
10.49
43.27
28.44
43.65
1.36
21.76
61.26
53.06
76.80
4,7
SLiC
26.52
12.45
42.33
33.74
33.74
55.38
48
77. 17.35
25.87
11.52
40.59
28.15
42.15
1.25
27.14
60.84
45.44
77.58
Mistral-7B
KTO
27.51
12.03
43.66
29.45
43.17
2.34
38.51
62.37
56.60
77.27
2,5
Base
SimPO
27.13
10.63
42.94
29.03
39 19.68
2.49
22.21
62.63
50.68
77.54
3.8
SimPER
27.84
15.83
43,99
30.12
43,95
2.57
33.02
63,50
53.64
76.25
DPO
31.58
33.61
47.80
32,23
40.48
4.53
38.67
64.42
53.48
76.80
SLiC
31.11
32.37
46.53
33.29
40.55
3.92
48.82
61.43
54.95
77.27
39.58
Lm
KTO
31.16
37.10
47.98
33.72
40.21
4.14
38.97
63.14
55.76
76.09
4.0
Base
CPO
30.95
38.57
33.15
41.59
$4.29
SimPO
31.61
37.55
48.38
33.22
40.08
4.23
31.54
65.19
59.
76.32
31PPRR
31.99
41.78
48.62
33,80
46.03
4,61
51.02
67.06
62,59
76.24
1,3
DPO
26.81
22.89
45.46
28, 19
46.43
1.89
35.25
66.89
68.40
76.32
SLiC
25.69
29.53
45.24
27.04
43,90
1.95
39.65
$9,90
65.30
76.32
Mistral-7B Instruct
KTO
27.46
35:42
45.34
28. 19
45.77
2.35
38.80
65.72
68.43
75.91
3.2
CPO
26.85
36.81
45.01
28.15
43.28
2.28
38.74
63.23
67.38
76.80
SimPO
27.10
37.52
45.70
28.04
44.71
2.19
35.25
66.89
68.40
76.32
SimPER
21.85
39.84
46.17
28.36
44.92
2.51
40.11
63.
68.78
76.40
1.5
DPO
35.86
44.57
48.31
31.04
39.02
8.23
49.81
63.99
59.01
74.66
SLiC
33.25
44.01
47.55
90.52
38.10
8.29
66.57
61.26
53.23
76.16
4.6
LLamm3-88
KTO
35.00
40.12
47.11
29.20
38.
7.63
57.01
63.57
58.15
73.40
Instruct
CPO
34.56
44.08
48.51
50.08
38.81
7.75
67.40
62.29
54.01
73.72
SimPO
35.09
31.29
39.15
50.72
62.80
60.70
73.32
3ixPBB
36.68
46.06
49.51
31.71
39.35
8.99
68.61
62.37
55.71
75.72
1,7
Baselines. We compare SimPER with the following preference optimization methods: DPO (Rafailov al., 2024), SLiC (Zhao al., 2023), IPO (Azar al., 2024), KTO (Ethayarajh al., 2024b), CPO (Xu et al., 2024) and SimPO (Meng et al., 2024). As shown in (Meng et al., 2024), hyperparam- eter tuning crucial for achieving optimal performance preference optimization methods. We thoroughly tuned the hyperparameters for each baseline and reported the best performance. More details of baselines and the hyperparameter search space can be found Appendix B.2.
4.1 MAIN RESULTS ON BENCHMARKS
Results Instruction-follovine Benchmarks. In Table 2, we report the performance the widely- used instruction- folllwtiionooooownng benchmarks: MT-Bench and AlpacaEval A notable improvement AlpacaEval 2, is observed. Notably,
in the performance of SimPER MT-Bench well the Llama3 fine-tuned by SimPER surpassed the performance of best baseline by 4.9 to 5.2 points on the AlpacaEval win rate across base and instruct settings, demonstrating that SimPER, while
7

---

Published as a conference paper at ICLR 2025
Chosen
Rejacted
Margins
-2.0
-2.4
Likelhood
--4
3.0
1.5
-2.
-3.0
10-
-366
Steps x207
Steps
Steps
SImPER SimPO
SimPO
SimPO 1.2
SimPO
Figure 5: The training dynamics during training of SimPER and SimPO with different hyperparameters on the Lllmm3-8B-Ban We can observe that SimPER exhibits the least decline chosen likelihoods, while still achieving the most significant increase in likelihood margins of rejected and chosen, compared to SimPO across various hyperparameters, and better performance as shown in Table 2.
eliminating the need for hyperparameters and a reference model, still achieves strong performance. Moreover, SimPER consistently achieves superior performance on I.C win rate, demonstrating that it can generate high-quality response without substantially increasing response length. We find that MT-Bench exhibits poor separability across different methods, likely due to the limited scale of its evaluation data and its single-instance scoring protocol. This finding aligns with observations reported in (Meng et al., 2024; Li et al., 2024). Nevertheless, on MT-Bench, SimPER can still consistently achieves superior or comparable performance across various models. Additionally, we provide examples generated by both SimPO and SimPER in Appendix These examples demonstrate that SimPER shows strong promise in aligning language models, ensuring that the generated responses are not only high-quality but also better structured and more reasonable.
Results on Downstream Tasks. In Table 3, we present comparative analysis of performance gains in down- stream tasks on the HuggingFace Leadboard, along with comparative ranking across all tasks. As shown in the table, SimPER, despite its simplicity, achieves remark- able improvements over SimPO and DPO, particularly on challenging reasoning benchmarks such as IFEval, MMLU-PRO and Math. Our results demonstrate that SimPER is highly effective in improving performance across various models. Notably, SimPER outperforms the best baseline by 1.48 to 4.2 points on IFEval across various models. The average improvements over SimPO are especially notable in Mistral-Base and Llama3-Base, SimPER, despite its simplicity, achieves the best over- all performance. These consistent and significant im- provements highlight the robustness and effectiveness of SimPER. In particular, SimPER outperforms the recent SimPO by 19.48 points in GSM8K and 4.23 points in IFEval on Llama3-Base. On GSM8K, SimPER consistently achieves superior performance, though it occasionally surpassed by CPO on Mistral-Base. We hypothesize that these improvements over SimPO can be attributed to the reduced decrease in the likelihood of chosen responses; As the likelihood of chosen response decreases, it results in suboptimal performance, particularly in mathematical reasoning and coding tasks, where the chosen responses are very likely ground-trute answers, as also shown (Pal al., 2024: Yuan al., 2024b). These observations suggest the strong potential of SimPER in real-world applications due to the elimination of hyperparameters and the reference model in the loss function, making more computationally and memory efficient.
Win rate vs. Chosen Response
0.6
Win rate
0.5
0.4
Baseline
SimPER
0.3
DPD
IPO
SinPO
0.2
0.25
0.50 0.75
1.00
Sampling temperature
Figure 4: The win rates, computed by GPT- 4, in comparison to the chosen responses of test prompts in the Anthropic-HH dataset.
Results on Safety Alignment. The benchmarks used above focus mainly on helpfulness. To further evaluate the effectiveness of SimPER in safety alignment on Pythia-2.8B following (Rafailov et al., 2024), we use the Anthropic-HH dataset. Figure 4 shows the win rates computed by GPT-4 over the chosen responses in the test set of Anthropic- HH. Remarkably, SimPER aligns better with human preferences than SimPO, DPO, and IPO, achieving win rates of approximately 60% against the chosen responses, indicating that SimPER shows strong promise in terms of aligning language models and can ensure that the generated responses not only high-quality but also safe and harmless.
8

---

Published as a conference paper at ICLR 2025
Mistrai-7B-Ben
Llama-3-8-BBae
Mistral-7B-Base
Llama-3-88 3-8--Base
0.1
SimPER
SimPER
SimPO
SimPO
Dersity
62
SimPER (pm0,43)
SimiPER (pm0 18)
64
33
DPO 1p-0pp000000 45)
1pm0
DPO (p-0.50)
(p=0 17)
60
20
25 51
25
21.
125
25.
Response length Figure 6: Analyzing perplexity density and response length correlation on Mistral-BB-Base and
Perplexity
Perpiexity
Response length ly]
Llama-3-8B 3-8B-Base. SimPER not only achieves lower perplexity but also does not exploit length bias.
Table 4: Ablation Study Results on Mistral-7B-BB
and -lama--8--Base Evaluating the Effects of
Removing Length Normalization (w/o LN) and Incorporating Reference Model (#/ Ref ))
Open LLM Leaderboard v2
Open LLM Leaderboard v1
Method
MMLU Pro
IFEval
BBH
GPQA
MUSR
MATH
GSMBK
ARC
TruthfulQA
Winograd
SimPER
27.84
15.83
43.99
30.12
43.95
2.57
31.92
63.50
53.64
76.25
Mistral-7B
Base
w/o LN
25.37
1.66
41.76
29.87
44.84
3.25
28.73
59.81
40.57
77.03
w/ Ref
27.31
16.32
43,99
30.54
40.74
2.11
26.91
61.86
43.62
76.72
SimPER
31.99
41.78
48.62
33.80
46.03
4.61
51.02
67.06
62.59
76.24
Llama3-8B Base
wwo LN w/ Ref. 32.06
30.08
36.31
48.62
32.05
45.37
3.17
44.28
60.58
46.87
76.64
36.71
49.30
31.71
41.53
4.61
45.19
63.40
50.84
75.77
4.2 ABLATIONS AND FURTHER ANALYSIS
Results on Ablation Study. In Table 4, we evaluated the effects of removing length normalization (w/o LN) and incorporating reference model (w/ Ref in SimPER the Open LLM Leaderboard. Removing length normalization generally leads a decline in performance across most tasks, except for a slight improvement in the Winograd. Incorporating reference model typically results in performance decrease. However, it does lead to slight improvements in specific tasks such as IFEval, GPQA, and BBH for the Mistral-7B- Base model, and in MMLU Pro, BBH, and MATH for the Mlmma3-B-Bass model. Based on this ablation study, we can conclude that maintaining length normalization and removing the reference model generally enhances the performance our method.
Analysis on Perplexity Density. As shown in Figure 6, the analysis of perplexity density on the held-out test set indicates that SimPER consistently achieves lower perplexity compared SimPO. Specifically, on Mistral-7B-Bane Basee, the density peak of SimPER is reduced by approximately relative SimPO, and on the Llama-3-8B-Ban it reduced by approximately 2. These suggest that SimPER is capable of generating more predictable and coherent responses by directly optimizing the perplexity.
Analysis on Response Length Correlation. Spearman correlation between response length and log probabilities in Figure 6 shows that SimPER exhibits significantly lower correlation compared to DPO, and is comparable to SimPO, suggesting that SimPER does not exploit length bias and generate longer sequences. In contrast, SimPER results in spearman correlation coefficient similar to the SimPO (Meng et al., 2024). Our results demonstrate that SimPER, despite its simplicity, consistently and significantly outperforms existing approaches without substantially increasing response length.
5 CONCLUSION
In this paper, we propose a simple yet effective alignment method for large language models, named SimPER. SimPER eliminates the need for both hyperparameters and a reference model while achieving strong performance. The key idea of SimPER is directly optimize the reverse perplexity of both the chosen and rejected responses in the preference dataset. Specifically, we minimize the perplexity over the chosen response while maximizing the perplexity over the rejected response, ensuring that the model produces responses with high preference scores. We also provide theoretical understanding of SimPER, demonstrating that it can mitigate the problem gradient imbalance between the chosen show that SimPER implicitly exhibits
and rejected responses during optimization. Furthermore, mode-seeking behavior, leading to strong alignment performance. Extensive experiments on widely used benchmarks demonstrate that SimPER significantly outperforms state-of-the-art methods.

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
The work of Vasant G Honavar and Teng Xiao was supported in part by grants from the National Science Foundation (2226025, 2225824), the National Center for Advancing Translational Sciences, and the National Institutes of Health (UL1 TR002014).
REFERENCES
Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew Leavitt, and Mansheej Paul. Perplexed by perplexity: Perplexity-bast baased data pruning with small reference models. arXiv preprint arXiv:2405.20541 2024.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In AISTATS, pp. 4447 4455, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0560 2022.
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open Ilm leaderboard. Hugging Face, 2023a.
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open Ilm leaderboard (2023-2024). https:// http:://huugnggfaee http://wwggnnggaccccccaaeaeee/e/ -eeaepppeap--eemdeaaaaaraeeeeeee oopee 11m_ lem__eaderboard, 2023b.
11m- leaderboard-old/
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Allah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: suite for analyzing large language models across training and scaling. In ICML, pp. 2397-24301 2023.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, pp. 324 345, 1952.
Huayu Chen, Guande He, Hang Su, and Jun Zhu. Noise contrastive alignment of language models with explicit rewards. arXiv preprint arXiv:2402.05369, 2024a.
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-cone context large language models. ICLR, 2024b.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. NeurlPS, 30, 2017.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. 05457, 2018.
arXiv preprint
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reñichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ulrrafeddbcck: Boosting language models with high- feedback. arXiv preprint arXiv:2310.01377. 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The laama herd of models. arXiv preprint arXiv:2407.21788 2024.
10

---

Published as a conference paper at ICLR 2025
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: case study on ppo and trpo. In ICLR, 2020.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model alignment as prospect theoretic optimization. ICML, 2024a.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. ICML, 2024b.
Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open Ilm leaderboard v2. htttp::/huuginngface leaderboard, 2024.
11m_
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In ICMI,, pp. 10835-10866, 2023a.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. A framework for few-shot org/recordd/1025688.. language model evaluation, 2023b. URL htttps://zenoos
Hila Gonen, Srini Iyer; Terra Blevins, Noah A Smith, and Luke Zettlemoyer. Demystifying prompts in language models via perplexity estimation. EMNLP. 2023.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https://arxtie http:/iww.rg/ab///////ee 03874.
Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691, pp. 5, 2024.
HuggingFace. Preference tuning: Fine-tuning language models with human feedback. ht // //buggingface. co/ blog/por pref-tuning. 2023.
Fred Jelinek, Robert Mereer, Lalit R Bahl, and James K Baker. Perplexity- measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, pp. S63-S63, 1977.
Haozhe Ji, Pei Ke, Zhipeng Hu, Rongsheng Zhang, and Minlie Huang. Tailoring language generation models under total variation distance. In ICLR, 2023.
Haozhe Ji, Cheng Lu, Yilin Niu, Pei Ke, Hongning Wang, Jun Zhu, Jie Tang, and Minlie Huang. Towards efficient and exact optimization of language model alignment. arXiv preprint arXiv:2402.00885 00856, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 2023a.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LIm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.0255 2023b.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980. 2014.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, april 2024. URL https://mmsys. ngg/llog/024-004 14--1---ena--aaaeeee 2024.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsumori B. Hashimoto. AlpacaEval: An automatic evaluator of instruction-folloom following models. Gitfuu repository, 2023a.
11

---

Published as a conference paper at ICLR 2025
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: An automatic evaluator of mnstructionfoolooiing models, 2023b.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), ACL, pp.
2022.
Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares- López, Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathieu Blondel. Decoding-time time realignment of language models. ICML, 2024a.
Xin Liu, Muhammad Khalifa, and Lu Wang. Litcab: Lightweight language model calibration short-and long-form responses. ICLR, 2024b.
Yixin Liu, Pengfei Liu, and Arman Cohan. Understanding reference policies in direct preference optimization. arXiv preprini urXiv:2407.13709, 2024c.
Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang. Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining iims at scale. ArXiv, 2023.
Costas Mavromatis, Petros Karypis, and George Karypis. Pack of Ilms: Model fusion at test-time via perplexity optimization. arXiv preprint arXiv:2404. //531, 2024.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference- free reward. arXiv preprint arXiv:2405. 14734, 2024.
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. NeurlPS, 2024.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. NewrlPS, pp. 17730-27744, 2022.
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. Arxiv, 2024.
Richard Yuanzhe Pang. Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason preferenceoptimiinn
Weston. Iterative reasoning preference optimization. arXiv preprint WxXi:2240.119333 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. NeurlPS, 36, 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google poogle-proof q&a benchmark, 2023. URL https: ha/paaxivvvvro/.. abs/2311 12022.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communications of the ACM, pp. 99-106, 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. Musr: Testing the limits of chain-of- thought with multistep soft reasoning. 23/0. 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. NeurIPS, pp. 3008-3021, 2020.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether cfain-of-thought can solve them. 22 22/0.0926), 2022.
12

---

Published as a conference paper at ICLR 2025
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon. Chelsea Finn, and Aviral Kumar. Preference fine-tuning of Ilms should leverage suboptimal, on-policy data. arXiv preprint arXt::2404.14367 2024.
Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. Fine-tuning language models for factuality. arXiv preprint arXiv:2311.08401 2023.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310. /6944, 2023.
Simon Valentin, Jimmiao Fu, Gianluca Detommaso, Shaoyuan Xu, Giovanni Zappella, and Bryan Wang. Cost-effective hallucination detection for Ilms. arXiv preprint arXiv:2407.21224, 2024.
Ramon Van Handel. Probability in high dimension. Lecture Notes (Princeton University), pp. 2-3, 2014.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang. Alex Zhuang, Ronggi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: more robust and challenging multi-task language understanding benchmark, 2024. URL https::///// // arxiv. org/ hts//bb//24000 01574..
Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. f-divergence minimization for sequence-level knowledge distillation. In ACL, pp. 10817-10834, 2023.
Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. beta-dpo: Direct preference optimization with dynamic beta. Arxiv, 2024.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. In JCML, 2024.
Teng Xiao and Donglin Wang. A general offline reinforcement learning framework for interactive recommendation. In Proceedings of the AAAI Conference Artificial Intelligence, 2021.
Teng Xiao, Mingxiao Li, Yige Yuan, Huaisheng Zhu, Chao Cui, and Vasant G Honavar. How to leverage demonstration data in alignment for large language model? a self-imitation learning perspective. arXiv preprint arXiv:2410. 2.. 10093, 2024a.
Teng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li, and Vasant G. Honavar. Cal-dpo: Calibrated direct preference optimization for language model alignment. Advances in Neural Information Processing Systems, 37, 2024b.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of l1m performance in machine translation. In ICML, 2024.
Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback. NeurIPS, 2024a.
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, et al. Advancing Ilm reasoning generalists with preference trees. arXiv preprint -=Xxv:2444.00077 2024b.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Salch, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM-as-a-juue a-jdge with MT-Bench and Chatbot Arena. In NeurIPS Datasets and Benchmarks Track, 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. NeurlPS, 36, 2024.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. instuuction-oolloiing evaluation for large language models. 23/9,07911, 2023.
13

---

Published as a conference paper at ICLR 2025
A PROOFS OF THEOREM 3.1
To prove Theorem 3.1 in the main paper, we first present the following Lemmas:
Lemma A.1. Given the model conditional distribution To x) = [I To (y: x, y.) and
data distribution Taata (y x) = - Tdata (y, x. Yet), then we have the following relationship between the sequence-level TVD objective and its token-level factorization:
by Σ
(T (y:), (y) (xil))
TV (y x), To x)) S By Tiate (xrix)
(14)
i=1
where a data (y:) and To (yys) are shorts for Adata (y: x, vii) and ne (y: x,), respectively.
Proof. We start by re-writing the TVD loss in the following recursive form:
1 E
TV (y Toly x)) yey Tchosen (y x) To xx]
(15)
yy
1
Σ
data (yr yx,)
II
no (y: x,y<
(16)
2
Viyl
i=1
ly
ly data (y:)
1
x
(y:)l
(17)
2
Y1s" Yirrl
i=1
ly
2
datta (y:))
data (y:)
T
<i
(y:))
Σ
II
T (y())
(18)
1=1
1=
1=1
Yith, Niyl t=i+1
y
Σ
Σ
n data Taa T0 "xx())
(19)
2 yiy!" jmm
y
By coneadata (x Th T6'(xi1)
(20)
2 i=1 y;
lyl
E Tusta
TV (y:),
(21)
where Equation (16) breaks the sequence-level summation into the steps and Equation (18) applies the following triangle inequality (Wen al., 2023; Ji et al., 2023):
T
II
II b
<
Σ
a.
bul
II a:
II by j-t+l
(22)
and Equation (19) marginalizes out variables yit1s! .Yyl
Theorem 3.1. Minimizing SFT with respect to 0 is approximately minimizing the KLDD between no
and the distribution of the chosen response in the preference dataset, while minimizing our SimPER is approximately minimizing the TVD.
min LSFT = min KL KTehoooeen (y (T x)) = Σ Eyey Tchosen (y xx) log
Wehowen (y x)
(23)
To x)
min CSmPERR + min (T(Thhoeen (y ())l x)) = Σ VEY TThosee (y x) To x)
(24)
0
14

---

Published as a conference paper at ICLR 2025
Proof. Given that Bywngg (xx) data (y:), where y, represents the observed one-hot distribu- tion with only the w-th index of the observed token being 1 and the others being we have
Taaaa (y:). T8 (y:)) = 2
Σ Taata (y:) - T (y:)l
(25)
Σ
yovr data
[tvel= T6 (y:))
(26)
2
Y1
You aos (yrs)
Σ
ys - nh (y:))] = E yoa
To 7f
(27)
2
yt
=B, Bywrgit (r.)
min To (y:)) - (y)).
(28)
Combing the above with Lemma A.1, we have:
y
(TTastay x),TT({ To x)) S By Byo
TV (Taara (v:), (y)) (y:))
(29)
-y
Σ
no (y: xs yec))
(30)
Recall that the objective of SimPER, without length-averagin is:
LSiMPER (0; D) = exp (log No (y x)) + exp (log mo x))
(31)
Σ - To (Yw w x) 2 (chhoose (y x), To x)).
(32)
Combining Equation (32) completes the whole proof.
B EXPERIMENTAL DETAILS
B.1 DOWNSTREAM TASK EVALUATION
This section is for the detailed introduction to the evaluation of downstream tasks. For models fine-tuned on UltraFeedback Binarized, following previous works (Rafailov al., 2024; Tunstall et al., 2023), we evaluated them on HuggingFace Open LLM Leaderboard v1 and v22 (Beeching al., 2023b; Fourrier et al., 2024), encompassing ten tasks: MMUL-PRO (Wang et al., 2024), IFEval (Zhou et al., 2023), BBH (Suzgun et al., 2022), GPQA (Rein et al., 2023), MUSR (Sprague et al., 2024), MATH (Hendrycks et al., 2021), GSM8k (Cobbe et al., 2021), ARC (Clark al., 2018), TruthfulQA (Lin et al., 2022) and Winogrande (Sakaguchi et al., 2021) provided by the Language Model Evaluation Harness library (Gao et al., 2023b). In addition to the Icaderboard, benchmarks AlpacaEval2 (Li al., 2023b)
also evaluated them on two mnstruction-ollowiii and MT-Bench (Zheng et al., 2023). For models fine-tuned on Anthropic- HH for safety alignment, following previous works (Rafailov et al., 2024), which is shown to be consistent with human judgments.
GPT-4 for zero-shot pair- mire evaluation,
To evaluate the safety of the Anthropic HH trained model, the Anthropic HH test set is used as the evaluation dataset. The preferred responses within this test set serve as the baseline. GPT-4 is then employed determine the winner, which shown be consistent with human judgments. We use gpt-4 4401-44014 for all our experiments. The specific prompts used for the GPT-4 evaluation are detailed below.
Safety Evaluation:
B.2 BASELINE METHODS
This section provides detailed introduction to state-of-the-art basclines for preference fine-tuning, with an emphasis on the usage of hyperparameters in their loss functions.
11m leaderboard
bedeerrrrrrrb.ree.eeeeee 11m, leaderboard
15

---

Published as a conference paper at ICLR 2025
For the following dialogue history a chatbot, which response more helpful and harmless?
Dialogue history: [dialogue history)
Response [Response 1}
Reaponse [Response 2}
FIRST provide one-sentence comparison of the two responses and explain which you feel helpful and harmless. SECOND, to indicate which response is more helpful and harmless. Your response should use the format:
line, state only "1" "2"
Comparison: <one-sentence comparison and explanation>
More helpful: <"1" "2">
Table 5: Prompt for GPT-4 evaluation for the safety alignment task on the Anthropic-Hle dataset. (dialogue history). (Response }} and {Response 2} are placeholders.
DPO Direct Preference Optimization (Rafailov et al., 2024) uses log-like log-likelihood differences to implicitly represent the reward function, eliminating the need for explicit reward model like RLHE. DPO involves one tunable hyperparameter, 3, which controls the deviation from the reference model.
π (y
x)
TT x) x)
CDPO(fs D) = B. log (33 log
3 Blg
Tyet Fyer(ww w x)
Teef (y:
IPO Identity Preference Optimization (Azar et al., 2024) minimizes squared loss regression problem by defining an alternative reward function, avoiding unstable RL training. IPO involves one hyperparameter, 3, to adjust the reward margin.
(1Og
To To X) (y:
x)
1
LIPO(e: D) E (xyx,>>>))>)
Th (y:
x) Tzet (y
x) 2,3
CPO Contrastive Preference Optimization (Xu al., 2024) uses log likelihood as the reward and is trained alongside Supervised Fine- Tine-TTuning (SFT) objective. CPO involves two hyperparameters: B, which scales the log probabilities, and X, which weights the SFT component.
Lcro (0; D) - llo a (3 log No (y x) - log ( x)) - Alog To x).
SLiC Sequence Likelihood Calibration (Zhao et al., 2023) directly uses log-likelihood and includes a SFT objective. SLiC involves hyperparameters: 8, which sets the margin for the ranking loss, and x, which weights the SFT component.
(f D) max (0, - log To x) + log Ty(y; x)) Alog (y x).
SimPO Simple Preference Optimization (Meng et al., 2024) eliminates the need for reference model and optimizes a manghh-regulaiieed probability of response pairs. SimPO involves two hyper- parameters: scale the log probabilities and y to adjust the reward margin.
Lsimro (0; D) E(x.y..))) log a log no (y x) Bog moly; x)) 7]
KTO Kahneman- Tversky Optimization (Ethayarajh al., 2024b) learns from non-paired preference data. KTO involves three hyperparameters: 8, which controls the deviation from the reference model; An and λ₁. which weight the preference components for winning and losing responses, respectively.
To To x)
To x)
LKTO D) ~> a (Blog
Zzef
+ No
Zref
3log
(ve
x)
Trery x)
(33)
where Frof = EE KL (T
x) (Teerre x))]
16

---

Published as a conference paper at ICLR 2025
B.3 IMPLEMENTATION DETAILS
Training Hyperparameters. For general hyperparameters. we adhered strictly to the settings used in SimPO. We applied the following hyperparameters: For the SFT stage, 2 x 10-5. For both the SFT and the preference optimization stages, we use a batch size of 128, maximum sequence length of 2048, and cosine learning rate schedule with 10% warmup steps for one epoch, all through the Adam optimizer (Kingma, 2014). We maintain these settings consistently ensure uniformity and comparability across experiments.
a learning rate of
For method-specifte hyperparameters, we also followed the search strategy from SimPO, noting that our method does not require any additional hyperparameters beyond the learning rate. Specifically, for different baseline methods, each with its own set of additional hyperparameters, the search strategy is Table Each method is individually search for the learning rates [3e 7, 5e 7, 6e 7, le 6].
Table Search ranges for hyperparameters across baseline methods
Method Hyperparameter Search Range
DPO
[0.01, 0.1]
IPO
T 0.1, 1.0)
x= 1.0
CPO
8 [0.01, 0.05, 0.1]
A 00.1, 0.5, 1.0, 10.0) 8 [0.1, 0.5, 2.0,
SLiC
A Aw = 1.0 [0.01, 0.05, 0.1]
KTO
E [2.0, 2.5) E [0.3, 0.5, 1.0, 1.2, 1.6
SimPO
Evaluation Hyperparameters. The hyperparameters for evaluation on AlpacaEval adhere to SimPO, with temperature of 0.7 for the Mistral-Base setting, a temperature of 0.5 for the Mistral- Instruct setting, and temperature of 99 for both Llama3 settings. The hyperparameters for evaluation on MT-Bench are according to the official decoding configuration, which specifies distinet sampling temperatures for various categories.
Computation Resources. All the training experiments in this paper were conducted on 4 x NVIDIA A100 (80G) GPUs with 128 batchsize, based on the alignment-handbood repo.
C
CASE STUDY
In this section, we present examples of comparisons between SimPER and the bascline SimPO.
handbook
17

---

Published as a conference paper at ICLR 2025
Prompt Does the second rule actually exist?
SimPER
The "5 second rule" is urban legend that claims that if food falls the floor, it is still safe to eat long you pick it up within seconds. However, this rule fact, there is no evidence to support it. Here are some reasons why the 5-second rule is not a reliable method for determining food safety:
acientifically proven
Bacteria and viruses can transfer quickly: Bacteria and viruses can transfer from surfaces to food within seconds. The amount of time that passes between a food item's contact with contaminated surface and its being picked up affect the likelihood of bacterial viral transmission.
2. Contamination levels on floors and surfaces: The level of contamination on floors and surfaces vary widely depending the environment, the cleanliness of the area, and the presence of meisture. Even surface dry, bacteria viruses can still be present.
3. Personal hygiene: The cleanliness of your hands, the aurface you place the food on, and the utensils you all play role in food safety. Picking up food within 5 seconds does not ensure that it is clean enough to eat.
pick up the food
4. Temperature: The temperature of food also affect the risk bacterial or viral contamination. Food that has been room temperature for a prolonged time, even if picked up quickly, may still harbor dangerous bacteria.
Therefore, it best follow proper food handling and preparation practices, such as washing hands before and after handling food, using clean utensils and surfaces, and cooking food to the appropriate temperatures to ensure its safety.
SimPO
The "5-second rule" refers to a popular myth or belief that suggests touching food on the ground for less than five seconds makes it safe without contamination. However, this rule does not exist scientifically. According food safety guidelines, bacteria can transfer from contaminated surfaces to food within seconds, regardless of the elapsed time. The duration of contact between the food and contaminated aurfaces plays crucial role in determining the likelihood of bacterial transfer, but it's not limited five seconds. Cross-contaminatioon occurs when bacteria from unwashed hands, dirty utensils, or contaminated surfaces come into contact with food, leading to potential foodborne illnesses. Therefore, adhering to proper hygiene practices, such washing hands, cleaning surfaces, and cooking food thoroughly, remains essential prevent food contamination.
summary, there is no acientific basis for the "5-second rule," and relying solely on this belief could increase the risk of consuming contaminated food."
Table An AlpacaEval 2 generation from the Llama-3-8B SimPO.
model trained with SimPER and prefers SimPER over SimPO. Compared to SimPO, the generation by SimPER better structured, with hierarchical discussions that make more helpful.
18

---

Published as a conference paper at ICLR 2025
Prompt How make escargot?
SimPER Here's general recipe for making escargot:
Ingredients:
1 pound fresh smails (escargots)
1/4 cup butter, at room temperature
clove garlic, minced
2 tablespoons finely chopped pareley
- 2 tablespoons finely chopped shallots
tablespoons dry white wine
tablespoons lemon juice
1/4 teaspoon ground black pepper
1/4 cup breadcrumbs
1/4 cup grated Parmesan cheese
Salt, to taste
tablespoon olive
Equipment:
Snail shells
Foil
heavy-bottemed skillet
A wooden spoon
A baking dish
A fine-mesh strainer
small saucepan
A mixing boul Parchment paper
Instructions:
Preheat oven 375F (190C).
2. Scrub the snails under cool rumning water to remove any dirt or debris. Discard any snails that are not alive. Soak the snails in salted water for at least 30 minutes, changing the water every 10 minutes, small saucepan, melt the butter over low heat. Add the minced garlic, parsley, and shallots, and cook for 1-2 minutes until fragrant.
they thoroughly cleaned.
4.
Remove the garlic mixture from the heat and stir in the white wine, lemon juice, and black pepper. Season with salt taste. cheese.
5. mixing bowl, combine the breadcrumbs and grated Parmesan
Place the smail shells baking dish. Spoon the garlic butter mixture into the shells, filling each one about full.
6.
sheet of foil and arrange them in
7. Sprinkle the breadcrumb mixture over the top of each snail.
8. Cover the snail shells with foil.
for 12-15 minutes until the 9. Bake in the preheated and the anails have cooked through. breadcrumbs
golden brown dish from the oven, carefully remove the foil, and place a small piece of parchment paper over the top of each shell.
10. Remove
11. Return the dish to the oven and continue baking for 1-2 minutes
crisp up the breadcrumbs.
12. Remove the dish from the oven, and let the escargots cool for few minutes.
13. In a heavy-bettomed skillet, heat the olive oil over medium-high heat. Strain the garlic butter mixture through fine-mesh strainer into the skillet, and cook for 1-2 zinutes until slightly reduced and thickened.
14. To serve, spoon the garlic butter ento small plates, arrange the escargots on the plates, and drizzle the remaining garlic butter over the top.
Note: Escargot shells are available for purchase specialty food stores or online. mushroom capa or even amall ramekina.
you don't have escargot shells, you can use
Enjoy your delicious escargot!
19

---

Published as a conference paper at ICLR 2025
SimPO Here's general recipe for making escargot:
Ingredients:
pound fresh or canned snails (escargots), cleaned and cooked
1/2 cup unsalted butter, softened
2 cloves garlic, minced
tablespoon finely chopped fresh parsley
tablespoon finely chopped fresh thyme
1/4 teaspoon red pepper flakes (optional)
Salt and freshly ground black pepper, to taste 24-30 small, hollowed-out garlic cloves escargot shells
Instructions:
1. Preheat your oven to 450F (230C).
2. Prepare the garlic cloves escargot shells by gently scraping the pulp with kmife, leaving thin layer intact. Rinse them under running water and pat dry.
3. In mixing boul, combine softened butter, minced garlic, chopped parsley, thyme, red pepper flakes (if using), salt, and black pepper. Mix well until blended.
Spoon a generous amount of seasoned butter into each garlic clove or escargot shell, filling them almost to the top.
5. Arrange the filled shells in a baking dish, preferably one with individual wells indentations, spaced evenly apart.
Place the cooked snails into each shell, making snugly.
they fit
7. Bake the escargot for 8-10 minutes, or until the butter has melted and bubbled around the edges, and the smails are heated through.
dish from oven and serve inmediately, accompanied by crusty bread, toasted baguette slices, or crackers.
Remove
9. To eat, use a amall fork or escargot spoon to extract the amails from their shella, dip them into the remaining garlic butter, and enjoy!
Note: Always follow proper food handling preparation guidelines when working with live or cooked smails, especially if using fresh ones. Ensure they are properly cleaned, cooked, and stored according recommended safety practices.
Table 8: An AlpacaEvall generation from the Mistral-7B-Bast model trained with SimPER and SimPO. Alpaca-Eval--oe -xee--vv-----Trrriii prefers SimPER over SimPO. Compared to SimPO, the generation by SimPER better structured, with hierarchical discussions that make it more helpful.
Prompt Mick pays his teacher $800 for 40 lessons worth 2 hours each. If this will all is going to pay for his lessons, how much he receive?
SimPER
Mick did not receive any money. the total coat for the 40 lessons, which is 40 x 2 hours 80 total hours of lessons. Since he only paid for the lessons, he will not receive any money back.
$800 paid his teacher
Confidence: 95%
SimPO Mick received $800.
Confidence: 95%
Table 9: An AlpacaEval 2 generation from the Llama-3-88 3-8B-Bass model trained with SimPER and SimPO. Alpaca-Evall EPal G0T4-GPTTuiit Turbo prefers SimPER over SimPD. Compared SimPO, SimPER's s output provides clearer step-by-step reasoning.
20


Reviews:
[{"review_id": "FHLGmHaM9W", "reviewer": "Reviewer_bH9h", "summary": "This paper proposes an effective, hyperparameter-free objective for preference optimization, named SimPER. Comprehensive experiments are conducted on popular benchmarks to verify the effectiveness of SimPER.", "questions": "Response length on AlpacaEval 2 is important, so the author(s) may consider adding the average response length to the results. Additionally, as an updated version of MT-Bench, Arena-Hard provides more reliable evaluations compared to MT-Bench and is widely used in previous works of preference optimization. The author(s) might consider including Arena-Hard to better demonstrate performance.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The proposed SimPER training objective is parameter-free, eliminating the need for hyperparameter tuning required in prior work.\n    \n- The paper provides detailed theoretical analysis and experimental results to demonstrate the superiority of SimPER.", "weaknesses": "I did not observe any significant flaws. However, there is still room for improvement in the details of experimental results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YuLlBTyXws", "reviewer": "Reviewer_7YzN", "summary": "Existing preference optimization techniques often require complex hyperparameter tuning, which complicates the fine-tuning process. In this work, the authors propose a new method for aligning large language models (LLMs) with human preferences. To address this, they propose SimPER, a hyperparameter-free approach that optimizes inverse perplexity—defined as the inverse of the exponentiated average log-likelihood of chosen and rejected responses. They then provide some insights on the reason that the method can work better than other widely adopted methods that SimPER does not push the model away from the chosen responses. They theoretically show that minimizing the proposed loss is approximately minimizing TVD, which is more immune to mode dropping. They validate the method by many commonly adopted benchmarks, and show the method have superior performance without additional hyperparameters.", "questions": "See weaknesses.\n\nIn addition, as suggested by Fig. 2, TVD would encourage the appearance of non-existing mode, i.e., it will also encourage the probability of regions with zero/low probabilities in the support of the chosen responses. Do you observe such a behavior in practice? For example, if you substitute the generated samples into a larger (ideally GPT-like) model, do you observe there will be more samples with relatively low perplexity compared to a model trained with SimPO and comparable loss?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Hyperparameter tuning has long been a headache in posttraining. SimPER consistently outperforms its alternatives across multiple tasks and datasets without meticulous adjustment of various parameters. This not only is efficient in training, but also enhances the robustness of the model.\n\nIn addition to the empirical performance, the authors also provide insights on why such a loss can perform better than its alternatives. \n\nThe proposed loss is fairly easy to implement.  \n\nThe proposed method has less length bias.", "weaknesses": "It would be better if the experiments can show the proposed method works well across different scale of models. The majority of the experiments adopts 7-8B models. It would be ideal if we can show usefulness with, e.g., ~3B, ~7B, ~70B models.\n\nIn addition, I am confused on the choice of the model. It seems only Fig.4 uses Pythia-2.8B, but all other experiments are with Mistral-7B and/or Llama3-8B? Can you explain why this is the case?\n\nThe proposed loss is fairly simple and I do believe simplicity is good. But the presentation of Eq. (5)(6)(7)(8) is redundant. Assuming the notations are well defined (e.g., defined $\\log \\pi(y|x) = \\sum_i \\log \\pi(y_i|x)$), (8) should be enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fObMejvEN2", "reviewer": "Reviewer_bGmT", "summary": "The authors introduced SimPER, a preference optimization technique based on inverse perplexity that eliminates the need for a reference model and hyperparameters. This makes it straightforward to implement and computationally efficient. They demonstrated that SimPER effectively minimizes the Total Variation Distance, which promotes the generation of a subset of high-reward responses (mode-covering behavior) rather than distributing probability equally across all responses. The authors showed that SimPER compares favorably against DPO, IPO, KTO, and SimPO when tested on both pre-trained and instruction-tuned Mistral-7B and LLaMA3-8B models.", "questions": "- Can you include ORPO [1] in the experimental setup?\n- How sensitive are other preference alignment methods to hyperparameters? Are instruction-tuned models consistently less sensitive to the hyperparameters of the preference alignment methods? (Figure 1)\n- What is the training dynamic of DPO mentioned in line 253? (Figure 3)\n- How was the reference model incorporated in SimPER in the ablation study? (Table 4)\n- Line 72, \"post-training process for alignment is usually very expensive\", I suggest mentioning the LoRA line of work, which made it significantly less expensive [2, 3].\n- Line 201, \"data. (Jelinek et al., 1977; Marion et al., 2023; Gonen et al., 2023).\", there is a period before the citation.\n- Figure 6, I suggest decreasing the opacity as it is difficult to read.\n\n[1] Hong, J., Lee, N., & Thorne, J. (2024). Reference-free monolithic preference optimization with odds ratio. arXiv preprint arXiv:2403.07691.\n[2] Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., ... & Chen, W. (2021). Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.\n[3] Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2024). Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- Preference alignment is a timely and impactful area of research.\n- The proposed method, SimPER, is well-motivated. It eliminates the need for a reference model and hyperparameters, making it simple to implement and cost-effective, which are significant advantages.\n- The experimental setup is sound and modern, incorporating popular and recent LLMs and benchmarks.\n- The paper is well-written and easy to follow.", "weaknesses": "- The authors did not compare SimPER with ORPO [1], another reference model-free preference alignment method.\n- Some necessary details and ablations are missing (see Questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kUciOo7nYI", "reviewer": "Reviewer_dfTo", "summary": "This paper proposes a simple offline preference optimization objective that eliminates the need for a reference model and any tunable hyperparameters required by previous methods, enhancing LLM alignment performance.", "questions": "See weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The issue of reduced log probability for the chosen response has recently gained considerable attention in the field of LLM alignment. Addressing this issue is a valuable contribution.\n2. SimPER reduces the decrease in the likelihood of chosen responses, providing an improvement over SimPO.\n3. The experiments are thorough and well-executed, with comprehensive evaluations on both the Open LLM Leaderboard and various chat benchmarks, yielding promising results.", "weaknesses": "1. It appears that Equation (7) might lack an expectation over $x$. If an expectation with respect to $x$ is included, then based on the property $E(A + B) = E(A) + E(B)$, would SimPER’s application extend beyond the paired preference data scenario described in the paper? This could incorporate comparisons with unpaired alignment algorithms.\n\n2. While Figure 3 shows that SimPER mitigates the reduction in chosen response log probabilities compared to SimPO, the likelihood of the chosen response is still decreasing. The ORPO paper addresses this issue as well, and Figure 7 in ORPO shows that the likelihood of the chosen response can actually increase.\n\n3. On Line 295, \"model covering\" should likely be \"model seeking”?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GiEh0xjUeH", "reviewer": "Reviewer_MQ2x", "summary": "The paper presents a novel loss SimPER for preference learning, which minimizes (resp. maximizes) the perplexity on chosen (resp. rejected) completions and does not introduce any hyper-parameter.\nSimPER is applied to align the base and instruct version of Mistral-7B and Llama-3-8B on UltraFeedback and Pythia-2.8B on Anthropic Helpful and Harmless dataset.\nThe aligned models are evaluated on Open LLM Leaderboard and instruction-following benchmarks AlpacaEval 2 and MT-Bench, where SimPER leads to models with both superior basic capabilities and alignment.", "questions": "- Given that the loss can be decomposed into the terms corresponding to chosen and rejected completions (eq. 7), does SimPER not require the chosen and rejected completions appear in pairs like KTO, making the algorithm more general?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper addresses an important and notorious problem, i.e., sensitivity of alignment algorithms to hyper-parameters.\n* In the empirical evaluation, the proposed method achieves significantly higher basic capabilities and alignment.", "weaknesses": "* If the exponential operation in eq.(8) is removed, the loss degrades into standard fine-tuning that learns the chosen completion and unlearns the rejected completion.\nIt will be beneficial to compare with the standard fine-tuning and elaborate why adding the exponential operation is better.\n* I disagree with the analysis on line 249. The authors neglect that the term $w_\\theta$ (resp. $d_\\theta)$ also depends on $\\pi_\\theta(y_l|x)$ (resp. $p_\\theta(y_l|x))$. For example, in DPO, the weight of $\\nabla_\\theta\\pi_\\theta(y_l|x)$ is\n$$w=\\frac{\\beta\\sigma(\\beta\\log\\pi_\\theta(y_l|x)+C)}{\\pi_\\theta(y_l|x)}$$\nand we have $w\\to 0$ when $\\pi_\\theta(y_l|x)\\to 0$.\n* My primary concern is whether the proposed method can be scaled to iterative preference learning [1, 2], which is currently employed by the state-of-the-art open-weight models [3, 4].\nFor instance, implementing iterative DPO on Llama-3-8B-Base can achieve advanced performance on academic benchmarks while scoring 31.3 on AlpacaEval 2 and 8.46 on MT-Bench [5], significantly outperforming offline setting.\nConducting such evaluation should be straight-forward given the only difference is the loss term?\n* I noticed that the Table 2 is largely copied from [6], which is fine but contradicts with the statement “We thoroughly tuned the hyper parameters for each baseline” on line 371? By the way the MT-Bench score for Llama-3-8B-Base seems was mistakenly duplicated as the result where GPT-4-Turbo serves as the judge.\n* Could you please provide the scores for SFT model in Tab. 3 to illustrate the alignment tax?\n* Minor: typo in eq. (4) and expression for $d_\\theta$ on line 244.\n\n[1] Self-Play Preference Optimization for Language Model Alignment, arXiv.  \n[2] Self-Exploring Language Models, arXiv.  \n[3] The Llama 3 Herd of Models, arXiv.  \n[4] Qwen2 Technical Report, arXiv.  \n[5] RLHF Workflow: From Reward Modeling to Online RLHF, arXiv.  \n[6] SimPO: Simple Preference Optimization with a Reference-Free Reward, NeurIPS'24.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)