SimPER: A Minimalist Approach to Preference  Alignment without Hyperparameters

Authors: Teng Xiao, Yige Yuan, Zhengyu Chen, Mingxiao Li, Shangsong Liang, Zhaochun Ren, Vasant G Honavar
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Large Language Model, Alignment, RLHF

Abstract:
Existing preference optimization objectives for language model alignment require additional hyperparameters that must be extensively tuned to achieve optimal performance, increasing both the complexity and time required for fine-tuning large language models. In this paper, we propose a simple yet effective hyperparameter-free preference optimization algorithm for alignment. We observe that promising performance can be achieved simply by optimizing inverse perplexity, which is calculated as the inverse of the exponentiated average log-likelihood of the chosen and rejected responses in the preference dataset. The resulting simple learning objective, SimPER, is easy to implement and eliminates the need for expensive hyperparameter tuning and a reference model, making it both computationally and memory efficient. Extensive experiments on widely used real-world benchmarks, including MT-Bench, AlpacaEval 2, and 10 key benchmarks of the Open LLM Leaderboard with 5 base models, demonstrate that SimPER consistently and significantly outperforms existing approaches—even without any hyperparameters or a reference model. For example, despite its simplicity, SimPER outperforms state-of-the-art methods by up to 5.7 points on AlpacaEval 2 and achieves the highest average ranking across 10 benchmarks on the Open LLM Leaderboard. The source code for SimPER is publicly available at: https://github.com/tengxiao1/SimPER.

Decision: accept (poster)
Reviewer scores: 8, 6, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730193649986

thread_type:
official_review

<thread_object>
creator: reviewer_mq2x

<content>
summary:
The paper presents a novel loss SimPER for preference learning, which minimizes (resp. maximizes) the perplexity on chosen (resp. rejected) completions and does not introduce any hyper-parameter.
SimPER is applied to align the base and instruct version of Mistral-7B and Llama-3-8B on UltraFeedback and Pythia-2.8B on Anthropic Helpful and Harmless dataset.
The aligned models are evaluated on Open LLM Leaderboard and instruction-following benchmarks AlpacaEval 2 and MT-Bench, where SimPER leads to models with both superior basic capabilities and alignment.

strengths:
* The paper addresses an important and notorious problem, i.e., sensitivity of alignment algorithms to hyper-parameters.
* In the empirical evaluation, the proposed method achieves significantly higher basic capabilities and alignment.

weaknesses:
* If the exponential operation in eq.(8) is removed, the loss degrades into standard fine-tuning that learns the chosen completion and unlearns the rejected completion.
It will be beneficial to compare with the standard fine-tuning and elaborate why adding the exponential operation is better.
* I disagree with the analysis on line 249. The authors neglect that the term $w_\theta$ (resp. $d_\theta)$ also depends on $\pi_\theta(y_l|x)$ (resp. $p_\theta(y_l|x))$. For example, in DPO, the weight of $\nabla_\theta\pi_\theta(y_l|x)$ is
$$w=\frac{\beta\sigma(\beta\log\pi_\theta(y_l|x)+C)}{\pi_\theta(y_l|x)}$$
and we have $w\to 0$ when $\pi_\theta(y_l|x)\to 0$.
* My primary concern is whether the proposed method can be scaled to iterative preference learning [1, 2], which is currently employed by the state-of-the-art open-weight models [3, 4].
For instance, implementing iterative DPO on Llama-3-8B-Base can achieve advanced performance on academic benchmarks while scoring 31.3 on AlpacaEval 2 and 8.46 on MT-Bench [5], significantly outperforming offline setting.
Conducting such evaluation should be straight-forward given the only difference is the loss term?
* I noticed that the Table 2 is largely copied from [6], which is fine but contradicts with the statement “We thoroughly tuned the hyper parameters for each baseline” on line 371? By the way the MT-Bench score for Llama-3-8B-Base seems was mistakenly duplicated as the result where GPT-4-Turbo serves as the judge.
* Could you please provide the scores for SFT model in Tab. 3 to illustrate the alignment tax?
* Minor: typo in eq. (4) and expression for $d_\theta$ on line 244.

[1] Self-Play Preference Optimization for Language Model Alignment, arXiv.  
[2] Self-Exploring Language Models, arXiv.  
[3] The Llama 3 Herd of Models, arXiv.  
[4] Qwen2 Technical Report, arXiv.  
[5] RLHF Workflow: From Reward Modeling to Online RLHF, arXiv.  
[6] SimPO: Simple Preference Optimization with a Reference-Free Reward, NeurIPS'24.

questions:
- Given that the loss can be decomposed into the terms corresponding to chosen and rejected completions (eq. 7), does SimPER not require the chosen and rejected completions appear in pairs like KTO, making the algorithm more general?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730250145845

thread_type:
official_review

<thread_object>
creator: reviewer_dfto

<content>
summary:
This paper proposes a simple offline preference optimization objective that eliminates the need for a reference model and any tunable hyperparameters required by previous methods, enhancing LLM alignment performance.

strengths:
1. The issue of reduced log probability for the chosen response has recently gained considerable attention in the field of LLM alignment. Addressing this issue is a valuable contribution.
2. SimPER reduces the decrease in the likelihood of chosen responses, providing an improvement over SimPO.
3. The experiments are thorough and well-executed, with comprehensive evaluations on both the Open LLM Leaderboard and various chat benchmarks, yielding promising results.

weaknesses:
1. It appears that Equation (7) might lack an expectation over $x$. If an expectation with respect to $x$ is included, then based on the property $E(A + B) = E(A) + E(B)$, would SimPER’s application extend beyond the paired preference data scenario described in the paper? This could incorporate comparisons with unpaired alignment algorithms.

2. While Figure 3 shows that SimPER mitigates the reduction in chosen response log probabilities compared to SimPO, the likelihood of the chosen response is still decreasing. The ORPO paper addresses this issue as well, and Figure 7 in ORPO shows that the likelihood of the chosen response can actually increase.

3. On Line 295, "model covering" should likely be "model seeking”?

questions:
See weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730399644028

thread_type:
official_review

<thread_object>
creator: reviewer_bgmt

<content>
summary:
The authors introduced SimPER, a preference optimization technique based on inverse perplexity that eliminates the need for a reference model and hyperparameters. This makes it straightforward to implement and computationally efficient. They demonstrated that SimPER effectively minimizes the Total Variation Distance, which promotes the generation of a subset of high-reward responses (mode-covering behavior) rather than distributing probability equally across all responses. The authors showed that SimPER compares favorably against DPO, IPO, KTO, and SimPO when tested on both pre-trained and instruction-tuned Mistral-7B and LLaMA3-8B models.

strengths:
- Preference alignment is a timely and impactful area of research.
- The proposed method, SimPER, is well-motivated. It eliminates the need for a reference model and hyperparameters, making it simple to implement and cost-effective, which are significant advantages.
- The experimental setup is sound and modern, incorporating popular and recent LLMs and benchmarks.
- The paper is well-written and easy to follow.

weaknesses:
- The authors did not compare SimPER with ORPO [1], another reference model-free preference alignment method.
- Some necessary details and ablations are missing (see Questions).

questions:
- Can you include ORPO [1] in the experimental setup?
- How sensitive are other preference alignment methods to hyperparameters? Are instruction-tuned models consistently less sensitive to the hyperparameters of the preference alignment methods? (Figure 1)
- What is the training dynamic of DPO mentioned in line 253? (Figure 3)
- How was the reference model incorporated in SimPER in the ablation study? (Table 4)
- Line 72, "post-training process for alignment is usually very expensive", I suggest mentioning the LoRA line of work, which made it significantly less expensive [2, 3].
- Line 201, "data. (Jelinek et al., 1977; Marion et al., 2023; Gonen et al., 2023).", there is a period before the citation.
- Figure 6, I suggest decreasing the opacity as it is difficult to read.

[1] Hong, J., Lee, N., & Thorne, J. (2024). Reference-free monolithic preference optimization with odds ratio. arXiv preprint arXiv:2403.07691.
[2] Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., ... & Chen, W. (2021). Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
[3] Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2024). Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730589945696

thread_type:
official_review

<thread_object>
creator: reviewer_7yzn

<content>
summary:
Existing preference optimization techniques often require complex hyperparameter tuning, which complicates the fine-tuning process. In this work, the authors propose a new method for aligning large language models (LLMs) with human preferences. To address this, they propose SimPER, a hyperparameter-free approach that optimizes inverse perplexity—defined as the inverse of the exponentiated average log-likelihood of chosen and rejected responses. They then provide some insights on the reason that the method can work better than other widely adopted methods that SimPER does not push the model away from the chosen responses. They theoretically show that minimizing the proposed loss is approximately minimizing TVD, which is more immune to mode dropping. They validate the method by many commonly adopted benchmarks, and show the method have superior performance without additional hyperparameters.

strengths:
Hyperparameter tuning has long been a headache in posttraining. SimPER consistently outperforms its alternatives across multiple tasks and datasets without meticulous adjustment of various parameters. This not only is efficient in training, but also enhances the robustness of the model.

In addition to the empirical performance, the authors also provide insights on why such a loss can perform better than its alternatives. 

The proposed loss is fairly easy to implement.  

The proposed method has less length bias.

weaknesses:
It would be better if the experiments can show the proposed method works well across different scale of models. The majority of the experiments adopts 7-8B models. It would be ideal if we can show usefulness with, e.g., ~3B, ~7B, ~70B models.

In addition, I am confused on the choice of the model. It seems only Fig.4 uses Pythia-2.8B, but all other experiments are with Mistral-7B and/or Llama3-8B? Can you explain why this is the case?

The proposed loss is fairly simple and I do believe simplicity is good. But the presentation of Eq. (5)(6)(7)(8) is redundant. Assuming the notations are well defined (e.g., defined $\log \pi(y|x) = \sum_i \log \pi(y_i|x)$), (8) should be enough.

questions:
See weaknesses.

In addition, as suggested by Fig. 2, TVD would encourage the appearance of non-existing mode, i.e., it will also encourage the probability of regions with zero/low probabilities in the support of the chosen responses. Do you observe such a behavior in practice? For example, if you substitute the generated samples into a larger (ideally GPT-like) model, do you observe there will be more samples with relatively low perplexity compared to a model trained with SimPO and comparable loss?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730624619025

thread_type:
official_review

<thread_object>
creator: reviewer_bh9h

<content>
summary:
This paper proposes an effective, hyperparameter-free objective for preference optimization, named SimPER. Comprehensive experiments are conducted on popular benchmarks to verify the effectiveness of SimPER.

strengths:
- The proposed SimPER training objective is parameter-free, eliminating the need for hyperparameter tuning required in prior work.
    
- The paper provides detailed theoretical analysis and experimental results to demonstrate the superiority of SimPER.

weaknesses:
I did not observe any significant flaws. However, there is still room for improvement in the details of experimental results.

questions:
Response length on AlpacaEval 2 is important, so the author(s) may consider adding the average response length to the results. Additionally, as an updated version of MT-Bench, Arena-Hard provides more reliable evaluations compared to MT-Bench and is widely used in previous works of preference optimization. The author(s) might consider including Arena-Hard to better demonstrate performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732069898343

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response to Reviewers

comment:
We sincerely thank all the reviewers for their insightful comments and helpful suggestions. We are encouraged to see that all reviewers agree our paper is "simple and effective," "well-motivated," and demonstrates "solid analysis and good empirical results."

In this work, we address the limitations of current preference alignment methods, particularly their sensitivity to hyperparameters, which poses a critical and practical  challenge in alignment. To tackle this, we propose a surprisingly effective hyperparameter-free objective for alignment through perplexity optimization, a contribution that reviewers found particularly compelling.

We have made every effort to address all comments, suggestions, and misunderstandings in our responses and have prepared a point-by-point reply for each reviewer. Below, we summarize the major responses while addressing the comments of each reviewer individually:

- We provided additional results, including comparisons with ORPO.
- We clarified several details to address some misunderstandings from the reviewers.
- We also further evaluated SimPER in the online iterative scenario suggested by Reviewer MQ2x.

We have updated the submission to include these additional results and clarifications (marked in red text).

We extend our sincere thanks to all the reviewers once again for their time and efforts.

Best regards,

Authors of “SimPER: Simple Preference Fine-Tuning without Hyperparameters by Perplexity Optimization”
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces SimPER as a state-of-the-art method for preference optimization in LLM alignment.  

- Pros: Its key contribution is a hyperparameter-free objective based on inverse perplexity, which the authors demonstrate exhibits mode-seeking behavior that effectively aligns with human preferences. Extensive experiments highlight SimPER's superior performance compared to common baselines across multiple benchmarks.  

- Cons: The experiments focus primarily on 7–8B models, with limited exploration of larger scales. Additionally, while SimPER achieves strong alignment, it comes at the cost of reduced response diversity.  

SimPER's simplicity, robustness, and strong empirical results make it a notable advancement in preference optimization for LLMs.