AlphaPO: Reward Shape Matters for LLM Alignment
Authors: Aman Gupta
Shao Tang
Qingquan Song
Sirou Zhu
Jiwoo Hong
Ankan Saha
Viral Gupta
Noah Lee
Eunki Kim
Siyu Zhu
Parag Agrawal
Natesh S. Pillai
Sathiya Keerthi
Venue: ICML 2025 poster (2025)
Keywords: llm
large language models
deep learning
alignment
preference tuning
post training
reward shaping
Primary area: deep_learning->large_language_models
TL;DR: Use a parameter alpha to control the shape of the reward function for alignment training, in order to improve performance

Abstract: Reinforcement Learning with Human Feedback (RLHF) and its variants have made huge strides toward the effective alignment of large language models (LLMs) to follow instructions and reflect human values. More recently, Direct Alignment Algorithms (DAAs) have emerged in which the reward modeling stage of RLHF is skipped by characterizing the reward directly as a function of the policy being learned. Some popular examples of DAAs include Direct Preference Optimization (DPO) and Simple Preference Optimization (SimPO). These methods often suffer from likelihood displacement, a phenomenon by which the probabilities of preferred responses are often reduced undesirably. In this paper, we argue that, for DAAs the reward (function) shape matters. We introduce \textbf{AlphaPO}, a new DAA method that leverages an $\alpha$-parameter to help change the shape of the reward function beyond the standard log reward. AlphaPO helps maintain fine-grained control over likelihood displacement and over-optimization. Compared to SimPO, one of the best performing DAAs, AlphaPO leads to about 7\% to 10\% relative improvement in alignment performance for the instruct versions of Mistral-7B and Llama3-8B while achieving 15\% to 50\% relative improvement over DPO on the same models. The analysis and results presented highlight the importance of the reward shape and how one can systematically change it to affect training dynamics, as well as improve alignment performance.

AlphaPO: Reward Shape Matters for LLM Alignment
Aman Gupta" 1
Shao Tang"
Qingquan Song'
Sirou Zhu
Jiwoo Hong",
Ankan Saha'
Viral Gupta'
Noah Lee? Eunki Kim?
Siyu Zhu'
Parag Agrawal
Natesh Pillai
S. Sathiya Keerthi
Abstract
1. Introduction
Large language models (LLMs) (Vaswani, 2017; Brown et al., 2020; Dubey et al., 2024) have ushered in new for artificial intelligence. LLM training can be broadly split into two stages pre- and post-training. An important step in the post-training stage is "alignment" which involves improving the model's ability to follow instructions, human values and style. This step crucial helping bridge the gap between the raw ability of pre-trained models and the immense utility of post-trained models.
Reinforcement I Learning with Human Feedback (RLHF) and its variants have made huge strides toward the effective alignment large language models (LLMs) to follow instructions and reflect human values. More recently, Direct Alignment Algorithms (DAAs) have emerged in which the reward modeling stage of RLHF is skipped by characterizing the reward directly as a function of the policy being learned. Some popular ex- amples of DAAs include Direct Preference Op- timization (DPO) and Simple Preference Opti- mization (SimPO). These methods often suffer from likelihood displacement, phenomenon by which the probabilities of preferred responses are often reduced undesirably. In this paper, we argue that, for DAAs the reward (function) shape mat- ters. We introduce AlphaPO, new DAA method that leverages an a-parameter to help change the shape of the reward function beyond the standard log reward. AlphaPO helps maintain fine- tineggraiedd control over likelihood displacement and over- optimization. Compared to SimPO, one of the best performing DAAS, AlphaPO leads to about 7% to 10% relative improvement in alignment per- formance for the instruct versions of Mistral-7B and Llama3-8B while achieving 15% to 50% rela- tive improvement over DPO the same models. The analysis and results presented highlight the importance of the reward shape and how one can systematically change it to affect training dynam- ics, as well as improve alignment performance.
To this end, researchers have developed several techniques under the umbrella of Reinforcement Learning with Human Feedback (RLHF) (Ouyang et al., 2022; Ziegler et al., 2020), RLHF involves a three stage process - supervised fine-tuning (SFT), reward modeling and reinforcement learning (RI.)- based fine-tuning to learn the optimal policy. Learning is typically performed on preference data, which include pre- ferred and dispreferred responses the same imput prompt. While achieving impressive results, RLHF involves several stages and can be cumbersome to train (Casper et al., 2023).
To simplify the aligmment process, methods such as Di- rect Preference Optimization (DPO) (Rafailov al., 2023) and Simple Preference Optimization (SimPO) (Meng et al., 2024) have been developed. DPO optimizes directly for the optimal policy, bypassing the reward modeling step. The optimal policy is identified as the solution to an expected reward maximization problem while ensuring that the policy does not diverge too much from a reference policy. SimPO achieves the same goal with a different reward function nor- malized by the length of the generation, and also introduces margin term to better separate preferred and dispreferred responses. Both belong to a class of methods known Di- rect Alignment Algorithms (DAAs) (Rafailov et al., 2024a).
DAAS include DPO and its variants (Rafailov et al., 2023; Park et al., 2024a), SimPO (Meng et al., 2024) and methods like CPO (Xu et al., 2024) and ORPO (Hong et al., 2024) among others.
'Equal contribution 'LinkedIn Corporation, CA. USA AKAIST AI, KAIST, South Korea !ork done while Linkedln Corporation. Correspondence to: Aman Gupta
com>, Shao Tang
Several studies have shown that DAAs tend to widen the margin between preferred and dispreferred responses while simultaneously reducing the probabilities of preferred re- sponses. Although slightly lower completion likelihood is known to improve output diversity and generalization, ex-
com>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

AlphaPO: Reward Shape Matters for LLM Alignment
Generalized Reward Shaping with AlphaPO
[y)x(y|x) w/b
Ta Py;;;; x) B
Pin y; x) = S (x M
PSimPO xy:x)
log ()(y/x)
Pinv-lin (y;x) = B 1 (-maayyxx
1//al +
a
a= - 0.5
a 0
ar = 0.5
a =
1
-3
-3
-3
0.0
02
0.4
0/6
0.8
0.0
0.2
1.0
3.0
0.4
05
0.5
3.0
0.0
0.2
0.4
0.6
0.0
0.2
0.
3.0
(y/(x)
Figure 1. The generalized reward shaping paradigm of AIphaPO. By adjusting the parameter in the reward function ra (y;x), (0yxx)AAppp AlphaPO induces various reward shapes, leading different preference optimization dynamics and downstream performances.
cessive focus on increasing the reward margin can degrade actual performance, as measured by human evaluations. This is known as reward over-optimization (Shi et al., 2024). Similarly, the reduction in preferred response probabilities, known likelihood displacement (Razin et al., 2024), has been linked to unintended misalignment, further harming performance. Extended training exacerbates both issues, leading to deterioration in the overall effectiveness of pref- erence optimization.
proved. We note that although AlphaPO's a reward shapes are inspired by f-DPO et al., 2024a), they are dis- tinctly different since we use length normalization in the reward and f-DPO does not. Moreover, we demonstrate that non-zero values of a improve generalization performance while f-DPO fails to demonstrate that for any value of except 0 (which is standard DPO).
Through gradient analysis and experiments, we show how the shape the newly introduced reward function affects the aggressive or conservative nature of likelihood displace- ment. For varying values of a, then demonstrate how AlphaPO's performance evolves evaluation benchmarks such as AlpacaEval 2.0 and ArenaHard (Li et al., 2024). The critical factor is the combination of the a- t-parameterized scoring function with length normalization. We demon- strate that AlphaPO comprehensively outperforms SimPO and DPO across models such as Mistral- Instruct (Jiang et al., 2023a) and Llama3-8B Instruct (Dubey et al., 2024), while remaining competitive for models like Gemma2-9B Instruct (Team et al., 2024). In particular, AlphaPO achieves relative improvement of 7% to 10% over SimPO (and 15% to 50% over DPO) on AlpacaEval 2 for Llama3-8B and Mistral-7B. Additionally, integrating AlphaPO with the SPPO method (Wu al., 2024) yields strong improve- ments on AlpacaEval 2.0 with length controlled win rate of 47. 17.42% for PairRM- based regeneration (Jiang et al., 2023b) of the UltraFeedback dataset (Cui et al., 2024), bettering the results of SimPO with SPPO applied on top.
Although the aforementioned issues may be concerning, in practice, there are typically model checkpoints along the training trajectory that demonstrate strong alignment perfor- mance. These checkpoints can be identified through early stopping. SimPO, known for its strong alignment capabil- ities, uses just one epoch of training (Meng et al., 2024). By varying three hyperparameters--eeteee scaling), and y (reward margin shift) SimPO generates three-dimensional manifold of trajectories. These hyperpa- rameters fine-tuned enhance alignment performance on benchmarks such as AlpacaEval 2.0 (Dubois et al., 2024), optimizing the win rate to select the final model of choice.
rate, 3 (reward
In this paper, we introduce a new method AlphaPO based on a fourth important dimension the reward function shape. Most DAAS, including SimPO, are based on the log reward function. We point out that changing this reward shape yields new types of trajectories with distinctly different mar- gin and preferred-likellhood behaviors. We take inspiration from Wang et al. (2024a), use the reward function arising from the a-divergence idea and apply length- normalization to to generate an interesting class reward shapes that are parameterized by a as depicted in Figure (a yields the log reward shape for SimPO, making it a special case. More value different
2. Preliminaries
Let us consider the setup of alignment training. x [21, ..... denotes single input prompt with se- quence of tokens. The objective of an LLM is to generate relevant and consistent response y [V1, V2, Us. ...] to x.
details provided later.). By choosing from zero, alignment performance can be significantly im-
2

---

AlphaPO: Reward Shape Matters for LLM Alignment
LLM learns a policy =e parameterized by 0 where To (x) denotes the probability assigned to y.
_ log ne (y(x)))) where yy| is the length of y. SimPO also introduces an additional margin term in the
"simpo x) |y]
BT objective: p(y > y:) = (r((w:: - x()xx) - 7).
RLHF aims to fine-tune the policy xo to maximize the re- ward r(-) without excessively diverging from the initial
This results in the following loss for SimPO:
policy Tee (Christiano et al., 2017; Ouyang et al., 2022).
In the offline alignment learning scenario, the data consist of the prompt x and a pair of (preferred, dispreferred) re- sponses Wo Yt)" of sequences, with preference denoted by Y Yt (Rafailov et al., 2023). This triplet is often denoted as Yw, RL-F-basedd methods consist of using supervised fine-tuning (SFT) on a dataset, reward modeling and then reinforcement learning to finally learn an aligned model. Methods such as Direct Preference Op- timization (DPO) (Rafailov et al.. 2023). SimPO (Meng et al., 2024), CPO (Xu et al., 2024), and ORPO (Hong et al., 2024) use the Bradley-Terry (BT) (Bradley & Terry, 1952) setup: p(y > vr) (r((::: x) ()), where (x) = 1/(1+ + exp(-x)) is the sigmoid function. These method directly specify the reward function as a function of the optimal policy, as described in the subsequent sec- tion. These methods are also known as Direct Alignment Algorithms (DAAs) (Rafailov et al., 2024a). DAAS have achieved immense popularity recently for their simplic- ity and effectiveness at achieving strong alignment perfor- mance (Lambert et al., 2024).
B
LsipO = - -Exx.y
log a
log To (y
[y
log To (y:(x) -7) )
B yi
(2)
While the parameter Y influences the size of margin values, the parameter 3 determines how close (3z)) is to the hinge loss, i.e., max 2, 0}. SimPO demonstrates its effectiveness as robust and efficient alignment method for LLMs due to several key innovations. Unlike DPO, SimPO eliminates the reliance on reference model dur- ing training, reducing memory and computational overhead while ensuring reward maximization aligns with inference. Furthermore, SimPO establishes length normalization, first introduced by ORPO, as a critical factor for alignment performance. This technique enables SimPO to outper- form DPO and other methods on benchmarks like AlpacaE- val2.0 (Dubois et al., 2024), achieving superior results with- out increasing response length. Notably, SimPO's length normalization has inspired enhancements in other methods, such as Tulu (Lambert et al., 2024), underscoring its impact.
2.1. Direct Preference Optimization (DPO)
DPO (Rafailov et al., 2023) is a recent popular method that solves the problem of maximizing the expected reward with an added KL penalty between the sequence probabili- ties of the model being trained and a reference model Fee (trained using SFT). DPO uses closed-fon foreed-fomm expression for the reward r by leveraging the optimality conditions of the problem: xy;
By fixing training epochs to one and carefully tuning / and 7, SimPO achieves strong alignment performance. A key aspect of its robustness lies in its constant penalization Y, which addresses limitations in DPO. While DPO parame- terizes the optimal human preference distribution under the BT model with a maximum likelihood objective, it relies on divergence penalties from reference policy Tiets which can be suboptimal when preference samples are sampled from arbitrary policies (Tunstall et al., 2024) or Tiet itself (Meng et al., 2024; Lambert et al., 2024). SimPO miti- gates this issue by replacing the variable divergence penalty -8 - log (N (x)/ (y) in LDFO with a constant therehy improving robustness under noisy preference sam- ples.
3 log (mо(y|²)/ / /<r((y|))) (x)) 3 log(Z (x)),, where To refers to the policy being trained and (x) is the partition function. Plugging this into the BT model, we get the following loss function for DPO:
To TY (x)
LOPO- -E(x.ww..))D
log
B log Tret (yuo [x)
(1)
To ( x)
3 log
Teel [y: (y:(x)
where D denotes the preference dataset.
3. Reward Function Matters
2.2. Simple Preference Optimization (SimPO)
In this section, dive deep into the shape of reward func- tions and its relationship with likelihood displacement. The main premise of this paper is that by systematically mod- ifying the log reward function, it is possible to adjust the intensity of the likelihood displacement and, in particular, improve the alignment performance of SimPO.
SimPO (Meng et al., 2024) is a modification of DAAS like DPO. SimPO scales the reward by the length of the output?:
Some papers refer to this as the (chosen, reject) pair. 'The idea of using average log probability for reward was originally proposed in Hong et al. (2024, ORPÓ).
3

---

AlphaPO: Reward Shape Matters for LLM Alignment
3.1. Likelihood displacement
et al., 2008). However, in the DPO context, the paper does not find much benefit from this extension since standard DPO (a 0) yields the best alignment performance.
Let us define the length normalized margin as (1/1yw/) Tw - (1/lyal) log ny, where Tw To (x), T - :o(y (x). Clearly, minimization of the preference loss in (2) would (in expectation) lead to an increase in the length normalized margin. Although this seems to imply that training decreases log ny and increases log T usually log xy rapidly decreases while also dragging the preferred likelihood log Tw down. This phenomenon called likelihood displacement (Razin al., 2024). Although several recent papers (Pal et al., 2024; Liu al., 2024; Pang al., 2024; Yuan et al., 2024; Rafailov et al., 2024b; Tajwar et al., 2024) mention this behavior and discuss its ramifications, (Razin et al., 2024) dissect this in detail. They make a key observation - since both log Te and log कं decrease, other responses (denoted by 2) increase in likelihood. If is as preferable as Y (e.g.. = is similar in meaning Yo.), then the likelihood displacement is benign. However, if 2 is poor response, then likelihood displacement is labeled as catastrophic and leads to unintentional consequences.
f-DPO operates by allowing alignment to incorporate vari- ous divergences, which determines the implicit base reward function. We consider a similar idea in the context of SimPO we use the reward function corresponding to a-divergence, but crucially modify it with length- :agtt-noormalization:
ry: (o(y:))=1 - 3 (1 1
To (y(x) a/ly|
a
This yields the following preference loss in the context of the margin-based BT model:
3 To
LllpaaP -(xxy=-)))
logo a
Trul
B + [To(yilx) 191 )
(4)
We refer to this new method, derived from utilizing the new reward function, as AlphaPO. In the limit a - 0, the AlphaPO reward funetion yields SimPO's log reward. Specific choices a 1 and a -1 yield the inverse-linear and linear reward functions, respectively: Tinv Finv innyx) 3 (- ( + 1) and rin x)) B (Tee(y)x) - 1), where ( x) To ( 1/xl
A closely related idea is discussed in (Shi et al., 2024), where the authors note that high likelihood of preferred responses and big gaps between the likelihood of preferred and dis- preferred responses can lead to reward over-optimization resulting in better margin but worse generalization. A controlled lowering of completion likelihood is shown to be beneficial for output diversity and generalization.
AlphaPO differs from f-DPO in the following ways:
The ideas listed above indicate the importance of balancing the optimization of the reward margin with the likelihood of the preferred and dispreferred responses. To avoid confu- sion, use the phrase "likelihood displacement" to refer to all the related ideas described above.
(1) Length normalization AlphaPO uses length normal- ization a crucial element in improving SimPO's 's perfor- mance over DPO.
(2) Fewer restrictions on the value of a f-DPO requires the estimation of the normalization constant (zz for some divergences. Specifically for a-divergence, f-DPO restricts the value of a E (0, 1) for the normalization constant to cancel out. No such restrictions are required for AlphaPO.
While excessive likelihood displacement (over- optimization) is detrimental, controlled likelihood displacement offers clear benefits, as demonstrated by methods like SimPO, which employ early stopping and limit training to one or a few epochs. Building on these insights, we explore how DAAS can be designed to shape the margin and preferredd/iisppeferred likelihoods.
(3) Improved generalization with tuning of a While f-DPO fails to demonstrate that any value of a except 0 (which is standard DPO) yields improved generalization, we demonstrate in the experiments section that non-zero values of a actually improve generalization performance.
3.2. The AlphaPO reward function
It turns out that a simple condition on yields reward func- tion shapes that encourage likelihood displacement. For de- tails, please see Lemma A. 1 and Figure 4 in Appendix A.5.
Most DAAS are based on the log reward function. We begin by discussing f-DPO (Wang et al., 2024a) where the authors point out that standard DPO corresponds to using reverse KL. as the divergence and that other divergences can lead to improvement in properties like generation diversity. One parametric divergence proposed by them is the a-divergence. This divergence has the reverse Kullback-Leibler (KL) di- vergence limiting case when + 0, and the forward KI. divergence as limiting case when a + 1 (Cichocki
3.3. Understanding AlphaPO using gradient analysis and experiments
We now perform gradient analysis and conduct experiments to study the three key length-normalized quantities: pre- ferred likelihood (///ywl) log Tw, dispreferred likelihood
4

---

AlphaPO: Reward Shape Matters for LLM Alignment
(1/lx=)) log m and margin (1/ly²) w - (1/ly=l) log m.- This will help us understand the degree of likelihood dis- placement that happens across methods, the role of and the effect on alignment performance.
As a consequence, large |o| values impose a regularization effect on alignment training due to the vanishing gradi- ent for samples with positive length-normalized margins. The regularization effects are stronger for positive values (super-exponential) compared to negative a values (expo- nential); see the proof for details. The following numerical illustration adequately demonstrates the limiting behavior in Theorem 3.1.
Let us begin with a gradient analysis of AlphaPO for a single training example. For y, x such that (y(y)) > O, we have
(-a log (x(ix) by
(T)(((()))) - (r)(y: B Bexp (y)x)
Illustration 1 Positive margin. Consider toy exam-
(
ple with 3 = 1, 7 = 0, yy lyrl = 1, 8 0ōv 0m/OO = 1. log Te = -1 and log ग = -2 Substi-
(5)
where the last expression on the right hand side has been written with the purpose of clearly separating out the role of a. Let be generic weight parameter. Define non- negative parameters Ce- =
tuting these values, we find [āl/dv| = (1 + cxp((e22
ex)) /a)) |ea+1 - e2a+21. For the values of a = -2, 0, 0.25, 1, and 2. the corresponding magnitudes of the deriva- tive |0t/0o| are 0.11, 1.26, 1.63, 0.44, and 2.15 10- respectively. The associated values for T1 are 0.49, 0.27, 0.19, 0.01, and 5.60 x 10-11, while the T2 values are 0.23, 4.67, 8.69, 47. 47.21, and 383.34.
Jog ma Ty , Q
log me Define Tyrl
5r ra (TT) ra(m)) Based on (3) and (5), the magnitude of the gradient of per-sample loss l with respect to u is:
σ 0v
@Tw Ov
ami de
= (é' - -7)1
I'm
ri
(6)
Illustration 2 Negative margin. Consider the toy example, where log Tw Tm magnitude of the derivative is given by: |0l/Oul (1 + exp ((e* - -=22)/aa)) l - cl+a| same a values, the computed jal/du] values are 0.12, 3.41, 7.05, 46.77, and 383.34, respectively. The corresponding values for Ti are 0.51, 0.73, 0.81, 0.99, and 1.00, while the T3 values remain identical to those in Illustration 1.
= -2 and log = -1. the
where the reward derivatives for the dispreferred and pre- ferred responses are: rf _ (m)) B exp(oci)/(() lyrl) and r/e A r (m)) = B eex((cc.)/ / a I) and ór = # [exp(aci) cxp( Substituting these quantities into Equation (6), we obtain |Øt/au|- (a). (a)) where
c2+2a
Using the
B
Tiaa=
These results demonstrate non monotonic behavior in the relationship between |aE/0v /dol and a (see Appendix A.10 for holistic illustration).
1 exexp
@ exp((ccw)
exp(acc)) exp(acw))
exp(ac:)
T2(a)=
Tw yw
dr
x
du
The key takeaways from the analysis are: (a) reward func- tions significantly impact preference-based alignment learn- ing, and (b) changes in a notably influence the training process by altering gradients and modifying the relative im- portance assigned to preferred and dispreferred responses. To gain a deeper understanding, we examine the evolution of Tate) (Vw xx) during training under gradient flow. Specifi- cally, we formalize how this relative importance influences the dynamics of To(e) (y.lx)) over training time 2 O, where ((t) denotes the model parameters at time t. For brevity, we define Twe A (V T. a To(s)
We are interested in studying how affects /0e/Oul. It turns out that Ti (a) and T2(a) exhibit markedly different behay- iors as functions of a. This discrepancy leads to [0e/Oul behaving in a non-monotonic fashion with respect to a. The limiting behavior is established in Theorem 3.1, whose proof is deferred to Appendix A.1.
Theorem 3.1. Let B > 0 and
7 2 0 be fixed constants. Let
A The
8s1a Then: Be
de
m2
and T A VociyTwe VT 2 Vo(IYTT
1. lim
0.
a 00
Theorem 3.2 (Evolution of Rw) Consider a single training
example (x, V.yy..i) Vi). where yw and yi are the preferred and dispreferred responses, respectively. Consider Rw, mi (0, 1). Assume that |TTll +0.1 If
2. As a + +o0, the limit of
0E de
depends on the sign of
the margin:
If the margin is positive, then lim 00++00
dr
0.
('(Tw)
VTW, Vm)
(9)
((())
VT
If the margin is non-positive,
lim
+o0
+-0+
then under gradient-flow dynamics. the probability of the preferred response Tu increases over time, i.e. d di Tw 2 0.
except when the margin is zero and
A = 03.
'If the margin is zero and A 0, T2 is identically zero, i.c.
(a) = and thus jae/ = 0.
The proof in Appendix A.2.
5

---

AlphaPO: Reward Shape Matters for LLM Alignment
If (VTT Vm) O, then (9) holds trivially for all at and Tw increases. A more interesting case is where (Viye, Vm) > O, i.e., the gradient step will increase or decrease the prob- abilities for both Tw and T.. In this senario, we have the following Corollary (proof in Appendix A.3).
4. Experiments
4.1. Experimental setup
Models We conduct all our experiments using the in- struct versions of three popular families of models Llama 3(Dubey et al., 2024), Mistral (Jiang al., 2023a) and Gemma 2 (Team et al., 2024). We make this choice for two reasons- (1) The aforementioned models represent the state-of- of-the-rrt and (2) they have been used in several recent works, making comparisons with baselines easier.
Corollary 3.3. Under the conditions of Theorem 3.2, sup- pose that (V VT)) > 0. If the margin is negative, then for a 2 oro, the probability The increases over time, Con- versely, if the margin is positive, then for a S Co. the probability To increases over time, where Og defined as
Datasets We chose the UltraFeedback (UF) dataset (Cui et al., 2024) for all experiments. Previous works (Meng et al., 2024; Wu et al., 2024; Zhao et al., 2024) have demon- strated that using an on-policy setting for the instruct setup helps mitigate the distribution shift between off-the-shelf instruct variants of these models and the preference opti- mization process. Following (Meng et al., 2024), specifi- cally, we regenerate five responses for every prompt in the UF dataset using a sampling temperature of 0.8. We then use two reward models- PairRM (Jiang et al., 2023b) and ArmoRM (Wang et al., 2024b) to rank the 5 responses. The highest scoring response is labeled Yw and the lowest scor- ing response is labeled yt- We use the PairRM-basee dataset to conduct experiments for Llama 3 and Mistral, and lever- age the ArmoRM- based dataset for Llama and Gemma 2- based experiments.
log
|V (V N...VVm) mZ)
Tu July
0g0
1 yul
log Tw
log I
(10)
Iyal
As pointed out by (Fisch et al., 2024), methods such as DPO have global optima that include policies capable of shifting nearly all probability mass to responses that never appear in the training set and even assigning near-zero probability to all training data responses that correspond to winning gen- erations. In Theorem 3.2 and Corollary 3.3, we demonstrate that such likelihood displacement can be mitigated by en- suring a sufficiently large ratio of r' (TT)// (Ti), controlled by selecting appropriate values of a.
The above analysis is at the level of a single example. A detailed analysis of the interaction of many examples is hard. Therefore, we conduct training experiments using the instruct versions of the Mistral-7B and Gemma-2-9B mod- els on the UltraFeedback dataset with 1 epoch of training (detailed setup in Section 4). Specifically, we vary a from -2.0 to 2.0, while keeping other hyperparameters fixed.
Hyperparameter tuning AlphaPO introduces new pa- rameter as described in equation (3). We tune for each different LLM and note the effect of tuning alpha in sec- tion 4.2. The exact values of learning rate, a, B, and other hyperparameters used in the experiment iterations are detailed in Appendix A.6.
Line plots for the length-normalized quantities of the Mistral model can be found in Figure (a similar figure for Gemma 2 (Figure 7) and the corresponding box plots Figures 15 and 16 are in Appendix 1.1 As shown in the figure, the influence of a on dynamics is indeed non-monotonic. Specifically, there are larger displacements for a values close to and smaller displacements for larger |a] values. In particular, larger |a|s lead smaller quartile values of margin toward the end of training, which could help mitigate the over-optimization. Scatter plots of preferred likelihood vs. margin in Figures 13 and 14 tell a similar story. For large absolute values of a (-2 and 2), we notice a clear reg- ularization effect for both margin and preferred likelihood.
Baselines We compare AlphaPO primarily against SimPO and DPO since they represent the the
Evaluation. We evaluate trained models using two popu- lar benchmarks - AlpacaEval 2.0 (Dubois et al., 2024) and Arena-Hard (Wang et al., 2024b) (referred to as AF2 and AH, respectively, hereinafter). AE2 consists of 805 prompts sourced from a variety of datasets, gearing it towards mea- suring the ability of models to follow diverse and complex instructions. For AE2, we report WR (win rate) and LC (length-controllec win rate). LC is specifically designed to discourage models from using verbose answers, since GPT4 is known to favor longer responses when judging for instruction following (Wang et al., 2023; Park et al., 2024b). For AH, we only report WR since it doesn't offer LC.
During the initial phase of training, many samples exhibit an undesired ordering, characterized by negative margin. According to Corollary 3.3, selecting a sufficiently large a should lead to an increase in T.. This behavior is evident in epochs 0 to 0.1 in Figures 2 and 7 (for a >> -2).
4.2. Results
AlphaPO outperforms SimPO and DPO on most bench- marks as demonstrated in Table 1. Focusing on PairRM- results, AlphaPO outperforms SimPO and DPO across
Additionally, we explore in Appendix A.4 what happens when a reference model is included in the SimPO and Al- phaPO training objectives.
6

---

AlphaPO: Reward Shape Matters for LLM Alignment
Statitics
0.8
Median Max Whisker
-0.5
Dispreferred
0.4
Preferred
-1.0
Margin
0.0
-+
2r 0
-15
-0.2
-D- ==
-20
-2.5
-0.6
0.0
022
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.8
0.8
1.0
0.0
0.2
0.4
0.6
0.0
1.0
Epoch Number
Epoch Number
Epoch Number
Figure 2. Tracking various statistics for nortth-normalized preferred likelihood, dispreferred likelihood and margin over an epoch for the Mistral instruct model. The min and max computed after removing outliers.
Table I. Comparison of DPO, SimPO, and AlphaPO for on-policy data created using PairRM and ArmoRM.
PairRM
ArmoRM
Llama-3-88-Instuuc
Mistral-7B-Innst
Llama-3-8B-Instrac
Gemma-2-9B-Instruuc
Method
AE2
AH
AE2
AH
AE2
AH
AE2
LC (%)
WR (%)
WR (%)
LC (%)
WR (%)
WR (%)
LC (96)
WR (%)
WR (%)
LC (%)
WR (%)
WR (%)
DPO
39.18
36.84
34.1
21.96
22.16
14.8
47.19
45.61
35.5
67.83
65.77
60.7
SimPO
42.05
36.90
32.8
29.71
31.12
21.5
51.66
46.54
35.4
73.72
67.36
AlphaPO
45.37
40.97
34.6
33.03
34.12
21.7
53.01
47.64
36.0
74.13
65.89
57.7
both AE2 and AH for the instruct versions of the Llama 3 and Mistral models. The results are especially pronounced for AE2 where AlphaPO relatively improves over SimPO by 7% to 10% for LC, and over DPO by 15% for Llama 3 and 50% for Mistral. Notably, AlphaPO doesn't increase the generation length significantly for all the models when compared to SimPO and DPO (more in Appendix A.7).
Table 2. Language modeling benchmark evaluation results for
SimPO and AlphaPO. We evaluated in 10-shot setting for Hel- laSwag.
Mlama-3-8B-Instiee
Histral-7B-Inntruee
SimPO
AlphaPO
SimPO
AlphaPO
HellaSwag TruthfulQA
0.7576
0.7694
0.8610
0.8638
0.6078
0.6142
0.7061
0.7127
For ArmoRM-based results, Llama 3 AlphaPO outperforms both DPO and SimPO. For Gemma 2, we note that AlphaPO has a better LC and a slightly lower or comparable WR for AE2 when compared to SimPO and DPO. The lower WR for AB2 is also mirrored by AH results where SimPO and DPO are slightly better. This is consistent with the observation that GPT as a judge has a bias towards longer responses, and AH doesn't have way control for length when mea- suring rewards (Wang et al., 2023; Park et al., 2024b). More detailed results can be found in Appendix A.7. From Fig- ure 5 in Appendix A.8, we demonstrate that the number of AE2 samples where AlphaPO outperforms SimPO is higher than the number of samples where SimPO does better.
the peak value. The drop-off less steep on the positive side. This is not surprising, since a positive a results in less aggressive likelihood displacement when compared SimPO (which lowers both preferred and dispreferred likeli- hoods) and subsequently achieves higher AE2 performance. A negative a results in more aggressive likelihood place- ment compared to positive values a. See section 3.3 for more discussion.
Effect of tuning y on generalization performance We examine the impact of the margin parameter 7 on AE2 performance. To this end, we tune y while keeping other hyperparameters fixed the optimal settings reported in Table 3. We use PairRM-basee data for all models, except for Gemma 2, which employs ArmoRM M-based-a data.
Effect of a on generalization performance To under- stand the effect of model generalization, mea- sure the AE2 performance of various models with changing a. Results for the Mistral model are in Figure (results for other models are similar and are Figure in Ap- pendix A.1.). A slightly positive value of a achieves the best AE2 performance, with a drop-off on either side of
Results for the Mistral model appear in Figure 3 (findings for other models are in Figure 9 in Appendix A. A.11). Consis- tent with the findings of (Meng et al., 2024), increasing ?

---

AlphaPO: Reward Shape Matters for LLM Alignment
improves AE2 performance a certain level, beyond which larger values hurt performance. This indicates that there exists an optimal gamma that has to be tuned depending the model family and the quality of response generation (de- termined by AE2 evaluation) is not determined by the mar- gin alone. On the other hand, increasing 7 leads to longer response lengths, as it dominates the length-normalizee quantities r( - r(y;:) x) in the loss function.
models on HellaSwag (Zellers al., 2019) and Truth- fulQA (Lin et al., 2021). HellaSwag is multiple-choice commonsense reasoning benchmark whereas TruthfulQA is crrestion-answringg benchmark designed to evaluate a model's propensity to generate truthful versus misleading or false answers. Results are presented in Table 2. AlphaPO outperforms SimPO across all settings.
5. Related work
Effect on reward scores To better understand the quali- tative impact of various alignment methods, we look at the reward distribution for the test set of UltraFeedback using PairRM. We compare AlphaPO SimPO and SFT for the Mistral in Figure 3. It is evident that AlphaPO yields reward distribution that is better or on par with SimPO (the strongest baseline in our experiments). Since the intent of alignment methods is to match human preferences, this test a good proxy for measuring the performance (see the right plot Figure 12 in Appendix A.11 for the Llama 3 result).
Reinforcement learning with human feedback RL.F aligns language models to human preferences by leveraging parameterized reward models as proxies and reinforcement learning (RL) algorithms (Christiano et al., 2017: Ziegler et al., 2019). Having parameterized reward models trained with the Bradley-Terry model (Bradley & Terry, 1952), al- gorithms like proximal policy optimization (Schulman et al., 2017) and REINFORCE (Williams, 1992) are applied for language models to maximize the reward with online gener- ations (Ziegler et al., 2020; Ouyang et al., 2022; Ahmadian et al., 2024; Kazemnejad et al., 2024). While typically being used to align the style of generations to human preferences (Ziegler et al., 2020; Ouyang et al., 2022), the paradigm of RLHF is being applied to specialized tasks like complex mathematical reasoning (Kazemnejad et al., 2024; Lambert et al., 2024) and coding (Gehring et al., 2024).
Evolution of KL Divergence with training In Figures 10 and 11 of the Appendix A. we present the evolution of KL. divergence (relative to the SFT checkpoint) and the AB2 LC during training for AlphaPO and SimPO applied to both Mistral and Llama 3 instruct models. We observe that both methods remain close to the instruct model, exhibit- ing minimal divergence even without explicit regularization to the reference policy. Notably, AlphaPO achieves a KL. divergence comparable to that of SimPO while attaining better LC values, suggesting that excessively high or low KL divergence can impede generalization. Furthermore, al- though KI. divergence continues to increase and eventually plateaus, a larger KL does not necessarily lead to improved LC. These findings highlight the importance of early stop- ping and careful tuning of the total number of training steps to enhance preference generalization performance.
Direct alignment algorithms Direct alignment algo- rithms (Rafailov et al., 2024a, DAAs) as methods for di- rectly aligning the language models to human preferences hypass RI. in RLHF (Yuan et al., 2023; Zhao et al., 2023;
Rafailov et al., 2023; Wang et al., 2024a; Xu et al., 2024; Gheshlaghi Azar et al., 2024; Ethayarajh et al., 2024; Wu et al., 2024; Hong et al., 2024; Meng et al., 2024) either through optimizing expected rewards penalized by diver- gence from a reference policy (e.g.. DPO (Rafailov et al., 2023), SPPO (Wu et al., 2024)) or directly as a function of the policy (e.g., ORPO (Hong et al., 2024), SimPO (Meng et al., 2024)).
Combining SimPO//llhaPOO and SPPO SimPO and Al- phaPO use BT modeling to diverge from reference model, achieving strong alignment through optimal hyperparame- ters. In contrast, SPPO (Wu et al., 2024) cautiously moves toward a Nash equilibrium, For the PairRM-based (Jiang et al., 2023b) experiments, we select the best performing checkpoints for SimPO and AlphaPO and train both further using SPPO (see Appendix A.9 for details). Without exten- sive tuning, SPPO improves the AE2 LC of both SimPO (from 42.05 to 45.06) and AlphaPO (from 45.37 to 47.42). This demonstrates that methods like SPPO can be orthogo- nally used to improve these methods
While these score functions effectively guide fine-tuning with alignment data, they often exhibit over-optimization, excessively widening the margin between chosen and re- jected outputs, sometimes lowering the quality of chosen responses. Recent research like f-divergence Preference Op- timization (Han et al., 2024, f-PO) offers theoretical insights addressing overooptimization by minimizing f-divergences.
6. Conclusion
Additional experiments on other datasets To under- stand the impact of AlphaPO on datasets other than those used traditionally for alignment, we compare AlphaPO and SimPO-based checkpoints of the Mistral and Llama
Altering the reward function shape to enhance the alignment performance of SimPO is a novel insight introduced and demonstrated in this paper. Since reward function (via the choice of divergence measure) occurs in other methods such
8

---

AIphaPO: Reward Shape Matters for LLM Alignment
34
2,200
34
0.1
SFT
SimPO
33
2,150
32
AlphaPO
32
2,100
30
31
30
2,050
28
0
0.25
0.5
0.75
0.5
0
0.5
-10
0
10
7
Reward
Figure 3. Ablation studies the Mistral instruct model. Left: Effect of the margin parameter y on the AE2 LC (bluc) and response length (red). Middle: Effect of the parameter a on the AE2 LC. Right: Reward distributions for the SFT, SimPO, and AlphaPO checkpoints.
as DPO and RLHF, whether it is possible that changing its shape can provide value for them too is a worthwhile future research direction.
L., Hase, P., BIyIk, E., Dragan, A., Krueger, D., Sadigh, D., and Hadfield-Memell, D. Open problems and funda- mental limitations of reinforcement learning from human feedback, 2023. URL. https: /ttp/aaavvvv//..///vv//eee///.aa www.aii.w.oo/aa..e..e.oeeeeeeeee 2307. 15217.
Impact Statement
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Christiano, F., Leike, J., Brown, T., Martie, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Gareett R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https::/prooceedings neurips.. cc cc/paper_files//peeeeeeeeeeseeee /tl/aper_flee/pppp////////// dis2ccoadadd03eee111e999200000d0 pdf.
References
Ahmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A., and Hooker, S. Back to basics: Revisiting REINFORCE -tyyle optimization for learning from human feedback in LLMs. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers), pp. 1 Bangkok, Thailand, August 2024. Association for Com- putational Linguistics. doi: 10.186655/1/2022444--ngg 662. URL https:// http::://cccanntoooooyoo//////// acl Long. 662.
Cichocki, A., Lee, H., Kim, Y.-D,, and Choi, S. Non- negative matrix factorization with a-divergence. Pattem Recognition Letters, 29(9): 14l0-1443444444 1440, 2008.
Cui, G., Yuan, L., Ding, N., Yao, G., He, B., Zhu, W. Ni, Y., Xie, G., Xie, R., Lin, Y., Liu, Z., and Sun, M. Ultrafeedback: boosting language models with scaled ai feedback. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org. 2024.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345 1952.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., al. The llama 3 herd of models. 2024. URL https://arxiv.orr /ttp///s.2222.... 21783...
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.0444 04475, 2024.
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., Wang. T., Marks, S., Segeric, C.-R., Carroll, M., Peng. A., Christoffersen, P., Damani, M., Slocum, S., Anwar, U.. Siththaramjan, A., Nadeau, M., Michaud, E. J., Pfau, J., Krasheninnikov, D., Chen, X., Langosco,
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Model alignment as prospect theoretic optimiza- tion. Forty-first International Conference on Machine

---

AlphaPO: Reward Shape Matters for LLM Alignment
Learning, 2024. URL U
Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, I., J. V., Liu, A., Dziri, N., Lyu, S., Gu, Y., Malik, S., Graf, V., Hwang, J. D., Yang, J., Bras, R. L., Tafjord, O., Wilhelm, C., Soldaini, L., Smith, N. A., Wang, Y., Dasigi, P., and Hajishirzi, H. Tulu 3: Pushing frontiers in open language model post-training, 2024. URL https: //axiiv...iv/////aa/aai .rgg /at///aa//// 2411. 15124.
Fisch, A., Eisenstein, J., Zayats, V., Agarwal, A., Beirami, A., Nagpal, C., Shaw, P., and Berant, J. Robust preference optimization through reward model distillation. arXiv preprint arXiv:2405. 193/6, 2024.
Gehring, J.. Zheng, K., Copet, J., Mella, V., Cohen, T., and Synnaeve, G. Rlef: Grounding code lims in execu- tion feedback with reinforcement learning. 2024. URL https: /arxiv. org/ abs/2410, 2410. 02089.
Li, Chiang, W.-L., Frick, E., Dunlap, L., Wu, T., Zhu, B., Gonzalez, J. E., and Stoica, I. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline, 2024. URL https: /ttp:://axxxvorooo abs/ 2406. 11939.
Gheshlaghi Azar, M., Daniel Guo, Z., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In Dasgupta, S., Mandt, S., and Li, Y. (eds.), Proceedings of The 27th Interna- tional Conference on Artificial Intelligence and Statis- tics, volume 238 of Proceedings of Machine Learn- ing Research, pp. 4447-4455. PMLR, 02-04 May 2024. URL /ttpa//://cccceeppoeeee mlr......... .press/ html.
Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint 2021.
Liu, Z., Lu, M., Zhang, S., Liu, B., Guo, H., Yang, Y., Blanchet, J., and Wang. Z. Provably mitigating overopti- mization in rlhf: Your sft loss is implicitly an adversarial regularizer, 2024. URL https: /awxivivvvvv./////// org/ /bs/ 2405.16433 16436.
v238 h htt//ghhshaahhaaaaaaaaa
Han, J. Jiang, M., Song, Y., Leskovec, J., Ermon, S., and Xu, M. f-po: Generalizing preference optimization with f-divergence minimization, 2024. URL https: //arxixv www.rioovogooooeeeee...o..ooo abs /0bss/2410 2410. 21662.
Meng, Y., Xia, M., and Chen, D. SimPO:
Simple preference optimization with a reference-free reward. In The Thirty- eighth Annual Conference cessing Systems, 2024. URL httpt::t/////teet /otp:ree//ppeeeviee. net / forum? 3dmmmiddd-3-ils
Neural Information Pro-
Hong, J., Lee, N., and Thorne, J. ORPO: Monolithic pref- erence optimization without reference model. In Al- Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 11170-11 189, Miami, Florida, USA, November 2024. Association for Compu- tational Linguistics. doi: 10.18653w1/22224ccmm... 626. URL https:/ ////cclaant htt////annnhllly.orrgg/o//////// emn m-anl-mmai... 626.
Ouyang, L., Wu, J.,, Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., al. Training language models follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744 2022.
Pal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., and White, C. Smaug: Fixing failure modes of preference optimisation with dpo-positive, 2024. URL https:/ // arxiv. org/ 2402.12022 13228.
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, I.., Lavaud, I.. R., Lachaux, M.- A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023a. URL https: //arxiv. org/ abs/ /20/2/3111 2210..1001000 06855.
Pang, R. Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., and Weston, J. Iterative reasoning preference optimiza- tion, 2024. URL https: http//xxvvvv//.i//ee org/ 19733.
htgg/bb/////aao///e///gg////b abs/ 2404.
Jang, D.. Ren, X., and Lin, B. Y. .lmedeender: Ensembling large language models with pairwise comparison and generative fusion. In Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics (ACI. 2023), 2023b.
Park, R., Rafailov, R., Ermon, S., and Finn, C. Disentan- gling length from quality in direct preference optimiza- tion. arXiv preprint arXiv:2403. 19159, 2024a.
Park, R., Rafailov, R., Ermon, S., and Finn, C. Disen- tangling length from quality in direct preference opti- mization. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computa- tional Linguistics: ACL 2024, pp. 4998-5017, Bangkok, Thailand, August 2024b. Association for Computa- tional Linguistics. doi: 10. 186138811/0224 10553//1/222.nndings--aa
Kazemnejad, A., Aghajohari, M., Portelance, E., Sordoni, A., Reddy, S., Courville, A., and Roux, N. L. Vineppo: Unlocking rl potential for Ilm reasoning through re- fined credit assignment, 2024. URL https: //ari//a...iviii org/abs/ 2410. 01679.
10

---

AlphaPO: Reward Shape Matters for LLM Alignment
297. URL hwtph:tpp///cm petttp:://acaattinn /tR./utteee//eel.antoooooooyyrr/ findings-acl.2977
J. P., Carrasqueira, J., Iljazi, J., Becker, J., Fernandez, J., van Amersfoort, J., Gordon, J., Lipschultz, J., Newlan, J., yeong Ji, J., Mohamed, K., Badola, K. Black, K., Milli- can, K., McDonell, K.. Nguyen, K., Sodhia, K., Greene, K., Sjoesund, I., L.,, Usui, I.,, Sifre, I.., Heuermann, I., Lago, L., McNealus, L., Soares, L. B., Kilpatrick, L., Dixon, L.,, Martins, L.,, Reid, M., Singh, M., Iverson, M., Görner, M.. Velloso, M., Wirth, M., Davidow, M.. Miller, M., Rahtz, M., Watson, M., Risdal, M., Kazemi, M. Moynihan, M., Zhang, M., Kahng, M., Park, M., Rahman, M., Khatwani, M., Dao, N., Bardoliwalla, N., Devanathan, N., Dumai, N., Chauhan, N., Wahltinez, O., Botarda, P., Barnes, P., Barham, P., Michel, P., Jin, P., Georgiev, P., Culliton, P., Kuppala, P., Comanescu, R., Merhej, R., Jana, R., Rokni, R. A., Agarwal, R., Mullins, R., Saadat, S., Carthy, S. M., Cogan, S., Per- rin, S., Amold, S. M. R., Krause, S., Dai, S., Garg, S., Sheth, S., Ronstrom, S., Chan, S., Jordan, T., Yu, T., Eccles, T., Hennigan, T., Kocisky, T., Doshi, T., Jain, V., Yadav, V., Meshram, V., Dharmadhikari, V., Barkley, W., Wei, W., Ye, W., Han, W., Kwon, W., Xu, X., Shen, Z., Gong, Z., Wei, Z., Cotruta, V., Kirk, P., Rao, A., Giang, M., Peran, L., Warkentin, T., Collins, E., Bar- ral, J., Ghahramani, Z., Hadsell, R., Sculley, D., Banks, J., Dragan, A., Petrov, S., Vinyals, O., Dean, J., Has- sabis, D., Kavukcuoglu, K., Farabet, C., Buchatskaya, E., Borgeaud, S., Fiedel, N., Joulin, A., Kenealy, K., Dadashi, R., and Andreev, A. Gemma 2: Improving open language models at a practical size, 2024. URL https: / aawivv.iivvv.///vv///..a/...aava . org g/ / abs / 240 00118.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. In Thirty- seventh Conference on Neural Information Processing Systems, 2023. URL ptttps::/oope /ttps////ppeeeeeeeeeeee / forum?id=HPP US IXJaa9.
Rafailov, R., Chittepu, Y., Park, R., Sikchi, H., Hejna, J., Knox, B., Finn, C., and Niekum, S. Scaling laws for reward model overoptimization in direct alignment algo- rithms. arXiv preprint
2024a.
Rafailov, R., Hejna, J., Park, R., and Finn, C. From r to q²: Your language model is secretly q-function, 2024b. URL https:/ http///axvvv... org/ abs/ / 2444 12358.
Razin, N., Malladi, S., Bhaskar, A.. Chen, D., Arora, S., and
Hanin, B. Unintentional unalignment: Likelihood dis- placement in direct preference optimization, 2024. URL https: arxiv org/ abs/ / 2210.01... 08847.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algo- rithms, 2017. URL https: //arxi///aiivvaeaaaeeaaeeeeraa org/oa/g//oooo /bs/ 2707.06347
Shi, Z., Land, S., Locatelli, A., Geist, M., and Bartolo, M. Understanding likelihood over-optimisation in direct alignment algorithms. arXiv preprint arXiv:2410.11677, 2024.
Tunstall, L., Beeching, E. E., Lambert, N., Rajani, N. Rasul, K., Belkada, Y., Huang, S., Werra, L. V., Four- rier, C., Habib, N., Sarrazin, N., Sanseviero, O., Rush, A. M., and Wolf, T. Zephyr: Direct distillation of LM alignment. In First Conference on Language Modeling, 2024. URL https: /wwpnreveeeeweeeeeeeeeeeeereeewe net forum? id=akkAwZB6.v,
Tajwar, F., Singh, A., Sharma, A., Rafailov, R., Schneider, J., Xie, T., Ermon, S., Finn, C., and Kumar, A. Preference fine-tuning of Ilms should leverage suboptimal, on-policy data, 2024. URL https: arxiv. org/ abs///b///// 2404. 14367.
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C. I.., Jerome, S., Tsit- sulin, A., Vieillard, N., Stanczyk, P., Girgin, S., Momchev, N., Hoffman, M., Thakoor, S., Grill, J., Neyshabur, B., Bachem, O., Walton, A., Severyn, A., Parrish, A., Ahmad, A., Hutchison, A., Abdagic A., Carl, A., Shen, A., Brock, A., Coenen, A., Laforge, A., Paterson, A., Bastian, B., Piot. B., Wu, B., Royal. B., Chen. C., Kumar. Perry, C. Welty, C., Choquette-Choo, C. A., Sinopalnikov, D., Weinberger, D., Vijaykumar, D., Rogozińska, D., Her- bison, D., Bandy, E., Wang. F.,, Noland, E., Moreira, E., Senter, E., Eltyshev, E., Visin, F., Rasskin, G., Wei, G., Cameron, G., Martins, G., Hashemi, H., Klimezak- Plucinska, H., Batra, H., Dhand, H., Nardini, I., Mein, J., Zhou, J., Svensson, J., Stanway, J., Chan, J., Zhou,
Vaswani, A. Attention all you need, Advances Neural Information Processing Systems, 2017.
Wang, C., Jiang, Y., Yang, C., Liu, H., and Chen, Y. Be- yond reverse KL: Generalizing direct preference optimiza- tion with diverse divergence constraints. In The Twelfth International Conference on Learning Representations, 2024a. URL https: /oppnrevieweeeeoeooooee net/ forum? Ad=2cRzmWXK9n
Wang, H., Xiong, W., Xie, T., Zhao, H., and Zhang, T. Interpretable preferences via multi-objective reward mod- eling and mixture-of-experts, 2024b. URL https: farxxi..ogg/bb//24666.66
Wang, Y., Ivison, H., Dasigi, P., Hessel, J., Khot, T., Chandu, K., Wadden, D., MacMillan, K., Smith, N. A., Beltagy,

---

AlphaPO: Reward Shape Matters for LLM Alignment
I., al. How far can camels go? exploring the state of instruction tuning open resources. Advances in Neural Information Processing Systems, 36:74764-74786, 2023.
Williams, J. Simple statistical gradient-following algo- rithms for connectionist reinforcement learning. Machine learning, 8:229-2550 1992.
Wu, Y., Sun, Z., Yuan, H., Ji, K., Yang, Y., and Gu, Q- Self-play preference optimization for language model alignment, 2024. URL https:// /tttt:://axii..../// wrg./brgggao. / 2405. 00675.
Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L., Durme, B. V., Murray, K., and Kim, Y. J. Contrastive preference optimization: Pushing the boundaries of LLM perfor- mance in machine translation. In Forty-first International Conference on Machine Learning, 2024. URL ht tts: //openreview net
Yuan, H., Yuan, Z., Tan, C., Wang. W., Huang, S., and Huang, F. RRHF: Rank responses to align lan- guage models with human feedback. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: http:::://ppp.eeiiiee net forum? id=EdIGMCHk41.
Yuan, L., Cui, G., Wang, H., Ding, N., Wang. X., Deng, J., Shan, B., Chen, H., Xie, R., Lin, Y., Liu, Z., Zhou, B., Peng, H., Liu, Z., and Sun, M. Advancing I1m reasoning generalists with preference trees, 2024. URL https: //aaxiv..orr// wwwpxivvorr/aeooo abs/ 444 02078.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sentence? 2019.
arXiv preprint arXiv:1905.07880
Zhao, H., Winata, G. I., Das, A., Zhang, S.-X., Yao, D. D., Tang, W., and Sahu, S. Rainbowpo: A unified framework for combining improvements in preference optimization. arXiv preprint arXiv:2410.04203, 2024.
Zhao, Y., Khalman, M.. Joshi, R., Narayan, S.. Salch, M., and Liu, P. J. Calibrating sequence likelihood im- proves conditional language generation. In The Eleventh International Conference on Learning Representations, 2023. URL atttps:/ae httttss://operreee// http//ppereeviieeeeee htwp://wwww.e//oorrrreee id=0qSOodKmJaN,
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint (9t.:199000893.. 2019.
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences, 2020. URL https: hetps://arxivv wwg/abs/////oooooooooo..oo... 909. 08593.
12

---

AlphaPO: Reward Shape Matters for LLM Alignment
A. Appendix
A.I. Proof of Theorem 3.1
Proof. For Tws iTy E (0, 1), recall Cw log Fay 0,
log I Mi
> 0.
Part I: o + -C.
Since &t > 0 and Cu 0, both exp(a ci) and exp(a cw) go to 0. Hence
exp(ac)) exp(ac.) + 0, a (explae)eexp(cca)))
- exp(acc))
7 0.
Therefore
B
[i)a) = 1 exexp exp(acew) -7)
+ 1te
a finite and positive constant.
For (2)):):
Tww
am Ti yil du
(a) = exp((a cw)
exp(aci)
Ta y
du
Each term contains factor exp(a G;)
which goes to 0, so (2 + 0. Thus the product satisfies
lim
86
(i) 0. a
lim
Part 2: a + +00.
Case 2(a), Positive margin: aj Gu= Since q Cws exp(act) >> exp(ocw) as +00. Thus:
exp((c) exp(acw)] 8 exp(ac)).
(11)
a
The denominator of T) (a:) satisfies:
( [xxp(ac) exp(acw)] -7) cxp (8 (xp((cc)
cxp
(12)
Since exp(ac) >> exp(acw), the term exp(ac;) dominates:
T2(a)) ~ exp(ac)) mm
(13)
Ou
Combining the asymptotics of Ti (a) and T2(a), find:
exp(ac))
Om
[1(a)
x
(14)
exp 2 exp(aci)
T lyil
Ov
Bexp(acy)
mmi
+ 0.
(15)
exp
2 exp((ac))
T. m||l
Ov
Thus, we conclude:
lim +00
0, when >> Car-
(16)
Ou
Case 2(b), Negative Margin: < Cw. Then exp(a cw) dominates exp(a and
exp(a<) cx)( x exp(a Cw))
13

---

AlphaPO: Reward Shape Matters for LLM Alignment
Hence
B a ~ exp(acw) -00, @ (x) - exp(a
so
exp(a Gw)) -7) O, Ti(a) + B. exp - f cxp(-
In Tb(),, the exp(a ow) tern dominates, so [2) grows on the order of exp(acul Thus
T2(a) +o0, (i(a) 8 exp(a Gw) to0,
ae 8v = too when q <
implying lima-++o0
Case 2(c), Zero Margin: q Cw. Then
B exp(ac)) (expla) 0, Ti(a) 1 exp( (-))
which is positive constant. In I2(a),
1 Ouw
1 0m
T2(a) eep((cc)
exp(a c)
Tw yy
Or
m lyi
dv
1
1
0m
cxp(acc
The yol
yrl
Ou
If A
8m
+ 0, then T2(a) grows exponentially since exp(a +00. Hence the product with
To y
t. I dh
the finite positive (a:) diverges to +oc.
Thus
lim
88 80
+o0
when A + 0.
a00000
A.2. Proof of Theorem 3.2
Proof. As the model parameters evolve according to gradient flow:
d @(t) -=
(17)
dt
where E(0) is a differentiable loss function. By the chain rule. the time derivative of ग. is
d dt Tw = (Vzw: d (()) -V VE(O(()))
(18)
dt
Recall the loss gradient
Ve((())) - (r (F) 76)
(19)
where (ōr 7)) is negative. Applying the condition Substituting into the expression for 4 म T gives
d Tw - 7) -1 (.)))))
(20)
Factoring out (T)), we obtain
d
['((w) V 12-- (ViT, Val) (mm)
dt Tw - (m))
(21)
Recall (5), (5) r(m) 0. Since -8r -7) 0, the sign of a Tw is determined by
r' (TT))
Vnwll (ViTw Vm).
(22)
14

---

AlphaPO: Reward Shape Matters for LLM Alignment
Rearranging the inequality condition, require
VT
VTi
(23)
V Two ||
to ensure that the bracketed term is nonnegative. Under this condition, it follows that
d
T 0.
(24)
di
Thus, the probability Tho of the preferred response increases over time.
A.3. Proof of Corollary 3.3
Proof. Under the assumption that (V Twa the ratio a. The ratio parameterized by a
VT)) > O, the condition in Theorem 3.2 can be satisfied by appropriately choosing
r' (TT)) Myld
log Tw yw
log m Iyal
exp
-a
(25)
r' (m))
Tu. Yw
Substituting (25) into the inequality (9), obtain
m Vil
log ww
log n yy
V Tw. Vi()) V
exp
-a
(26)
n y
y
Rearranging terms, we have
log Tw
log ni lyil
V I1?
Tan V
exp
(27)
ywwl
V Twu Vx()
Taking the natural logarithm of both sides yields
log T
log TI
V ||
Tan ywl x
log
(28)
y ml
Tww VT()
Solving for o.
log Tan
log m yıl
S
if
(positive margin),
(29)
yywl
and
log The
log
a
2 000
if
Ywe
yy
(negative margin),
(30)
where or is defined as
V 1|
log
TV Tso
V m3
m. yil
a0
Tu:
logar Tyd
(31)
Ty
A.4. AlphaPO with Reference Policy
Another interesting observation what happens when reference model is included in the SimPO and AlphaPO training objectives. Interestingly, the SimPO variant reduces to SimPO with a per-response-pice margin, whereas the AlphaPO variant reduces to AlphaPO with per-response weights for the preferred and dispreferred responses.
The inclusion of the reference policy in SimPO (2) results in the loss
LSiPPO ref -E(xyy. log B
(X( lywl Ter((
3
log To(y)(xx) ]
log
lydl
B
--E(xyyosi)w log
log no (yxl)))))
log (y()))) - 1 YY v1.2)
[y
ly
15

---

AlphaPO: Reward Shape Matters for LLM Alignment
Therefore, IiIpPO ver reduces SimPO with the per-response-pair margin > (Y Yt x)
B
=
log (y xx)
log (y)
y
y1
In contrast, with the reference policy, the AlphaPO (4) becomes
-8 To (y
-0/1xwl
To (y(((xx)
-a/lsrl -
AlphaPO ref = -E(x.ym..)
logo
+
T (yr((y1
3' (y x)
3' B'y1,, x
To T -0//brl -
--
log
To w -0/1y// xx(1vlx)-o//1m +
a
where
(y x) 3 Trerly m: /x)a//xll
8 x) = B ( (x)
Therefore, LAlpharo set reduces to AlphaPO with per-response weight B3 (y x) and 3 xy).
16

---

AlphaPO: Reward Shape Matters for LLM Alignment
A.5. Likelihood displacement with AlphaPO
turns that simple condition on yields reward function shapes that encourage likelihood displacement (and subsequent decrease in the likelihood of both, preferred and dispreferred responses). The following lemma describes this. Lemma A.1 (Monotonically decreasing derivative of ra (T)). Let π & (0, 1] be the probability of a response y for parameters 0, and let a, 3 E R with > 0. Also, let yy 2 1. Then ra (TT) is monotonically decreasing in π if and only if 2 -lyl-
We relegate the proof to Appendix A. In Lemma A.I, we use slight abuse of notation where refer to the reward function as ra (y)(()))) instead of (y|x). Since in general, ly| > 1 (and potentially much larger than 1) for responses, most practical values of a will satisfy a 2 - y|- Thus, by Lemma 1, lower likelihood values will have higher derivative values for the reward function. This encourages preference optimization to move the likelihood of preferred and dispreferred responses to lower values.
Figure 4 illustrates this for the log reward function (a 0). Notice that for the same difference in likelihood, one can achieve much higher reward difference if the likelihood of preferred and dispreferred responses are both reduced.
The log reward function
0.1
022
0.3
0.4
0.5
06
0.7
0.9
09
Reward value
(0:00.
-0.11)
(0.65,-0453
Likelibood
2
(0.10 15, 11990)
(0.10. -2.30)
log(a)
-5
Figure The log reward function (y) as a function of the completion likelihood (x). From the plot, is evident that similar change in likelihood, the difference in reward is much higher when the likelihood changes from 0.1 to 0.15 (~00.40), rather than from 0.6 to 0.65 (~00.08).
A.5.1. PROOF OF LEMMA A.1
Proof. The derivative of the reward function with respect to π is given by
8 r (T) = y π (1)+
(32)
To determine when r'((T) monotonically decreasing, we compute its derivative with respect to m:
(T)
d
( 3
# -
( a +1 x (1) +2)
(33)
dæ
do
[
Iy
y
Since 3 > 0, ly| 2 1, and π (5) +2) > 0 for ग E (0, 1], the sign of dr' (T) dz
is determined by the term (0) + +1).
For r'a (T) to be monotonically decreasing in т,, we require:
dr (T)
I 0.
dm
This inequality holds and only if
2 0
+
2
-lyl-
17

---

AlphaPO: Reward Shape Matters for LLM Alignment
A.6. Implementation Details
Training nyper-parameter tuning Following the recommendations of SimPO, we adopt global batch size of 128, maximum sequence length of 2048, and cosine learning rate schedule with warmup ratio of 0.1 for one epoch across all training settings. For DPO, we use the best and learning rate values reported in SimPO github.
Although AlphaPO introduces an additional hyperparameter, a, compared to SimPO, the need for extensive grid searches can be mitigated if the optimal parameters for SimPO have already been determined. Instead of performing a full grid search, we can leverage the best parameters identified for SimPO and conduct a coordinate-wise hyperparameter search in greedy manner. This approach allows us achieve improved performance rapidly, often within few search iterations. Notably, the optimal parameters of B, y and learning rate for AlphaPO show only minor deviations from those of SimPO. The hyperparameters that differ from the SimPO settings are highlighted in bold in Table 3. Follow the practice from the SimPO repo, report 7/B (instead of 7) in Table 3.
Table Best hyperparameters for training.
Method
Model
7/B
Learning Rate
DPO
Mistral-Instruct
0.01
x 10
Llama-3-Instruct
0.01
x 10 10-7
Llama-3- Instruct (ArmoRM)
0.01
10-7
Gemma-2-Instruc
0.01
x 10-7
SimPO
Mistral- Instruct
2.5
0.1
x 10-7
Alama---Innsrrus
2.5
0.55
10-6
Llama-3-Instrun (ArmoRM)
10.0
0.3
10
Gemma-2-Instruct
10
0.5
x 10-7
AlphaPO
Mistral-Instruct
0.25
2.5
0.1
7 10
3lama-3-Inntrrue
0.25
2.5
1.0
1 10
Llama-3-Instreee (ArmoRM)
0.25
10.0
0.3
1.1 10-6
Gemma-2-Instruct
0.1
10
0.5
10
Decoding hyperparameters For AlpacaEval 2.0, we adopt the default settings for AlpacaFval 2.0 with 4.turbo the annotator and use gpt 4_turbo as the reference model. We sampling decoding strategy to generate responses, with temperature of 0.5 for the Mistral-Instrus setting and temperature of 0.9 for Llamm-3- settings following from the SimPO configs. We use tempreture of 0.7 following from the WPO-HB config for better reproducibility. For Arena-Hard, we use the default greedy decoding for all settings and methods.
Computation All the training experiments in this paper were conducted on 8xA100 GPUs with the adamw torch optimizer based on the alignment-handbook. The training time for Mistral-Instrue and -lama-3-Instruus I-sttrutt is around 2.3 hours, while Gemma-2 1ommaa--IInstruel requires 3 hours.
Open Sourced Models used in Experiments The list of open-sourced LLMs used in our experiments and their Hugging- face IDs are listed in Table 4.
Model
Huggingface ID
Mistral-Instrun SFT Llama-3 Hoima-3-Instren InIstruttt SFT
mistralai/Miitral--TT htnp:iisrrraaa----nnutt-t-2222
Meta-llama//Mta-L googlelgemma-229bbii
Gemma-2-Instren SFT
Table 4, List open-source models used experiments.
18

---

AlphaPO: Reward Shape Matters for LLM Alignment
A.7. Detailed results for AlphaPO and SimPO
The full results of the best runs are reported in Table 5.
Table 5. Detailed results of AlpacaEval and Arena- Hard for the best run. LC means length-controlle win rate, WR means raw win rate, STD means standard deviation of win rate over different instructions. Length the average generation length. For Arena-Hard, we report the win rate and 95% confidence interval.
AlpacaEval 2
Arena-Hard
Models
LC (%)
WR (%)
STD (%)
Length
WR
95 CI high
95 CI low
Length
Mistral--B-Instruet
DPO
21.96
22.16
1.25
2034
14.8
16.4
13.1
667
SimPO
29.71
31.12
1.37
2330
21.5
23.6
19.8
551
AlphaPO
33.03
34.12
1.40
2097
21.7
24.0
19.9
503
Llama-3-8B-Instrucc
DPO
39.18
36.84
1.40
1885
34.1
35.9
31.5
539
SimPO
42.05
36.90
1.42
1759
32.8
34.9
30.5
485
AlphaPO
45.37
40.97
1.45
1820
34.6
37.1
32.9
508
Llama-3-8B-Instrun
(ArmoRM)
DPO
47.19
45.61
1.47
1974
33.4
38.0
35.5
587
SimPO
51.66
46.54
1.47
1806
35.4
37.8
33.3
515
AlphaPO
53.01
47.64
1.47
1794
36.0
38.7
33.9
507
Gemma 2-9B-Instruct
DPO
67.83
65.77
1.39
2042
60.7
63.3
58.3
729
SimPO
73.72
67.36
1.37
1832
59.0
61.5
56.7
720
AlphaPO
74.13
65.89
1.41
1802
57.7
60.4
55.6
689
A.8. Win Rate Heatmap
In Figure 5, we create AE2 win rate heatmaps of AlphaPO and SimPO for the Mistral instruct and Llama 3 instruct models.
Observations on Win Rate: It is observed that the percentage of instances where AlphaPO outperforms the base GPT model, but SimPO does not, significantly higher compared to the percentage of instances where SimPO outperforms the base GPT model, AlphaPO does not.
100
100
SimPO
A < B
57.5%
12.3%
SimPO
A < B
51.7%
12.9%
50 de
50 8
1
A B
8.2%
22.0%
A
8.1%
27.3%
B
A AlphaPO
A
A A AlphaPO
(a) Mistral-Instruct.
(b) Llama-3 I-ntruct.
Figure 5. Win rate heatmap of Mistral-Instruct and Llama-3-Instruct on AlpacaFval 2. B represents the reference model (i.e.,
gpt 4.turbo).
19

---

AlphaPO: Reward Shape Matters for LLM Alignment
A.9. Training Details of SPPO
We use a 20k sampled subset from the UltraFeedback dataset (Cui al., 2024) and follow the same hyper parameter setting in the SPPO except for changing the learning rate and number of epochs. We set the learning rate to be 10 with linear warm up and decay and beta to 0.001 for both methods. We use 6 epochs for SPPO and 4 epochs for AlphaPO.
A.10. Illustration of the Non-Monotoniss of [0e/aul
To illustrate the non-monotonic behavior of the gradient magnitude (āe/uul, examine simple example with
0m
5, 7 0, y 1yrl
1
log Tw -5, log मं
-10.
Ov
dr
Graddent
Length Lake
2
33
-4
Figure 6. Surface plot of the gradient magnitude [āε/au| (z-axis, log scale) as function of the parameter a (z-axis) and the common norm y Iywl ly:] (g-axis).
A.11, Extra plots
20

---

AlphaPO: Reward Shape Matters for LLM Alignment
Stetssiis
1.5
Median
Whisker
Preferred
Dispreferres
1.0
Margin
-+-- =
-Q-. 0 0.25
-g- 0=2
-0.5
-1.0
-.0
0.6 Epoch Number
044
0.8
0.0
0.4
0.8
0.8
0.2
0.4
0.6
1.0
Epoch Number
Epoch Number
preferred likelihood, dispreferred likelihood and margin over an epoch for the Gemma instruct model. The min and max computed after removing outliers.
Figure 7. Tracking various statistics for
Mistral 7B Instruct
Llama 8B Instruct
Gemma 9B Instruct
34
47
76
33
45
74
Win rate
32
43
72
31
41
70
30
29
39
68
28
37
66
-.5
-0.25
0
0.25
0.5
0.75
-0.5
-0.5
0.25
0.5
0.75
-0.5
-0.25
0.25
0.5
0.70
Figure Effect tuning for AlphaPO. The highest AE2 length-controlled win rate achieved for >> 0.1 Gemma2 0.25 for the other two models.
Mistral 7B Instruct
Llama 8B Instruct
Gemma 2 9B Instruct
34
2,200
50
1,900
80
2,300
2,200
Sanee esuodsae
33
70
win rate
2,150
45
1,850
2,100
32
60
2,000
2, 100
40
1,800
31
50
1,900
30
2,050
35
1,750
40
1,800
0
0.25
0.5
0.75
0
2.5
5
7.5
5
10
15
Y
7
Figure 9. Effect of margin parameter on AE2 length--ontrooled win rate and response length.
21

---

AlphaPO: Reward Shape Matters for LLM Alignment
0.6
SimPO
SimPO
AlphaPO
AlpacaBval 2 LC
30
AlphaPO
KL Divergence
0.4
25
0.2
20
100
200
300
400
500
100
200
300
400
500
Training Steps
Training Steps
(a) KI. Divergence evolution during training compared to orig-
(h) AlpacaEval length-controlled win rate evolution with training steps.
inal reference Mistral instruct model.
Figure 10. Side-by-side comparison of two metrics for Mistral 7B Instruct Model: KL Divergence (left) and AE (right).
SimPO
SimPO
AIphaPO
AlphaPO
AlpacaEval 2 LC
KL Divergence
40
0.5
30
0
100
200
300
400
500
100
200
300
400
500
Training Steps
Training Steps
(a) KL. Divergence evolution during training compared to orig-
(b) AlpacaEval length-controlled win rate evolution with training steps.
inal reference Mistral instruct model.
Figure Side-by-side comparison of two metrics for Llama3 Instruct Model: KL Divergence (left) and AE (right).
Mistral Reward Distribution
Llama Reward Distribution
Mistral-7BB-Lnstreeee
Llama-3 B-B----------cc
0.1
SimPO
0.1
SimPO
AlphaPO
AlphaPO
Density
Density
10 -
10 10-2
-10
0
10
10
0
10
Reward
Reward
Figure 12. Comparison of reward distributions for PairRM for Mistral (left) and Llama 3 (right). AlphaPO yields reward distribution either on par or better (i.e., right-shifted) when compared to strong baselines SimPO.
22

---

AlphaPO: Reward Shape Matters for LLM Alignment
-2.0, epo
0.1 a=
-2.0, epo
0.3 a -2.0, epo
0.5 a -2.0, epo 0.7 a= -2.0, epo 0.9
0.0
LN log prob
-1.
-1.5
-2.0
0, epo
0.1
epo 0.3
0.5
0, 0.7
O, 0.9
0.0
-0.5
log prob
-1.5
-2.0
0.25, epo 0.1 0.25, epo
0.3 0.25, epo
0.5 0.25, epo 0.7 0.25, epo 0.9
0.0
log prob
00.
-.0
-1.5
a
2.0,
epo
0.1
2.0, epo
0.3
a 2.0, epo
2.0. epo
2.0, epo
0.0
LN log prob
-0.5
-1.0
-1.5
-2.0
0.5
0.0 Margin
-0.5
Margin
Margin
Margin
Margin
Figure 1.3. Scatter plot of length normalized preferred likelihood (y-axis) vs. length normalized margin (z-axis) for varying values of alpha the Mistral instruct model. a alpha parameter, epo epoch number. Outliers were moved before plotting. Clearly noticeable is the regularization effect of large absolute values of
23

---

AlphaPO: Reward Shape Matters for LLM Alignment
a= -2.0, epo 0.1 a -2.0, epo
0.3 = -2.0, epo
0.5 a -2.0, epo
0.7 -2.0, epo = 0.9
0-
LN log prob
-2
-3
0, 0.1
0, 0.3
epo 0.5
0, epo 0.7
O, epo
0.9
log prob
-3
0.25, epo
0.1 0.25, epo
0.25, epo
0.5
0.25, epo
0.7 0.25, epo
0.9
log prob
-1
-2
2.0, epo
2.0, epo
0.3
2.0, epo
0.5
a 2.0, epo
0.7 a 2.0, epo
LN log prob
-1
-2
-4
Margin
Margin
Margin
Margin
Margin
Figure 14. Scatter plot of length normalized preferred likelihood (y-axis) vs. length normalized margin (z-axis) for varying values of alpha Gemma instruct model. a the alpha parameter, epo is epoch number. Outliers were moved before plotting. Clearly noticeable is the regularization effect of large absolute values of
24

---

AlphaPO: Reward Shape Matters for LLM Alignment
log (y
log (y x)
log my ww
x)
log re(yy x)
1.0
[y]
0.5
-2
2
-0.5
-5
0.0
0.5
1.0
0 (SimPO)
0.5
-2
-2
-0.5
-5
-1.0
0.0
0.1
0.2
0.5
0.6
0.7
0.8
0.9
0.5
0.6
0.7
0.8
0.2
0.3
0.4
0.5
0.6
0.9
1.0
0.25
-2
-2
0.0
-0.5
-5
1.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8 0.9
1.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
1.0
-2
-2
a=7
0.0
-5
1.00
0.0 0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.0
0.1
0.2
0.3
0.4
0.5
0.7
0.8
0.9
1.0
0.0
0.1
0.2
0.3
0.4 0.5 0.6 0.7 0.8 0.9 1.0 Epoch
Figure 155. Mistral: Box plots illustrating the distribution of length-normalized preferred (dispreferred) likelihood and length-
margin across different values. The plot provides detailed view of variations scatter intensity and performance metries as function of a.
25

---

AlphaPO: Reward Shape Matters for LLM Alignment
log (y
log Ty x)
log re w
x)
log re(yy x)
Yvl
yj
2.5
[y]
2.0
1.5
1.0
-0.5
-1.0
-5
0.0
0.5
1.0
a= 0 (SimPO)
1.5
1.0
-2
-3
-3
-0.5
-1.0
-1.5
0.0
0.1
0.2
0.6
0.7
0.8
0.9
1.0
0.4
0.5
0.6
0.7
0.8
0.0
0.2
0.3
0.4
0.5
0.6
0.9
1.0
2.5
1.5
1.0
0.5
-3
-3
-0.5
-1.0
-5
-1.5
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
D.7
0.8
0.9
1.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
2.5
1.0
2
0.5
-3
-3
-1.0
-5
1.5
0.0 0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.0
0.1
0.2
0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Epoch
Figure 16. Gemma 2: Box plots illustrating the distribution of length-normalized preferred (dispreferred) likelihood and length-normalized margin across different values. The plot provides detailed view of variations scatter intensity and performance metries as function of a.
26


Reviews:
[{"review_id": "BxHj47eVZz", "reviewer": "Reviewer_G3bY", "summary": "The paper introduces AlphaPO, a variant of f-DPO that adopts $\\alpha$-divergence and length normalization. The paper shows that varying $\\alpha$ affects the shape of the implicit reward. With an appropriate value of $\\alpha$, AlphaPO can mitigate the over-optimization issue of Direct Alignment Methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> In section 3.3 the authors mention \"large values impose a regularization effect ...\n\nWe thank the reviewer for their insightful comment. The reviewer is right to point out that IPO and SLiC are important methods. The reasons we did not include them in the paper are (1) the SimPO paper already compares to SLiC and IPO, and demonstrates that SimPO is substantially better. (2) Both SLiC and IPO do not contain length normalization, which is key for generalization. (3) SLiC contains a separate term for log likelihood for the winning response, which can affect likelihood displacement (some likelihood displacement is actually necessary for good generalization). IPO, on the other hand, enforces its margin constraints solely via the L2-loss, which lacks a mechanism to encourage the controlled likelihood displacement. \n\nWe will update the paper to include these details in the draft.\n\n> Section 3.2 states that length normalization is a crucial element.....\n\nWe do want to highlight that we indeed study the effect of generation length on generalization (1) We highlight in Figure 3 (left panel) and Figure 8 that longer responses usually result in a lower quality (AE 2.0 length-controlled win rate) as we vary . (2) Section 2.2 of the SimPO paper comprehensively discusses that length normalization is a key element in preventing the reward formulation from resulting in a bias towards generating longer but lower-quality sequences. \n\nThe reviewer is right to point out that Illustration 1 uses a fixed length of 1. We create a few more illustrations with length > 1.  We create a 3D plot, with gradient norm in log scale on the z-axis, alpha on the y-axis and length of y_w, y_l on the x-axis. We use the following parameters for the plot - log_pi_w = -5, log_pi_l = -10, beta = 5, partial π_w / partial v = partial π_l / partial v = 1. The resulting plot can be found [here](https://i.imgur.com/cxM7PFX.png). From the plot (a) The gradient goes to zero whenever alpha goes to + / - infinity, as proved in Theorem 3.1 (b) In general, the gradient is not a monotonic function of alpha. \n\nWe will be happy to include this improved illustration in the camera-ready version. \n\n> AlphaPO introduces an additional hyper-parameter α. ....\n\nPlease see our response for Reviewer **2gDt** for a detailed treatment.\n\n> The paper mainly focuses on RLHF tasks. How does AlphaPO perform on reasoning tasks, e.g., math or coding?\n\nThank you for the question. Our evaluation is on **AlpacaEval 2.0 (AE2.0)** and **ArenaHard**. Both benchmarks include math and coding questions. Below are one example for each category from each benchmark.\n\n\n\n## AE2.0\n\n### Coding\n\n- **Question 1:**  \n  Write a C++ function that takes a reference to a `std::string` containing markdown formatted text and returns a `std::string` containing html formatted text.\n\n### Math\n\n- **Question 1:**  \n  Given that *f(x) = 5x³ - 2x + 3*, find the value of *f(2)*.\n---\n\n## ArenaHard\n\n### Coding\n\n- **Question 1:**  \n  I have a Python script that scrapes a webpage using Playwright. Now I want to start ten instances of that script in parallel on one AWS EC2 instance, but so that each script binds to a different IP address. How can I do that with Terraform?\n\n### Math\n\n- **Question 1:**  \n  What is the 95% confidence interval for the sum of 100 fair six-sided dice?\n\n\n\nBased on the reviewer’s questions, we conducted additional experiments:\n\n- **HellaSwag** – A commonsense reasoning benchmark.\n- **TruthfulQA** – Carefully designed questions to test models' susceptibility to common misconceptions and their factual accuracy.\n\nWe compare **AlphaPO** to **SimPO** on these datasets:\n\n### HellaSwag (10 Shot)\n\n| Model                  | AlphaPO | SimPO  |\n|------------------------|---------|--------|\n| Llama3-8b-instruct     | 0.7694  | 0.7576 |\n| Mistral-7b-instruct    | 0.8638  | 0.8610 |\n\n### TruthfulQA\n\n| Model                  | AlphaPO | SimPO  |\n|------------------------|---------|--------|\n| Llama3-8b-instruct     | 0.6142  | 0.6078 |\n| Mistral-7b-instruct    | 0.7127  | 0.7061 |\n\nBased on these results, it is clear that **AlphaPO outperforms SimPO**. \n\n> According to figure 2, the median of the margin stays around 0 ..\n\nIn Figure 2, we plot the margin of length-normalized probability, i.e., $\\frac{\\log \\pi_w}{y_w} - \\frac{\\log \\pi_l}{y_l}$, instead of the reward margin (with an additional factor of $\\beta$), to better track the training dynamics. That is why the absolute value of the margin is small. For $\\alpha=0$, our results align with the training curve open-sourced by the SimPO author [link](https://wandb.ai/yumeng0818/simpo/runs/4w25j650?nw=nwuseryumeng0818) for the Gemma2 model (the mistral training curve is not published by the authors).\n\nAt the end of training, the SimPO authors reported the average eval margin of 0.48 and our average eval margin is 0.528 and the median is 0.372 (Figure 6). Similarly, the positive but small margin of length-normalized probability is as expected.", "claims_and_evidence": "Yes.\n1. The authors perform extensive experiments over 2 RLHF benchmarks and 3 base models. The effects of crucial hyper-parameters ( $\\alpha$ and margin $\\gamma$ ) are studied.\n2. The authors theoretically analyze the effect of varying $\\alpha$ on the gradient scale and the likelihood of preferred responses.\n\nThere are a few questions though:\n1. In section 3.3 the authors mention \"large $|\\alpha|$ values impose a regularization effect on alignment training due to the vanishing gradient for samples with positive length-normalized margins\". Methods like IPO [1] and SLiC [2] also limit the margin between preferred and dispreferred responses through L2-loss or hinge loss. It would be better if the paper includes a discussion of these works.\n2. Section 3.2 states that length normalization is a crucial element. But the importance of length normalization is not thoroughly discussed in the paper. For example in Illustration 1, the length is fixed to be 1.\n\n[1] Azar, Mohammad Gheshlaghi et al. “A General Theoretical Paradigm to Understand Learning from Human Preferences.” ArXiv abs/2310.12036 (2023): n. pag.\n\n[2] Zhao, Yao et al. “SLiC-HF: Sequence Likelihood Calibration with Human Feedback.” ArXiv abs/2305.10425 (2023): n. pag.", "methods_and_evaluation_criteria": "Yes. The authors demonstrate that AlphaPO with an appropriate $\\alpha$ regularizes the magnitude of positive margin and thus mitigating over-optimization.", "theoretical_claims": "I have checked the main claims, which looks alright to me.", "experimental_designs_or_analyses": "The paper follows standard evaluation protocol of RLHF. The effects of crucial hyper-parameters ( $\\alpha$ and margin $\\gamma$ ) are studied.", "relation_to_broader_scientific_literature": "The paper studies the problem of over-optimization, a well known issue in Direct Alignment Algorithms.", "essential_references_not_discussed": "See \"Claims And Evidence\".", "other_strengths_and_weaknesses": "**Strengths**\n- There is a significant improvement on AlpacaEval 2 and Arena-Hard over previous methods like DPO and SimPO.\n\n**Weaknesses**\n- AlphaPO introduces an additional hyper-parameter $\\alpha$ . It seems that the optimal value of $\\alpha$ depends on the base model and must be found using grid search.", "other_comments_or_suggestions": "I don't have other comments for this draft.", "questions_for_authors": "1. The paper mainly focuses on RLHF tasks. How does AlphaPO perform on reasoning tasks, e.g., math or coding?\n2. According to figure 2, the median of the margin stays around 0. But a well-trained model should have a positive margin between preferred and dispreferred responses, if I understand correctly.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ojJ9XQne4v", "reviewer": "Reviewer_2gDt", "summary": "The paper introduces AlphaPO, a novel preference training algorithm designed to improve the alignment of large language models (LLMs) by modifying the shape of the reward function used in preference optimization. \n\nUnlike existing methods like Direct Preference Optimization (DPO) and Simple Preference Optimization (SimPO), which often suffer from likelihood displacement and reward over-optimization, AlphaPO leverages an \\(\\alpha\\)-parameter to adjust the reward function shape, allowing for better control over these issues. \n\nThe authors demonstrate that AlphaPO achieve improvements in alignment performance, with relative gains of 7% to 10% over SimPO and 15% to 50% over DPO on models like Mistral-7B and Llama3-8B. A comparable performance with SimPO is obtained on Gemma models.\n\nThrough gradient analysis and experiments, the paper shows that AlphaPO's reward function shapes influence the training dynamics, leading to better generalization and reduced over-optimization. The results highlight the importance of reward function design in LLM alignment and suggest that AlphaPO's approach can be extended to other alignment methods.\n\n\n## update after rebuttal\n\nMy position stilll stand as weak accept", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The main weakness is that the absolute improvement on SimPO is limited. If the paper can address the stability issues of SimPO like method (choice of hyper-parameter), it would be making a more impactful contribution.\n\nWe thank the reviewer for the comment. We would like to point out that our improvements over SimPO are actually significant and in-line with other well-established papers:- Reviewer **G3bY** pointed out that **AlphaPO** provides a significant improvement on **AlpacaEval 2** and **Arena-Hard** over methods like **DPO** and **SimPO**.\n- We also compared our gains to the gains of established, popular methods like **ORPO** [1] and **SimPO** [2].\n  - **ORPO** achieves a relative gain of ~10% over the **Zephyr** model, which was the SOTA model at the time of publication (Table 1 in the ORPO paper).\n  - From Table 4 of the **SimPO** paper, it is clear that many established methods such as **SLiC-HF** [3], **IPO** [4], **ORPO** [1], and **KTO** [5] perform at par or worse than both **DPO** and **SimPO** on many benchmarks.\n- Thus, improving upon **SimPO**'s performance is a challenging problem.\n\nWe thank the reviewer for the important comment about stability issues for SimPO related to choice of hyperparameters. We address this in the responses below.\n\n\n> Can you discuss more on how you made the hyper-parameter choices for your experiment. Given that the differences between your result is very close to SimPO, could hyper-parameters be the differing factor?\n\nAs discussed above, we would like to emphasize that our improvements over SimPO are significant and consistent with observations from other papers (please refer to the discussion above).\n\nFrom **Table 1** in our paper, we observe the following:\n\n- For **Mistral-Instruct**, AlphaPO achieves an LCWR of **33.03**, while SimPO results in **29.71**. This is a substantial improvement.\n- For **LLaMA-Instruct**, AlphaPO yields an LCWR of **45.37**, compared to **42.05** from SimPO — again, a significant improvement.\n\nTo tune AlphaPO, we started with the best-tuned SimPO baseline and then lightly tuned the `gamma` and learning rate (LR). In many cases, we did not need to modify `gamma` and LR at all. As for `alpha`, both theoretical insights and ablation studies indicate that extremely high positive or negative values suppress the gradient. In practice, a small positive value — typically around **0.1** or **0.25** — consistently improves performance over SimPO.\n\nWe believe AlphaPO could perform even better with more extensive tuning, which we were unable to pursue due to compute limitations.\n\nWe appreciate the reviewer's question and conducted an additional experiment to investigate further. Specifically, we took the SimPO-tuned hyperparameters, changed only `alpha` to a positive value, and trained an AlphaPO model. Note that this setup is not necessarily optimal for AlphaPO. The results are:\n\n**LLaMA-3-Instruct**:  \n\n- AlphaPO (SimPO HPs, `α = 0.25`) → **LCWR: 43.33** , **WR of 38.51**\n- SimPO→ **LCWR: 42.05**,  **WR: 36.90**\n\n**LLaMA-3-Instruct ArmoRM**:\n\n- AlphaPO (SimPO HPs, `α = 0.25`) → **LCWR: 52.91**, **WR: 47.21**\n- SimPO → **LCWR: 51.66**, **WR: 46.54**\n\nThese results demonstrate that simply modifying `alpha`, without changing any other hyperparameters, can lead to clear performance gains over SimPO.\n\n> Also, does reward shaping help with stabilizing training, because find right hyper-parameter for SimPO has been a pain point for people.\n\nTuning the hyperparameters for SimPO has long been a challenge. While the reward shaping introduced in our method does help stabilize training by regularizing the gradient (see Theorem 3.1), empirically, SimPO with α = 0 typically exhibits the most aggressive increase in the margin (as demonstrated in Figures 14 and 15). However, despite its stabilizing effects against reward over optimization, this does not always translate into improved generalization. Therefore, while it aids training stability, reward shaping does not fully alleviate the need for hyperparameter tuning to achieve optimal generalization performance. Reward shape helps to further enhance the generalization on top of the SimPO.\n\n\n\n[1] Hong, Jiwoo, Noah Lee, and James Thorne. \"Orpo: Monolithic preference optimization without reference model.\" arXiv preprint arXiv:2403.07691 (2024).\n\n[2] Meng, Yu, Mengzhou Xia, and Danqi Chen. \"Simpo: Simple preference optimization with a reference-free reward.\" Advances in Neural Information Processing Systems 37 (2024): 124198-124235.\n\n[3] Zhao, Yao, et al. \"Slic-hf: Sequence likelihood calibration with human feedback.\" arXiv preprint arXiv:2305.10425 (2023).\n\n[4] Azar, Mohammad Gheshlaghi, et al. \"A general theoretical paradigm to understand learning from human preferences.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2024.\n\n[5] Ethayarajh, Kawin, et al. \"Kto: Model alignment as prospect theoretic optimization.\" arXiv preprint arXiv:2402.01306 (2024).", "claims_and_evidence": "* Reward shaping affects the training dynamics, and a properly shaped reward can alleviate the likelihood displacement issue. \nThis claim is supported by analysis in Figure-2 tracking the likelihood for different alpha values.\n\n* The paper claims better generalizability after reward shaping for SimPO method. The claim is supported by superior results on widely acknowledged benchmarks in Table-1,", "methods_and_evaluation_criteria": "makes sense. AlpacaEval and ArenaHard are the right benchmarks for preference alignment.", "theoretical_claims": "I check the proof for Theorem 3.1, and it looks good.", "experimental_designs_or_analyses": "The experiment design is good.", "relation_to_broader_scientific_literature": "It relates to offline preference training literature.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "The main weakness is that the absolute improvement on SimPO is limited. If the paper can address the stability issues of SimPO like method (choice of hyper-parameter), it would be making a more impactful contribution.", "other_comments_or_suggestions": "NA", "questions_for_authors": "Can you discuss more on how you made the hyper-parameter choices for your experiment. Given that the differences between your result is very close to SimPO, could hyper-parameters be the differing factor?\n\nAlso, does reward shaping help with stabilizing training, because find right hyper-parameter for SimPO has been a pain point for people.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jR5g4eeWyI", "reviewer": "Reviewer_CYmG", "summary": "Authors step from likelihood displacement, a phenomenon in which both preferred and dispreferred responses drop likelihood during optimization. Derived from f-DPO, authors add length normalization to the alpha-divergence and show that alpha controls the likelihood displacement strengths through gradient analysis: larger alpha will be less aggressive.\n\nResults show that AlphaPO outperforms DPO and SimPO, with a proper positive alpha achieving the best performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The theorem does not provide proof ...\n\nWe acknowledge that the theorem can be made more clear. In particular, monotonicity can be proved for $T_1(\\alpha)$; see the summary table [here](https://i.imgur.com/aW2jjkf.png). In general, the gradient magnitude is not monotonic as presented in Illustrations. We present a 3D plot wrt the relation of the gradient, response length, and $\\alpha$ [here](https://i.imgur.com/cxM7PFX.png), which demonstrates non-monotonicity clearly.\n\nWe will update the draft with the proof of $T_1(\\alpha)$ monotonicity.\n\n> It is unclear why the range of alpha in ...\n\nIn order to derive an explicit formula for the reward as a function of $\\pi$ without normalization, Wang et al. require that 0 is not in the domain of  $f'(u)$, which constrains the allowable range of $\\alpha$. In contrast, our reward construction—similar to SimPO—is not based on divergences and is thus not subject to that same constraint. Moreover, the AlphaPO reward function involves length normalization—something Wang et al. do not consider. \n\n> Does the author apply hyperparam ...\n\nYes, we performed comprehensive tuning. The published HPs from the SimPO authors proved optimal for all three models, and our AE and AH results closely match.\n\n>Is the hyperparam searching ...\n\nHP search was done independently for every model. The instruct models use on-policy data, so datasets across models are not directly comparable (details in section 4.1).  \n\nHPs introduced in SimPO are required to be tuned for every model. However, extensive tuning of α is not required. See response to reviewer **2gDT** for additional details.\n\n> Reducing the learning rate ....\n\nAlthough small LR can mitigate the drop in the probabilities of preferred responses, it may result in worse generalization. We verify this by decreasing the optimal LR for Mistral to $6.0e-7$. Result - LC 29.36 / WR 30.28, compared to LC 33.03 / WR 34.12 obtained with LR = $7.0e-7$. A similar behavior was observed in the SimPO paper (Table 17) for the Gemma-2-9b model.\n\nFrom reference [2], the authors propose a smaller LR based on gradient analysis in the paragraph “On-policy sampling and small learning ....” Furthermore, they explicitly confirm: “A more comprehensive understanding of the training dynamic of the direct preference learning algorithms remains largely open.” - this is a key contribution of our paper.\n\nAdditionally, incorporating an NLL loss term [1] harms generalization because it inhibits likelihood displacement of preferred responses. Methods such as RRHF, SLiC-HF, CPO, and ORPO include an NLL loss term, yet their LC / WR scores in Table 12 are considerably lower than those of SimPO. Moreover, as noted in Appendix H of SimPO under the paragraph “Incorporating SFT regularization in SimPO,” while NLL regularization may benefit certain tasks, it leads to a decrease in performance on AE2.\n\nAdaptive Margin [3] is an interesting idea. We will acknowledge this in the paper.\n\n> The introduction of length normalization is intuitive but...\n\nThe reward function is designed—rather than derived from a divergence—based on its impact on training dynamics and generalization. This approach is similar to SimPO [1] and ORPO [2].  R-DPO [3] use length penalty - similar in spirit to [1][2]. Tulu3 [4] uses the length normalized variant of DPO successfully. Although length normalization is appealing and empirically beneficial, literature has not yet succeeded in deriving it as an emergent property from any specific divergence.\n\n**References:**\n[1] \"Simpo\" \n[2] \"Orpo\" \n[3] \"Disentangling length from quality in direct preference optimization.\"   \n[4] \"Tulu 3: Pushing frontiers...\"\n\n> Authors say that prior work (Wang et al 2024) shows that α ....\n\nDue to length normalization, the effective α’ in the fDPO exponent becomes α divided by response length. Based on Theorem 3.1 and our empirical study, a small positive α leads to improved generalization. In the context of fDPO, this translates to a tiny effective α’—especially considering that responses can be as long as 1000 tokens, which the authors of fDPO did not consider\n\nWe also refer the reviewer to Figure 2 of the SimPO paper which also applies to AlphaPO. Due to length normalization, AlphaPO, like SimPO, consistently achieves a positive reward margin for all response pairs (irrespective of length differences), and consistently improves the margin over the SFT model. Therefore, this is another key reason for AlphaPO doing better than without LN (fDPO)\n\n> Experiments are only conducted on chat tasks..\n\nDetails + additional experiments can be found in the response to Reviewer **G3bY**. \n\n> It is better to add label to y axis...\n\nWe have fixed this.\n\n> My main concern is that the AlphaPO is less...\n\nPlease see above our response to your specific questions on the theoretical aspects of AlphaPO. Also, please see our answer to Reviewer **G3bY** for experiments on reasoning and other tasks where AlphaPO still outperforms SimPO.", "claims_and_evidence": "### alpha in AlphaPO controls the strengths of likelihood displacement: larger alpha gives less agressiveness in likelihood displacement.\nEvidence: experiment figure 2 and Theorem 3.1.\n* The theorem does not provide proof of monotonicity. Instead,it analyzes the extreme cases where alpha reaches -inf or inf. Therefore, the claim is not fully supported by the theorem. However, the experiment looks good.\n\n### AlphaPO has less constraint on the range of alpha\nEvidence: Not found\n* It is unclear why the range of alpha in AlphaPO is wider, given its theory foundation is the same as Wang et al. 2024a\n\n### AlphaPO performs better than DPO and SimPO, and corresponding side claims in ablation study\nEvidence: Table 1, and other related figures\n* This is well supported by the experiment.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I check Thm 3.1, 3.2, and Cor 3.3. I read Thm 3.1 proof and had a quick glance at Thm 3.2 and Cor 3.3.\n\nThe proof is correct, to my knowledge. For the claim correctness, please see above sections.", "experimental_designs_or_analyses": "The experiment is overall sound and complete.\n\nSeveral things that are unclear to me:\n* Does the author apply hyperparam tuning for SimPO too?\n* Is the hyperparam searching applied to each dataset? Or do authors apply hyperparam searching on one dataset and apply it on another dataset without searching again?", "relation_to_broader_scientific_literature": "Prior works solve this problem either by adding NLL loss [1], reducing learning rate [2], and adaptive margin [3].\n\n\n[1] https://arxiv.org/pdf/2404.19733\n[2] https://arxiv.org/pdf/2409.02392\n[3] https://arxiv.org/pdf/2402.13228", "essential_references_not_discussed": "Reducing the learning rate can mitigate preferred responses probability drop[2], which is not discussed in the paper.", "other_strengths_and_weaknesses": "Other Strengths:\n* The gradient analysis is interesting and gives a picture of how AlphaPo is working.\n\nOther Weakness:\n* The introduction of length normalization is intuitive but less theoretical. Specifically, it is unclear what divergence the modified objective is representing, and whether it makes sense from the theory perspective.\n*  Authors say that prior work (Wang et al 2024) shows that $\\alpha$ does not bring much performance gain but their experiment shows that different $\\alpha$ affects results much. What causes the differences? Is it because of length normalization? More discussion are needed here.\n* Experiments are only conducted on chat tasks, making it unclear if the methods are still good for other tasks such as reasoning.", "other_comments_or_suggestions": "* It is better to add label to y axis in Figures 3 for easier understanding without reading texts\n\n\n## update after rebuttal\n\nI raise my score to 3 since the authors address my major concerns.", "questions_for_authors": "My main concern is that the AlphaPO is less theoretically convincing (See above reviews). Also, given that the experiment was conducted on chat tasks, it is hard to convince me that the AlphaPO is generally useful.\n\nI would be happy to raise scores if the authors can address my concerns.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: ## Summary
The paper you provided, titled **"AlphaPO - Reward Shape Matters for LLM Alignment"**, proposes a new method called **AlphaPO** that falls under the category of **Direct Alignment Algorithms (DAAs)** for aligning large language models (LLMs) to human preferences. The paper shows that varying $\alpha$ coming from the $\alpha-$divergence affects the shape of the implicit reward. The authors demonstrate that AlphaPO improves alignment performance, with relative gains of 7% to 10% over SimPO and 15% to 50% over DPO on models like Mistral-7B and Llama3-8 B. A comparable performance with SimPO is obtained on Gemma models.

## Decision 

Overall, the paper is well-written. The topic is timely and important. The reviewers have raised several concerns and provided feedback to the authors. The authors have done a good job during the rebuttal, addressing the concerns of the reviewers. Thus, some of the reviewers have increased their scores. The results reported in the paper are good; there is a significant improvement on AlpacaEval 2 and Arena-Hard over other methods like DPO and SimPO. The proposed approach is reasonable. I would recommend that the authors address the main issues raised by the reviewers during the rebuttal papers in the final version of the paper. This paper presents a new approach with a simple modification over the pre-existing SimPO loss; thus, I believe it is a meaningful contribution to the community. As a result, I recommend this paper for acceptance.
Decision: Accept (poster)