Information Theoretic Guarantees For Policy Alignment In Large Language Models
Authors: Youssef Mroueh
Apoorva Nitsure
Venue: Accepted by TMLR ()

Abstract: Policy alignment of large language models refers to constrained policy optimization, where the policy is optimized to maximize a reward while staying close to a reference policy based on an $f$-divergence like $\mathsf{KL}$ divergence. The best of $n$ alignment policy selects the sample with the highest reward from $n$ independent samples. Recent work shows that the reward improvement of the aligned policy scales as $\sqrt{\mathsf{KL}}$, with an explicit bound on the $\mathsf{KL}$ for best of $n$ policies. We show that this $\sqrt{\mathsf{KL}}$ bound holds if the reference policy’s reward has sub-gaussian tails. For best of $n$ policies, the $\mathsf{KL}$ bound applies to any $f$-divergence through a reduction to exponential order statistics using the Rényi representation. Tighter control can be achieved with Rényi divergence if additional tail information is known. Finally, we demonstrate how these bounds transfer to golden rewards, resulting in decreased golden reward improvement due to proxy reward overestimation and approximation errors.

Published in Transactions on Machine Learning Research (May/202)
Information Theoretic Guarantees For Policy Alignment In Large Language Models
Youssef Mroueh IBM Research
mrooue@hu..imm
Apoorva Nitsure IBM Research
Apoorva. Apoorva.Nitsurreee Witbsureutuurebmmmmmmmmmm
Reviewed on OpenReview: https: // openreview. norum? id- U29J7Riul
Abstract
Policy alignment of large language models refers to constrained policy optimization, where the policy is optimized to maximize a reward while staying close to a reference policy based on an f-divergence like KL divergence. The best of n alignment policy selects the sample with the highest reward from n independent samples. Recent work shows that the reward improvement of the aligned policy scales as vKL, with an explicit bound on the KL for best of n policies. We show that this vKL bound holds the reference policy's reward has sub- -abbsaassiinn tails. For best of n policies, the KL bound applies to any f-divergence through a reduction to exponential order statistics using the Rényi representation. Tighter control can be achieved with Rényi divergence additional tail information is known. Finally, we demonstrate how these bounds transfer to golden rewards, resulting in decreased golden reward improvement due to proxy reward overestimation and approximation errors.
1 Introduction
Aligning Large Language Models (LLMs) with human preferences allows tradeoff between maintaining the utility of the pre-trained reference model and the aligmment of the model with human values such safety or other socio-technical considerations. Aligument is becoming a crucial step in LLMs training pipeline, especially as these models are leveraged in decision making as well as becoming more and more accessible to the general public. Policy alignment starts by learning a reward model that predicts human preferences, these reward models are typically fine-tuned LLMs that are trained on pairwise human preference data (Christiano et al., 2017; Stiennon et al., 2020; Ouyang al., 2022; Bai et al., 2022). The reward is then optimized using training time alignment i.c via policy gradient based reinforcement learning leading to the called Reinforcemnent Learning from Human Feedback (RLHF) (Christiano et al., 2017). RLHF ensures that the reward is maximized while the policy stays close to the initial reference policy #ref in the sense of the Kullback-Leibler divergence KL(T]|Trer). Other variants of these training time alignment have been proposed via direct preference optimization (Rafailov et al., 2024) (Zhao et al., 2023) (Ethayarajh et al., 2024). Another important paradigm for optimizing the reward is lest time alignment via best sampling from the reference policy and retaining the sample that maximizes the reward. The resulting policy is known as the best of n policy. The best of n policy is also used in controlled decoding settings (Yang & Klein, 2021; Mudgal et al., 2023) and in fine-tuning LLMs to match the best of n policy responses (Touvron et al., 2023).
Gao et al. (2023) and Hilton & Gao (2022) studied the scaling laws of reward models optimization in both the RL and the best of n selups. Gao et. al. (2023) distinguished between "golden reward" that can be thought of as the golden human preference and "proxy reward" which is trained to predict the golden reward. For proxy rewards Gao et al. (2023) found experimentally for both RL and best of policies that the reward improve- ment on the reference policy scales as / /KL(T||[Tee)) Similar observations for reward improvement scaling in RL. were made in (Bai et al., 2022). For golden rewards, Gao et al. (2023) showed for both RL and best of n.

---

Published in Transactions on Machine Learning Research MMy/2223)
policies that LLMs that optimize the proxy reward suffer from over-optimization in the sense that as the pol- icy drifts from the reference policy, optimizing the proxy reward results in deterioration of the golden reward.
This phenomena is referred to in (Gao et al., 2023) (Hilton & Gao, 2022) as Goodhart's law. A qualitative plot of scaling laws discovered in (Gao al., 2023) is given Figure 1. For the best of n policy, most works log(n) -=1 (Stiennon et al., 2020; Coste et al., 2024; Nakano
in this space assumed that (L Tret) et al., 2021; Go et al., 2024; Gao et al., 2023). Recently Beirami et al. (2024) showed that this is in fact an inequality under the assumption that the reward is one to one map (a bijection) and for finite alphabets.
Mit Rewards for Different Modils
Lo rossee Lange model
Pracy rewa's Lange Mood
50
43
2.0
1.5
1.8
20
80
100
K00 may
Figure 1: Qualitiative plot of centered rewards vs. KL of Proxy and Gold Rewards for both Best of n and RL policies. (See Fig. a) and b) in (Gao et al., 2023) for scaling laws in policy alignment).
The main contributions of this paper are:
1. In Theorem 1 (Section 2), we provide a new proof for the best of n policy inequality KL(T]] Tret) S log(n) - n Beirami et al. (2024)'s setup of one-to-one rewards and finite alphabets to surjective rewards and beyond finite alphabots, also giving conditions for equality and generalizing to f-divergences and Rényi divergences.
2-1 showing it results from the data processing inequality of KL. We extend this beyond
2. In Section 3, show that the policy improvement scaling laws in (Gao et al., 2023) are information- theoretic upper bounds, derived from transportation inequalities with KL under sub-gaussian reward tails. We discuss how the KL dependence is driven solely by the reward tails and can only improve they are fatter than sub-gaussian (c.g., sub-gamma or sub-exponential)
3. In Theorem 4, we examine the tightness of these bounds when the optimized policy's tails are known, deriving new transportation inequalities for Rényi divergence D, for a E (0, 1). We show that the /KL upper bound cannot be met, reflecting Goodhart's law (Gao et al., 2023).
4. We show in Section 3.3, that VKL policy improvement upper bounds for best of n polices with bounded and calibrated rewards are loose and lighter upper bounds can be obtained leveraging our best of n upper bounds for the total variation distance.
In Section 4, we study the transfer of transportation inequalitics from proxy to golden rewards, proving that golden reward improvement is limited by the proxy reward's overestimation, as reported empirically in (Gao et al., 2023).
Our work answers positively the following question:
"Without performing alignment, can we predict on upper bound on the expected reward for given KL level given the distribution of the reword of the reference model?" Indeed from the tails of the reward under the reference model can predict the upper bound for alignment as shown in Figures 7 and 8.
2

---

Published in Transactions on Machine Learning Research (May/2025)
2
The Alignment Problem
2.1 RLHF: A Constrained Policy Optimization Problem
Let A' be the set of prompts and y be the set of responses y E y from a LLM conditioned on a prompt # E X. The reference LLM is represented as policy ( ie as a conditional probability on given a prompt x E X. Let px be distribution on prompts, and a a reward, r: x: x + E, represents a safety or alignment objective that is desirable maximize. Given a reference policy wet, the goal of alignment is to find a policy x' that maximizes the reward and that it is still close to the original reference policy for some positive A >> 0:
Ty/l arg max Ez-px B (y
Hyla
s.t /
SA
(1)
where ([(ae))))]l = (y log
Teat(|/o3) With some abuse of notation, we write T(x, y) = x (v)))xx(x) and (re y) = (orr( px))). Tet. P(x x y) be joint probability defined on x y that has
px marginal X. Hence we can write the alignment problem equation 1 in more compact way follows:
/ |dd s.t (L(T]|TTee) s A. sup xEP(Ax))
(2
For > 0, we can also write penalized form of this constrained policy optimization problem as follows!
/ rdw 1 (TTTT|TTre)
sup mEP(Xx>)
It is easy to see that the optimal policy of the penalized problem is given by:
exp(yr(1, Tret (yl=r)
(yk=)) J exp((r((x v) dort (ulæ) pxalmost surely.
(3)
The constrained problem equation 2 has similar solution (See for egg (Yang et al., 2024)):
Aa A)ao( exp( ((xx,))) The (Ulxz)
Axalmost surely,
(4)
exp(
((a-v) Jd
As
where Aa > is a lagrangian that satisfies fx KL(TA (xl== (il =A A.
2.2
Best of Policy Alignment
Let X be the random variable associated with prompts such that Law(X) = px. Let Y be the random variable associated with the conditional response werl given X. Define the conditional reward of the reference policy:
R(Y) == r(X,Y( rXx,,) whore (re((- we assume that X admits CDF denoted as FRiYijx and let FR be its quantile:
F (-1) (p) = inf Fnaxxx()) 2 for p [0.
Let .....
... Yn be independent samples from (.)() We define the best of n reward as follows:
R(n) (Y) max R(Y)) 1=1...
(5)
this the maximum of iid random variables with common CDF FR(Y)X- The best of n policy corresponds to (x) arg maxi=l.... r(X, Yt). We note x(n))
(.IX) the law of y [ (n) is referred to as the best of alignment policy. We consider two setups for the reward: 7
The regularizer 1 is usually referred as B.
3

---

Published in Transactions on Machine Learning Research (May/2025)
Assumption 1 h IR + y such that (r((
We assume that the reward r is one to map for fixed 2, and admits an inverse
This assumption was considered in (Beirami et al., 2024). Nevertheless this assumption is strong and not usually met practice, we weaken this assumption to the following:
Assumption 2 We assume that there stochastic map Hx such that Hx(Ry) Y|X and (x(B)o) x
Under Assumption the reward can be surjective which is more realistic but we assume that there is stochastic map that ensures invertibility not point-wise but on distribution level. Our assumption means that we have conditionally on X: R|X + Y|X form markov chain i. exists A,Y X) that Pr|x- R,, PRIx, and Pyooxx A(Y Prmpp- Nole that Assumplion 2, will always hold from Bayes rule as long as this Markov kernel A is well defined.
Best of n Policy KL Guarantees: A reduction to Exponentials and Data Processing Inequalities In what. follows for random variables 7, 7' with laws Pz,P2 we write interchangeably: KL p2) = KL(Z)
Our goal in this section is to relate the Kullback--eiblerr (KL) divergence (L(X(")|XXXX) which measures the difference between the best of n policy (X())X) and the reference policy (Y|X), to (Y)))((()))) which quantifies the divergence between the reward distribution under the best of n policy (Y))) and the reward distribution under the reference policy (R(Y)[X). Another objective to express this latter divergence as a function of n. To achieve this, we will leverage a powerful tool in information theory known as Strong Data Processing Inequalitics (DPI).
In information theory, the Data Processing Inequality (DPI) (Polyanskiy & Wu, 2023) establishes a funda- mental limit how information evolves under transformations. It states that the Kullback-Leibler (KL) divergence between two random variables, U and U" cannot increasc after applying the same transformation to both. This transformation can be either deterministic function h or probabilistic mapping, commonly referred to as a "channel" and denoted by = Pviu- Formally, DPT asserts that:
KL(VIIV') < KL(UIJU").
(6)
where V and V are the transformed versions of U and U' respectively. This ensures that information content, as measured by KL divergence, cannot increase through processing. DPI is tight if h is a one to one function (Polyanskiy & Wu, 2023).
U
V
h or Pviu
U'
V'
KL(VIIV") S KL(UIIU")
Figure 2: Data Processing Inequality for the KL divergence.
Under Assumption 1, the reward function is a bijection, allowing us to transform the rewards R!n) (Y)JX- ((Y(")) and R(Y)JX into ( and Y|X via the inverse mapping hx of r(X..). On the other hand, under Assumption 2, this transformation is achieved through the Markov kernel A({)R,

---

Published in Transactions on Machine Learning Research (May/202)
These transformations illustrated in Figure 3. By applying the strong Data Processing Incquality for the KL divergence, we obtain:
KL
T (r.zeell - (L((X")/XXI)>> & KL((BM)))IR
(7)
Inequality equation 7 is tight under Assumption 1.
(Y)
you
= lx or A(Y R)
R(Y)
Y|X
Figure 3: From rewards to outputs via r-1 under Assumption 1 or via the stochastic map (Markov kernel A) under Assumption 2.
Turning to (T)R(((()[[((())))) our goal is to bound or express it as function of n. To achieve this, we aim to leverage the Data Processing Inequality (DPI). Specifically, we seek to transform a pair of random variables, for which the KL divergence has a known analytical expression in terms of n, into R(n) (Y)) and R(Y)|X, respectively.
Transporting one random variable to another is at the core of optimal transport theory. In the following, we demonstrate that exponential random variable E and the maximum of n i.i.d. exponentials, E(m), allow such transformation via the optimal transport map.
Let E ~ Exp(1). The optimal transport map Tx = FRrix Fe from the exponential distribution E to R(Y) (sce, for example, Theorem 2.5 in (Santambrogio, 2015), noting that E is atomless while R(Y) can take discrete values) allows to express:
R(Y) & (E))),
(8)
where denotes equality in distribution. This representation illustrates that the target reward distribution can be generated through the optimal transport map Tx = =FRryxx o FE as illustrated below:
FE(e)
FR (u)
Exponential E ~ Exp(1)
U ~ Uniform(O,
Target R.
Figure 4: Optimal tansport map from an Exponentially distributed random variable E to the random variable Rt representing the reward.
On the other hand, let R (Y)X)) S ... < R(n) (Y)] be the order statistics of the rewards of n independent samples Y, 1...n, n, Y, (-)) The order statistics refer to sorting the random variable from the minimum (index (1)) the maximum (index (n)). Consider n independent exponential E E,, where E, FIp(1), and their order statistics E < ((2) .... F(n), The Rényi representation of order statistics

---

Published in Transactions on Machine Learning Research (May/2025)
(Rényi, 1953), similar to the Optimal Transport (OT) representation allows to express the distribution of the order statistics of the rewards in terms of the order statistics of exponentials follows:
(Y) Rd (X)))
[Rin)x Fe.. FRIOIx FF(E
(9)
The central idea in the Rényi representation is that the mapping Tx = FR(Y)X FF is monotonic and hence ordering preserving and by the OT representation each component is distributed as R(Y) See (Boucheron & Thomas, 2012) for more account the Rényi reprosentation of order statistics. Note that could have used uniform random variables instead of exponential, Rényi representation exponential order statistics have nice properties.
exponentials to stay faithful to
Thanks to the mapping Tx, we can transform an exponentially distributed random variable E as well as the maximum of n iid Exponentials E(n) to X and (Y) respectively. This transformation is illustrated Figure
E(n]
(Y)
Tx
E
R(Y)
Figure 5: Transforming exponential and maximum of exponentials to the reward and maximum reward random variables via the optimal transport map Tx, by virtue of the OT representation equation 8, and the Rényi representation equation 9 respectively. We recognize Tx as the "channel" in Data Processing Inequalities.
Hence using the OT representation in equation 8 and the Rényi representation of the maximum equation 9 and the data processing inequality, we can bound the KL between the rewards to KL on functions of exponentials and their order statistics:
KL (L(R(T) (M)R(()))) (E
KL(E(") E),
(10)
the tightness of this inequality depends on the properties of the map Tx and the space V. Hence chaining the two data processing inequalities given in equation and equation 10 we obtain finally
KL Teree
[Ter| < KL (L (Y)II(())) < (L(E(")|LE)
(11)
tightness of each inequality depends on the tightness each DPI.
The following Lemma gives a closed form expression for KL(E(")[E):
Lemma 1 (KL Between Exponential and Maximum of Exponentials) Let E Exp(1), and Ei.... En be iid exponentials and (()) their marimum, we have:
KL(E(1) E) log(n) n
(12)
n
The following result establishes the the best of policy KL expression bounds in terms of n

---

Published in Transactions on Machine Learning Research (May/2025
Theorem The best of n policy satisfies under (i) Assumption 1 (reward to one) and for finite y under (1i) Assumplion 2 (existence of stochastic "inverse")
(le S (L(E log(n) n
(13)
Under Assumplion 1, for infonite y and assuminn FR(YIX) is continuous and strictly increasing for all X we have:
KL KL (T (leelllTrett) = (L(E = log(n)
(14)
n
Divergence
f(z)
Bound on Dy r,
TTof
KL
x: xlog(())
log n
Chi-squared
(x - 1)²
[n
1)
2n
Total Variation
f(x) = x 1
(
Hellinger distance
(1 )x)
[1
1
Forward KL
log(())
71
log(ng
a Rényi Divergence
NA
log
12
[a-
a(n 1)+1
Table 1:
Best of n policy J-Divergence and a Rényi Divergence Bounds.
Beirami et al. (2024) showed this result under condition (i) which is not a realistic setting and used the finiteness of y to provide direct proof. Our analysis via chaining DPI and using OT and Rényi repre- sentations to reduce the problem to exponentials allows us to extend the result to more realistic setup under condition (ii) i. the existence of a stochastic "inverse', without any assumption on V. Furthermore our analysis sheds the light on the underlying assumptions of invertible reward and continuous space y that was implicitly made previous works that stated this bound as an equality (Stiennon et al., 2020) (Coste et al., 2024; Nakano et al., 2021; Go et al., 2024) (Hilton & Gao, 2022) (Gao et al., 2023). Singh al. (2025) recently built this result to analyze the KL divergence of best of in diffusion models.
Our approach of reduction to exponentials using Rényi representation of order statistics and data processing inequalities extends bounding the f-divergence D ( [rer well the a Rényi divergence. The Rényi divergence for (0, 1) U (1, oo) is defined as follows:
(.(I))) (a 1a-1) log (fD(
the limit - 1 coincides with KL, ie:: D2(PI10)) = KL(PIIQ). These bounds are summarized in Table 1. Full proofs and theorems are in the Appendix.
Best of n-Policy Dominance on the Reference Policy. The following proposition shows that the best of n policy leads to an improved reward on average:
Proposition 1 R(m) dominates R in the first order dominance that is R(n) dominates R on all quantiles: (t))(() QR(1) Vt e [0. 1]. It follows that have ER(n) > ERR.
Best of Policy and RL Policy The following proposition discusses the sub-optimality of the best of n policy with respect to the alignment RL objective given in equation 1:
Proposition 2 Assume bounded reward in [-M, For A > O and exp(A) the best of policy (n)
W "rref and the A Construined RL policy TAair (given in equation 4) satisfy:
V2TM 22 - 1)
exp (-1)
A
Xa

---

Published in Transactions on Machine Learning Research (May/202)
A similar asymptotic result appeared in (Yang et al., 2024) for A 5o, showing as oo, - O, we provide here a non asymptotic result for finite n and finite A. This result shows that the best of policy can be seen as approximation of the RL policy for n exp(A), where A is the desired KL level.
3 Reward Improvement Guarantees Through Transportation Inequalities
Notations Let X be real random variable. The logarithmic moment generating function of X is defined as follows for À E R: dxx (A)= = log ExxxxXEx)) X said to be nab-Ganssian with variance o² if ()x(A) X²a for all À E R. We denote SubGauss(a") the set of sub-Gaussian random variables with variance of X is said to be sub-Gammn on the right tail with variance factor o² and scale parameter >> if X²o² (A) S I[1-exylor every À such that 0 À < J. We denote SubGamma(o", .) the set of left and right tailed sub-Gamma random variables. Sub-gamma tails can be thought an interpolation between sub-Gaussian and sub-exponential tails.
Scaling Laws in Alignment Tt. has been observed empirically (Coste et al., 2024; Nakano al., 2021; Go et al., 2024; Hilton & Gao, 2022; Gao et al., 2023) that optimal RL policy Aaai satisfy the following incquality for constant ofet estimated from data
E r - Br S
V k KL (ITaAt.rlllTm
A similar scaling for best of n policy
V 20 logn 12 - 1
E (n) r Br,.e &
and those bounds are oftentimes tight even when empirically estimated from samples.
The case of Bounded Rewards and Pinsker Inequality This hints that those bounds are information theoretic and independent of the alignment problem. Indeed if the reward was bounded, a simple application of Pinsker incquality gives rise to /KL scaling. Let TV be the total variation distance, we have: TV(T, Rret)
1 sup]|i|la Si Ear ExTer r S / [KLL(T||TT (vrr). Hence we can deduce that for bounded rewards r with norm
infinity ||r|l that:
Esr E S 1 KL (Tet).
Nevertheless this boundedness assumption on the reward not realistic, since most reward models are unbounded: quoting Lambert et al. (2024b) "implemented by appending a linear layer to predict one logit or removing the final docoding layers and replacing them with a linear layer" and hence the reward is unbounded by construction. We will show in what follows that those scalings laws tied to the tails of the reward under the reference policy and instances of transportation inequalities. Note that the reward can be rescaled and transformed to be bounded, nevertheless Pinsker incquality remains loose, with transportation inequality we aim at replacing the || by a second moment or a standard deviation a, and typically if a & [|r|]so this leads to tigther bounds.
3.1 Transportation Inequalities with KL Divergence
For policy и E P()) and for a reward function r y RR, we note ryz, the push-forward map of æ through r. The reader is referred to Appendix D.1 for background on transportation inequalities and how they are derived from the so-called Donsker- Varadhan variational representation of the KL divergence. The following Proposition hinges on Lemma 4.14 in (Boucheron et al., 2013)):
Proposition 3 (Transportation Inequalities) The following inequalities hold depending on the tails of
Tel
8

---

Published in Transactions on Machine Learning Research (May/2025)
1. Assume that ryaree E Sub For any E P())) that is absolutely continuous with respect to Tref, and such that KL(
[ (ret)
< oc then we have:
F R r| s 20 (T((l(TTT)
Assume that ret E SubGamma(dPet c). For any x E P(Y) that is absolutely continuous with respect lo Weef, and such that [L(TI||Trod then have:
E - Em rl & 20 KL KL Tret) + cKL(T] Tret)
In particular we have the Tollowing Corollary:
Corollary 1 (Expected Reward Improvement) If Ti E SubGauss(ofed) the following holds for the optimal RL policy TAa. and for the best of policy T.reet
For the optimal RL policy AAa. we have:
BRN Exrar S 2 (L( ((Tret) S / 2022 A.
For any feasible policy aRL of the RL problem equation (for example the one obtained via gradient descent), we have:
B Ex S v 20 KL K (L(TII S 20
3. For the Best of n policy aret under Assumption 2 we have:
0 E (m) r Euar S KL (maeelll Tret) S /2022 (og 1
A similar statement holds under sub-gamma tails of the reward of the reference model. Item (1) in Corollary shows that the KL provides an upper bound on the reward improvement of the alignment under subgaussian tails of the reference reward. Under subgaussian tails of the reference, this information theoretic barrier cannot be broken with better algorithm. On way to improve on the VKL ceiling is by aiming having reference model with reward that has subgamma tails to improve the upper limit to V KL+KKL or to subexponential tails to be linear in the KL. Item (2) can be seen as refinement on the classical /20 log(n) upper bound on the expectation of maximum of subgaussians see for e.g Corollary 2.6 in (Boucheron el. al., 2013). If in addition r is positive and for = Ty - EuTr" we have for <>> P(X: > f) 2 P(gl > t), where g ~ A/(0, of) (where of variance), then we have matching lower bound (n)
for 7y that scales with Voz log(n) for sulficiently large (See (Kamath, 2015)).
Remark 1 (Tightness of KL Transportation Inequality) If we assume that r is one to one and that rewards gaussian with same variance o² Ty _N,,,02 and TgT (((1'00) then have
[L(T|| = (L(rgalIrgTree) = (m-x')² Assume 2 nl, in that case we have: E,r- - Ennay 1 1 KL Fret), and the inequality is tight in this case.
28"
We turn now to providing bound in high probability on the empirical reward improvement of RL. The following Theorem gives high probability bounds for the excess reward when estimated from empirical sam- ples:
Thcorem (High Probability Empirical Reward Improvement For RL) Assume T: E SubGauss(o2 Let y and to 0. Let Ha. be the optimal policy of the penalized RL problem given in Equation equation 3. Let Riy and Ri 1... be the rewards evalualed al samples from

---

Published in Transactions on Machine Learning Research (May/2025)
Wy, and Tre-- Assume that the 7y-énnyi divergence Dy (1 and ll( both finite. The
mig e-((--1)60
following inequality holds with probability at least. 1 20
1 R
1
m
Dy [I (Tett KL My.l Tref) + 2to-
re S 22 (eett +
m
m
7
Note that in Theorem 2, we did not make any assumptions on the tails of Ty and we that this results in a biased concentration inequality with a non-negative bias D_(T,,lllset) =l(m,,-llTTat > 0. For the best of policy, if the reward positive and has folded normal distribution (absolute value of gaussians), Boucheron & Thomas (2012) provides concentration bounds, owing to subgamma tails of maximum of absolute Gaussians.
3.2 Tail Adaptive Transportation Inequalities with the Rényi Divergence
An important question on the tightness of the bounds rises from the bounds in Corollary 1. question by considering additional information on the tails of the reward under the policy т, and we obtain tail adaptive bounds that are eventually tighter than the one in Corollary Our new bounds leverage a variational representation of the Rényi divergence that uses the logarithmic moment generating function of both measures hand.
We answer this
Preliminaries for the Rényi Divergence The Donsker-Varadalh representation of KL was crucial in deriving transportation incqualities. In (Shayevitz, 2011) the following variational form is given for the Rényi divergence in terms of the KL divergence, for all a R
(1 (1-1)D.(PIIQ) inf aKL(R) 11--1(11-aa))0))) ()(L)R)
(15)
A similar variational form rediscovered in (Anantharam, 2018). Finally Donsker- Varadahn-addaaaaaaaaaaaa representation of Da was given in (Birrell et al., 2021). For all a E IR+ + 0, we have
(a(I() = sup kEH log (Epela-11)) log (E
(16)
where H caa- < 5f db <c). Birrell et al. (2021) presents direct proof of this formulation without exploring its link to the representation given in equation 15, we show in what follows an elementary proof via convex conjugacy, the duality relationship between equations equation 15 and equation 16.
Theorem For < a < 1 Equations equation 15 and equation 16 are dual of one another. For a they are Toland Dual.
We collect in what follows elementary lemmas that will be instrumental to derive transportation inequalities in terms of the Rényi divergence. Proofs are given in the Appendix.
Lemma 2 Let a E (0, (1, (c), and define H (hle (a-th- hdP) LP(P),, (00)) ( E (2)).. We have for all h E H and for E (0,1) oo)
J hhaP - |haP-
hiQ < 1 -. - a - log (ʃ (flla-Di hdP)) ap)
1
(fatt-Shhdl dQ)
+ log
Lemma 3 The following limit holds for the Rényyi divergence lim 1 D. (P(I() KL(Q)IP)
10

---

Published in Transactions on Machine Learning Research (May/2025)
Transportation Inequalities with Rényi Divergence. The following theorem shows that when con- sidering the tails of we can obtain tighter upper bounds using the Rényi divergence that is more tail adaptive:
Theorem (Tail Adaptive Transportation Inequalities) Let a E (0, 1). Assume ryt E SubGauss(a²) and Fre E SubGauss(ofe) then we have for all 10.
aot ( Trel)
B - Exr S 2(((1
a)oz + 22((1a))oz
(17)
a
KL(T) Tret) then the tail adaptive upper bound given in Equation equation 17 is tighter than the one provided by the tails of ree only i.c (LT] Note that this is possible because Da is increasing in E (0, (van Erven & Harremos, 2014), i.e (T((Tt) V [L(m||/rrr) and (1-a)e S 1. Note that taking limits + 0 (applying Lemma 3) and a + 1, and laking the minimum of the upper bounds we obtain:
In particular there exits a E (0,1) such that Da(a||Fret) I (I-a)o?+ook Tnt
og?
ao
Eg
Ess min(o? Reed (LTT|Trrr)) KL KL (Treer/=)))
this inequality can be also obtained by applying Proposition twice: the tails of and &ref respectively.
Another important implication of Theorem 4, other than lighter than KL upper bound, is that if we were to change the RL aligument problem equation 1 to be constrained by Da. a E (0, instead of KL, we may end up with a smaller upper limit on the reward improvement. This D, constrained alignment may lead to a policy that under- peater-erforms when compared to a policy obtained with the KL constraint. This was indeed observed experimentally in (Wang al., 2024a) that used constraints with a- divergences for a (0, 1) (that are related to Rényi divergences) and noticed a degradation in the reward improvement w.r.t policies obtained using KL.
3.3 TV-rransportation Inequalities for Bounded and Calibrated Rewards
While in the previous sections we focused on potentially unbounded rewards and analyzed their tail behavior to derive transportation inequalities linking rewards to KL Rényi divergences, in this Section, we turn to establishing sharp transportation inequalities for the best of n policy under bounded rewards. Such rewards may be inherently bounded, as in the case of win rates (Azar et al., 2024), or they may be obtained through transformation or calibration. For instance, one approach is to use the cumulative distribution function (CDF) of the reward under the reference model, Fraues to calibrate the reward of the best of n policy (Beirami al., 2024; Balashankar al., 2024; Nitsure al., 2024; Belgodere et al., 2024).
The total variation (TV) distance, being both an integral probability metric and f-divergence, satisfies the data processing inequality. Under Assumption 2, demonstrated in Table 1, have:
+
TV( (n) (Tett (
1
sup E Eo0 r T- Er,ar TV(E(")IE) [E)
2 |1
72
Hence, for any bounded reward:
1 -
E o
r - ErTer 22
12
We note that our bounds are translation-invariant. If E [0, 1], defining F P=r- -r-s such that F E (- 1 and || we obtain:
1
--
E (n) r
Engey
E
(a) F- Br,,e < 2 R.eeet
=
11

---

Published in Transactions on Machine Learning Research (May/2025)
In particular, by calibrating the reward using Fr. or, obtain bounded reward where (Fr, o is uniform. Since Breee (Pr,n rr) (by uniformity). we derive the following bound for the best of n policy:
E (n) (Fr or) rryed
TV(E
(18)
72
The upper bound in equation 18 is exactly TV(E(n)I and is Lighter than what would be obtained using the subgaussianity of the calibrated reward. Since (Frixtat o Tre SubGauss( {), Corollary (2) yiclds:
Discussion
E (n) (Fr rr) - log(n) "n1) ryref
KL(EW)))B)
(19)
Comparing equation 18 and equation 19, that TV provides tighter upper bound. By Pinsker's inequality:
TV(E(*ILE)
which shows that equation 19 provides looser bound. Moreover Pinkser incquality is known to be vaccuous whenerver KL > 1, tighter upper bound for KL can be derived using the Fhebagnolle-Huber inequality Polyanskiy & Wu (2023):
TV(E**ILE) & (1-e-KL(EEN))
Substituting KL(E(M) [E) from Lemma into equation 18, we obtain:
E
(Friszat rr) S Kyyoe
1 72
-
- loR(n)+
-
(20)
We illustrate these bounds in Figure for uniform and beta distributed random variables. We see that TV provides the sharpest upper bound followed by the Bretagnolle-Hubee upper bound and finally by the KL upper bound.
listsinhh11
dpussh A41-00
(a) Bounds for uniform random variable.
(b) Bounds for Beta distributed random variable
Figure 6: Information Thcoretic Upper Bounds for Best of n the bounded case.
4 Transportation Inequality Transfer From Proxy to Golden Reward
As we saw in the previous sections, the tightness of [L(T]||Taa) upper bound in alignment can be due to the tails of the reward the aligned policy ग (Theorem 4) and to the concentration around the mean finite sample size (Thcorem 2). Another important consideration is the mismatch between the golden reward r" that one desires maximize that is expensive and difficult obtain (for example human evaluation) and proxy reward that approximates r". The proxy reward is used instead of y in RL and in best of
12

---

Published in Transactions on Machine Learning Research (May/2023
policy. While may know the tails of the reward r of the reference and aligned model, we don't have access to this information on the golden reward r". We show in this section how to transfer transportation inequalities from r to r" for RL and Best of n policy.
Proposition 4 (r" Transportation Incquality for RL Policy) ) The following inequality holds:
(fat rdsteet f dduc)) dry..r))
BT Fi r S E Exi ?
log
Tre E , log (s (selv-r- Indtzett -f ddinet) dryy.0 2 8KL [| then we have:
Assume
SubGauss(o and there exists
0
such that:
KL B Ensarr" 2 ( Tret) ( Trer).
Note that log ( s [[Sritees f dry,.r
interpreted here as an interpolation between the mean and the maximum of its argument the support of Ny,-" (Proposition 9 in (Feydy et al., 2018)).
Indeed as + O, this boils down to the mean on J'r r"),,,,.- (J rdwıet - r _frrd and y - co this f dr=dd)tet) Our assumption means that r overestimates r" and the overestimation is accentuated as we drift from &eet which was learned. If r overestimates r", there exists A > O such that: r r" - (ʃ rdTrel f dTrett 2 A By Jensen inequality we have:
boils down to maXsuppa,,." fr rr (J rddtzet -
I log (fe( rd (=ddss))) do 2 1 (1(r-r'- (S (drrree f dTtrde
2 A. Hence our assumption is on the overestimation error A = ( This assumption echoes findings in (Gao et al., 2023) that show that the transportation inequalities suffer from overestimation of proxy reward models of the golden reward (See Figure 8 in (Gao et al., 2023)).
7
7
Note that in Proposition 4, we evaluating the golden reward r* improvement when using the proxy reward optimal policy ,y... We see that the golden reward of the RL policy inherits the transportation inequality from the proxy one but the improvement of the reward is hindered by possible overestimation of the golden reward by the proxy model. This explains the dip in performance as measured by the golden reward depicted in Figure 1 and reported in (Gao et al., 2023).
Proposition 5 (r" Transportation Inequality for Best of n Policy) Let E > O. Let r be a surrogate reward such that ||r -rrllllo S and assume TifTeef E SubGauss(o (o+)) then the best of n policy π (n) salisfies:
E (m) (r") - Em. (r*) < 200 (1og(n)---1 + 28 Fy.eet
n n-
(4 --1
Transportation inequalities transfers for the best of n policy from r to p" and pays only an additional error function of is given in Table
term || -r--re 1' TV (Tee) 1. As mentioned earlier, if we have lower bounds on the tail of the reference reward, then we also have a lower bound on the reward improvement that scales like C CV of log(n) 2s ((t)== (+)= This is in line with empirical findings in (Hilton Gao, 2022) (Gao et al., 2023) that showed that best of n policy is resilient as the reward model gets closer to r*
upper bound of this total variation
Practical Implications. As the proxy reward may overestimate the golden reward, (Wang al., 2024b) proposed reward transformation that reduces the tails of the reward distribution. This, in turn, improves the golden reward by avoiding shortcuts that exploit the reward and target the unreliable tails of the distribution, which overestimate the golden reward with high values. In our theory, this would reduce in Proposition 4, leading to less catastrophic Goodhart
5 Numerical Results
Prompts Dataset, LLMs and Reward Models We consider the attaq dataset (Kour et al., 2023) consisting of 1.4k prompts that triggers undesirable behaviors in LLMss. We use Reward model FSFAIRX-
13

---

Published in Transactions on Machine Learning Research (May/202)
LLAMA3-RM-VOL, (Dong et al. 2023; Xiong et al., 2024) this is among the best reward model for measuring helpfulness, safety, instruction following and lack of toxicity (Lambert et al., 2024a). We use three LLMss from each we sample with top-p sampling and a temperature 7 100 responses for each prompt, models are: MERLINITE (Sudalairaj et al., 2024) (a base model not aligned), MIXTRAL--X7E-INTRRCC (Jiang et al., 2024) and LLAMA-2-3BB-CM. (Touvron al., 2023) (aligned models with different reward).
Alrovrthh
n.0300
Figure Histogram of reward and best of n reward with -EALIRX--LLMA--REE
Moan N bes Moar Ne!
best
muxtal rosuud-vOL,
tempp,. oor.op..... ILina
Ia1ma-2-11b Ib-chat y 0.9 Jamo, best Jow
0.5
1.0
3.5
logtro
Figure 8: (y-axis) The centered reward using SSPAIRX-LLAMAB 1--MA------.1111 reward model best of n policy
Ey a T - Enge versus (x-axis) KL(TP.ret Tret) policy computed as log(n) - R-1, We also plot fitted curves me
y == a/z + bz. Coefficients are given in Tab. 3.
Results and Discussion We plot Figure 7 the histograms of the reward under the original LLM policy (first panel) and under the best n policy for 10, 50, 100 (second fourth pancls). We observe that the reference rewards are not heavy-tailed and follow sub- aub-Gaasaaanssbaaaanna provides tests such q-q plots to probe this). Table 2 presents statistics of these distributions. We see that
/ssu-Gammma distributions ( (appendix G
14

---

Published in Transactions on Machine Learning Research (May/202)
Mixtral achieves the highest mean reward using the best of n for 100, while Merlinite shows the greatest improvement. over the original policy.
This is due to the fact that Merlinite is a base model and not an aligned one. In Figure 8, we plot the centered best of n rewards against the KL divergence the best of n policy to the reference policy, and observe that. it follows the form ayT+ + ba, indicating a sub-Gamma tail as shown in Proposition 3. Looking the estimated coefficient in Table 3 and the standard deviation of the reward under reference models in Table (denoted all), we observe that is slightly larger than the standard deviation of the reward under the reference policy, which aligns with theory. If we apply Pinsker's inequality, the upper bound on the reward would scale with the maximum reward, also given in Table 2, which is much larger than resulting in a loose bound. Thus, the reward/Ke KL plots are governed by the tail properties of the reference policy.
Model and best of
Mean
Std
Max
MERLINITE
-1.42
2.66
9.91
MERLINITE n 10
2.33
1.81
8.44
MERLINITE 50
1.07
1.45
9.74
MERLINITE = 100
4.76
1.40
9.91
MIXTRAL ALL
1.96
1.86
12.57
MIXTRAL n 10
4.74
1.31
8.86
MIXTRAL 50
6.21
1.03
12.57
MIXTRAL n = 100
6.78
0.98
12.57
LLAMA M 3BCHAT ALL
1.98
1.58
9.92
LLAMA 3BCHAT 10
4.35
1.23
9.12
LLAMAI3BCHAT n = 50
5.74
1.09
9.92
LLAMAI3BCHAT n 100
6.31
1.04
9.92
Table 2: Comparison of Mean, Std, and Max values for the reward model PSPAIRX-LLAMA--RN-VVV.
evaluating samples from different models using various best of policies.
Model
Merlinite-7B
3.01309273
0.12227016
Mixttral8xx7--instrre 2.03057845
0.26265606
Llama 13B-chat
1.57822599
0.36367755
Table Best fitted curves coefficients y = a. + bz
Reward Versus Rényi divergence for Best of Varying n, we compute the centered expected reward of the best of policy versus the a Rényi divergence. The Rényi divergence of the best policy is computed using the expression given in Table 1. For a 10, for smaller values of the Rényi divergence we achieve higher reward than KL (See Figures 22, 23 and 24). For > 1, for same value of KL, achieve higher reward using KL than using the Rényi divergence for > 1, (See Figures 25, 26 and 27). This suggests that for same value of the divergence, a-Rényi divergence for a > may allow for less reward hacking. This observation was used in (Huang et al., 2024) using the Chi-Squared regularizer (a = 2) in addition to KL to fight reward overoptimization.
Calibrated Reward versus TV and KL We follow the notations of Section 3.3 for contered calibrated rewards using the CDF Fromer of the reward under the reference model. In Figure 9, we plot the centered calibrated rewards of the best of policy for the three LLMs considered using the reward model FSFAIRX- LLAMA3-RM-vLl against (a) the TV upper bound given in equation 18, (b) the / / upper bound given in equation 19, and (c) the VI-e -K upper bound given equation 20.
From Figure 9, we observe that (a) TV provides the sharpest upper bound, showing a linear correspondence between the centered calibrated rewards of the best of policy and TV. The next tightest bound is (c)
15

---

Published in Transactions on Machine Learning Research (May/2025)
Callerd Sowed viee
380
(a) Calibrated Rewards TV
(b) Calibrated Rewards V (c) Calibrated Rewards v1 e
Figure 9: (y-axis) The Calibrated centered reward using
reward model of best
of policy Exg, Fri or- 2 versus on (x-axis) (a) TV upper bound given in equation 18, (b) V kL upper KC upper bound given equation 20.
bound given equation 19 (c) v1 e-KC
e derived via the Bretagnolle- Huber inequality, followed by (b) V1KL, obtained via Pinsker's inequality, which, as expected, is the loosest among them.
6 Conclusion
We presented this paper a comprehensive information theoretical analysis of policy alignment using reward optimization with RL and best of n sampling. We showed for best of n a bound on KL under realistic assumptions on the reward. Our analysis showed that the aligument reward improvement, is intrinsically constrained by the tails of the reward under the reference policy and controlling the KL divergence results in upper bound of the policy improvement. We showed that the KL bound may not be tight if the tails of the optimized policy satisfy a condition expressed via Rényi divergence. We also explained the deterioration of the golden reward via overestimation of the proxy reward.
References
Venkat Anantharam. A variational characterization of rényi divergences. IEEE Transactions on Information Theory, (4(11):6979-696) 2018.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint rrXii:2204.05800 2022.
Ananth Balashankar, Ziteng Sun, Jonathan Berant, Jacob Eisenstein, Michael Collins, Adrian Hutter, Jong Lec, Chirag Nagpal, Flavien Prost, Aradhana Sinha, et al. Infalign: Inferencc-aware language model alignment. Xixiv preprint 1rXiv:2412.100 197992 2024.
Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander D'Amour, Jacob Eisenstein, Chirag Nagpal, and Ananda Theertha Suresh. Theoretical guarantees on the best-of-n alignment policy, 2024.
Brian Belgodere, Pierre Dognin, Adam Ivankay, Igor Melnyk, Youssef Mrouch, Aleksandra Mojsilović, Jiri Navratil, Apoorva Nitsure, Tnkit. Padhi, Mattia Rigotti, Jerret. Ross, Yair Schiff, Radhika Vedpathak,
16

---

Published in Transactions on Machine Learning Research (May/202)
and Richard A. Young. Auditing and generating synthetic data with controllable trust trade-offs. 1EEE Journal on Emerging and Selected Topics in Circuils and Systems, 14(4):773-788) 2024.
Jeremiah Birrell, Paul Dupuis, Markos A Katsoulakis, Lue Rey-Bellet, and Jie Wang. Variational repre- sentations and neural network estimation of rényi divergences. SIAM Journal on Mathematics of Data Science, 3(4):1093 1116, 2021.
Stéphane Boucheron and Maud Thomas. Concentration inequalities for order statisties. Electronic Comman- nications in Probobility, 2012.
Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration Inequalities A Nonasymplolic The- ory of Independencee Oxford University Press, 2013. ISBN 978-0-199993325-55 doi: 10. 10.1093/ACPROP, 0SS0/9730199335555000.00 0001.
URL https:/ http:///doo.org/////// http:///.////aaooooooooooo//////
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment learning from human preferences. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fer- gus, S. Vishwanathan, and R. Garnett (eds.), Advances in Newral Information Processing Systems, vol- newrips..i.... wttpppprccppppppppppppp http:/ppeepeeeeeeeeeeeeeeeeeee/e files/paper/ ume 30. Curran Associates, Inc., 2017. URL https://proceedin nerureedings.nuurpss.... Paper. .pf...r..ff..... pdf.
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. In The Twelfth International Conference on Learning Representations, 2024. URL
https://openrreview net//oorummiddddcettWYkp
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint (XXiv:334.06767 2023.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXin preprint arXiv:2402.01306 2024.
Jean Feydy, Thibault Séjourné, Xavier Vialard, Shun ichi Amari, Alain Trouvé, and Gabriel Peyré. Interpolating between optimal transport and mmd using sinkhorn divergences, 2018.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866, PMLR, 2023.
Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, and Marc Dymetman. Composit preference models for aligning LMs. In The Twelfth International Conference on Learning Representations, 2024. URL tttpa://openrew/// wetiiiewweeeeeeeeeee aeaee/ffommmiittiiiittt"
Jacob Hilton and Leo Gao. Measuring goodhart's 's law, 2022.
Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D Lee, Wen Sun, Akshay Krishnamurthy, and Dylan J Foster. Correcting the mythos of klrreullarizaionn Direct alignment without overoptimization via chi- squared preference optimization. arXiv preprint arXiv:2407.1339, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint crXiv:2401.0408: 2024.
Gautam Kamath. Bounds on the expectation of the maximum of samples from gaussian. URL http://www/ gautamkamath. hom//rtting////// /ggussion mar. pdf, 10:20-30, 2015.
George Kour, Marcel Zalmanovici, Naama Zwerdling, Esther Goldbraich, Ora Nova Fandina, Ateret Anaby- Tavor, Orna Raz, and Eitan Farchi. Unveiling safety vulnerabilities of large language models. arXiv preprint ar xrXi:22311.01 1,4,124, 2023.
17

---

Published in Transactions on Machine Learning Research (May/2025)
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Duiri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Reward- bench: Evaluating reward models for language modeling. https://nuggingfacee httpeegiinaaaccccaaaeaeeee ct/ taapppppeaaleeeeeee reward-bench, 2024a.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannanch Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024b.
Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, et al. Controlled decoding from language models. arXiv preprint Xri::22300.1 7022, 2023.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human leedback. arXiv preprinl or Xrxiu:2112 12.09332. 2021.
Apoorva Nitsure, Youssef Mroueh, Mattia Rigotti, Kristjan Greenewald, Brian Belgodere, Mikhail Yurochkin, Jiri Navratil, Igor Melnyk, and Jarret Ross. Risk aware benchmarking of large language models. In International Conference Machine Learning, pp. 38264 38297. PMLR, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Newral Information Processing Systems, 35:27730-277 27744, 2022.
Y. Polyanskiy and Y. Wu. Information theory: From coding to learning, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Newral Information Processing Systems, 36, 2024.
Alfréd Rényi. On the theory of order statistics. Acla Mothemalica Academiae Scienliaram Hungarica, 4: 191-231, 1953. URL httpss//app..snnatttccoooooooo www.nticscoollaata....oooaoaaaaa wwwpaaaiicccoolaaooo/oo/rrrssuss org/CorppsII: :ttp:/II::1211111777 123132570.
Filippo Santambrogio. Optimal Thansport for Applied Mathematicians: Calculus of Variations, PDEs, and Modeling. Birkhäuser, Cham, 2015. ISBN 9783319208275. doi: 11.1007/978-3-311 20828-2..
Ofer Shayevitz. On rényi measures and hypothesis testing. In 2011 IEEE International Symposium on Information Theory Proceedings, pp. 894 898, 2011. doi: 10.1109 201.......1..63426666
Anuj Singh, Sayak Mukherjee, Ahmad Beirami, and Hadi Mamaal--Rad Code: Blockwise control for denoising diffusion models, 2025. URL https: / /ttpx//xxvvv////////////////aa./ . orgo / 00968.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with humaa feedback. Advances in Neural Information Processing Systems, 33:3008 33:3008-3021, 2020.
Shivchander Sudalairaj, Abhishek Bhandwaldar, Aldo Pareja, Kai Xu, David D Cox, and Akash Srivastava. Lab: Large-scale alignment for chatbots. arXiv preprint arXiv:2403.0100 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiu:2307.09288 2023.
Tim van Erven and Peter Harremos. Rényi divergence and kullback-leibler divergence. IEEE Transactions on Information Theory, 60(7):3797-3820, 2014. doi: 10.1109/TIT .2014.2300000.
Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse KL: Generalizing direct preference optimization with diverse divergence constraints. In The Twelffh International Conference on Learning Representations, 2024a. URL http://pppprreeeeeeeeeeoorrraiaa
18

---

Published in Transactions on Machine Learning Research (May/202)
Zihao Wang, Chirag Nagpal, Jonathan Berant, Jacob Eisenstein, Alex D'Amour, Sanmi Koyejo, and Vic- tor Veitch. Transforming and combining rewards for aligning large language models. arXiv preprinl arXiv:2402.00742, 2024b.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Tterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint, 2024.
Joy Qiping Yang, Salman Salamatian, Ziteng Sun, Ananda Theertha Suresh, and Ahmad Beirami. Asymp- totics of language model alignment, 2024.
Kevin Yang and Dan Klein. FUDGE: Controlled text generation with future discriminators. In Kristina Toutanova, Anna Rumshisky, Luke Zetilemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technolo- gies, pp. 3511 3535, Online, June 2021. Association for Computational Linguisties. doi: 10.18653/v1/ 2021 naacl- main. 276. URL https: /atee://allaaaaa http///aaannoooo.orrrrrrr
www.:aacclaaaaaa masel.mmmain..... 276.
Yao Zhao, Mikhail Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu. Calibrating sequence likelihood improves conditional language generation. In The Eleventh International Conference on Learning Representations, 2023. URL http:://ppreeeeeeeeee/e//oddddas
19

---

Published in Transactions on Machine Learning Research (May/2025)
A Broader Impact and Limitations
We believe this work explaining scaling laws for reward models and alignment will give practitioners insights regarding the limits of what is attainable via alignment. All assumptions under which our statements hold are given. We don't see any negative societal impact our work.
B Proofs For Best of n Policy
B.1 Best of Policy KL Guarantees
Proof 1 (Proof of Lemma 1)
(L(E(M) = 1o fgo (x) log [E de
+00
fE(Ø)
We have fx(x) = e x200- Note that the CDF of marimam of exponential FEO (2) = (1 =-=1 120, and
hence (T) = n11 - --n--- = "x200 Hence we have:
n(1 e-0jn-1e-
(L(E(")
((EE()EE) |E)
n(1 c (11---m--e-- log
dr
n(1 e --m-- log (n(1 - - do
Let u = - - =,, we have du = e- dx. It follows that
KL(E(1)|(E) [' nu- log (nz""11) du
S/" nu (log(n) + (n (n-18 (n-1)loge log(l)) du = =log(n) S du + (n (n- //" nu n- log(u)du
llo(n)) + (n (n- f d(u" logu- u"
log(n) 1
n
Proof 2 (Proof of Theorem 1) Recall that Tx = F о FE, Fg is one to one. If the space y is finile, R(Y has a discontinuous CDF hence not strictly monotonic. follows that its quantile FRbyyx is not a to one map and Tx result is not one to map and hence we have by DPI (that is an inequality in this case since Tx is not one to one):
(E(TX ((Tx(EE)) S KL(E(")ILE)
(21)
If the space y is infinite and we assume that XX is continuous and strictly monotonic then Futyix is a one to one map, and as a result Tx is one to one map and the DPI an equality in this case:
(E
= (L(F())
(22)
Hence under Assumption 1 and for y finite combining equation 11 and equation 21 we have:
(n)
meref S (L(F()) F),
(23)
and under Assumption and for y infinite and assuming FR(UXX is continuous and strictly monotonic, combining equation 11 and equation 22 we have:
KL (x)
Tee XX) (L(F(()) E).
(24)
20

---

Published in Transactions on Machine Learning Research (May/202)
Under the more realistic Assumption 2 we con also apply the DPI on the stochastic map Hx, since DPI also holds for stochastic maps ( (under our ussumplion R|X Y|X see for example (van Brven & Horvemos, 2014) Example 2)
KL(TTnooll =
$ (X(((((())))( = (I
(25)
and hence under Assumption 2 regardless whether Tx is one to one map or not, thus we have: (L (e - (L(E())JE)
B.2 Best of n Policy f divergence and Rényi Divergence
Best of Policy f divergence and Renyi divergence Guarantees Given that our proof technique relies on DPT and Rényi representation, we show that similar results hold for any f-divergence and for the Rényi divergence:
Dy (P1IQ) = f
q(t))
p((z) g(x)
dx,
(26)
where f is convex and f(1) = 0. Hence we have by DPT for f-divergences:
Theorem Under Assumption the best of n policy satisfies for any f-divergence:
Dy ( S (/¹ (nu"~1) du
(27)
Proof 3 (Proof Theorem 5)
lo X) D2(r())
(n)
=Dy(Hx (X()))XX((())XX
(D)()(()))((())) By the data processing inequality
(28)
10 -D (TTx( Renyi and Optimal Transport Representotions equation
DD(EW)) [E) since Tx is a monotonic bijection DPI is an equality
(29)
too (fesw(t) dx fov SE(x) (JE(()
(30)
{ (e>*) ( du
(31)
J/ f(num-2)ddu
(32)
In porticular we have the following bounds for common divergences:
For f(x)= = x log(2) we obtain the KL divergence and we have the result:
S nu nu n log(nnn (L(F = log(n) n
For [(x) (xr - 1)2 we oblain the chi-squared divergence and we have: ft (nu=~1 1) du x²
So ((2 2-1 1 + 1)du
-n-7 u 2n 2n + w/b -n 2 +
n²
32=2n+1
(n-1)²
2n-1 2n-I
21

---

Published in Transactions on Machine Learning Research (May/202))
For f(z) 2- obtoin the total variation distance (TV) and we have: 2. S0 nu--1 - 1 du
2(5o (1 - nu"=1) du (fr- (nu"-! - 1) du) (u" (m), where
1, Le ut" =
-Hence the TV is (#) ( T=T.
-2 For f(x)) (1- (1= we have the hellinger distance: So (vru++ 1) du -o (nu= 12 2 =
(1-vvn) 1)du = - -2vn" nil + u 2(1 2
For f(x) -loo(x), we obtain the forward KL and we have So fnnu"-))du n log(n).
Guarantees with Rényi Divergence Turning now to the Rényi divergence for a E
(P( = (a (a-1) log (fr(a)--a))))
the limit as a D2(PIIQ)) KL(PIIQ)
Theorem 6 Under Assumption 2 the best of policy satisfies:
n2a a(n 1) +
Da Da(T (itor) S (a-1) 1) log
(33)
Proof 4 (Proof of Theorem 6) Applying DPT that holds also for the Rényý divergence twice from Y, y
to R. R(a) and from R(m) E, E(a) we obtain
(n) Da [I < (E((
(a 1) log (fo n²(1 - = 1)e 10aeaar are e- -011-a) -o)
Da (a(F(())1E))
(ax 1) log (fo to0
1
n² --
Let 1 - have du =x
log (f" na
Da (E(())EE)
(a 1)
logn log f' na(nn
(a 1)
log + +log aln - 1) + 1 110) 1!)
19+1
(a - 1)
na a(n 1)+1
log
(a -1] 1)
From Renyi to KL guarantees Let (a) = (a - 1) and s2(a) = log a(n-1)+1
n we have
(E())
s. (a) we have (L(E(*)IE)) lim D. [E [E lim s2(a)
(a)
g. hence applying
L L'Hôpital rule we have: limg-11 (a(a
lim ((aa (a)
lim log(n) - ..
log(n) n-1 Hence we
(()
recover the result for the KL divergence.
22

---

Published in Transactions on Machine Learning Research (May/2025
B.3 Best of Dominance
Proof 5 (Proof of Proposition 1
(x) (Fr(z))* I Fr(2), Vx 2 O, which means also that F (t) 2 (F) (t) (0, 11, which means that E(n) dominates E in the first stochastic order E(n) o E FSD
which means there erists coupling belween E(n) and E, I(E("), E), such that E 2 e, for all e) ~ On the other hand By Rényi and Monge map representations have: R(w) FR FE(E!m)
and R FR o FE(E), given that T = FR o FE is non decreasing the same coupling т guarantees that T(E) 2 T(e), for all (E,c)= ~ I and Hence R(a) > R.
FSD
Corollary 2 Best of n-polciy has higher erpectation
ER(M) 2 ER.
and is safer policy, let the Tail Value at Risk be:
TVAR, = (e((aam) Qn(()dt
_ p 1///-
We have
TVAR,(R*) 2 TVAR, (R), E [0,1]
Proof 6 (Proof of Corollary 2) First order dominance implies second order dominance (i.e by integrating
quantiles). Expectation obtained for ==
C Best of n and RL Policy
Proof 7 (Proof of Proposition 2) We fir here 7 As
(L( = / क mne (vl log
Traeet (3)2x)
(n)
/ a on (vlz)) log
(312z)
Tt. (v)(x))
Teae (y)r)) Z,(z)
agr(x.N)
= (m (m)
(Tet) log (Brue e0 -r / rdd
(n)
On the other hand by optimality of nyr we have:
KL (Trarll(Tee) 7 / rdway log (f E diyee
(feddred
and hence we have:
KL (Tretr((Tret) + +7 ( /rdmyy f redriiid
KL(Zineell KL (L( (Tor) (T/e)erl(TTrrt)
We choose such that
KL KL( (T Tret) S log(n) n < KL (T (rer) = A
and we conclude choosing n e therefore for that choice of n that:
(L( (n) [T,..) 5 ( (rdmirr - rndrfnnd
23

---

Published in Transactions on Machine Learning Research (May/2025)
On the other hand we have:
/rdtnr - f
rdwyy r.ref
(n)
/ rexp(1r) -TToff - maxx((r)dtrro((i)) ... (en)
Zy
/ n
r(x;) exp(1r(xx))
maxr(x:) diree(211) ... deter(zn))
Zy
/
r(z;) exp((r(z;)) E exp(vr(za)) - maxer(s)) - dmeellos) ... dercot (In)
n
Eis exp(yr(z;))
Zy
/
Ei= exp(pr(ai)) -1) - (o dTver(In))
max (2;;)
Z
M 1 E
exp(yr(x:)) Zy
Z
where we used the following fact, followed by Jensen inequality
12 r(z,) (xp(((x(()2(
Σ E.
ExTesp((r(x)))) exp(rr((x:) S max)(x;).
Assume that the reward is bounded hence have by Hoeffding inequalily
P
exp(tr(xx)) Zy 21 < 2e ((captrM expl xM632
==1
Hence we have:
x exx(>> - exp(--7M)
exp(>r(x;)) - Zn s 22
2
vn
i=1
(L( (exp(A)) r,ref (TA.
M
(exp((M)) exp(->Mi) /exp(-A)
Aa Z//a
D Transportation Inequalities and KL Divergence
D.1 Transportation Inequalities with KL
The following Lemma (Lemma 4.14 in (Boucheron et al., 2013)) uses the Donsker-Varadhan representation of the KT. divergence to obtain bounds on the change measure, and using the tails of Tref-
Lemma 4 (Lemma 4.14 in (Boucheron et al., 2013)) Let th be conver and continuously differen-
liable function sb on a possibly unbownded intercal [0, b), and assume (00)) (/(() = 0. Define for every
x 2 0, the conver conjugate 30r (x) = supxe(0,b) Ax - (A) and let (t) inf (a 2 0: (z))z)) > t). Then the following statements are equivalent:
(i) For X E b)
log (fM < ((A)
(it) For any probability measure P that is absolutely continuous with respect to Q and such that KL(PIIQ) D0:
|ráp- f.dddd rdQ < 30' (KL(P(())
Lemma 5 ( Inverre of the conjugate (Boucheron et al., 2013)) have for 2 0 ( 22
1. If Q SubGauss(o²)),
24

---

Published in Transactions on Machine Learning Research (May/202)
2. If Q Subgamma(o?,c)) .), we have for t 2 x0 (t) V2 + ct.
We give here a direct proof for the subgaussian case:
Proof 8 By the Donsker Varadhan representation of the KL we have:
KL(PIIQ) = sup / hdP - -og (ferat)
h
Fir and M > 0 and define for 0 < X M
hx(y) X ( 4))
(x
We omit what follows x and y, but the reader can assume from here on that π and tee are conditioned on x. Note that Rref x (r(z, (rr and we assume Reet/æ subgaussian. Note that
E B e = E Breellrt) (M)
where M Rustl x the moment generating function of the reward under the reference policy. Brei is subgaussian we have for all X R:
224² -e Se m3.2
Breelee Se
<00
Hence 1y E and we have for all a < Tret and for all 0 < M < xo and 0 < < M:
A Ex) S (L(T||/Tret + log E (Exnolee Al- Brreet (eat
or equivalently:
Exjar T & (L(r||Tnal + log E e -
1
Finally we have for a < Tret for all 0 < X < M:
E E - E E S x (TL + x log (M
(34)
Being a subgaussian, the MGF of Retl is bounded as follows:
X20²
log (e S
2
Hence we have for
2a²
Exjar E Erwwlr 1 S [xToo/)T)o(((z) +
2
Integrating over x we obtain for all x & &eef and all 0 < x < >:
2o
E ExreT P < KL(T|| Tref) +
2
Define:
1 6(A) =1 (L(Tl Xo
mindmizing the upper bound 5(2) for X (0, taking derivative 5'(2) KL(x) (sees)
o²
0 gives
X ()) Taking M = 2X*, X is the minimizer. Pulling this in the bownd we have finally for all rewards for all
Eyr Brrer S 20 [
(35)
25

---

Published in Transactions on Machine Learning Research (May/2025)
Proof 9 (Proof of Corollary 1) (i) This follows from optimality of Tha and applying the transportation inequality for goussion loil.
(ii) This follows from applying Corollary (best of n policy has larger mean ) and Theorem 1 for bounding the KL.
Proof 10 (Proof of Theorem 2) For the penalized RL we have by optimality:
J |rdmyr
- -KL(TTy
1
(feTdru
log
7
1
e diyet) + rddtier
log
7
It follows that
1 log (felt-- rddantt) drreft )
/ rddue
1 KL (lret))))
-
(36)
On the other hand by the variational representation of the Rényí divergence have:
f (rdt / rddtet
Dy (Ty,
(ʃ [ rdv, r dry
log
1 + - log 7
(f (feto-- rdarat) dirret
(37)
Summing Equations equation 36 and equation 37 obtain bound the moment generating function at of Tp (this is not uniform bound, il holds only for 7):
1
log (f (fen--11 f rdz,,)) dryyer
(Ty (LTTTTlllTTe
(38)
Let assume y > 1 we have therefore the following bound the logarithmic moment generation function at -
Vr (7 1) 2- 2 (
(L(mgrll|Feet)
Let Ri, F 1...m. the reward evaluation of indepenident samples of Naaer we have:
Run - / rdbty.r) > mt) P( 60 -f rida..)) > m((--))r
P
(7- m (7-1)
5 (7- 1)me 2- (Da, (eer)) KK (I
m((--))(( - -K Kesey Syot)
(39)
Lel to > 0, hence we have for 7 > 1:
P
Rin / rdmay + lo
Dy (y(TTyrllFree)) - KL ((Toe))
S Se- se-m((7-)4m
7
Now turning to Rrof Fret since Rref E SubGauss(o2a) have for every to > 0
m²
Riret > / yddrzee + to s
P
20
m
1=
26

---

Published in Transactions on Machine Learning Research (May/2025)
mi
Hence we have with probability at least 1 1-
20
m
m
Rixee S / rdwyer
/
rditee + 2ta + (T ( - KL (Tryr(((Tret)
Ri,7
m
m
DT KL(
2 KL + 2ta +
E Proofs for Transportation Inequalities and Rényi Divergence
Proposition 6 (Fenchel Conjugate Propreties) Let F and G be conver functions space E. and F G be their convex conjugates defined E*. We have:
Let (=(x)= F (x) we have:
(x) = 7F (?)
(40)
2. Duality:
min F(x) + G(x) = max -F -P))
(41)
ZER
PEE
Toland Duality:
min F(x) min G (p) FP) ZEP.
(42)
Proof 11 (Proof of Theorem 3) Let > , let (R) = >KL(R|PP) the Fenchel conjugate of FP.1 (.) is defined for bounded and measurable function as follows (h) = log Epc follows by 1) in Proposition 6 that (h) (
log E
For 0 D<ack 1: The objective function in equation 15 is the sum of convez functions: FP, + FQ,1-a (R), by (2) in Proposition 6, we have by duality:
(1 - Da(P()))) inf FP.a(R) + FQ.1-a(R)
sup - -F--Paaf (P - (2)
NEH
sup aa log Epe - (1 - (1- log Eger NEW
Replacing by (1 (a)) does not change the value of the sup and hence we obtain:
(1mi(aahh
(1 a)) log Eqe (11
sup -a log Epe (1 - ()I1())
kel
sup a log Epe (1-a)h (1 a) log Egeah MCk
dividing by |[|-a] both sides obtain for 0 < a 1:
(PII)) sup heR
log Epc
(1-a)h Epe--1-a)a a
1 log Egeah
a
For a a>> 1: The objective function in equation 15 is the difference of conver functions: FP) F by Tolond Duality (3) in Proposition we have:
(1 (1- (L-a)I((P(1Q) = inf FP. (R) - FQ.a-1(R)
R
hEH F2 (h) Fie. (h)
inf Fx
inf (a - 1) log Eqee ta - a log Epe
hEH
27

---

Published in Transactions on Machine Learning Research (May/2025)
The inf does not change when we replace h by a(a 1)h, hence we have:
inf (a 1) log Ege ta-15 log Epe ala afa-11h 11h
aie-1th
(a ()-1))a(Pll))
sup a log Epe 1yh (ax log Boe NEW
dividing both sides by ala-1) we obtain for a 1:
(P(())))=
1
log Epe la-1k log Egeah
sup WEH O
Proof 12 (Proof of Lemma 2 Adding and subtracting in the exponential I hdP and I hdQ resp obtoin the result: a log (ʃ ela-1))dd) _/ log (J eahdQ) a log (J (fe(a-1)(n- haF' hdP) hdP)) dP)
1 log (feath-S geotn-hhoQ]] hdl2) dQ2)
= S hdP [hdQ + a log (fela-1)(n hdP) dP)
1 log (faath-Shd))
I-a Proof 13 ( (Proof of Lemma 3) Note that we have for < a < Da(PIlQ) - -1-a----11111 (See Proposition 2 in van Erven & Harremos (2014)). Taking limits we obtain lima PP = PQ] = PL.
Proof 14 (Proof of Theorem 4) For 0 < < 1, we have for all h H
1 Da(l(() + 1- log (fc (f hdP) dP) + log (f do) (43) I hhap- ---hahaa <
Assuming r is bounded < then we have (r)₂ and (r):Q-Bu are sub- Gaussian with porameter
²² 6. Hence we have for X E R:
A2o2 2
12022 2
Ere rdP) I exp
and Exe (rQ) exp
Fir finite M > O. For 0 < <<> M and P = x|z and Q = Treelæ, consider hx = Xr, thanks to subgaussianity
and boundedness of X, ha E for all X E (0, M). Hence we have by Equalion equalion 43 for all X E M):
rd1 1 (P
log (f Ma- (rali--sdppppp ap))ap)
+ log (f ( dQ
1 a
a
we have by sub- Gaussianity:
I-a log (ʃ saPp)
1 X²(1 - -0)
2 a)oz
<
1
2
2
log (feaat-Ssnne (rdd) dQ
1 22 20
X200
a
a
It follows that for all X M)
> (f rdmtf - -f rddrroli
22 a)
+ X2
<
D. | [//////Teett (:)
2
2
X²((1 - o)o0 + oo
D. (
28

---

Published in Transactions on Machine Learning Research (May/2025)
Integrating over obtain:
> (f rdt
1rd= - rdtrred = a
Da(a]](rett)
X² a)op + a0²))
2
Finally we have:
J rdt - / rdtrret S Xa
1((1 ((1--)033 a) aoo)
Da-aa(T (=(lTttt) +
2
minimizing over X E (0, M): we oblain X gives that X is the minimizer and hence we have for all a E (0, 1):
[( [1-a)ot +a0gla M is free of choice, choosing M = 2X",
V
I (dx - rdTTeet $ V
1((11- a)o + ao Do (T)
F
Goodhart Laws
Proof 15 (Proof of Proposition 4) We have by duality:
log (fer drtee) sup / r'dv (L(v|(Tret)
1
?
hence for Zy.r we have:
log (f dizeet 2 r rr
- (L(TT [Tett (
?
Hence:
J (rddyy log (f ( diyeet ( seet
On the other hand by optimality of nyy have:
KL (Trwrll(Teet = 7 / rdday log (ʃ e diyee
Hence have:
f Ird S log (Sore dirret) + rddT, 7
(fordmmet)
1 log (fer (orrdmee S $ rddT, 1 log
[Ser"day doreet
+ 7
S
It follows that:
f f diy. - I diredt I Indtarr / rditel + log [Seiii- of diseet) dirret
e rdttent) drieet
S Se -f ddddrt)) drveet
- r= rddred) e dndee I do
I ( rddreet doteet
= -f ( (f-(f=ddtoo-- rd dig..
29

---

Published in Transactions on Machine Learning Research (May/2025)
Hence have finally:
I r'
('d dxyo - f r (ddrect dorett
9 (ʃ Seeti'- (fr r'''d - rdsrec) dayer
rddarr rd + log
f f f doinnt $ /rdtnss -
f rdttrot
log (f (frtrri- rndtrottJ -f dasee)) dmy..r"
The proof follows from using the subgaussianity of Tee and the assumption on the soft max. Proof 16 (Proof of Proposition 5)
r" 100 TV(T,, Rzet) Er -r) - Enres (r- S 2/1r
For (n)
we have:
(n) (r*) Faet
FrTeer (r*) R
(r)
EF FT (r) + 2/1r r- TV riref Tref)
and
(v) (r*) BITeor (r*) (ne) (r)
E
ETTTor (r) 21/r 121 TV r.reff Wref)
By the data processing inequality we have: subguassian tails under Tret than we have:
(M Fret) S TV(R(() nieef R) = ( (2) - (2) If r has
E ())) (r*) Ex (r') 20² (log(n) - n 1=11
+ 2 1)
100 -1)
Fyret
n
En(n) (r*) - ETTeo (r²) S 222 [og(m) - n -1) --11)
1) + 2 inf ||r--"1l r~
( -6,
rEw
30

---

Published in Transactions on Machine Learning Research (May/202)
G Supplementary Figures and Experiments
G.1 Tails of Reward model FsfairX-LLaMA3-RRR -M--00011 evaluated on Popular LLMs
0000
2000
4000
Did Baiic
030
0.2
Jamna PA
102
422
115
-15
100
Figure 10: Reward evaluated on LLama2-7B. We see that the reward follows more Gaussian a gamma random variable and it is not heavy tailed. The Moment generating function (MGF) follows quadratic The Hill index is not meaningful in this case.
G.2 Reward versus KL on other LLMs with best of n policies
31

---

Published in Transactions on Machine Learning Research (lay/2025)
1100
1000
3150
Der
Dut
4.14
0.0
4.3
0.2
6:54
404
0.0
6.0
0.7
0.0
NeenstO and
Fepeertel
-00
Figure 11: Reward evaluated on Merlinite 7B. The reward follows a gamma distribution as is almostly perfectly matching it in the q-q plots (quantile plots).
32

---

Published in Transactions on Machine Learning Research (lay/2025)
2000
000
4.20
020
0.0
0.0
NevwstO and
Terme or
Figure 12: Reward evaluated on Mixtral8x7b. The reward follows a gaussian or a gamma distribution and is heavy tailed.
33

---

Published in Transactions on Machine Learning Research (May/2025)
iitellema, Lemo 3tt.nttrdddd..... pem
nos tamu pana peep
mile, 3me-33/6--h--- www.0m.mmm..m...mmmm 0.5
premte-biaa.. 3.3 Jamp, 335
inte-lema,, ume
te 2. a.d
ooste-za-dtara samo
iitellema, amod wt..33tt--ssttt...co l.
crente-Tbien. 2.P 2ammp.1
1g220-1822h
Figure 13: Centered Reward (ISPAIIX-LLAMA3RRM--00)) versus KL of best of n policy for various LLM
best Rd
merlintte77ap 0.9 Sump.__pp... bast
merlinte--Tbpp 9.t_pppp___mp_..
9x76 irstnuct v01_p
lamc-2:13b-chat,
Bama-2-1 3b-hh __
wwwp:/nnn__...pm____ee 0.00 best line
0.5
1.0
3.5
2.0
2.5
3.0
35
l (1-11
Figure 14: Best fit. for y = a v for centered best of reward
KL. We that this fit is not good v + b hinting subgamma tail rather than subgaussian.
34

---

Published in Transactions on Machine Learning Research (May/202))
H Experiments with OAS Reward Model Opuuksssttn//mowcccnmdddeeeeeeee
v2
histogrems
All
100
6030
500
300
5000
230
300
130
240
2000
100
1020
-5.0
-2.5
5.0
25
1.2
-2.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
Liama 706 Chatcig q best of 'nn rewards histegrams
All rewards
B 10
n 50
- 100
540
300
5030
250
5030
253
200
4000
150
200
150
2000
100
50-
5.0
Merlinite best rewards histograms
All rewards
n 10
n= -00
= 1000
10000
500
500
400
5000
340
300
200
200
2000
100
100
-3.0
Figure 15: Histograms of OAS Reward for reference and best of policies.
Model and
Mcan
Std
Max
Mixtral
-3.25
2.11
7.8
Mixtral =10
-0.34
1.5
6.23
Mixtral =50
1.1
1.57
7.8
Mixtral =100
1.76
1.64
7.8
Llama2 70b chat all
-1.47
2.17
6.48
Llama2 70b =10
0.95
1.15
5.29
Llama2 70b =50
2.12
1.09
6.48
Llama2 70b =100
2.63
1.01
6.48
Merlinite all
-3.47
1.6
6.46
Mcrlinite =10
-1.23
1.19
5.07
Merlinite n =50
-0.14
1.08
6.46
Merlinite =100
0.31
1.2
6.46
Table 4: Statistics OPENASSISTANT/RRRS /Ww.:////D.D........E.AA.AAAAAA.
reward evaluated for ref-
erence and best of policies for different values.
35

---

Published in Transactions on Machine Learning Research (lay/2025)
Wean Maee Mean her
marésite Nu. D.S. Sung. 0.95
merinbee7o, tmme ber, it:
u-1ttructu011 p tamp
10-mmtucci-o111.1 a tamp best
lame-2-100--de chai- p
mala, lame- chai-uuauuuu 0.0.aempp tre
0.5
1.0
1/5
2
2.5
3.0
15
loghtt mndds
Figure 16: Centered Reward (OAS) versus KL best of n policies, with best fit y =avI + br. The fit hints to subgamma tails.
5
Moan bast: Moan Ret
mariibe 20...0....000. w.0.stemmppee 0.55
merbibe-Ph, Ph_stee--hpp.u hh.... temo beet St,
lump. best
chai u... a.m.m..
meto, dha. aa..emo_.a..... Cesp_.oaeeesssseCe
1.0
1.5
25
2.5
3.0
15
109201 mils
Figure 17: Centered Reward (OAS) versus KI. best of n policies, with best fit. = ay.. This su subgaussian fit provides an upper bound that is not as tight as above.
36

---

Published in Transactions on Machine Learning Research (May/2025)
Model
a
Merlinite-7B
1.74119977
0. 1.12554422
Mixtral-8x7B
2.06674977
0.28401807
Llama2-70B chat
1.83327634
0.15764552
Table 5: OAS Reward model: av + bæ best fitted coefficients for centered reward versus KL. is
slightly larger than std.
Model
a
Merlinite-77 7B
1.9550695
Mixtral-8x7B
2.55054775
Llama-700 Maaiaa-7OB-Chaae Chat
2.10181062
Table 6: OAS Reward model: avx best fitted coefficients for centered reward versus KL.
Open Assistant Rewards Curve
mate Jemed--ti-mntr at. jtt___mm--1b---aaaa..
0.35
mata kama Jume3-/0b- intruct samp 0.95
matra-t.mhhnemmrtstt _a,3 3.9_empppp0 p.95
modimns 70_p_8.9.tcre__1 1.0
gnante-1 tb-chtt22-ttt 0.9 temp 0.85
meta Bacoo_ttosssssaa3 Jam.o-130-ccor 1-b--chr____00 wtt._______eppppp.00 2.mmp_0.95
mata tama J.ma3.700 nstruct 2.0.p.ompp110
med moniinne-oou..0
Joma instr
Newn Noeett
0.5
1.0
1.5
35 2.0
23
3.0
Ingini-
Figure 18: OAS Reward: Centered Rewards versus KL best of n policies for various models.
Reward Versus Rényi In Best of N
37

---

Published in Transactions on Machine Learning Research (lay/2025)
EngiosteC feadd
+1085
-000
-3000
Neru D821
0.0
030
Eoanert Duenia Rot
Danna O.
Figure 19: subgamma tails of the OAS reward for Merlinite 7B as seen in the q-q plots.
38

---

Published in Transactions on Machine Learning Research (lay/2025)
EnglosteC
1900
2000
4.2
Neru DAU
www.rrrddddd Pad
018
0.2
090
034
0.7
Eoanert Duenea Rot
Danna OU
-10
Figure 20: Abbgaussian/subgaaaa tails of the OAS reward for Maama2-7B--aa
39

---

Published in Transactions on Machine Learning Research (lay/2025)
EnglosteC Readd
2009
3000
1.000
500
Neru (831
Pod Balk Ng1
4.50
3.2
022
615
Nomsrouurd Net
Eounert Dunnei Ror
Sanna Ourrde
-10
-12
Figure 21: Subgamma tails of the OAS reward of Mixtral-8x7b-inm
40

---

Published in Transactions on Machine Learning Research (l25/2025)
FSFair Rewards for merlinite-7b. Sactiitte-b______e0 0.9
1a_pr_b_____e___eee temp__eee 1.0
6
5
Mean N best Mean Ref
4
3
Alpha 0.01
Alpha = 0.26
Alpha = 0.50
Alpha 0.74
Alpha 0.99
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Renyi
Figure 22: Centered Reward for Best of N versus Renyi divergence for a E (0, Merlinite
41

---

Published in Transactions on Machine Learning Research (l25/2025)
FSFair Rewards for mintral-bbbb-intttuttto00....... 009__empp00.9
4.5
4.0
Mean N best Mean Ref
3.5
3.0
2.5
2.0
Alpha 0.01
Alpha = 0.26
1.5
Alpha = 0.50
Alpha 0.74
1.0
Alpha 0.99
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Renyi
Figure 23: Centered Reward for Best of N versus Renyi divergence for a E (0, 1) -Mixtral
42

---

Published in Transactions on Machine Learning Research (l25/2025)
FSFair Rewards for Hama-2-13b-chee p p_0.9 temp_0.99 0.95
4.0
3.5
Mean N best - Mean Ref
3.0
2.5
2.0
Alpha 0.01
1.5
Alpha = 0.26
Alpha = 0.50
Alpha 0.74
1.0
Alpha 0.99
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Renyi
Figure 24: Centered Reward for Best of N versus Renyi divergence for a E (0, 1) -LLama
43

---

Published in Transactions on Machine Learning Research (225/2025)
FSFair Rewards for merlinite-7b_p__ 1tt_:_________m____________
temp_1. 1.0
Alpha 0.99
6
Alpha 1.24
Alpha = 1.50
Alpha 1.75
5
Alpha 2.00
Mean N best Mean Ref
4
3
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Renyi
Figure 25: Centered Reward for Best of versus Renyi divergence for a > 1 Merlinite
44

---

Published in Transactions on Machine Learning Research (l25/2025)
FSFair Rewards for mintral--x/b-innttuutt-oo0.__ _09__emmm_0.95
Alpha 0.99
Alpha 1.24
4.5
Alpha = 1.50
Alpha 1.75
4.0
Alpha 2.00
Mean N best Mean Ref
3.5
3.0
2.5
2.0
1.5
1.0
0.5
1.0
1.5
2.0 Renyi
2.5
3.0
3.5
4.0
Figure 26: Centered Reward for Best versus Renyi divergence for a -Mixtral
45

---

Published in Transactions on Machine Learning Research (l25/2025)
FSFair Rewards for llama-2-11 1ima-2213b-chat, P pttpt__tt__ppppppp__1
Alpha 0.99
Alpha 1.24
4.0
Alpha = 1.50
Alpha 1.75
3.5
Alpha 2.00
Mean N best Mean Ref
3.0
2.5
2.0
1.5
1.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Renyi
Figure 27: Centered Reward for Best of versus Renyi divergence for a > -LLama
46


Reviews:
[{"review_id": "MHMnKHC2w2", "reviewer": "Reviewer_y2Mw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n\nThe paper provides rigorous analysis of the theoretical guarantees on various alignment methods. Moreover, the theoretical results either provides theoretical justifications on some of the widely-used practical guidelines (e.g., the KL divergence between best-of-n policy and the base policy), or provides insights on some of the empirically observed phenomena in alignment (e.g. the square root dependence between the reward improvement and KL divergence). \n\nThe result are new to the best of my knowledge. I appreciate the rigorous approach that the authors take on studying these problems and I believe these results will be of interest to both the LLM alignment community and the information theory community.\n\nI didn't verify all the proofs but the results look believable to me. \n\n*Weaknesses*\n\nThe results on reward improvement are mainly upper bound results. It would be nice if the authors could also comment on the lower bound side to provide a more complete picture and justify the tightness of the result. For example, Proposition 4 shows that the upper bound on the reward improvement could be hurt by the discrepancy between the proxy reward and the real reward. While this hints on that the reward improvement might also be hurt, however, this does imply this fact. The argument holds similarly to other results such as proposition 3 as well. \n\nThe writing of the paper could be improved. This is mainly a stylistic suggestion. The paper contains a list of results with different techniques and assumptions. So there can be a few switching of topics through the paper. While I believe each one of the results is interesting, emphasizing on the a few and providing sufficient discussions on their implications might help the readers grasp the main message of the paper.\n\n*Minor comments*\n\n1. Can the authors comment on when Assumption won't hold?\n2. The authors make the assumption that the optimization problem in KL-RL will be solved to its optimality, which may not be true in practice. This is mainly a comment on the limitation. I don't view this as a major weakness since many papers made the assumption.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies information-theoretic guarantees of various LLM alignment methods, including training-time method KL-regularized reinforcement learning (KL-RL), and test-time method best-of-N. The main contributions are the following:\n\n(1) Prove an upper bound on the KL divergence of the best-of-n policy $\\pi^{(n)}$ against the base policy $\\pi_{ref}$, which relaxes the assumption imposed in previous work.\n\n(2) Prove an upper bound on the improvement of the reward for the aligned policy which depends on $\\sigma_{ref}\\sqrt{KL(\\pi \\|\\|\\pi_{ref})}$, where $\\sigma_{ref}$ is the subgaussian proxy of the reward under the base model, improving the trivial bound which has the max norm of the reward instead of $\\sigma_{ref}$. The paper also provide a tail-dependent upper bound on the improvement of the reward that depends on the Renyi divergence.\n\n(3) The paper also studies the case where the learned reward is not the same as the golden reward, and provide an upper bound on the reward improvement taking the discrepancy between the two rewards into account.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See previous comment on weakness. \n\nAlso there seems to be a many typos in the paper. So I would suggest the authors to do another round of proofreading.\n\nList of typos I found:\n1. The line after equation (1), missing parentheses in the subscript of the expectation symbol.\n2. Section 2.2. First equation, after where, Y should be italicized.\n3. Assumption 2: Right parentheses do not match left ones.\n4. Beginning of page 4: conditionnaly -> conditionally or conditioned?\n5. Lemma 1: $E \\sim exp(1) $  -> $E \\sim Exp(1) $.\n6. After lemma 1: best n -> best of n.\n7. The sentence is broken in proposition 1.\n8. Page 6, notations and 'Scaling Laws in Alignment': $\\sigma_{ref}$ is used without being defined.\n9. Page 11, end of the hanging paragraph: missing a period.", "broader_impact_concerns": "I don't see any particular ethical concerns of the work that needs additional discussion.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UwKWq8phXu", "reviewer": "Reviewer_JnBY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The major strengths of the paper are the three aforementioned major contributions that are highly novel and non-trivial.\n\nThe major weakness of the paper are:\n- The paper might be hard to follow for a non-expert in information theory. Given that this is a journal paper with no page limit, I think the authors can teach some of the basic concepts and their relevance before giving the results.\n\n- Another weakness of the paper is that the majority of the results are derived for unbounded rewards that are sub-Gaussian. While this is a very reasonable assumption for the raw rewards, the actual golden reward of interest might indeed be bounded in [0, 1] and well-behaving, e.g., when the reward is calibrated through the CDF function such that the expected calibrated reward gives win rate (see Eq. (28) in (Beirami et al., 2024)).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents several information-theoretic guarantees on policy alignment. \n\n- The first contribution of this paper is to establish an upper bound on the KL divergence (and other f-divergences) of best-of-n policy and reference policy under fairly general assumptions and relating these to the data processing inequality.\n\n- Second, the authors characterize upper bounds on the improvement in expected reward as a function of the KL divergence (for sub-Gaussian distributions) and establish this to be an information-theoretic upper bound corroborating the existing empirical findings in the literature.\n\n- They use their results to establish upper bounds on the reward improvement merely given the distribution of the reward.\n\nOverall, the paper makes significant contributions to the theory of alignment and would be a nice addition to the literature.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the aforementioned weaknesses. A more detailed list of changes to address these issues is provided below.\n\n- I think it would be very nice if the results are explicitly extended to the case where reward is calibrated in the sense of (Beirami et al., 2024, Eq. (28)). In this case reward is bounded on (0,1) and in fact is uniformly distributed over [0,1] so many of the results in this paper should simplify. In fact, these would imply guarantees on win rate (which is an important quantity of interest). Further, it would be nice to contrast the upper bounds on the calibrated reward with (Beirami et al., 2024, Conjecture 7.2) which basically says that n/(n+1) as a function of log(n) - (n-1)/n is an upper bound on the win rate.\n\n- When discussing the unboundedness of the reward, in Section 3 end of Page 6, it would be good to mention that this may be an artifact of how we construct rewards in practice, and we may actually have to transform rewards to make them bounded. Besides, IPO (Azar et al., 2024) has already proposed to use win rate directly as reward which makes the reward bounded so a connection with IPO would also be nice. So I think the quote of Lambert et al. (2024b) might also be an overclaim in general.\n\n- Page 2, end of Section 1: \"Can we before doing any alignment predict the upper bound of any alignment from looking only on the histogram of the reward of the reference models?\" -> \"Without performing alignment, can we predict an upper bound on the expected reward for a given KL level given the distribution of the reward?\"\n\n- Please use \\citet and \\citep correctly. For example, when the sentence starts with a citation it should be \\citet. Also, please consider using a different color than black for hyperlinks.\n\n- Beginning of Section 2: \\cal X and \\cal Y are technically sets, not spaces.\n\n- The constant before the KL regularizer is usually \\beta in the literature whereas it is chosen to be 1/\\beta in this paper. While this is fine given the paper is self-contained, I suggest to use \\lambda or some other notation instead and in a footnote explain that 1/\\lambda is usually called \\beta in other papers.\n\n- Please use \\quad in Eq. (2), Eq. (3), etc to make the constraints more readable.\n\n- End of page 3, not usually meet -> met\n\n- Please expand on the context around (6); I think this is very hard to digest for a typical reader who doesn’t have the necessary background.\n\n- Eq. (7) does not need to be broken in two lines.\n\n- Please provide more context on DPI. I think majority of the readers of this paper are unfamiliar with (Polyanskiy & Wu 2023).\n\n- I think the biggest value of Theorem 1 is that it applies to continuous distributions, e.g., diffusion models. In fact, the techniques of (Beirami et al., 2024) cannot be extended to handle this case. For example, (Singh et al., 2025) already build on Theorem 1 to analyze the KL divergence of best-of-n in diffusion models, which would not have been possible without Theorem 1 in this paper. Singh, Anuj, et al. \"CoDe: Blockwise Control for Denoising Diffusion Models.\" arXiv preprint arXiv:2502.00968 (2025).\n\n- After Theorem 1, it would be nice to connect the conditions for equality to the derivations of (Hilton & Gao), which basically implicitly assumes those conditions. Also, I think it would be good to mention that for LLMs with discrete outputs, those conditions cannot be satisfied so the inequality is always strict.\n\n- The Eq. in Prop 4 does not need to be broken in two lines.", "broader_impact_concerns": "The paper studies the theory of alignment and I think broader implications have been sufficiently addressed by the authors", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DXtlNFX0I3", "reviewer": "Reviewer_M4q8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\nThe authors did a great job in the field of LLM alignment. This paper is well-written and organized. While the math in this paper can sometimes be difficult to understand, given the difficulty of the content, it is easy to follow.\n\nThe main strength of this paper is its comprehensive information and theoretical analysis of policy alignment. The paper also provides a clear explanation of the technical material, making it accessible to a broad audience. I consider that some portion of this paper seems incremental compared to Gao+ (2023) or Beirami+ (2024), but I do feel that this paper poses solid contributions and it is an advantage to discuss the differences from such existing literature.\n\nThe theoretical analyses are interesting and will attract much attention from the research community. Best-of-$n$ test-time alignment is a promising paradigm, and this paper will contribute to understanding its influence and limitations theoretically.\n\nThough the empirical evaluation is limited, it is very interesting to see that there is some consistency between theoretical and empirical results.\n\n**Weakness**\nI think this paper should be read by many researchers not only from ML theory but also from the empirical side of ML or NLP. The current form of this paper is partly hard to follow and could be written in a way friendly to a wider range of audiences. For example, Section 2.2 could be written to be reader-friendly. In my honest opinion, the quality of Section 2.2 is much worse than that of other sections. I even feel that this section is written in a hurry.\n\nThe empirical results are minimal. I feel that the empirical results are sufficient to support the authors' claims, it would be helpful to provide more results regarding different divergences presented in Table 1.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper makes several theoretical contributions to Large Language Models (LLMs) alignment. The first contribution is to provide new proof for the best of $n$ policy inequality, extending previous work to more realistic settings and generalizing it to $f$-divergences and Rényi divergences. Second, it shows that the policy improvement scaling laws presented in Gao+ (2023) are information-theoretic upper bounds, derived from transportation inequalities with KL under sub-gaussian reward tails. Third, the authors examine the tightness of these bounds when the optimized policy’s tails are known, deriving new transportation inequalities for Rényi divergence. Finally, this paper studies the transfer of transportation inequalities from proxy to golden rewards, proving that golden reward improvement is limited by the proxy reward’s overestimation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think this paper is sufficiently well-written in its current form, but I would like to request the following changes if possible:\n\n- Rewrite Section 2.2 to be more reader-friendly. In my opinion, Beirami+ (2024) is a good reference for writing this section.\n- Add empirical results for other divergences than reverse KL and see if we can observe similar consistency between theory and practice", "broader_impact_concerns": "This paper is purely theoretical, so I do not see any ethical concern in this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission got clear positive reviews from all three reviewers, all recommending acceptance and stating that the theoretical contributions of this work will be of high interest to many ML and information theory researchers. The contributions are centered around a topic of high interest, which is understanding the expected improvements from policy alignment using existing popular techniques.
Decision: Accept (poster)