BOND: Aligning LLMs with Best-of-N Distillation
Authors: Pier Giuseppe Sessa
Robert Dadashi-Tazehozi
Leonard Hussenot
Johan Ferret
Nino Vieillard
Alexandre Rame
Bobak Shahriari
Sarah Perrin
Abram L. Friesen
Geoffrey Cideron
Sertan Girgin
Piotr Stanczyk
Andrea Michi
Danila Sinopalnikov
Sabela Ramos Garea
Amélie Héliou
Aliaksei Severyn
Matthew Hoffman
Nikola Momchev
Olivier Bachem
Venue: ICLR 2025 Poster (2025)
Keywords: LLM
Alignment
RLHF
Best-of-N
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a novel RLHF approach to align LLMs via online distillation of Best-of-N sampling policies.

Abstract: Reinforcement learning from human feedback (RLHF) is a key driver of quality and safety in state-of-the-art large language models.
Yet, a surprisingly simple and strong inference-time strategy is Best-of-N sampling that selects the best generation among N candidates.
In this paper, we propose Best-of-N Distillation (BOND), a novel RLHF algorithm that seeks to emulate Best-of-N but without its significant computational overhead at inference time. Specifically, BOND is a distribution matching algorithm that forces the distribution of generations from the policy to get closer to the Best-of-N distribution. We use the Jeffreys divergence (a linear combination of forward and backward KL) to balance between mode-covering and mode-seeking behavior, and derive an iterative formulation that utilizes a moving anchor for efficiency. We demonstrate the effectiveness of our approach and several design choices through experiments on abstractive summarization and Gemma models.

Published as a conference paper at ICLR 2025
BOND:
ALIGNING LLMs WITH BEST-OF-N DISTILLATION
Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Nino Vieillard Alexandre Ramé, Bobak Shahriari, Sarah Perrin, Abram L. Friesen, Geoffrey Cideron Sertan Girgin, Piotr Stanczyk, Andrea Michi, Danila Sinopalnikov, Sabela Ramos Amélie Héliou, Aliaksei Severyn, Matt Hoffman, Nikola Momchev, Olivier Bachem
Google DeepMind
ABSTRACT
Reinforcement learning from human feedback (RLHF) is a key driver of quality and safety in state-of-the-art large language models. Yet, surprisingly simple and strong inference-ting time strategy is Best-of-N sampling that selects the best generation among N candidates. In this paper, we propose Best-of-N Distillation (BOND), a novel RLHF algorithm that seeks to emulate Best-of-N but without its significant computational overhead at inference time. Specifically, BOND is a distribution matching algorithm that forces the distribution of generations from the policy to get closer to the Best-of-N distribution. We use the Jeffreys divergence (a linear combi- nation of forward and backward KL) to balance between mode-covering and mode- secking behavior, and derive an iterative formulation that utilizes a moving anchor for efficiency. We demonstrate the effectiveness of our approach and several design choices through experiments on abstractive summarization and Gemma models.
INTRODUCTION
Reid et al. 2024) and GPT-4 (OpenAl 2023) are generally trained in three stages. First, LLMs are pre-trained on large corpora of knowledge using next-token prediction (Radford al. 2018 2019). Second, the pre-trained models are fine-tumed to follow instructions via supervised fine tuning (SFT) RRffel et al. 2020: Wei et al.) 2022). Lastly, reinforeement learning from human feedback (RLHF) (Christiano) ct al. 1017:Ziego
State-of-the-art large language models (LLMs) such as Gemini (Gemini Team 2023
et al. 2019 [Stiennon et al.) 2020) is used to further increase the quality of generations. The RLHF step generally consists of learning a reward model (RM) (Ouyang al.. 2022) on human preferences and then optimizing the L.M to maximize predicted rewards using reinforcement learning algorithms.
RLHF algorithms and their challenges. Fine-tuning LLMs with reinforcement learning (RL) is challenging (Casper et al. 2023), notably since it can cause forgetting (French 1992) of pre-trained knowledge, and since loopholes in the RM (Clark & Amodei 2016 Pan et 2022) can cause reward hacking (Askell al., 202T Skalse et al. (022). The standard strategy to use policy-gradies methods (Williams) 1992) with KT. regularization towards the SFT policy. Those RI. algorithms seek Pareto-optimal policies with high reward at low KT., to preserve the general capabilities of the original model and tackle the misalignment (Ngo et al. 2022) concerns.
Best-of-N sampling. In practice, a surprisingly simple inference-time approach is often used to improve the quality of generations: Best-of-N sampling (Stiennon et al. 20220). It consists of drawing N candidate generations from the reference (typically, supervised fine-tuned) model and selecting the one with the highest reward according to the RM. This strategy empirically achieves excellent
reward-KL trade-offs (Nakano et al.) 2021 Gao et al. 2023 computational cost by factor of N.
Touvron et al.) (023) but increases the
BOND. In this paper, we propose BOND (Best-of-N Distillation), a novel RLHF algorithm that learns policy that achieves the strong performance of Best-of-N sampling but, crucially, requires only
1 Corresponneecce Pier Giuseppe Sessa applergg@goog com>

---

Published as a conference paper at ICLR 2025
Best-of-N sampling
BOND
camples
rewards
BoN
pxompt
prompt
LLM
BOND
0.6
Figure I: Best-of-N is an inference-time strategy that selects the best generation among N candidates from reference LLM policy, improving quality at the cost of large computational (need to sample and score N times from the model). contrast, the proposed BOND approach aims at obtaining a fine-tuned policy that can directly sample the Best-of-N generation. This would inherit the quality of Best-of-N sampling, while requiring single sample at inference time. We achieve this by distilling the Best-of-N strategy into the policy via online distribution matching.
single sample at inference time, as depicted in Figure [ Our key idea is to cast the alignment of the policy as distribution matching problem, where fine-tune the policy emulate the Best-of-N distribution. To achieve this, we first derive an analytical expression for the Best-of-N distribution. This allows us consider and optimize different divergence metrics. We first show how minimize the forward KL divergence using samples from the Best-of-N strategy, leading to a standard imitation learning setup with mode covering behavior. We also show how minimize the backward KT., leading to new form of quantile-based advantage, which does not depend on the reward scale, and corresponds to a mode seeking behavior. Then, we propose to minimize a linear combination of forward and backward KLL, also known as Jeffreys divergence, which retains the best of both approaches. Furthermore, to optimize performance while keeping a reduced sample-complexity we propose iterative BOND approach which consists of iteratively distilling the Best-of-N of moving anchor policy. Finally, based on the aforementioned ideas, we propose J-BOND (J for Jeffreys), a novel, stable, efficient and practical RLHF algorithm align LLMs.
Experiments. We first demonstrate the effectiveness of BOND and of our design choices on the abstractive summarization XSum (Narayan et al. 2018) task. Then, in Section 6 apply J-BOND to align Gemma (Gemma Team 2024) policies. J J-BOND does not require committing to a specific regularization strength, but it continuously improves the reward displaying stable optimization and better reward/KL trade-off compared to standard RL. algorithms. This translates into a higher quality and improved scores on popular real-world benchmarks.
PROBLEM SETUP
We consider a LLM based on the transformer (Vaswani al., 2017) architecture, defining policy T (Tr,) by auto-regressively generating token sequences y from the prompt x. Given a pre-trained and typically supervised fine-tuned reference policy ritts seek to further align it to human preferences. To achieve this, throughout the rest of the paper we assume access to a reward model (RM) which we denote as r(-), trained to reflect human preferences.
Standard RLHF Most RL algorithms optimize a linear combination of the expected reward and a KL divergence between the current and reference policy:
TRL argmax, E3 [r()) 3RL. KL(TT] || Teet),
(I)
with regularization strength ARI. 2 0. This KI, regularization forces the policy to remain close to
its initialization Tee GGest et al. 2019: Lazaridou et al. 2020). reducing forgetting (French) 1992)
and reward hacking Skalse et al.
2022). Equation [I] is usually optimized with online algorithms,
as they perform better than their offline counterparts Tang et al. 2024). Moreover, simple methods have demonstrated the best results, e.g., REINFORCE Williams 1992) with a sampled baseline for variance reduction (Li al.
2023 Ahmadian et al. 2024 outperform PPO (Schulman et al.) 2017).
Best-of-N. complementary alignment strategy is Best-of-N, which is an inference sereenceeiimeeeeee strategy that involves sampling multiple times from मee and selecting the generation with highest reward
2

---

Published as a conference paper at ICLR 2025
according to the RM r. In contrast to RLHF strategies, Best-of-N does not fine-tune the weights of the LLM, but instead modifies the inference procedure. Best-of-N was empirically shown to be efficient (Tovroon et al. 2023) when looking reward/KL trade-offs, and comes with theoretical guarantees (Qiping Yang et al. 2024) in terms of Pareto-optimality, Unfortunately, Best-of-N comes at significantly higher inference cost which increases linearly with N, since producing N generations is (in general) N times more costly than sampling a single one.
Motivated by the above considerations, for Best-of-N Distillation. The goal of BOND is to distill the Best-of-N stralegy into the policy. This allows the policy to reach the strong performance of Best-of-N sampling, while requiring only single sample at inference time. We outline our overall approach in the next section.
propose novel alignment method which we name BOND
3 THE BOND APPROACH
We formulate the BOND approach in two main steps. First, we derive an analytical expression for the Best-of-N distribution (3.ction Second, using the derived expression, we phrase the problem as a distribution matching problem (Section 3.2 i.e., we want to steer the policy closer to the Best-of-N draw insightful connections between BOND and standard RLHF.
distribution. In Section 3.3
3.1 THE BEST-OF-N DISTRIBUTION
In this section, we derive the exact analytical distribution of Best-of-N sampling and study its properties. For simplicity, we drop the context r from all notation without loss of generality and assume that the reward r(y) induces a strict ordering on all generations : following main theorem (proof in Appendix Theorem 1. For any generation y, les
We can affirm the
Pe(v)= = (y''' < r(v)]
denote the probability that a random generation y' from Tie is strictly worse than y and let
(z()) P (rl') S r(y)],
the probability that of is not better than (thus including the equality case). Then, the probability that y is the output of Best-of-N sampling given by
TB)N(y) = TTy((() x pP( N-1
Pe
(4)
Ps (v)
(A)
Interpretation. Theorem | provides intuitive explanation the behavior of Best-of-N sampling: it essentially reweights the original sampling distribution Tref, by the multiplicative terms (A) and (B).
The term (A) corresponds a penalty exponential in N based on the fraction of generations (for the same prompt) that are worse or equal to the considered generation Intuitively, this ensures that we sample exponentially less from bad generations when increasing N.
The term (B) is an additional correction factor due to the potential of collisions among generations.
Importantly, it is at most linear in N is always bounded within [1,
N Σ 1=2
N
N
P-
151+
ISSN.
(5)
==1
1=1
It achieves its minimum at 1 for the worst generation y- since we have p<(y-) 0 by definition.
This is not surprising, as we need to sample y- exactly N times in a row and which corresponds TBoN(y)) Tee((-)> ) (note that p> (y-) ) m ( )). In contrast, if the likelihood of individual generations are low and such generations are good, then P<(y) is almost P> (y) and term (b) is close N. Intuitively, this corresponds to the case where sampling generation y multiple times is unlikely. In the extreme case when Tie is continuous distribution, term (B) is constant and equal N (see Appendix A.2
To distinguish between generations with the same reward, ties broken by any arbitrary strict ordering.
3

---

Published as a conference paper at ICLR 2025
3.2 THE BOND OBJECTIVE
The analytical characterization of the Best-of-N distribution allows us to formulate BOND as distribution matching problem. That is, want solve the objective:
ToOND arg min D( [BBN)):
(6)
where DI- is a divergence metric steering the training policy и towards TBoN. For this, a toolbox of possible divergences exist in the literature including. e-g., forward and backward KL (Kullback) 1959). Moreover, we can employ existing distribution matching techniques to estimate D from online and offline samples. We defer the choice of divergences and resulting BOND algorithms to Section
3.3 CONNECTION WITH STANDARD RLHF
In this section, we draw important connections between the two seemingly different objectives of standard RLHF (Equation and BOND (Equation (
It is well known (see, e.g.. Vieillard et al. RLHF objective from Equation
2020 Rafailov et al. 2023)
that the policy maximizing the
TRL (v) & (y) exp
r(y)
3 RL
From the derived expression of TonN in Theorem I we see that the Best-of-N sampling distribution co- incides with the optimal solution of standard RLHF when using the following specific BOND reward:
N
P
(2)
i-1
TRONo (1) log (1)
log
Ps (v)
(A)
(3)
and the specific regularization strength BBOND N-I
The term (B) corresponds to the correction
factor in Theorem which is bounded in 0, log N N-1 for all generations y. Instead term (A) lies
in (-o0,0, 0]. This provides two interesting insights for Best-of-N sampling:
1. Best-of-N sampling corresponds to the solution of a standard KL-regularized RLHF problem where the choice N determines the level of KL regularization.
2. Best-of-N sampling corresponds optimizing the expected log reward quantile, i.e., the log likelihood that the generation has larger reward than a random sample from the reference distribution. Interestingly, due the concavity of the logarithm, Ysovu (y) strongly encourages the model to avoid bad generations rather than encouraging to generate good ones. Moreover, TROMn (y) is invariant to monotone trunsformations of the reward r(.), since depends only on the rank among the generations. We conjecture that both these features make the BOND reward TooRD (y) more robust to reward hacking compared to standard RLHE.
The connection to RLHF also inspires the proposed approach in this manuscript: if we can compute the BOND reward or equivalently the Best-of-N distribution ToON, then we can steer the policy towards Best-of-N N via distribution matching. In the next section we explore different algorithms to tackle the main underlying challenges.
BOND CHALLENGES AND ALGORITHMS
Implementing the BOND approach induces the three following challenges: (/) how estimate the reward quantiles, (2) which is the appropriate divergence metric to use, and (3) how to choose the hyperparameter N representing the number of sampled generations in Best-of-N. We discuss and address these challenges in the next three subsections.
4.1 MONTE-CARI.O QUANTILE ESTIMATION
One key difficulty in estimating the =B distribution that we need to estimate the quantile
Ps (v) P. Py'NTTe (r(v') r(v)),
4

---

Published as a conference paper at ICLR 2025
of given generation y. The quantile ps (y) measures the quality of y compared to generations from Tet when conditioned on the same prompt (recall that we have suppressed the conditioning on x in our notation). A very simple but effective quantile estimation method is Monte-Catt sampling, sampling k generations from Tee and obtaining the following empirical estimate:
Σ s
(10)
We found this to be a very effective in our experiments, even with limited number of samples. In principle, though, one could also use alternative approaches, e.g., training a learned quantile model (as we explore in Appendix B.2).
4.2 JEFFREYS DIVERGENCE AS A ROBUST OBJECTIVE
The choice of the divergence metric used in BOND is of crucial importance: different divergences can steer the policy to very different solutions. Here, we propose the Jeffreys divergence as robust distribution matching objective.
The Jeffreys divergence (Jeffreys) 1946) between two distributions is defined as:
_ effreys
(p 11 4) (1 B) KL (lp) + + (l
(11)
Forward KL.
Backward KI.
The (generalized) Jeffreys divergence is a weighted average (with weight / E [0, 1)) between the forward and backward KL divergence. Notably, when fine-tuning policy p. the forward KL p) encourages that generations likely under are also likely under thus encouraging a mode-covering behavior. Instead, the reverse (L || q) is well-known to have mode-seeking effect, steering policy p to produce generations that have high likelihood according to 4 (Agarwal et al. (024). While the forward KL may produce over-spread distributions, the backward KL can lead to policy and entropy collapses. Instead, we empirically show that the Jeffreys divergence inherits the best of both divergences, producing better aligned policies.
In the context of BOND, this translates into minimizing the divergence Jellsys (* || TBON) which we can estimate using samples from the training policy π and reference policy Tea as follows.
Estimation of the forward KL. The forward KL defined
(LBT iT) Eng [log TBnN(y) log (yy)
(12)
can be estimated directly drawing samples from the TaaN (i.e., sampling N times from nee and selecting the best one) and can be seen supervised fine-tuning loss the Best-of-N samples:
(L || T) = -Ey V (
(13)
Estimation of the backward KL. The backward KL defined as
(L TBON) (log π (y) log BBN
(14)
can estimated from the policy samples (note the expectation w.r.t. T) and their estimated log-
likelihood under TBON- In particular, by the analogies drawn in Section 3.3 we show (in Appendix A.3
that its gradient coincides with a policy gradient (c.g., used by REINFORCE (Williams 1992) in standard RLHF):
V=KL(r TBaN) = -(N - 1) < log ग (FeoND (v) - Seou: (log (1)) log Teet (v))
(15)
with the equivalent reward FRоnD and regularization 3ROND defined in Section 3.3 Note that TeoN v
depends on the true unknown quantile p> (v) and on the correction factor (B) defined in Equation
In practice, we substitute the true quantile by its estimate, while we observed the correction factor
does not play significant role. Thus, we use FeoND (v) Pe (3). Moreover, to reduce the resulting variance, we use a policy gradient baseline (Sutton & Barto 1998) which we compute as the average return for the other generations in the batch.
Thus, the overall Jefrress loss is linear weighted combination between a supervised fine-tuning and policy gradient loss.
5

---

Published as a conference paper at ICLR 2025
Experiments. We consider the abstractive summarization XSum task (Nrayan et al. 2018 with Ted being a T5 supervised fine-tuned policy and r(-) being a T5 NLI reward model (Roit et al. 2023) We run BOND with Lelleys objective and / E {0, 0.5, l]. We use 16 MC samples (per prompt) to estimate quantiles during training. During eval, we use 32 MC samples to estimate the backward and forward KL divergences between the training policy and the TBoN distribution. We report these in Figure 7 (Appendix [.II) when setting N = 4, 8 and 16, respectively. The results confirm our intuition: the Jeffreys divergence (33 = 0.5) allows to minimize both divergences from TooN (left and middle plot), compared to when solely the backward (8 = 1) or the forward (B = 0) KI. divergence is minimized. In addition, we compute the reward log quantiles (averaged over the eval batches) of the training policy. Interestingly, BOND with / = 0.5 maximizes the quantiles similarly to the mode-seeking B choice, while the mode-covering forward KL (B = 0) lags behind.
4.3 ITERATIVE BOND
Finally, we discuss the choice of the parameter N. In practice, choosing N may be difficult for three main reasons: (1) As in standard RLHF, N plays the role of regularization (see Section (.3):: a large N improves downstream performance, but if N is too large it will eventually cause reward over optimization (Gao et al. )023). (2) The larger the N the more the estimate of TBNN is sensitive to errors in the estimated quantiles (since (B) x Pg P<(y) Ny) ). (3) Estimating the forward KI.
divergence requires sampling from TBoN which is prohibitive for large N.
To address the above challenges, we propose the iterative BOND approach. The approach is inspired by the fact that Best-of-N sampling from a Best-of-N distribution, coincides with Best-of-N² sampling from the original distribution. More generally. by informally defining BoN(-) as an operator that performs Best-of-N sampling from base distribution, we have:
BoN(- .. BoN(BoN(((( Tet))) = BoN (Tref).
(16)
M times
This suggests the key idea behind iterative BOND: if we know how to distill the Best-of-N distribution (i.e., via BOND), then we can apply BOND recursively (say M times), equivalently to distilling a Best-of-N of the initial distribution Tee. This allows fixing small (i.e., n 2) and running BOND (with N = n) in an iterative fashion, as an improvement operator. For this, we can introduce an auxiliary anchor policy Tanchor initialized as Tee-- We can then BOND against umccho (i.e., can distill the Best-of-n version of Tanchor) and, after a given number of distillation steps, update Tanchor to be the current training policy Tr. The overall approach is depicted in Figure and summarized in Algorithm]
In a nutshell, iterative BOND allows exponential scaling arbitrary large N (in fact, does not require setting N in advance) while keeping a reduced sample complexity and a stable optimization. The claim is validated in the results below.
Experiments. In Figure 3 we consider the same experimental setup of Section 4.2 fix the BOND objective j0.3 efffrey and run iterative BOND with n E {2, 4} where the moving anchor is updated every 1000 steps. We report the average reward (left plot) and average log quantile (middle plot) obtained during training and compare them to (non-iterative) BOND run with N E (4, 8, 16). As expected, both reward signals saturate early for non-iterative BOND (the smaller the N the earlier the reward saturates), while the iterative BOND approach continuously improves performance (the higher the n, the faster). Moreover, in the rightmost plot we plot the obtained log quantiles against the KT.
t aanhor
Tref
π
T = arg min D(T Best-of-22 - aachor ))
Figure 2: Iterative BOND approach. The policy म, is trained to iteratively distill Best-of-N (in the figure, N 2) version of moving anchor. This allows to continuously improve the policy performance without requiring to set a (large) N upfront. Moreover, it leads to better training stability and minimal computational complexity, since small n is used at each distillation step.

---

Published as a conference paper at ICLR 2025
Algorithm 1 Iterative BOND (meta algorithm)
Inputs: Tef. E N. Initialize To = Tref, T "anchor = Tref- for / = .,.... do
*/ iterations */ distill the Best-of-n version of Fachhrr
At+1 = arg min ((T Best-offnnTTa
anchor π
BOND. N
30
BOND, N
Ieeeaiive DOND, n
logg Tyl
iteraivee DONS) n
BOND, BOND. 16
BOND, 4 BOND, Rerative BOND. Rerative BOND.
-18-
Berative BOND, 2 kerathve BOND.
70
steps
KLnn (egd
Figure 3: Iterative BOND (with n = 2 and n = 4) compared to BOND with N 4, 8, 16. Iterative BOND continuously improves rewards (left plot) and log quantiles (middle plot), while they saturate for non non-iterative BOND (the smaller the N, the sooner). It allows achieving the same reward/KL trade- off (right plot) non-iterative BOND while keeping small and smoothly moving away from Tee-
from the reference policy. The plot shows that iterative BOND essentially has the same reward/KL. trade-off of the non-iterative BOND runs, but crucially allows keeping a small n and to smoothly but continuously move away from Wref-
THE J-BOND ALGORITHM
In this section we present J-BOND, a concrete and practical BOND algorithm motivated by the results discussed in the previous sections. We describe its main components below, and summarize in the pseudo-code of Algorithm
J-BOND follows the template of iterative BOND (Algorithm I with n 2, i.e., it fine tne-tunes policy Te to iteratively distill the Best-of-2 version of a moving anchor π anchor initialized as Tet The name J-BOND stands for Jeffreys divergence BOND because it uses the Jeffreys divergence as distribution
matching objective, i..., it minimizes Jeffeys (T Best-oof-2(TT (n as defined in Section 4.2
Minimal sample complexity. Compared to the BOND algorithms tested in the previous section, J-BOND has a minimal sample complexity: for each prompt in the batch it generates 1 sample from the policy मe and 2 samples from the anchor n. aapchorr While more anchor samples are generally useful for better divergence estimation (in Section 4 we used 16 MC samples), autoregressive sampling is the main bottleneck of online RLHF and we have thus opted for a practical approach working with a small number of samples.
Crude divergence estimate based on anchor samples. The policy and anchor samples are used to obtain a crude estimate of the forward and backward KL components of Jeftreys (T Besst-of-2(T (TT(et)))) as described next.
We can minimize the forward KL as described in Section 4.2 by doing supervised fine-tuning on the best of the 2 anchor samples. To minimize the backward KL, we utilize the policy gradient-style loss of Equation 15) replacing TEOND (w) with different reward which reason for this is that when only 2 anchor samples are available. the reward TnoNo (v) = log P. (v) would be quite uninformative due to p (y) being a very noisy MC estimate. Let y be the policy sample and {v] 1/3 be the corresponding anchor samples, we instead define To-scko (1) as
denote as r'J BoND (v). The
ro- (v) = f- -loo((16) if r(v) < min(r(s6))
(17)
0 otherwise
7

---

Published as a conference paper at ICLR 2025
Algorithm 2 The J-BOND algorithm
Inputs:
Prompt dataset D, reference policy Tet reward r()), [0,1], 0.
Initialize policy and anchor mo тI anchor for 1 O,...do do
Tief-
Sample batch of prompts Dr C D For each x Dt generate: policy sample y (e(2)) and 2 anchor samples / 8/2 anchor (x) +/ Forward KL
Extract Best-of-2 sample: V = arg max, / (w(.))))) r(y').
Compute forward KL gradient: Grw (z. TT) V. log Ty (x, Fo22
/ Backward KL
Compute To-Rcen (z. y) according to Equation
Compute return: R(z, 3) TO noar (z, y) (Log (z, 3) log unchorr (zr, y]).
[Optional] Compute baseline B, e.g. average return of the other generations in the batch.
Compute backward KL gradient: GBw(z,
V,, log me (2, (R(x,)) v) B).
*/ Additional KL regularization
KL regularization gradient: GReg(E, my) = V., log (z, v) (log (2 v) log मं anchor (a,
Overall policy update: Jeffreys divergence + KL regularization
*/
Update policy weights lc+1 with the overall stochastic gradient:
-1 -1. -1.EE,D,1L 3) GFw(x, me) - B GBw(2, mi) + GRog (I, me)l
/ Update moving anchor Update anchor weights with EMA:
o++1 anchor +- (1 1)) O Ognehor + n Oc+1
That is, generation y receives a negative reward of log(16) if has worse reward than both anchor samples, while receives 0 reward otherwise. The above definition is motivated by the following two main reasons:
We negatively reward y only if it is worse than both the anchor samples, to mimic the concavity of the ideal (and unknown) reward function FBOND = log (.).
(ii) We choose value log(16) to ensure that: af [r Tu-eow (v) log Ps (v) when Ps (v) = 0.5. The interested reader is referred Appendix A. A.44 for a derivation and illustration of this fact. In words, the value log(16) calibrates the reward function r: (-) so that, in expectation under the anchor samples, it matches with the ideal reward log Ps (-) for generations y that have median reward (i.e., when ps (v) 0.5).
Exponential Moving Average (EMA) anchor. An important component of J-BOND, which refines the vanilla iterative BOND of Section 4.3 is the use of an Exponential Moving Average (EMA) anchor. That is, instead of using periodically updated anchor, we update the anchor weights At anchor at each
fine-tuning step moving average of the policy weights 0,:
gt-1 <- (1 - 17) O² Oincher + 17. -0+11-
(18)
Consistently with WARP Ramé et al. 2024), we observed that this weight averaging procedure has a positive effect on training stability by reducing variance, and can improve the overall reward/KL
trade-off of J-BOND. We provide an ablation in Section b
Additional KL regularization. Finally, we further regularize the policy stay closer to the moving anchor via an extra KL regularization term, modulated by tunable hyperparameter 2 0. The scope is to further stabilize the policy updates, viewing the overall operator constrained optimization one:
T = min effreys ( Best-off2( ( (
+
KL
T
(19)
FEll
Note that KT regularization is already present in the backward KT. component of the Jeffreys divergence.
8

---

Published as a conference paper at ICLR 2025
85-
d-3
KL. (n Reed
HERPORCE (0000)
.21D VEVFORCE p=1 SENTORTT, SENIFORCE, 0-00
JOUNO REESFORCE, A-
T2Y ZEPIPORIE 5.0.1
-30
REMFORCEE (-00033
SENEFORIZ: 0.03
100
754
steps
KL.f II foal
Figure J-BOND (1) = 0.02) for Gemma 7B, compared to standard REINFORCE (with KL- regularized objective of Equation (1)) with regularization strength 3R.. E {0.001, 0.1,,0.1,10 J- BOND does not require committing to a specific regularization strength, but it continuously improves the reward displaying stable and linear KL increase and a better reward/KL trade-off.
EXPERIMENTS
We test J-BOND on relevant use cases with the following main goals. First, we ablate and showcase important aspects of J-BOND: the benefits of the EMA anchor, and the effects of the anchor speed and the additional KL regularization. Then, compare J- J-BOND to classical RLHF baselines using REINFORCE, demonstrating its efficacy and better performance.
Setup. We consider Gemma (2B and 7B) models (Gemma Team 2024 which we aim to fine-tune into better conversational agents. For this task, we consider a set of conversational prompts D, a reference policy Tet previously supervised fine-tuned on similar prompts, and large previously trained reward model r(-). We use a batch size of 128 and the Adam optimizer (Kingma & Ba 2015) with learning rate 3e-6 and 100 warm-up steps. For the Jeffreys divergence objective, set B = 0.5 (we ablate different Jeffreys divercences in Appendix B.3).
EMA vs. hard anchor updates. We ablate the benefits of using EMA moving anchor (Equa- tion 18] compared to the periodically updated anchor used in Section 4.3 For this, we run J-BOND with 7 0 and EMA coefficient 1/ = 0.02 on Gemma 7B, and compare it with its variant where the anchor is only updated every 50 steps. In Figure [i (in Appendix B.4 due to space constraints), we report the average reward of the policy during training (left plot), the KL from the reference policy Tee (middle plot), and the resulting reward/KI. trade-off for the two runs. Both runs produce the same reward increase profile (this not surprising since an EMA with v = 0.02 roughly corresponds to an update period of 50 steps) but, crucially, J-BOND with an EMA anchor displays a significantly lower KL increase and, as a result, a better reward/KL trade-off.
Anchor speed and KL regularization. We illustrate the effect of the anchor mixing parameter 3) [0, 1] and the benefits of the additional KL regularization parameter y 2 0 introduced in Equation 19). For this, consider Gemma 2B and first run J- J-BOND with y 0 and r7 {0.011 0.05, 0.1}. In the left plot of Figure I1 (relegated to Appendix B.4) we report the average reward of the policy along training. This illustrates that the larger the mixing parameter 7i (i.e., the faster the anchor moves), the faster the reward increases, as one could intuitively expect. Second, we fix r) 0.05 and run J-BOND with different regularization strengths 7 E {0,0.5, 1, We plot the results in the middle and rightmost plots of Figure II] As expected, the larger the regularization the more constrained are the policy updates and thus, the slower the policy moves away from The (middle plot). Importantly, the right plot shows that such a regularization has positive effect since it can ultimately improve the reward/KL trade-off.
Comparison with standard RLHE. We compare J-BOND against standard RLHF algorithms that aim at maximizing the KTgrlagllr objective of Equation (I). To optimize Equation ( we use REINFORCE (Williams) 1992) with 2 policy samples per-prompt and leave-one-out baseline (Ah) madian et al. 2024) for policy gradient advantages. For J-BOND we set the anchor mixing coefficient to 0.02. For REINFORCE, we test possible regularization strengths BRI. E {0.001, 0.01, 0.1, 1). In Figure 4 we plot the average reward of the training policy (left plot) and its KL divergence from the reference Tee (middle plot). As presumed, REINFORCE is quite sensitive the regularization coefficient 3 the larger the regularization, the lower the reward achieved by REINFORCE (and the lower the KI. from Te.). This highlights& a key advantage of J-BOND: it does not require committing

---

Published as a conference paper at ICLR 2025
to specific regularization level, but it continuously improves the reward displaying stable and linear KL increase. Moreover, in the rightmost plot we plot the corresponding reward/KL trade-off showing that J-BOND produces better reward/KL than all of the REINFORCE baselines. This translates into a higher quality (measured via side-by-side comparisons) and improved scores on popular cal-world benchmarks, as we report in Appendix B.33 B.5
7 RELATED WORK
Best-of-N was introduced in Stiennon et al. (2020) as a straightforward but costly inference method to optimize language generation against a given reward function. Further works established and refined analytical form for the KL divergence against the reference (i.c., Bol) policy (Hilton 2023 Beirami et al., 2024), provided an estimator for the average Best-of-N reward (Nakano et al., 2021).
made theoretical connections with KT--constriined RL (Yang al. 2024 and provided scaling laws
for Best-of-N alignment (Gao et al. 2023).
Matching Best-of-N for improved alignment is a strategy that was studied in different flavors in the literature. Dong et al. (2023) and Touvron et al. (2023) propose to fine-tune LLMs in a supervised fashion on Best-of-N data actually applying forward KT. minimization. Concurrently ours, Gui et al.) (2024) proposes to mimic the Best-of-N policy by applying a combination of supervised fine-tuning on best responses and direct preference optimization on best-and-wood worst response pairs. The latter is similar to common strategy in online preference optimization methods: Guo et al. (2024) use pairwise AI feedback on online generations obtain online preferences that are then optimized. while Calandriello et al. (2024) use dedicated preference reward model instead. Concurrently and closest to our work, Amini et al. (2024) also apply distribution matching in order to get the benefits of Best-of-N sampling with amortized cost. While their formalization is identical, we opt for a different divergence (i.e., Jeffreys) than the one they use (i.e., only backward KL), and propose an iterative procedure with dynamic anchor, which we show critical for optimal results. We compare J-BOND with the vBON approach of (Amini et al. 2024) in Appendix A.5 Best-of-N can also be used for seff-improvement in reward modeling, as evidenced in Pace al. (2024).
Using a contrastive advantage an option of J-BOND studied in prior works as well, which replaced a value estimate by the average Monte Carlo return of other samples. This was applied in the context of REINFORCE (Kool et al. 2019
Piint et al., 2023), for online RLHF (Ahmadian et al.
2024) offline RLHF (Flet-Berliac et 2024 and preference optimization (Wu et al. 2024).
Exponential moving average (EMA) of policy as reference in regularization, which we use in J-BOND, is an increasingly popular option. While most alignment approaches use static anchor, dynamic anchors bring the benefit of improving the flexibility of the policy space being explored (Munos et al., 22023] Gorbatovski et al., 22244 Ramé et al., 2024p, with the caveat that too slow updates limit optimization and too fast updates hinder stability.
Scaling post-training and iterated amplification. BOND hinges on the idea of investing more resources during training to ensure that computational demands during inference remain low, factor often overlooked in traditional scaling laws (Hoffmann al. [022]. Specifically, BOND incorporates the principles of iterated amplification (Christiano et al. 2018 Cotra 2018), where amplification in this context consists of producing multiple generations, comparing their rewards, and using these to iteratively improve the policy performance. In this regard, BOND is complementary to WARM (Ramé et al. 2024) and WARP (Ramé et al. (2024), which previously scaled postt-raiiing by training multiple reward models and policies, respectively.
8 CONCLUSION
We introduce BOND, a novel RLHF method that fine-tunes the policy via online distillation of the Best-of-N sampling distribution. We propose a concrete algorithm, J-BOND, that integrates multiple components to enhance its practicality and efficiency; Monte-Carlo quantile estimation, combination between forward and backward KL divergence objectives, and an iterative procedure with an exponential moving average anchor. J-BOND improves the KL-reward Pareto front of solutions, and compares favorably against state-of-the the-hartt baselines. We hope this work can help improve alignment of AI systems, making them safer and more reliable.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We thank Daniele Calandriello for insightful comments, as well as Gil Shamir, Bilal Piot, and Remi Munos for helpful discussions.
REFERENCES
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation language models: Learning from self- generated mistakes. ICLR, 2024.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting REINFORCE style optimization for learning from human feedback in LLMs. arXiv preprint, 2024.
Afra Amini, Tim Vieira, and Ryan Cotterell. Variational Best-of-N alignment. arXiv preprint, 2024.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-oodds, Danny Hernan- dez, Jackson Kermion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. general language assistant as a laboratory for alignment. arXiv preprint, 2021.
Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander D' A'mour, Jacob Eisenstein, Chirag Nagpal, and Ananda Theertha Suresh. Theoretical guarantees on the Best-of-N alignment policy. arXiv preprint, 2024.
Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, et al. Human alignment of large language models through online preference optimisation. In ICML, 2024.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, al. Open problems and fundamental limitations of reinforcement learning from human feedback. TMLR, 2023.
Paul Christiano, Buck Shlegeris, and Dario Amodei. Supervising strong learners by amplifying weak experts. arXiv preprint, 2018.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In NeurlPS, 2017.
Jack Clark and Dario Amodei. Faulty Reward Functions in the Wild. https:// com/ @seearch/
2016.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint, 2021.
Ajeya Cotra. Iterated distillation and otm/liictionnntttsss//llallllll wtt.meettt.m/itrttttttttt. iottiteeatee aatiisiilaatiinnnnn--aaaa oe
2018.
Thomas M Cover. Elements of information theory. 1999.
Will Dabney, Mark Rowland, Mare G. Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In AAAI Conference on Artificial Intelligence, 2017.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. TMLR, 2023.
Yannis Flet- Hiet-Berliac, Nathan Grinsztajn, Florian Strub, Eugene Choi, Chris Cremer, Arash Ahmadian, Yash Chandak, Mohammad Gheshlaghi Azar, Olivier Pietquin, and Matthieu Geist. Contrastive policy gradient: Aligning Iims on sequence-level scores in a supervised -eriiiddddd-ri fashion. arXiv preprint, 2024.
11

---

Published as a conference paper at ICLR 2025
Robert M French.
Semi-distributed representations and catastrophic forgetting in connectionist networks. Connection Science, 1992.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In ICML, 2023.
Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized markov decision processes. In ICML, 2019.
Gemini Team. Gemini: A family of highly capable multimodal models. 2023.
Gemma Team. Gemma: Open models based on gemini research and technology. arXiv preprint, 2024.
Alexey Gorbatovski, Boris Shaposhnikov, Alexey Malakhov, Nikita Surnachev, Yaroslav Aksenov, Ian Maksimov, Nikita Balagansky, and Daniil Gavrilov. Learn your reference model for real good alignment. arXiv preprint, 2024.
Lin Gui, Cristina Gârbacea, and Victor Veitch. Bonbon alignment for large language models and sweetness of Best-of-N sampling. arXiv preprini, 2024.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, and Mathieu Blondel. Direct language model alignment from online ai feedback. arXiv preprint, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In NeurIPS, 2021.
Jacob Hilton. KL divergence of max-of-n, 2023.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. In NeurlPS, 2022.
Harold Jeffreys. An invariant form for the prior probability in estimation problems. Proceedings of the Royal Society of London. Series A, Mathematical and Physical Sciences, 186(1007)4433-61, 1946.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088. 2024.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 REINFORCE samples, get baseline for free! 2019.
Solomon Kullback. Information Theory and Statistics. New York, 1959.
Angeliki
Lazaridou, Anna Potapenko, and Olivier Tieleman. Multi-agent communication meets natural language: Synergies between functional and structural language learning. ACI,, 2020.
Ziniu Li, Tian Xu, Yushun Zhang, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models, arXiv preprint, 2023.
Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback. arXiv preprint, 2023.
12

---

Published as a conference paper at ICLR 2025
Rejichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assister question-answerin with human feedback. arXiv preprint, 2021.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun' icch Tsujii (eds.), Conference on Empirical Methods in Natural Language Processing, 2018.
Richard Ngo, Lawrence Chan, and Soren Mindermann. The alignment problem from a deep learning perspective. arXiv preprint, 2022.
OpenAl. Gpt-4 technical report. 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. NeurlPS, 2022.
Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. West-of-n: Synthetic preference generation for improved reward modeling. arXiv preprint, 2024.
Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In ICLR, 2022.
André Susano Pinto, Alexander Kolesnikov, Yuge Shi, Lucas Beyer, and Xiaohua Zhai. Tuning computer vision models with task rewards. In ICML, 2023.
Joy Qiping Yang. Salman Salamatian, Ziteng Sun, Ananda Theertha Suresh, and Ahmad Beirami. Asymptotics of language model aligmment. arXiv, 2024.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020.
Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. WARM: On the benefits of weight averaged reward models. In ICML, 2024.
Alexandre Ramé, Johan Ferret, Nino Vieillard, Robert Dadashi, Léonard Hussenot, Pierre-Louis Cedoz, Pier Giuseppe Sessa, Sertan Girgin, Arthur Douillard, and Olivier Bachem. WARP: On the benefits of weight averaged rewarded policies. arXiv preprint, 2024.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap. lean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schritwiisser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint, 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
Paul Roit, Johan Ferret, Lior Shani, Roee Aharoni, Geoffrey Cideron, Robert Dadashi, Matthieu Geist, Sertan Girgin, Léonard Hussenot, Orgad Keller, et al. Factually consistent summarization via reinforcement learning with textual entailment feedback. In ACI., 2023.
13

---

Published as a conference paper at ICLR 2025
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint, 2017.
Joar Max Viktor Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. In NeurIPS, 2022.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford. Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. NeurlPS, 2020.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint, 2022.
Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, and Will Dabney, Understanding the performance gap between online and offline alignment algorithms. arXiv preprint, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Rémi Munos, and Matthieu Geist, Leverage the average: an analysis of regularization reinforcement learning. NeurIPS, 2020.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In ICLR, 2022.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Reinforcement learning, 1992.
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model aligmment. arXiv preprint, 2024.
Joy Qiping Yang, Salman Salamatian, Ziteng Sun, Ananda Theertha Suresh, and Ahmad Beirami. Asymptotics of language model alignment. arXiv preprint, 2024.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint, 2019. 2019.
14

---

Published as a conference paper at ICLR 2025
A SUPPORTING RESULTS AND DERIVATIONS
A. PROOF OF THEOREM
Consider N random generations y1, .2..... yN from Tiet and an arbitrary generation y among them. Let (y) denote the event that is the best sample (i.e., r(y) 2 r(y.) for all i) and that i is the lowest index for which yi = y. It is trivial to see that the the events (,,,),)11 are disjoint
and that their union corresponds to y being selected by Best-of-N sampling.
The event A.( occurs if and only if three conditions are met: r(y;) < r(y) for all j 3: = y. and rwy) S r(y) for all j < i. This allows us to derive the likelihood of the event (,))):
P[A(())= = II P(r(s,) < r(w)) mer(() x
II s=i+1
Pfr(u)) s r(vil
= =p<(y)* Te) x ps N-11
The likelihood that Best-of-N sampling selects the generation y is then given by
N
BoN
P[A[(v))
N
Σ [v=( -1 x mar(() x (=(
TTr(()) [p=(v))' x ())
Py(y))
- (2)
N-1
Ps (3)
A.2 LINK TO THE CONTINUOUS CASE
A noteworthy observation is that we can relate the Best-of-N expression to the case of continuous distribution, which case the term (B) in Equation (4) is constant and equal to N (which is intuitively natural p< (3) and ps (y) have the same value in this case).
Indeed, recall that the probability for a sequence y to be drawn from the Best-of-N distribution
(B) TTe((1) x P p2(
Pe
(20)
Ps (3)
(A)
(B)
Here, y is a discrete variable, as it lives in [1; where T is the number of tokens and L is the maximum length of sequence.
Now, we show why Equation (2)) matches the classic formula for the max of N continuous variables. Formally, let X be real valued random variable with density fx and cumulative distribution function Fx. Taking Y.,. i.i.d. variables with the same density, define XN = max ...Y) as the maximum over the N variables. Then, we have that
Fxx (v) = IP(Y S y. YN < = =x(D)
(21)
and thus
-x))
(22)
In Equation 22
recognize the Best-of-N formula in the case where the correction factor (B) is N. For the term (A), Fx(1) plays the role of (v), as by definition Fx (y) PP < y). Finally,
fx (y) is the density of X. which is analogous to the probability Ty)(y) in the discrete case.
15

---

Published as a conference paper at ICLR 2025
A.3 BACKWARD KL AND POLICY GRADIENT EQUIVALENCE
We formally show the analogy between the gradient of the backward KL divergence of Equation [14 and the standard REg.RREINFORE (Williams) 1992]) policy gradient of Kll-regularized RL.FF problem with equivalent reward TsoNn and regu lauzzatiiiioon BAOND-
The exact backward KL gradient can be derived as:
V (L || TBaN) = V By [log π (y) - log BoN
Σ (l)) T log BB))
v
Σ , (z)) log (B)) + (1) Tly)
y
Σ क (v) V log ग (log (y) - log ToN (v)) + ( V= log (V))
y
E y V log π (Iog T (v) - log ToON (8) + V log TV)
E y v log (y) (vy log ToN (TB)n())) (g))
Above, we have used the product rule of gradient, the rule V V (y) = ग (v)) V log (v) and the fact that Eyna v log (() O.
Equivalence with Policy Gradient RL. As anticipated, one can verify that descending the above gradient is equivalent up to constant scaling to rumning the RL policy gradient REINFORCE algorithm on the RL objective of Equation [ with = Tooro and 3RI. = Beowu- Indeed, we can use the expression for TBo break down the above gradient into:
By-= ~ log x (y) (yy log (B))]
N
Pe P (1) Ps (1)
1-11
Eyn
n log (t))
log T(y) - log (er((y) - (N log (v) log
2 i=1
TROND (v)
(10) )
Eyo
x log (v)
log (y) log Teer(y)
N
- (N Eynas! V log π (v) (TDOND (v) - BBOND (log x (g) log Tee (rr(())))]
gradient by REINFORCE
A.4 DERIVATION OF J-BOND REWARD
Here we provide theoretical explanation behind the design of the J-BOND reward function discussed
in Section 5
T-- (v) = {- 0
(o6(16) if r(y) < min(r(s1),
otherwise
Recall that ro-soND (-) is meant to approximate the true reward function FEcen (y) = log ps which unknown since (=(-) (-) in general requires knowing the reward distribution of nt anchorr (in J- J-BOND, we only take 2 samples ví 1/2
TTnchorr).
As mentioned in Section 5 we designed (- to assign a negative reward only if sample is
worse than both the anchor samples, to mimic the concavity of the log quantile log (-). In practice, we did not observe gains when rewarding also the intermediate case. The particular choice of value - logg( 16) is motivated by the following main reason.
We want that, when sample y has median reward compared to the anchor rewards' distribution
(i.e., Ps (v) 0.5), then in expectation Tu-woou (v) coincides with the true reward recsu (v) = log Ps (.) log(0.5). For this purpose, let us consider the parametrized function:
- (4) = fa if r(y) min(r(v/))r
otherwise
16

---

Published as a conference paper at ICLR 2025
Note that the stochasticity of ro-- (y) is due to the 2 random anchor samples ví 1/2 and its expectation can be computed as:
Fy 1/ antm I BOND (v) (((((1))) r(v)) (r(v6) > r(v)] + "-PPootherwiis"" (23)
=a. (1- (1-Ps(u))
(24)
where we have used the definition of ps = py g' (r(v) S r(y)]. Using the expression above, we can find the a for which E Vi. ~ hea p (v) = THOND (1) when ps (3) = 0.5:
0.5)² log(0.5) a log(16).
We illustrate this in Figure TBOND(Y) (y) as a function of ps
where we plot the expected ra-sooo reward and the true reward
0.0
0.5
-1.0
-3.0
Ideal reward: logP= (y) expected (-BoNo(y)
335
-1.0
0.0
03 oe
da
Ps (y)
Figure 5: Expected value of the J-BOND reward, i.e., E u/ wi (r (y)l (see Equation (23)). compared to the true (unknown) reward logps To-POND, the two curves coincide when Ps (3) 0.5, i.c., when y has median reward w.r.t. the anchor distribution.
Ps (3). as function of the quantile Ps (v). By design of
A.5 COMPARISON WITH VBON (AMINI ET AL. 2024)
In this section we compare J-BOND with the concurrent vBoN approach of Amini et al. (2024). Amini et al. 2024) consider the same BOND objective of Section 3.th 3.ction3.2 the main difference that only backward KL divergence is considered. Thus, according to the equivalence drawn in Equation (15). their approach can be seen as using standard RLHF algorithms to optimize the equivalent reward TDOND of Equation (8) with regularization strength ABOND differences with respect to J-BOND;
N-T- Below, summarize the main
(i) vBoN considers only backward KL divergence, while J-BOND uses Jeffreys divergence.
(ii) BBoN requires setting fixed N in advance (similar to regularization in standard RLHF), while J-BOND continuously improves the reward thanks to iterative nature.
(iii) To compute TEcNu (v), the vanilla version of vBoN presented in (Amini et al., 2024 requires estimating the quantile Ps (3) with several MC samples. Unfortunately. this is only doable when the number of prompts small and the estimation can be done offline, but it is prohibitive in more complex experimental setups with millions of prompts.
To overcome challenge (iii), we consider variation of vBoN where Tecmp (y) are estimated via the exact same crude estimate used in J-BOND, based on 2 anchor samples (Equation (17)). We consider the Gemma 7B finetuning experiment of Section 5 and compare J-BOND to vBoN for N [4, 8, 64, 128, 512]. The corresponding results are reported in Figure 6 Unlike vBoN where the reached reward and KL depend on the value of N defined in advance, J-BOND displays stable KL and reward increase yielding a better reward/KL trade-off. This attributed to its iterative approach and to the fact that both backward and forward KI. divergences minimized.
17

---

Published as a conference paper at ICLR 2025
140
rtyl
-97
JoonD
180% M1
<Tol.
10ot 61
1Den. -4
100% 17 120
47e8 N- 170
2003
2006
steps
Steps
602
KLin |I Radl
Figure 6: Comparison of J-BOND and our implementation of vBoN (Amini et al. 2024 for finetuning Gemma 7B. Compared to (Amini et al. 2024), we approximate the backward KI. objective via the crude reward estimate of Equation (17) and optimize it with standard policy gradient. This is because estimating the actual BOND reward of Equation (8) requires several anchor samples and
is not feasible in our online setup. Unlike vBoN, J-BOND does not require setting fixed N in advance and produces a stable and improved reward/KL trade-off.
B ADDITIONAL PLOTS AND EXPERIMENTS
B.1 BOND WITH JEFFREYS DIVERGENCE OBJECTIVE
In Figure 7 we report the results concerning the experimental setup described in Section 3.2) i.c. we run BOND with J Jefteys objective and {0, 0.5, 1}. The Jeffreys divergence (5 0.5) allows to minimize both divergences from TBoN (left and middle plot), compared to when solely the backward (B 1) or the forward (B 0) KL. divergence is minimized.
B.2 LEARNED QUANTILE MODELS
Monte-Carlo quantile estimation (Section 4.1 approximates the reward quantiles by sampling multiple times from the reference policy Tets for each observed context. While we found it to be very simple and effective, it may require many sample for an accurate quantile estimation and, in addition, it does not exploit any information about the given context. For instance, assuming have good quantile estimation for context x and are presented a new context x'. MC quantile estimates treat x' independently from x, although they may have very similar reward quantiles.
Motivated by this, in this section we explore an alternative approach that aims at learning a context- dependent quantile estimator Prg (.), parametrized by parameter 0. The idea is to view quantile ps (v) as the parameter of a Binomial random variable Z where Z 1(r(z)) S rry) for Tee-- Under such a view, we can interpret p-) the output of binary classifier and train via maximum likelihood estimation using the standard binary cross-entropy loss (Cover 1999):
L(O, x, (Ey log ps (,,,y))), ())l(((r. + log(1 P (2, (res
(25)
We test such an approach in the abstractive summarization task considered in Section 4 We
parametrize (-) with a LLM initialized as Tee and fine-tuned using the loss of Equation 25
Simultaneously, the policy ne fine-tuned using BOND and utilizing PSo, (-) as quantile estimator at each step. Notably, we approximate the expectation in Equation 25 via a single sample from Tet for each prompt in the batch. In Figure & we report the backward and forward KL divergences between the training policy and the ToN distribution as well as the average log quantiles, and compare them to the ones obtained when running BOND with MC quantile estimation. Note that in both cases, TooN approximated by 32 MC samples during evaluation. When using the learned quantile model (.), we observe BOND achieves very comparable KL divergences and log quantiles compared to using MC quantile estimaton. This illustrates that the use of learned quantiles is valid and promising, potentially offering interesting computational advantages in situations where, e.g., (+) can be re-used or learned offline with a fixed sample budget.
Finally, we remark that the explored approach is quite naive, and alternative learned quantile models can definitely be derived, e.g., further enforcing ordering in the predicted quantiles, using quantile regression (Dabney al. 2017), assuming pre-specified (e.g., Gaussian) rewards' distributions.
18

---

Published as a conference paper at ICLR 2025
233
μ=0
9.5.5
Mo055
pw1
|[iR]| (auw)
lag (
243
0.5
2020
190h 1290
1920
1960
3333
.530
1030
12K0
1603 2000 20020
steps
steps
steps
Am0
8-0.5
9.000
8-1
KLin tesse)
log Ey
01
10690
steps
stops
stops
8-0
-24
8-0.5
KLi mooe!
9
lgg- (y)
-2.
-24
0.5.5
743 180
1943 TuEE 2303
202
IEAS 1ZAE DARE LOON AEXI
steps
steps
steps
Figure 7: BOND with N 4 (top row), N = 8 (middle row), and N = 16 (bottom row), and different values of 3 for the Jeffreys divergence objective (ef. Equation (II) When using B = 0.5 (Jeffreys divergence), BOND minimizes both backward (left plots) and forward (middle plots) KL
divergences from TeN. achieving best both objectives (B and 3 1). Moreover, optimizes the reward quantiles (right plots) significantly more than when using B 0.
Learned quanties
Learned quantlees
Monte Carto quantiles
|Lin|| (ovvel
log ty)
-1
Learned quartiees quantikes
2543 2000 SEE DIEE T900 thi0 2003
LEAS UZAE DAE LOW ASX steps
steps
steps
Figure 8: BOND (with N = 8) using MC quantile estimates vs. a Learned quantile model.
B.3 J-BOND ABLATIONS: JEFFREYS DIVERGENCE
Here we ablate the effect of parameter / in J-BOND, i.e., we test J-BOND with different Jeffreys divergences. We consider the Gemma 7B finetuning experiment Section 6 and run J-BOND with different choices for E [0, 0.25, 0.5, 0.75, 1]. We report the obtained results in Figure 9 J-BOND with 3 = 0.5 (in red) displays the best reward/KL trade-off, highlighting the importance of minimizing Jeffreys divergences opposed only backward (5 1) forward (8 = 0) KL. The ablation complements the one of Appendix [B. Isooinng that Jeffreys divergence as objective is also beneficial for J-BOND achieve best Pareto solutions.
19

---

Published as a conference paper at ICLR 2025
135
140
180'0
-02
JB2ND) 022
JBUM,, A-0
160%0,8-02
JO3ND. 0-5.5
100ND,
18010, 6-035
100ND. 5-1
18010, 1-1
1083
2008
2008
430
2000
1000
steps
steps
KLIn |I Read
Figure 9: J-BOND with different Jeffreys divergences (parametrized by 9) when finetuning Gemma 7B. Employing Jeffreys divergence as an objective, i.e., B E (0, leads to a better reward/KI. trade-off (rigthmost plot) than when only backward (3 = 1) or forward (B 0) KI. are considered. In this example, the best reward/KL trade-off achieved by 0.5.
B.4 J-BOND ABLATIONS: EMA ANCHOR AND KL REGULARIZATION
Here we provide the plots associated to the J-BOND ablations discussed in Section 6 In partic- ular, Figure TO illustrates benefit of the EMA anchor compared to periodic anchor updates. Moreover, Figure TI illustrates the role of the EMA mixing parameter 7) and the regularization parameter y in J-BOND. See Section 66 for the detailed setups and related discussions.
-042
4012
-0.5
-6.12
hert ancher updanes each steps
ancher updanes ene depp
DMA unchher O
areter, 0.02
186
200
3000
200
1000 2003
1966
500
2003
steps
steps
Figure 10: J-BOND with periodic anchor updates (every 50 steps) vs. EMA anchor (n 0.02) on Gemma 7B. While attaining the same reward (left), using the EMA anchor displays a significantly lower KL than the reference policy (middle) and thus better reward/KL trade-off (right).
2-000
7-0.0
1w0.1
Rive
-052
VeD
-3.0
Vell
KL Madd
Figure 11: J-BOND: the role of the EMA mixing parameter and regularization 7. The left plot shows J. J-BOND with 0, {0.01, 0.05, 0.1}: the larger the 7 the faster the average reward increases. The middle and right plots show J-BOND with 77 = 0.05 and y E {0, 0.5, 1, 2}: the larger the regularization y, the slower the policy moves away from Meet improving the reward/KL. trade-off.
B.5 DOWNSTREAM EVALUATIONS
We report downstream evaluations the Gemma 7B policies finetuned with J-BOND and the REINFORCE baselines considered in Section In particular, we report:
Side-by-side comparisons: For each trained policy we generate candidate answers on held-out collection of prompts. We compare such answers with the ones generated
20

---

Published as a conference paper at ICLR 2025
by Mistral (Jiang al. 2023) and Mixtral (Jiang al. 2024 models. We use Gem- ini (Gemini Team 2023 as a judge and for each comparison we assign a score in 1-1,5,-1, 0.5,, 0, 0.5, 1.5] ranging from "much worse" (-1.5) to "much better" (1.5).
We then report the average score over all prompts.
Zero-shot performance on popular benchmarks including: GPQA RRin al. 2024). GSM8K (Cobbe et al. 2021). MATH (Hendrycks et al.) 202I) and Big Bench Hard (BBH) (Suzgun et al. 2022) to test our policies on different capabilities.
We evaluate multiple checkpoints for J-BOND and REINFORCE and in Table report the numbers corresponding to the best ones. J-BOND consistently and significantly outperforms the REINFORCE (standard RLHF) baseline both terms of quality, as measured by the side-by-side comparisons, and in terms of accuracy on real-world benchmarks.
Side-by-side comparisons
Mistral 7B v1
Mistral 7B v2
Mixtral 8x7B
Standard RLHF
0.34 0.36
0.16
0.04
J-BOND
0.17
0.07
Benchmarks
GPQA
GSM8K
MATH
BBH
Standard RLHF J-BOND
26.6
45.4
26.8
53.3
31.2
54.2
28
54.1
Table 1: Downstream evaluations of Gemma 7B finetuned with J-BOND compared to standard RLHE; ie. REINFORCE with leave-one-out baseline (Ahmadian et al. 2024).
21


Reviews:
[{"review_id": "yh6wouxHVw", "reviewer": "Reviewer_2AHH", "summary": "The paper is focusing on the RLHF alignment problem, in particular on emulating the Best-of-N distribution which is known to perform very well, but is very costly at inference time (for each prompt it requires drawing N candidate generations from the reference model and selecting the one with highest reward according to a reward model). The authors propose the BOND (Best-of-N Distillation) algorithm designed to force the distribution of generations from the finetuned policy to be close to the Best-of-N distribution, requiring the generation of just a single sample (instead of N). To this end, BOND regards the alignment problem as a distribution matching problem and distills the Best-of-N distribution by finetuning the reference policy to imitate the Best-of-N distribution. To stay close to the original reference model, the authors incorporate a KL regularization term that considers both the forward and backward divergence (Jeffrey divergence). In addition, they incorporate Monte-Carlo quantile estimation, and exponential moving anchor, resulting in the J-BOND algorithm. The authors conduct experiments on the abstractive summarization task (XSum dataset) and aligning GEMMA using J-BOND.", "questions": "How does J-BOND performance compare to (Amini et al, 2024) and other concurrent works?\n\nThere are three algorithms discussed in the paper, namely BOND, Iterative-BOND and J-BOND. Is it always preferable to use JBOND or do you recommend using each algorithm in particular situations?\n\nWill the code be made publicly available to serve the research community?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper makes multiple contributions, namely theoretical derivation for the Best-of-N distribution and a practical RLHF finetuning algorithm that distills the Best-of-N distribution into a policy which is sample efficient and requires just one single sample at inference time\n\nThe authors are making a lot of engineering design choices in their proposed model, and carefully analyze the role of each component in the performance of the proposed algorithm\n\nTo regularize the model and ensure it is not steering too far from the reference model (the supervised finetuned policy), the authors use a combination of both forward and reverse KL, namely Jeffrey divergence. While the forward KL ensures mode covering behavior, the reverse KL is used for mode seeking behavior; their combination results in better aligned policies that combine the advantages of both divergences\n\nApplying BOND recursively (Iterative BOND) improves the sample efficiency of the BOND algorithm and works for very small values of n (2, 4); its reward/KL tradeoff is comparable to the non-iterative BOND while being more sample efficient\n\nThe J-BOND algorithm presents better reward/KL trade-off compared to the REINFORCE algorithm with different values of \\beta and does not require using a specific regularization strength\n\nThe paper is well written, well-motivated, presents theoretically and experimentally sound insights that would benefit the research community", "weaknesses": "The paper combines a lot of distinct ideas already proposed in previous works - it would be good to actually clearly articulate what the novel contribution is. Besides, the comparison with concurrent works is not very clear, in particular the difference with (Amini et al, 2024), WARM, WARP (Rame et al, 2024). \n\nFigure 4 - It would be interesting to see how the performance of Best-of-N compares to the proposed algorithm J-BOND and REINFORCE\n\nAlgorithm 1, line 330 - \\pi_t is not defined\n\nLine 329 - \\pi \\in Capital \\pi -Capital \\pi  in Algorithm 1 is not defined\n\nLine 456 - “a large previously trained reward model r(.)” - please provide details\n\nLines 481-482 - there are not many details about the hyperparameters of the REINFORCE algorithm \n\nThe authors are conducting experiments on Gemma 2B and 7B models, while results are convincing it would be good to see if they hold with other models and other tasks than summarization", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7pariekAi0", "reviewer": "Reviewer_6VBe", "summary": "The paper introduces Best-of-N Distillation (BOND), a novel alignment tuning algorithm designed to emulate the Best-of-N sampling method in a more computationally efficient manner. BOND aims to achieve the same high-quality output as Best-of-N sampling without the inference-time computational overhead, by aligning model outputs to match the distribution of the Best-of-N candidates. \nIn addition, to ensure stability and scalability, the authors introduce an iterative approximation strategy that operates effectively even with a minimal sample size (e.g., 2 or 4). \nFurther, based on the two types of loss function derivation aiming forward and reverse KL respectively, the author leverages Jeffreys divergence proposing J-BOND, an enhanced algorithm incorporating iterative distribution matching with an exponential moving average (EMA) anchor. J-BOND demonstrates effectiveness in maintaining a stable training and superior KL-reward trade-off through experiments.", "questions": "In Section 4.2, the authors utilize 32 Monte Carlo samples to estimate the backward and forward KL divergences between the training policy and the reference distribution. Given the high dimensionality of these distributions, this sample size seems insufficient to reliably capture the divergence and may introduce substantial estimation variance. A sensitivity analysis showing how the estimator's variance changes with an increasing number of Monte Carlo samples would strengthen the results. Alternatively, using a larger sample size for these estimates could enhance the reliability of the reported divergences.\n\nWhile BOND’s benefits, such as improved KL/reward trade-offs and dynamic regularization, are discussed in the body of the paper, they are not clearly summarized in the introduction or abstract. A brief overview in these sections would effectively communicate BOND’s main advantages over traditional RLHF approaches, aiding readers in understanding its unique contributions and practical value.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is well-structured and clearly presents its methodology, with detailed explanations and algorithms that allow readers to follow the progression. From iterative BOND to the addition of KL regularization in Sections 4 and 5, the additional experimental results effectively support these methodological advancements. \nBOND is notable for its originality, offering a practical and computationally efficient alternative to traditional RLHF that achieves a superior KL-reward balance without requiring commitment to a specific regularization level. The work is significant in its potential impact on RLHF practices, as it provides a scalable solution for optimizing performance and efficiency while minimizing trade-offs between KL divergence from the reference distribution and reward maximization.", "weaknesses": "The paper relies heavily on the Jeffreys divergence without sufficient comparative analysis against alternative divergence metrics. The mode-covering and mode-seeking behavior property paper mentioned about are only observed in lower dimension such as multimodal distribution in 1-dimension. An inclusion of other divergence types, especially in the iterative stages, could offer clearer insights into the unique advantages of Jeffreys divergence. Further, relevant literature on divergence measures in alignment tuning, should also be cited to contextualize this choice.\n- Go, Dongyoung, et al. \"Aligning language models with preferences through f-divergence minimization.\" Proceedings of the 40th International Conference on Machine Learning. 2023.\n\nAs the paper discusses the method’s efficiency, the paper would benefit from explicit comparison of the computational cost saved by BOND relative to traditional Best-of-N sampling, or comparisons with sampling approaches used in RLHF. This would clarify BOND’s potential advantages in real-world applications.\n\nAdditionally, while the paper addresses the challenge of sampling size N through iterative approximation, showing practical advantages like non-saturation compared to non-iterative BOND, this helpful randomness raised in iterative BOND is solely introduced by approximation randomness, which lacks controls or specific directions. This calls into question whether the proposed algorithm genuinely achieves a distilled $\\text{Bo}N^M$ distribution. \nThe substantial difference in $r(y)$ between iterative and non-iterative BOND  in Figure3 suggests a potential vulnerability to reward hacking, as discussed in Gao et al.\n- Gao, Leo, John Schulman, and Jacob Hilton. \"Scaling laws for reward model overoptimization.\" International Conference on Machine Learning. PMLR, 2023.\n\n\nThe introduction combines related works and problem setup, which could be structured more effectively. Detailed discussions on RLHF and Best-of-N would be more suitable in a separate related works section or could be incorporated into the problem setup. In the introduction, it would be clearer to emphasize the limitations of existing methods and highlight the advantages of the proposed approach over current methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zLTmC1CmmU", "reviewer": "Reviewer_X2hV", "summary": "This paper introduces a distribution matching-based Best-of-N distillation method that simulates the Best-of-N distribution space, while reducing the time overhead of N inferences to just one. Starting from the theoretical distribution of BoN, the authors construct the Iterative BOND algorithm based on Quantile estimation and the choice of Jeffreys Divergence, and further propose the more practically meaningful J-BOND algorithm.", "questions": "1. Can the authors compare with more fundamental baseline methods, such as BoN or other BoN distillation algorithms?\n\n2. Can the authors supplement additional experiments, including downstream validation and more ablation studies as discussed in Weakness 3?\n\n3. Can the authors prove more clearly the advantages of the BOND algorithm over other Alignment algorithms, in terms of both performance and efficiency, to make the argument more convincing?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Rigorous Theoretical Analysis: This work rigorously analyzes the distribution characteristics under Best-of-N sampling and establishes its connection with standard RLHF, as well as the specific reward value $r_{BOND}$ under this correlation. This provides a reliable theoretical foundation for the work, rather than being based on naive assumptions.\n\n2. Some Degree of Novelty: Although there is some concurrent work, the idea of distilling distributions from Best-of-N is fairly novel and important.\n\n3. Consideration of Practical Efficiency: I appreciate the authors' consideration of the practical efficiency of the algorithm. The proposed J-BOND algorithm theoretically has lower sampling complexity, which should increase the efficiency of RLHF.", "weaknesses": "1. Lack of Important Baselines: Given that the main purpose of the paper is to distill Best-of-N sampling, BoN performance should straightforwardly serve as an important baseline to analyze pros and cons in terms of performance and efficiency. Moreover, other concurrent BoN distillation algorithms [1] should also be considered.\n\n2. Lack of Downstream Validation: The main metrics in the paper, such as reward value and KL divergence, cannot be directly equated to the model's performance on downstream tasks. For an RLHF method, it is necessary to conduct experiments on downstream tasks and present more intuitive metrics to demonstrate the model's alignment performance.\n\n3. Insufficient Experimental Setup: The paper lacks exploration of several issues. For instance, BoN sampling heavily depends on the Reward Model, and the influence of different RMs on the BOND algorithm is not investigated. Additionally, a more nuanced exploration of Jeffreys Divergence with smoother β variations could be included; and the comparison between J-BOND and standard Iterative BOND lacks investigation.\n\n[1] Variational Best-of-N Alignment, https://arxiv.org/pdf/2407.06057", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MWqxSp120S", "reviewer": "Reviewer_rggM", "summary": "The paper is essentially distilling inference-time best-of-N sampling into training time. Specifically, the authors propose to train the policy to match the best-of-N distribution (which is an analytical form derived by the authors). The distribution matching is done through minimizing a combination of forward and backward KL. The behavior of J-BOND appears better for reward vs. num steps as well as KL(policy || ref policy) vs. num steps, compared to REINFORCE with various beta values.", "questions": "Would the curves in Figure 4(a) converge to similar spots if you run the algorithms long enough?\n\nDo the authors expect that the conclusions would be similar for much larger models?\n\nWriting: \n- pi(x,y) makes it look like a joint distribution; do the authors mean pi(y|x)?\n- 3e-6 means 0.000003; 3e^(-6) means 0.007. Which one are you referring to?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The motivation of distillation is great and this direction should be deeply explored. The paper is written carefully. The algorithm is clearly written. I checked the derivations and they seem correct so far.\n\nREINFORCE is a natural baseline and the authors have attempted multiple beta values for the baseline.", "weaknesses": "Have the authors tried reward shaping techniques for RLHF baseline, e.g., making the reward values more peaky – either very large or very small? \n\nI’d appreciate a more comprehensive discussion on how much the authors expect this technique to benefit downstream tasks.\n\nIt’ll be great if the authors can include more discussion on whether the baseline B is important or not in the algorithm.\n\nWhat ablations on beta and gamma in Algorithms 2 (balancing among forward KL, backward KL, additional regularization) would likely benefit downstream tasks more? It's still unclear to me why we want to put an equal/large weight on backward KL. More motivation would be nice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3OJVsXZZuL", "reviewer": "Reviewer_NVi9", "summary": "This article models the alignment of LLMs as a distribution-matching problem. Specifically, it considers the distribution induced by Best-of-N sampling as the target distribution and optimizes the Jeffreys divergence with respect to it for balancing the characteristics of forward & backward KL-based optimization. Additionally, this work derives an iterative form, updating the reference model during training.", "questions": "#### Question 1:\nLine 298, why is the proposed approach \"equivalently to distilling a Best-of-$N^M$ of the initial distribution $\\pi_{ref}$\"? Is this a qualitative or quantitative statement?\n#### Question 2:\nFigure 4, seems like a linear relationship between # steps and r(y) for iterative BOND.\nFor Figure 5, it seems like another kind of trending between $r(y)$ and # steps or $KL(\\pi||\\pi_{ref})$.\nFigures 6 and 7, present a log-like trend between the KL and reward similar to the REINFORCE algorithm.\nA similar trend is also observed in http://arxiv.org/abs/2406.00832 and http://arxiv.org/abs/2407.06057.\nAs discussed in http://arxiv.org/abs/2204.05862 and my experience, there can be an approximately linear relationship between $\\sqrt{D_{KL}}$  and $r$ for BoN sampling. \nIt would be interesting if the author could provide some empirical or theoretical intuition about such a relationship.\nDoes this indicate that even though performing a BoN distribution match, it is still more similar to the general policy-gradient RL algorithm (which may try to match another distribution)?\n#### Question 3:\nWhy is there an approximately linear relationship between #steps and KL in, as presented in Figures 5 and 6 for the BOND algorithm?\nFor the REINFORCE algorithm, it seems a quite different trend.\n#### Question 4:\nFigure 4 presents consistent trending between KL and $\\log_{p\\le(y)}$ across varying $N$ and algorithm, which is quite interesting.\nIs there any explanation for this phenomenon?\n#### Question 5:\nIs there any analysis or comparison of the reward overoptimization of this algorithm?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "1. This work implements the LLM alignment problem through Best-of-N distillation, which can be a sound direction for the development of algorithms in this field.\n2. This work formulates and discusses the BoN distribution and its relationship with the general RLHF target. \n3. This work proposes to utilize Jeffreys divergence to balance the mode-covering and mode-seeking behavior introduced by forward- and backward-KL- KL optimization.\n4. This work further integrates their method with EMA techniques and proposes an iterative version.", "weaknesses": "1. My biggest concern is the absence of direct comparison with close-related works http://arxiv.org/abs/2407.06057, http://arxiv.org/abs/2406.00832. It would be much more convincing if there is a clear comparison like Figure 1 in http://arxiv.org/abs/2407.06057, rather than a simple baseline REINFORCE presented in Figure 7 in this work.\n2. More discussion about why BoN distribution as the target can be included.\n3. Other possible additional analyses are discussed in the Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)