BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling
Authors: Lin Gui
Cristina Garbacea
Victor Veitch
Venue: NeurIPS 2024 poster (2024)
Keywords: LLM alignment; LLM fine tuning; Best of n; RLHF; Optimal Policy
Primary area: generative_models

Abstract: This paper concerns the problem of aligning samples from large language models to human preferences using *best-of-$n$* sampling, where we draw $n$ samples, rank them, and return the best one. We consider two fundamental problems. First: what is the relationship between best-of-$n$ and other (RLHF-type) approaches to aligning LLMs? In particular, when should one be preferred to the other? We show that the best-of-$n$ sampling distribution is essentially equivalent to the policy learned by RLHF if we apply a particular monotone transformation to the reward function. Moreover, we show that this transformation yields the best possible trade-off between win-rate against the base model vs KL distance from the base model. Then, best-of-$n$ is a Pareto-optimal win-rate vs KL solution.
The second problem we consider is how to fine-tune a model to mimic the best-of-$n$ sampling distribution, to avoid drawing $n$ samples for each inference. We derive *BonBon Alignment* as a method for achieving this. Experiments show that BonBon alignment yields a model that achieves high win rates while minimally affecting off-target aspects of the generations.

BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling
Lin Gui", Cristina Gårbacea", and Victor Veitch
Department of Statistics, University of Chicago Data Science Institute, University of Chicago
Abstract
This paper concerns the problem of aligning samples from large language models human preferences using best-of-n sampling, where we draw n samples, rank them, and return the best one. We consider two fundamental problems. First: what is the relationship between best-of-n and approaches to alignment that train LLMs to output samples with high expected reward (e.g., RLHF or DPO)? To answer this, we embed both the best-of-nt distribution and the sampling distributions learned by alignment procedures in common class of tiltings of the base LLM distribution. We then show that, within this class, best-of-n is essentially optimal in terms of the trade-off between win-rate against the base model vs KI. distance from the base model. That is, best-of-n is the best choice of alignment distribution if the goal is maximize win rate. However, best-of-n requires drawing n samples for each inference, a substantial cost. To avoid this, the second problem consider is how to fine-tune a LLM to mimic the best-of-n sampling distribution. We derive BoNBoN Alignment to achieve this by exploiting the special structure of the best-of-n distribution. Experiments show that BoNBoN alignment yields substantial improvements in producing a model that is preferred to the base policy while minimally affecting off target aspects. Code is available at https:/ gigggt.ccoo. gl- ybnbxb/ Baarb/BBBBB.,..
Introduction
This paper concerns the problem of aligning large language models (LLMs) to bias their outputs toward human preferences. There are now a wealth of approaches to this problem [e.g., Ouy+22; Chr+17; Kau+223; Li+24; Raf+23; Aza+244,, Here, we interested in the best-of-n (BoN) sampling strategy. In BoN sampling, interest, and return the best one. This simple procedure is surprisingly effective in practice [Bei+24; Wan+24; GSH23; Eis+23]. We consider two fundamental questions about BoN:
draw samples from the LLM, rank them the attribute of
1. What is the relationship between BoN and other approaches to alignment?
2. How can we effectively train a LLM to mimic the BoN sampling distribution?
In brief: we find that the BoN distribution is (essentially) the optimal policy for maximizing win rate while minimally affecting off-target aspects of generation, and we develop an effective method for aligning LLMs to mimic this distribution. Together, these results yield highly effective alignment method; see Figure for an illustration.
LLM Alignment The goal of alignment is bias the outputs of an LLM to be good on some target attribute (e.g., helpfulness), while minimally changing the behavior of the model off off-target attributes (e.g., reasoning ability). Commonly, the notion of goodness is elicited by collecting pairs of responses to many prompts, and asking (human or AI) annotators choose the better response. Then, these pairs are used to define a training procedure for updating the base LLM new, aligned, LLM that outputs responses that are better in the target attribute.
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

of ploc?
Average Length (Response) vs. Win Rate
Prompt
90
Tplecti
Model
80
Win Rete (%6
plu
andial cadn on Oriino adoPice cued there nuts
edkke nuthenet mugite anaprrty mould
70
aue lurget oor pio svre lawe, yie
pin pike
IPO III
Method
60
RoM sempling
BoMBoN
uoe wa the samdaa Cael
DPO original IPO original HH 8 Reference Model
the
and eugrecinn nember mine oe other ninkers
88
80
100
120
140
160
carber Orear hope ording ausber proldd toddle be enitiie
Average Length of Responses
al Houd
whing up qemmmlerr Bis.
DPO HNI
Figure 1: BoNBoN alignment achieves high win rates while minimally affecting off-target attributes of generation. Left: Average length of responses versus win rate of models aligned using each method on the Anthropic helpful and harmless single turn dialogue task, using achieves an excellent win rate while minimally affecting the off-target attribute length. Moreover, the BoNBoN aligned model effectively mimics this optimal policy, achieving a much higher win rate at low off off-target drift than other alignment approaches. Right: Sample responses from models with similar win rates to BoNBoN. Other methods require higher off-targer deviation to achieve comparably high win rate. We observe that this significantly changes their behavior on ofi-target aspects. Conversely, BoNBoN only minimally changes off-target behavior. See section 5 for details.
As predicted by theory, best-of-n
There are main approaches. First, RLHF methods train explicit reward model on the pairs, and then align the model using reinforcement learning with this learned reward [e.g., Ouy+22; Kau+23]. Second, contrastive methods directly the preference data define objective function for Sine-tuning the LLM [Raf+23; Aza+24; Eth+24;; Xu+24; HLT24]. In both cases, the trade-off between alignment and off-target behavior is controlled by hyper-parameter that explicitly penalizes the divergence from the base LLM. For example, in the reinforcement learning setting, this is done by adding regularization term that penalizes the estimated KL divergence between the aligned model and the reference model.
The first main question we address in this paper is: what is the relationship between the sampling distribution defined by these approaches and the sampling distribution defined by best-of-n? This is important, in particular, because in principle just use BoN sampling. However, it is not clear when each option should be preferred.
could forgo the explicit alignment training and
Now, the comparison of training-aligned models and BoN is not fully fair. The reason is that producing a BoN sample requires drawing n samples from the base LLM (instead of just one). This is substantial computational cost. The second main question we address is: if we do in fact want to sample from the BoN distribution, how can we train a LLM to mimic this distribution? this can be done effectively, then the inference cost of BoN sampling can be avoided.
We answer these questions with the following contributions:
1. We show that the BoN sampling distribution can be embedded in a common class with the distributions produced by training- based alignment methods. Within this common class, we derive the distribution with the best possible trade-off between win-rate against the base model vs KI. distance from the base model. Then, we show that the BoN distribution is essentially equal to this Pareto-optimal distribution.
2. We then develop effective method for training a LLM to mimic the BoN sampling distribution. In essenee, the procedure draws best-of-n and worst-of-n samples as training data, and combines these with an objective function derive by exploiting the analytical form of the BoN distribution. We call this procedure BoNBoN Alignment.
3. Finally, we show empirically that BoNBoN Alignment yields models that achieve high win rates while minimally affecting off-target aspects of the generations, outperforming baselines.
2

---

Preliminaries
Given a prompt x, a large language model (LLM) samples a text completion Y. We denote the LLM by and the sampling distribution of the completions by ITy x).
Most approaches to alignment begin with a supervised ane-tuning step where the LLM is trained with the ordinary next-word prediction task on example data illustrating the target behavior. We denote the resulting model by To. and call it the reference model. The problem we are interested in is how to further align this model.
To define the goal, we begin with some (unknown, ground truth) reward function r(x,y) that measures the quality of completion y for prompt The reward relates preferences in the sense that Y1 is preferred to Yo if and only if (xx, > rox, Informally, the goal is to produce LLM 7T, where the samples have high reward, but are otherwise similar to the reference model.
The intuitive requirement that the aligned model should be similar the reference model is usually formalized in terms of KI. divergence. The context-onndttooal KL. divergence and the KI. divergence from n, To a prompt set are defined
log 7 [x)
Toly xx) (00)2))
(rt,l11 I x) = By x)
DaL (T, == Ex-D [ (T,, x)].
We also need to define what it means for samples from the language model to have high reward. Naively, we could just look at the expected reward of the samples. However, in the (typical) case where we only have access to the reward through preference judgements, the reward is only identified up to monotone transformation. The issue that expected reward value not compatible with this unidentifiability. Instead, we consider the win rate of the aligned model against the reference model. The idea is, for given prompt, draw a sample from the aligned model and sample from the reference model, and see which is preferred. This can be mathematically formalized by defining the context-conditional win rate and the overall win rate on prompt set D:
PR,zno x == IP ( r(x,
Pr, = Ex-D [
(r(x, ())].
Reinforcement Learning from Human Feedback (RLHF) The most studied approach to alignment is RLHE This procedure follows two steps. First, the reward function is explicitly estimated from preference data, using the Bradley-Terry [BT52] model. Second, this estimated reward function is used in Kl-regularized reinforcement learning procedure to update the LLM. Denoting the estimated reward function by f, the objective function for the reinforcement learning step is:
@RLHF(TTg) nT) Exx-Dy-maor [f(x, PDxx (TgllTo))
(2.1)
where D is a prompt set and B is hyper pyper-parameter to control the deviation of T. from the reference model Rg. The policy म, is learned by finding the minimizer of the objective function in (2.1); e.g., using PPO [Sch+17].
Contrastive methods Contrastive methods use the preference data D = ((x, yr} Vi. where the prompt, and Yw and yi are preferred and dis dis-preferred responses, directly define an objective function for fine- fine-tuning the LLM, avoiding explicitly estimating the reward function. For example, the DPO [Raf+23] objective is:
Toly 1x) - - log LNpo np) - Ex......>> logo (Blog
To(ri |x) 1
(2.2)
Toly x)
To6%i
The aligned model found by optimizing this objective directly (via gradient descent).
'Fundamentally, the expectation of the transformed reward is not the reward of the transformed expectation.
3

---

Bradley-Terry and Alignment Targets In RLHE, the reward function is estimated using the Bradley-Terry model, which relates noisy observed preferences to rewards by:
pαy > Yo x)= -xx=
(2.3)
where o--) is the sigmoid function. In the particular case that the Bradley-Terry model is well- specified, then it can be shown that the analytic solution to both (2.1) and (2.2) is:
TRRIHF (v x) oc exp {
((x.v)) Toly x).
(2.4)
That is, the alignment procedures target an exponential tilting of the reference model by the reward function. Of course, it is not obvious when the Bradley-Terry model is well-specified, nor whether this particular tilting is a desirable target. Other works have considered explicitly or implicitly transforming the reward function change the target distribution [Wan+ 24; Aza+224]. Nevertheless, these works also take the target distribution to be a tilting the reference distribution.
Best-of-n sampling The best-of-n procedure as follows. Given a prompt x, sample Y2.,,,....,,, independently from the reference model Toly x). Then, select the response with the highest reward r(x,y;) as the final response. That is,
y
such that r(x,y;) max rxx,,,). 1sJSR
(2.5)
3 Best-of-n is Win-Rate vs KL Optimal
The first question address is: what is the relationship between the best-of-n distribution, and the distribution induced by training- based alignment methods?
3.1 A Common Setting for Alignment Policies
We begin with the underlying distribution of best-of-n sampling. Let Q, denote the cumulative distribution function of rxx, Yo), where Yo oz x). Suppose r(x,)): e - R is one-to-one mapping and To(y(x) is continuous", then the conditional density of the best-of-n policy is
xx)
(3.1)
Compare this to the RLHF policy n,, RLHP in (2.4). In both cases, the sampling distribution is re-weighted version the reference model Tag where higher weights are added to those responses with higher rewards. The observation is that both of these distributions and most alignment policies- can be embedded a larger class of wwward-weighte models. For any prompt x and reward model can define the fx -aligned model as:
T, | x) x f,(r(x,)) xx),
(3.2)
where fx is a non non-decreasing function that may vary across different prompts.
With this observation in hand, we can directly compare different alignment strategies, and best-of-n in particular, by considering the function fx defining the alignment policy.
3.2 Optimality: Win Rate versus KL divergence
To understand when different alignment policies are preferable, need to connect the choice of fx with a pragmatic critcria for alignment. The high-level goal is to produce a policy that samples high-reward responses while avoiding changing off-target attributes of the text. A natural formalization this goal is to maximize the win rate against the reference model while keeping the KL divergence low.
"This a reasonable simplification since we only care about the distribution of r(x,y) ) in the one- dimensional space and we consider the scenario where diverse responses without dominant one are expected. More details refer appendix B for discussion.
4

---

K0. Divergance verous Win Rate
Vin Fate Gain from Optinal Policy ove Regi-dl-n Sampling
0-
00
0.0000
Method
0.022
00
Oitmml Paule
0.000
Ohergenoo
Figure 2: The BoN essentially the the optimal policy in terms of win versus KL divergence. Left: The win rate versus KL divergence curves BoN and optimal policy, Right: The win rate difference between optimal policy and BoN policy for different n.
Optimal Policy Our aim to find the policy with the highest possible win rate at each KI. divergence level:
max E Ex-D PY-nty 1x)%o-Too (x( rxx,
(3.3)
subject (=
Now, this equation only depends on Y through the reward function r(x, Defining (x(r(x)))) as the distribution of r(x, Y) under To x), we can rewrite the objective as:
max Ex- (x) [rxx(x subject to lDxr. ()) = d,
By duality theory [BTNO1], there is some constant > 0 such that this problem is equivalent to:
max E D. (x) (x)x(,(( (z)l
(3.4)
Now, immediately recognize this objective the same the RLHF objective in (2.1) with the transformed reward function f(x,y) Q2(r(x, Then, the analytic solution to this problem is
opptimal
(3.5)
where is constant determined by the KL divergence penalty.
The following theorem makes the preceding argument precise. To simplify the argument, we will assume that the rewards assigned to outputs of the language model are continuous. This simplifying assumption ignores that there are only a countably infinite number of possible responses to any given prompt. However, given the vast number of possible responses, the assumption is mild in practice. Refer to appendix B for more detailed discussion.
Theorem 1. Let optimal be the solution to (3.3). Then, for all x, the density of the optimal policy is optimally xx) Toly x) x)exp (cQ,(r(x, IZ;,
,,
(3.6)
where Z is the normalizing constant, and c is positive constant such that
(c 1)ee +
log
d.
(3.7)
e
Furthermore, the context-conditional win rate and KL divergence of this optimal policy are
1. Context-conditional win rate: Pa.oiiin
(c-1)e++1 d(F-1)
2. Context-conditional KI. divergence: DDx.. (eooptimol |I xx)
(c-1)e'+1 e-1
log (1-
Since for any prompt x, both the context conditional win rate and KL divergence are constants, the overall win rate P spind and KL divergence Dx1 (n.pptimal)) on any prompt set D are also these values. [Proof]

---

3.3 The best-of-n policy is essentially optimal Now, we'd like to use the previous result to understand when the best-of-n policy is desirable. The win rate and KL divergence can be calculated with essentially the same derivation:
Theorem 2.
The context-conditional win rate and KL. divergence of the best-of-n policy are:
1. Context-conditioal win rate: Pafl m+1
2. [JH22] Context-conditional KL divergence: Dea (,e,:)] |x) = log(n)) m_1
Since both are constants, the overall win rate Pat and Kl divergence Dx (Tt (To) on any prompts
the same values. [Proof].
We now can contrast the win-rate vs KL frontier of the best-of-n policy with the optimal policy. Figure 2 shows KI. divergence versus win rate values of best-of-n policy and the optimal policy. The maximum difference in win rates (at = 2) is less than percentage point. Larger values of approximate the optimal policy even more closely. In summary:
The best-of-n policy is essentially optimal in terms of win rate versus KL. divergence.
3.4 Implicit vs Explicit KL regularization
RLHF and contrastive alignment methods include a control the trade-off between KL. divergence and model reward. By contrast, best-of-n only controls the KL. drift implicitly. This can actually be substantive advantage. There are two reasons. First, it is generally unclear how well controlling K1. actually captures the real requirement of controlling the degree to which off-target attributes of the text are modified. There might be multiple possible policies with fixed KL level that have radically different qualitative behavior. Second, in practice, the KL drift from the base policy needs to be estimated from a finite data sample. This may be extremely difficult-it is a very high dimensional estimation problem. Mis Mis-estimation of the KL is particularly problematic when we are explicitly optimizing against the estimate, because this may let the optimizer exploit mis nittmmatina.oon Empirically, we find that measured KL can have poor correspondence with attributes of text that humans would judge to be salient (see section 5). In particular, we find large variation response length that is not reflected in estimated KL.
that attempts to explicitly
The best-of-n procedure avoids both problems, since it avoids the need to estimate the KL drift, and since it does not explicitly optimize against the KL. drift.
BoNBoN: Best-of-n fine tuning
From section 3, we know that the best-of-n policy is essentially optimal in terms of win rate and KL divergence. Accordingly, it is often a good choice for the alignment policy. However, the best-of-n policy has significant practical drawback: it requires drawing n samples for each inference. This is a substantial computational expense. We now turn to developing a method to train language model to mimic the best-of-n sampling distribution. We call this method BoNBoN Alignment.
Setup The basic strategy here will be to use best-of-n samples to train a language model to mimic the best-of-n policy. We produce the training data by sampling n responses from the reference model Tg, and ranking them. The best and worst data are the samples with highest and lowest reward. Their corresponding best-of and worst-of n sampling distributions are denoted as ne(()) and n.(1), The task is then set up an optimization problem using this sampled data such that the solution approximates the best-of-n policy. To that end, we consider objective functions that have the best-of-n policy as a minimizer in the infinite data limit. (In practice, as usual, we approximate the expectation with an average.)
Beirami et al. [Bei+24] discuss that since the distribution of the language model is discrete, n!(a) has different form from that in Theorem 2, and the actual KL divergence is smaller. However, due to large cardinality of the corpus and the low probability of each response, the actual density is very close to (3.1) and the KL divergence is almost its upper bound log(n) =1

---

SFT-BoN. The most obvious option is to train the model to maximize the log ligeliikl of the best-of-n samples. The associated objective is:
LSFT (TT;; To) = -E = XDD. [log To To(M()) x)],
(4.1)
and it is well- that the minimizer is rT(") The training procedure is simply minimize the sample-average version of this objective. We call this training method SFT-BoN because supervised fine-tuning best-of-n samples. Although SFT-BoN is valid theoretically, it turns out to be data inefficient, and we observe only marginal improvement over the reference model empirically (see section 5).
IPO-BoN. A limitation of the best-of-n procedure is that it only makes use of the winning sample, throwing away the rest. Another intuitive option to construct a pairwise dataset and train the language model by contrastive method. Concretely, construct the pairwise data by picking the best and worst responses. We want to construct an objective function using this paired data that has the best-of-n policy minimizer.
The key result we require is:
Theorem 3. For any fixed n,
Th I x)
loog TT((m)) x))
log
yoo yumme!
(1) 11, (1(1) x)
TT [x) x)] 2/;
where
B== 2(n 2(n-1) 1) 1- 1/k
(4.2)
[Proof].
Following this result, we define the contrastive objective function as:
Th ()m) x) TT xx) )²1
SipO-BoN(TT'' n2)) = Bx-D. m ymm=+(")
log
log
(4.3)
TT((()) x) TT()(a)) x) 2B:
The optimizer of this objective is a policy where the log- ligeliiel ratio of the best and worst samples is equal to that of the best-of-n policy. We call this training method IPO-BoN because essentially the IPO objective on the best-and the IPO hyper parameter. We emphasize that the IPO-BoN objective does not involve any hyper parameters, there is only one choice for B for each n.
samples, with particular choice for
We find in section 5 that IPO-BoN is much more data efficient than the SFT-BoN. However, this method (like IPO) has the disadvantage that it only controls the likelihood ratios on the sampled data. In particular, this means that the optimizer can cheat by reducing the likelihood of both the winning and losing responses, long as the loser's likelihood decreases more (so the ratio still goes up). Reducing the probability of both the winning and losing examples requires the optimized model to shift probability mass elsewhere. practice, we find that tends increase the probability of very long responses.
BonBon Alignment
We can now write the BoNBoN objective:
The BoNBoN alignment objective is:
LBONBON (TI; (Tg) = ( Fo) Lo To),
(4.4)
where ESFT-BON and LPONBBN are defined in (4.1) and (4.3), and a is hyper parameter that balances the SFT and the IPO objectives.
We call the procedure BoNBoN because is combination two objective functions that have the best-of-n policy as minimizer. Relative to SFT alone, BoNBoN can be understood improving data efficiency by making use of the worst-of-n samples. Relative to IPO alone, BoNBoN can be understood preventing cheating by forcing the likelihood of the best-of-n samples to be
7

---

high. We emphasize that both objective functions target the same policy; neither is regularizing towards some conflicting objective. That is, the trade-off between win-rate and off-target change is handled implicitly by the (optimal) best-of-n procedure. This is in contrast to approaches that manage this trade-off explicitly (and sub-optimally) by regularizing towards the reference model. Reflecting this, we choose so that the contribution of each term to the total loss is approximately equal.
5 Experiments
5.1 Experimental Setup
We study two tasks: a) single-turn dialogue generation, for which we conduct experiments on the Anthropic Helpful and Harmless (HH) dataset [Bai+22] and b) text summarization, for which we use the OpenAl TL;DR dataset [Sti+20]. Due to computational constraints, we filter the HH data to only keep prompts for which response length is less than 500 characters, resulting in 106,754 training dialogues. For TL;DR dataset, we discard instances where the input post length less than 90 characters, resulting in 92,831 (14,764 prompts) training posts. Each example in both datasets contains a pair of responses that were generated by a large language model along with label denoting the human- -amaan-referred response among the two generations.
We want to compare different alignment methods on their ground truth win rate, Accordingly, we need ground truth ranker. To that end, we construct data by using an off-the-shelf reward model* as our ground truth. (In particular, we relabel the human preferences).
As the reference model, we fine-tune Pythia- 23] with supervised fine-tuning (SFT) on the human-preferred completions from each dataset. For alignment methods other than BoNBoN, we draw π = 8 completions for each prompt, and we use the best and worst completions as training data for them. For BoNBoN, we vary n from 2 to 8.
We use DPO and IPO as baselines for the alignment task. We run both procedures on both the original (Anthropic HH OpenAI summarization) datasets, and the best--nd-worss-off completions. The former gives a baseline for performance using stronger responses, the latter gives baseline for using exactly the same data BoNBoN. Both IPO and DPO include hyper parameter B controlling regularization towards the reference model. We report results for each method run with several values of B. For BoNBoN, we value is chosen so that the SFT and IPO terms in the loss have approximatcly equal contribution. Further details can be found in appendix C.
0.005 for all experiments. This
5.2 BoNBoN achieves high win rate with little off-target deviation
We are interested in the win-rate off-targes deviation trade-off. We measure off-target deviation in two ways: (1) the estimated KI. divergence from the base model, and (2) the average length of model responses. Length is noteworthy because is readily salient to humans but (as see in the results) alignment methods can change it dramatically, and it is not well captured by the estimated KL divergence. We show win-rate off-target behavior for cach trained model in Figure 3. The main obscrvation is that BoNBoN achieves a much better win-rate vs off-target tradcoff than any other approach. In particular, DPO/IPO ß values that achieve comparable win- win-rates result in high off-target deviation-eat e.g. nearly doubling the average response length!
To further explore this point, examine sample responses from baseline models with similar win-rates to BoNBoN. Examples are shown in Figure and tables and 6. Other approaches can dramatically change off-target behavior.
5.3 BoNBoN mimics the best-of-n policy
Figure 3 shows SFT and IPO fine-tur ine-tuned on the best-of-n data. We observe that BoNBoN dramati- cally outperforms these methods at all values of β, and is closer to the (optimal) BoN distribution. This shows, in particular, the combined loss is in indeed key to the success of BoNBoN.
One substantial practical advantage of BoNBoN is that it is nearly the goal is to mimic the best-of-n distribution, which is known to be optimal, we do not need to sweep hyper-parameters parameters for the 'best' choice of win-rate vs KL. particular, the f term in IPO is analytically derived in Theorem 3. In Figure 5 we show the win rate off-target behavior
free. Because

---

Summarization
Divergence
Average Length (Reeponse)
100
A0hh Plae
40
123
100
Responsie
Method
BoN sampling thecrsfical
BolkBNN
DPO original H+
IDD originall HH Reference
SFT BoN
Bah
Helpful and Harmless
KL. Diver Divergence va. Win Rate
Avarage Length (Response) vs. Win Rate
We 53t (%)
90
60
10. Divergence
Ausraga Lengh- Rreperooe
Method
Bal sampling
BoN theoredcal
DPO BON
BaNBBN
DPO miginal HH
IPO BoN
IPO original H+ Referance Modal
SFT BoN
Figure 3: BoNBoN achieves high win-rates while minimally affecting off target aspects of generation. Each point model aligned with the indicated method. We measure using ground truth ranker. assess change in off-target behavior, we measure both estimated KL divergence (left) and average response length (right). Above: Comparison of BoNBoN with baselines for the summarization task. Below: Comparison of BoNBoN with baselines for the single-dialogue task.
against the base model
Prompt:
Human: list dirty
Assistant:
Reference
how ahout this?
BoNBoN (n=8)
I'm afrnid that's something support.
The word "dirty" considered very derogatory word. not something say to another person. I'm surry that you heard that term being used in that way.
DPO original HH
I'm sorry, don't support using offensive language, and I'm not aware of specific language that would be considered 'dimyy. Someone seeking to use an offensive word might feel understands why some words are considered offensive. hope this was helpful!
IPO original HH
being misunderstond, offended, but the truth that
Table 1: With similar win rates, only BoNBoN does not modify the off-target attributes. The responses of the same prompt drawn from models fine tuned by BoNBoN, DPO and IPO the original HH data with no sampling technique. The win rate of each model around 85%.
for several other choices for in the IPO term. We observe that, generally, the default B has an excellent win-rate vs off-target trade-off. Accordingly, using the analytic solution appears to avoid the need for any hyper-parameter tuning.
Discussion and Related work
Best-of-n BoN sampling is widely used for LLMS [e.g., Sti+200; Nak+21; Liu+23; Gul+23; Tou+23; GSH23]. Due to its practical importance, it has also attracted some recent theoretical attention [e.g., Mud+23; Bei+24; Yan+24; Jin+24]. Beirami al. [Bei+24] show closed form probability mass function of the BoN policy in discrete case and provide new KL estimator for
9

---

it. Yang et al. [4] define the optimality in terms of minimizing the cross entropy given an upper bounded KL, and show that BoN is asymptotically equivalent the optimal policy, which is in line with our findings. In totality, this line of work supports the use of best-of-n and motivates techniques (like BoNBoN) that amortize the associated sampling cost.
Fine-tuning using best-of-n data has also been tried in many existing works align LLMs with human reference. Dong et al. [Don+23]] and Xiong et al. [Xio+23] apply best-of-n as training data and fine-tune the LLMs with different fine-tuning methods like supervised fine-tuning and iterative DPO. Touvron et al. [Tou++23] draw best-of-n samples and do gradient updates in the iterative fine-tuning step to further reinforce the human- aaman-aligned reward.
LLM alignment There is extensive literature on aligning LLMs [e.g., 19; YK21; Qin+-22; She+23; Wan+23; Mud+23; Ouy+22; 2h;+23; Raf+23; Yua+23; Aza+24; Eth+24; Xu+24; HLT24; Wan+244; Liu+24; Par+24]. Broadly, this work uses preference-labeleed data to (implic- itly) define goal for alignment and optimizes towards while regularizing to avoid excessively changing off-target behavior. Relative to this line of work, this paper makes two main contribu- tions. First, we embed the best-of-n policy into the general alignment framework, showing is optimal in terms of win-rate vs KL. Second, we derive BoNBoN alignment as way of training an LLM to mimic the best-of-n distribution. Notice that this second goal is significant departure from previous alignment approaches that define the target policy through an objective that explicitly trades off between high-reward and changes on off- off-target attributes. We do not have any regularization towards the reference model. This has some significant practical advantages. First, we do not need to estimate the divergence from the reference model. As we saw in section 5, estimated KL. can fail to capture large changes in response length, and thus mis-estimate the actual amount of ofr-tt deviation. Second, we do not need to search for a that balances the conflicting goals. This hyper-paramete existing alignment methods. (We do need to select a, but this is casier since the aim is just to balance the loss terms rather than controlling trade-off in the final solution.)
search is significant challenge in
Alignment methods can be divided into those that operate online-in the sense of drawing samples as part of the optimization procedure- offline. The online methods are vastly more computationally costly and involve complex and often unstable RL-type optimization procedures [2he+23; San+23]. However, the online methods seem to have considerably better performance [c.g., 24]. The results in the present paper suggest this gap may be artificial. Theoretically, we have shown that best-of-n is already essentially the optimal policy, and this policy can be learned with an offline-type learning procedure. Empirically, we saw in section that BoNBoN vastly outperforms the IPO and DPO baselines run on the existing preference data (which is standard procedure). would be an interesting direction for future work to determine whether online methods have a real advantage over BoNBoN. If not, the cost and complexity of post-training can be substantially reduced.
Our empirical results also support the idea that alignment methods should use on-policy data even if these samples are relatively weak-we see aligning with best-of-n samples substantially outperforms aligning with the original HH summarization completions. Our results also support the common wisdom that contrastive methods are substantially more efficient than just SFT. Interestingly, we have found that the main flaw of contrastive methods they cheat by pushing down the likelihood of preferred solutions, leading drift on off-target ettributes-can be readily fixed by simply adding in an extra SFT term.
The results here can be understood studying particular choice of reward transformation used for alignment. Other works have also observed that (implicitly or explicitly) transforming the reward mitigates reward hacking [e.g., Aza+24; Wan+24; LSD24; Ska+22]. Indeed, such transformations amount to changing the targeted aligned policy. Our results show how to optimize win rate. However, this is not the only possible goal. For example, Wang et al. [Wan+24] take the target alignment policy as particular posterior distribution over the base model. Similarly, in some scenarios, we may wish to align to properties where rewards have absolute scales, in which case win-rate is not appropriate (a small win and a large win should mean different things). Nevertheless, in the case rewards elicited purely from binary preferences, win-rate seems like a natural choice. It would be an exciting direction for future work to either show that win-rate is in some sense the best one can do, or to explicitly demonstrate an advantage for approaches that use explicit reward scale.
10

---

Acknowledgements
Thanks to Alekh Agarwal for pointing out a typo in a previous version. This work is supported by ONR grant N00014-23-1-259 and Open Philanthropy.
References
[Aza+24]
M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. "A general theoretical paradigm to understand learning from human preferences". In: International Conference on Artificial Intelligence and Statistics. PMLR. 2024 (cit. on
pp. 1, 2, 4, 10).
[Bai+22]
Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, Ganguli, T. Henighan, et al. "Training a helpful and harmless assistant with reinforcement learning from human feedback". In: arXiv preprint arXiv:2204.05862 (2022) (cit. on p. 8).
[Bei+24]
A. Beirami, A. Agarwal, J. Berant, A. D'Amour, J. Eisenstein, C. Nagpal, and A. T. Suresh. "Theoretical guarantees on the best-of-n alignment policy". : In: arXiv preprint W/Xiv:240.0166 (2024) (cit. pp. 6, 9, 16).
[BTNO1]
A. Ben-Tal and A. Nemirovski. Lectures on modern convex optimization: analysis, algorithms, and engineering applications. 2001 (cit. on p. 5).
[Bid+23]
S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. Prashanth, E. Raff, et al. "Pythia: a suite for analyzing large language models across training and scaling". In: International Conference on Machine Learning. PMLR. 2023 (cit. on p. 8).
[BT52]
R. A. Bradley and M. E. Terry. "Rank analysis of incomplete block designs: i. the method of paired comparisons". In: Biometrika 3/4 (1952) (cit. on p. 3).
[Chr+17]] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. "Deep Advances in Neural Information
reinforcement learning from human preferences" Processing Systems. 2017 (cit. on p. 1).
[Don+23] Dong, W. Xiong, D. Goyal, Zhang, W. Chow, R. Pan, Diao, J. Zhang, K. SHUM, and T. Zhang. "RAFT: reward ranked finetuning for generative foundation model alignment". In: Transactions on Machine Learning Research (2023) (cit. on p. 10). [Eis+23] J. Eisenstein, C. Nagpal, A. Agarwal, A. Beirami, A. D'Amour, D. Dvijotham, A. Fisch, K. Heller, S. Pfohl, D. Ramachandran, et al. "Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking". In: arXiv preprint 23X292311.09224 (2023) (cit. on p. 1).
[Eth+24] K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. "Kto: model alignment as prospect theoretic optimization". In: arXiv preprint arXiv:240..01306
(2024) (cit. on pp. 2, 10).
[GSH23]
I. Schhulman, and Hilton. "Scaling laws for reward model overoptimization"
In: International Conference on Machine Learning. PMLR. 2023 (cit. pp. 9).
[Gul+23]
C. Gulcchre, T. L. Paine, S. Srinivasan, K. Konyushkova, L. Weerts, A. Sharma, A. Siddhant, A. Ahern, M. Wang, Gu, et al. "Reinforced self-training (rest) for language modeling". In: arXiv preprint arXiv:2308. 08998 (2023) (cit. on p. 9).
[HLT24]
J. Hong, N. Lee, and J. Thorne. "Reference- free monolithic preference optimization with odds ratio". In: arXiv preprint arXiv:2403. 07691 (2024) (cit. on pp. 2, 10).
[JH22]
L. G. Jacob Hilton. Measuring Goodhart's law. 2022 (cit. on pp. 6, 16).
[Jin+24]
Y. Jinnai, T. Morimura, K. Ariu, and K. Abe. "Regularized best-of-n sampling mitigate reward hacking for language model alignment". In: arXiv preprint #rXiv:240.00105 (2024) (cit. on p. 9).
[Kau+23]
T. Kaufmann, Weng, V. Bengs, and Hüllermeier. "A survey of reinforcement learning from human feedback", In: arXiv preprint arXiv:2312. 14925 (2023) (cit. on
pp. 1, 2).
[LSD24]
C. Laidlaw, S. Singhal, and A. Dragan. "Preventing reward hacking with occupancy measure regularization". In: arXiv preprint arXiv:2403.00 03185 (2024) (cit. on p. 10). K. Li, O. Patel, F. Viégas, H. Pfister, and M. Wattenberg. "Inference-tine intervention: eliciting truthful answers from a language model". In: Advances in Neural Information Processing Systems (2024) (cit. on p. 1).
[Li+24]
11

---

[Liu+23]]
T. Liu, Y. Zhao, R. Joshi, M. Khalman, M. Saleh, P. J. Liu, and J. Liu. "Statis- tical rejection sampling improves preference optimization". In: arXtv preprint (2023) (cit. on p. 9).
arXiv:2309.0666
[Liu+24]
T. Liu, Y. Zhao, R. Joshi, M. Khalman, M. Saleh, J. Liu, and J. Liu. "Statistical
rejection sampling improves preference optimization". In: The Twelfth International Conference on Learning Representations. 2024 (cit. on p. 10).
[Mud+23] S. Mudgal, J. Lec, H. Ganapathy, Y. Li, T. Wang, Y. Huang, Z. Chen, H.-I. Cheng, M. Collins, T. Strohman, ct al. "Controlled decoding from language models". In: arXiv preprint arXiv:2310. 17022 (2023) (cit. pp. 9, 10).
[Nak+21] R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. al. "Webgpt: browser- assisteed question question-answerte answering with human feedback". In: arXiv preprint arXiv:21 12. 2.09322 (2021) (cit. on p. 9).
Kosaraju, W. Saunders,
[Ouy+22]
I. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. E Christiano, J. Leike, and R. Lowe. "Training language models to follow instructions with human feedback". In: Advances in Neural Information Processing Systems. 2022 (cit. on pp. 1, 2, 10).
[Par+24]
R. Park, R. Rafailov, S. Ermon, and C. Finn. "Disentangling length from quality in direct preference optimization", In: arXiv preprint 2rXiv:2403.19115 (2024) (cit. on 10).
[Qin+22] L. Qin, S. Welleck, D. Khashabi, and Y. Choi. "Cold decoding: energy-based con- strained text generation with langevin dynamics". In: Advances in Neural Information Processing Systems. 2022 (cit. on p. 10).
[Raf+23]
R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. "Direct preference optimization: your language model secretly a reward model". In: seirth Conference on Neural Information Processing Systems. 2023 (cit. on pp. 1-3, 10).
[San+23]
M. Santacroce, Y. Lu, H. Yu, Y. Li, and Y. Shen. "Efficient rlhf: reducing the memory usage of ppo". In: arXiv preprint arXiv:2309. 000:2000000 (2023) (cit. on p. 10).
[Sch+17] J. Schulman, E Wolski, P. Dhariwal, A. Radford, and O. Klimov. "Proximal policy optimization algorithms". In: arXiv preprint arXiv: 10t::1777.000 1707001777 (2017) (cit, on p. 3).
[She+233] T. Shen, R. Jin, Y. Huang, C. Liu, W. Dong, Z. Guo, X. Wu, Y. Liu, and D. Xiong- "Large language model alignment: a survey". In: arXiv preprint arXiv:2309. 15025 (2023) (cit. on p. 10).
[Ska+22] J. Skalse, N. Howe, D. Krasheninnikov, and D. Krueger. "Defining and characterizing reward gaming". In: Advances in Neural Information Processing Systems (2022) (cit.
on P. 10).
[Sti+20]
N. Stiennon, Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. "Learning to summarize with human feedback". In: Advances in Neural Information Processing Systems. 2020 (cit. on pp. 8, 9).
[Tan+24] Y. Tang, D. Z. Guo, Z. Zheng, D. Calandriello, Y. Cao, E. Tarassov, R. Munos, B. Ávila Pires, M. Valko, Y. Cheng, and W. Dabney. Understanding the performance gap between online and offline alignment algorithms. 2024 (cit. on p. 10).
[Tou++3]]
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, Bhargava, S. Bhosale, et al. "Llama 2: open foundation and fine-tuned chat models". In: arXiv preprint 23X7:22307.09288 09288 (2023) (cit. on pp. 9, 10).
[Wan+23]
Y. Wang, W. Zhong, L. Li, E Mi, X. Zeng, W. Huang, L. Shang, X. Jiang, and Q- Liu. "Aligning large language models with human: survey". In: arXiv preprint arXiv:2307.11 12966 (2023) (cit. on p. 10).
[Wan+24]
Z. Wang, C. Nagpal, J. Berant, J. Eisenstein, A. D'Amour, S. Koyejo, and V. Veitch. Transforming and Combining Rewards for Aligning Large Language Models. 2024
(cit. on pp. 1, 4, 10).
[Xio+23]
W. Xiong, H. Dong, C. Ye, Z. Wang, H. Zhong, H. Ji, N. Jiang, and T. Zhang. "Iter- ative preference learning from human feedback: bridging theory and practice for rlhf under kl-constraint". In: ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models. 2023 (cit. on p. 10).
[Xu+24]
H. Xu, A. Sharaf, Y. Chen, W. Tan, L. Shen, B. Van Durme, K. Murray, and Y. J. Kim. "Contrastive preference optimization: pushing the boundaries of 1m performance
12

---

in machine translation". In: arXiv preprint arXiv:2401.08411 (2024) (cit. on pp. 2,
10).
[Y]
J. Q. Yang, S. Salamatian, Z. Sun, A. T. Suresh, and A. Beirami. "Asymptotics of language model alignment". In: arXiv preprint arXiv:2404.01730 (2024) (cit. on pp. 9, 10).
[YK21]
K. Yang and D. Klein. "FUDGE: controlled text generation with future discriminators".
In: Proceedings of the 2021 Conference of the North American Chapter of the Association
for Computational Linguistics: Human Language Technologies. 2021 (cit. on p. 10). Z. Yuan, H. Yuan, C. Tan, W. Wang, S. Huang, and F. Huang. "Rrhf: rank responses
[Yua+23]
to align language models with human feedback without tears". In: arXiv preprint arXiv:2304.08. 05002 (2023) (cit. p. 10).
[Zha+23] Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and P. Liu. "Slic-hf: sequence likelihood calibration with human feedback" In: arXiv preprint arXiv:2305. 10425
(2023) (cit, on p. 10).
[Zhe+23] R. Zheng, S. Dou, S. Gao, Y. Hua, W. Shen, B. Wang, Liu, S. Jin, Q- Liu, Y. Zhou, et al. "Secrets of rlhf in large language models part ppo". In: arXiv preprint arXiv:2307. 04964 (2023) (cit. on p. 10).
[Zic+19] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodci, P Christiano, and Irving. FFine-tuning language models from human preferences". In: arXiv preprint 1rXiv:1909.08993 (2019) (cit. on 10).
13

---

A Theoretical Results
This section contains all theoretical results of the paper. We start with elaborate some useful notations and lemmas, and then provide the proofs of all thcorems in the main text of the paper.
For simplicity of proofs below, we first define general reward allgned policy:
Definition 4 (Reward aligned model rτ.). For any prompt x, the reward aligned model n!! satisfies
, x)
Z, To (x)
(A.1)
where (f: f is increasing and 2 0} and 2, the normalizing constant.
This general policy class includes both the optimal policy rr.ptimal and the best-of-n policy. More specifically, the optimal policy n.. opoptiall is with the choice of exponential functions and the the best-of-n policy is with the choice of power functions.
Before proofs of the theorems, we first illustrate useful lemma.
A.1 A Useful Lemma
Lemma 5. For T, with the definition (A.1), the following conclusions hold:
L (f)ddu
1. Context-conditional win rate is Pr,
To (w)du
2. Context--onditional KL divergence is
So logl (u)) So fuudu
DxL (T1,1I (x)
-log
Furthermore, both win rate and KL divergence independent of distribution of
Proof. The context-conditional win rate is
Papsmol xx PYNn, (x)% ()) rxx rxx,
Toly mob1 Q,((,,))))))__0
n, (x))
J Toly (x)
[ Toly ())(0,,tr(x,
So of (u)du
J Foly x)) (Txx,
So (u)du"
where the last equation is because Q, (x(x, Yg)) ~ U(O,1) when Y, ~ noly x).
The context-conditional KL divergence is
T, |x) Toly x) dy
Dxa (Tt,1 | (x) TT,(y x)
=ʃ Taly (r(x,,))(
s(Q,(r(x,))))
log I (o00
dy
S Toly (rxx,
f(u) log
f(u) f'
du So (w))og(f(uud
log [ F(Wdy)
=]
fu)ddu
S/' fu)ddu
where the third equation uses the fact that (r Yx)) U(O, 1) when Yo~ xx((x)).
14

---

A.2 Proof of Theorem 1
Theorem 1. Let optimall be the solution to (3.3). Then, for all the density of the optimal policy is "Fe
Te
opptimal (y x)= moly x exp (cQ, (r(x)))) /Z,,
(3.6)
where Z is the normalizing constart,, and с positive constant such that
e ( d. (c 1)ee + - log
(3.7)
Furthermore, the context-conditionn win rate and KL divergence of this optimal policy are
(o-1)e'+1 ((6--))
1. Context-conditional win rate: Par 2no
2. Context-conditional KL divergence: Dxa.
opttimall Th
ITo xx)
(c-1)='+1
log (()))
e
Since for any prompt x, both the context conditional win rate and divergence are constants, the overall win rate P smari values. [Proof]. mis mno
and KL divergence Dxi (11 (1 on any prompt set D are also these
Proof.
Since (3.3) is equivalent to (3.4) with some O. Now have:
argmaxE (0] (rxx,((x)))) - (a (TI| d)
Qx(r(x,))) log x(y x)
argmax yy
To x)
rol xx)
1
B.ggiin 1x)
log
Qx (r(,,,)))))
Foly xx)
B
F xx)
argmin BxpninnBBBBBBeeeee 1x)
log
log Z
(A.2)
2 To exp (xxx(((x))))
< (x) 2 T ex) (xx,,(()))))
argmin D.- Bywnty 1x)
log
1
Toly x) p (2(0(())))))
argmin DxL mly
x))
where the second to last equation is because the normalizer Z is constant:
( dy = ('' e du = B
exp exp (()
Z :: Toly x)
Since the KL divergence is minimized if and only if the two distributions are identical, the minimizer of (A.2) is the II" satisfying that for any prompt x e D,
1 Z moly (x)exp(cQ,((xxx.))))
x'' |x) =
where c= 1/B.
Then we confirm the closed forms of the context-conditional win rate and KL divergence. It is straightforward deduction from Lemma 5 by just plugging / (u) ecu in the context-conditional win rate and KI. divergence.
Furthermore, we require
DxL. (TT'l = d.
15

---

Therefore, we have
d = D (TT)
exp (xx,,),(((x,,))) dy
Toly xx) (cQx
So' cen udu
log (Z)
(c 1)ee +1
log e" e--1
ec -6-1
c
This completes the proof.
A.3 Proof of Theorem
Theorem 2. The context-conditioal win rate and KL divergence of the best-of-n policy are:
1. Context-conditional win rate: Pal)>a, xx
m+1
2. [JH22] Context-conditional KL divergence: DR1. (IT x) = log(n)) n-1
Since both constants, the overall win rate Pap and Kl divergence Dxa (T.(7))) [| any prompts set are the same values. [Proof].
Proof. Plug (u) nı"11 in the win rate and kl divergence formats Lemma 5 and the theorem follows.
A.4 Proof of Theorem 3
Theorem 3. For any fixed n,
log To(((m)) x)) nT, (1(1) x) To(ra) x) 2/
(1(m x) (1)
log
X-D st
where
B== 2(n 1)n-1)Em11 1/k
(4.2)
[Proof].
Proof. Denote U(6) and U()) the order statistics of the uniform distribution. That is, suppose U, U,,",,,, are independently and identically from U(0, 1), and
Unm) max U, and U() min U,. isise
isisa
The value of B is derived as follows:
E3
x yaron h ( (mn0)))))))x)))
2
x ( x)
Th (v()) x)
log
-log
TolY(a) xx)
ToCFi x)
(rX, )
-By-Dyyor say =B
log
= (n 1)E,DD [log (Um) - log (Un)
(r,
=(n-1) 1).
nlog(u)u* du (n 1).
nlog(w)(1-ui -2) du
R-1
11 + (n-
n
Beirami et al. [Bei+24] discuss that since the distribution of the language model is discrete, re(a) has different form from that in Theorem 2, and the actual KL divergence is smaller. However, due to large cardinality of the corpus and the low probability of each response, the actual density is very close to (3.1) and the KL divergence is almost its upper bound log(n) 21
16

---

Therefore,
B 1
2(n 1) 11
1/k
Discrete versus Continuous Uniform Distribution
Since the cardinality of a corpus is finite and not all combinations of words is possible, the number of all responses should also be limited. Suppose given a prompt x, the set of all responses
y = [y;))=
and their corresponding probabilities and rewards p; and r; r(x, (;), 1...,,4 L. We assume the reward model good enough to provide different rewards for all responses. Without loss of generality, suppose the rewards of them are increasing the subscript rises, i.e.,
< 12 < <... .....L
For simplicity, we use Pıá to represent the sum Σj=r throughout this section. Moreover, when i 0, we define P1:0 = O.
B.1 Q₂ normalization in discrete case
We first revisit the distribution of (r(x,, Yo)) where Y, ~ Tooy x). In the continuous case, this one distributed from the uniform distribution U(O, 1). In the discrete case, since To x) is discrete, Q2(r(x Yo))'s distribution becomes discrete one with the following CDF:
Ú(u) = Σ pi (w2P)))
(B.1)
(u) is a staircase function with the i-th segment from the left having length of Pt. For all = Pı. with i from 1 to L, this new CDF still satisfies Úu)) Figure 4 shows two examples of CDF of Ū with small and large corpus. responses is large and the probability of each response is low, the discrete distribution is almost identical to the continuous uniform distribution.
obvious that when the number of all possible
3-
3-
(a) small
(b) large
Figure The CDF of discrete transformation is almost the uniform distribution when the number of responses is large. Left: CDF discrete case differs a lot from uniform distribution when is small. Right: The difference between the discrete CDF and the CDF of the uniform distribution negligible.
Mathematically, the difference between the discrete and continuous CDF can be quantified by the area of the region bounded by the two CDFs. Specifically, the area is
1
Arca =
p
udu
p; Pı:(i-))-
(B.2)
2
1=1
17

---

Since E P: P1:(-1) goes S udu max, P: 0, this area also converges to 0 as max; P: -
In practice, the difference between the two CDFs is negligible since the probability of any response low.
B.2 KL Divergence and Win Rate
In the discrete case, the PMF of the best-of-n policy has different form from (3.1). Specifically, its PMF
(y xx) (p (P=a i=
This is actually similar to its continuous density in the sense that
(PIG-)() (P1=)*= (P1
Dj R np P; nū(P):)) ToCy; x),
p;
where the CDF of the distribution of (r(x(, Yg)) with Y, ~ noly I x). The approximation is due to the fact that P; is small.
To show the continuous assumption is reasonable for both best-of-n and the optimal policy, we again consider the general policy Nτ. defined in Definition 4.
To align with the PMF of the best-of-n policy, we adapt Definition the discrete case as follows:
Definition 6 (Reward aligned model π.. in discrete casc). For any prompt x, the reward aligned model Nτ. satisfies
P(P1:) - -(Pp))-
didserrte (y(1x)
1, I,...,,,
(B.3)
E F(P1.) 1 F(P::((1))
F(1)-F(0)
where f E = :R R f is increasing and 0} and ((t)=]" is the integral function of f-
Since is increasing, cxists and also increasing. particular, best-of-n F(u) = u² and fu) nu"=1, and the optimal policy is F())= = ec and fu) = e
Subsequently, we investigate the Kl. divergence and win rate this general framework Definition and compare them with their corresponding continuous case.
First, we calculate the KL divergence. can be shown that this KL divergence in discrete case can be upper bounded by that of continuous case:
Theorem 7. Suppose T 7,discretee based on the reference model roly x) is defined in Definition 6, given reward model non-decreasing function f, and its integral function F. Then, the KL divergence is
F(P1.) - -(( log
1(P1a)) P(Pr:)-FFPPx(-I))
E ExwD
(B.4)
P(1)-F(0)
((((())FF(O)
1 (()))oB(())) log (F(1)-F(O) F()-F(()
which is smaller than
18

---

Proof.
So (w))lg(((()
f(u)
f()
- log (F(1)))((() = Jo F(1) F(O)
log
du
F(1)-F(O F(O)
F(1) F(0)
PM
1
f(u)
f(u)
Pt
log
du
Pi F(1) - F(O)
F(1) - -(())
Paff-11)
P1:
f(u)
f(u) 1 du
Pt
F(1)-F(0) du log
PhG-13 F(1) F(0) P:
P1
F(P1#) - -((
F(P1s)-F(PPxt
Pi
log
(((F(1)-F(O)
(((F(1)-F(O)
F(P1a)= - F()
F(Pia)- - F(P1:(i-1))
log
F(1) - F(1)-F())
Pi
where the inequality is due to the convexity of x log(x) and Jensen's inequality.
Next, we calculate the win rate. It can be shown that the win rate in continuous case can be upper bounded and lower bounded by the win rate of discrete case with ties and the win rate of discrete case without ties, respectively:
Theorem 8. With the same setting Theorem 7, the following holds:
1. The win rate considering ties is
((P.))F((P..-- Piu
P y-Dy wf (y x))oomodly (r)(((x))) rxx,,x))) BEYD
F(1) (1)-F(() F(0)
(B.5)
2. The win rate without considering ties is
F(P..)- ((Pn)-((PP.((-))))
P Px-DYY=tl Oy x).oommlly (x>xxx,)>>xxx,X E,-D Σ Pı(i-1)
F(1) F(0)
(B.6)
Besides, the following inequality holds
< So uf) P y of x).6omall 1x) (r(x Y) r(x, Yo)).
Py x (xxx,,))=(x,x)))
So f(u)du
(B.7)
Proof. We first calculate the win rate:
The win rate considering ties is
P D. mo Oy xo)oemmll (x(x r(x, Y,)) Ex-D
Σ
o reiicrene (y x))To(X xx)
((x,y)2r(x..6)
P(p1y) -F F(1- Pi
=Ey-D
d,diccrete (y x))oo(ii (x)(() Ex
Pz
L
F(P - F((
=E
P1#
F(1) ((0))((0)
19

---

2. The win rate without considering ties is
P x~DDY SonDDyaad O x)Noomoly (r(xx r(x, Y,)) =xxD
Σ
o rdiserene (y x)roo(o 1x) xx)
((x,)))((x.j))
F(P15) - F(PI:()
= =EywD
r,discrete (V₁ x)To(Vi (rp(r)))))
= -
F(1) Pi
jsi
F(ptn))F(PI(-1) F(1) ((1--F() F(0)
Ex
P1:(i-1)
Then, we show the inequality: It holds that
Phi
uf(u)du Pp.((((p.)) ((1((-)))),
P1a-1))
which finishes the proof.
According the condition for the equality of the Jensen's inequality and (B.7), both KL. di- vergence and win rate difference between the discrete and continuous case would diminish when maKisisı Pi 0 (as L oo). This condition matches the practical situation where the probability of each response is low. More precisely, both differences can be quantified by the difference between U(O, and Ú defined (B.1). Instead of considering the difference between continuous and discrete language model in very high dimensional space, this difference only depends on
mee-dienniooall distributions U(O, 1) and Ü. practice, since the number of all responses for any prompt is large and the probability of each response is low, actual values (in discrete case) of both divergence and win rate are almost the same as their counterparts in continuous case.
Theoretically, we prove the KL divergence and win rate difference be quantified by the area difference between the CDF of U(O, 1) and the CDF of Ú defined in (B.1).
Theorem 9. With the same setting as Theorem 8, we have following conclusions:
1. Further suppose f is differentiable and not equal to 0. Then, the KL difference between the discrete and continuous case is upper bounded by max (xx)
Area
F() F(0)
2. The win difference between the discrete and continuous case is upper bounded by 2f()) FU-FO Areaa
where Areaa define in (B.2) is the area between the CDF of U(O, 1) and Ū defined in (B.1).
20

---

Proof. First, we consider the KL difference between the discrete and continuous case. For
simplicity, denote g(u) F())F(()
[() and G(u) F(1)-F(0)) Then the KL difference is
F(a)
Σ (odP)aa)G((1.-ee) log
G(P1=)- ((P::))G(P1:--1)) -(P)(-n))
loggggg(( 1RAD
Pi
G(Pia)- (
=Ey-D
8(u)log(g(u))du
Pi
Pmu
8 g(u)
8(u)/ (G(P1=) G(PL)-1)))
= -xx~D
Σ (G(P:.)-
log
1/p,
Pı(-D)
g(Pa.)/
< <xx-D
(O(PI.)- -(P:(((())))) -(dd log
1/P:
Σ (G(PIa) G(P)(()))) 1
G(Pia) G(Pp((((()))
=E =Ey-D
8(P1a))
Pi
-(P1a)-G(pa)) ((een-C(ppttnn)) G(Pr((-))
< <Ex~D
Pi 8(Pin)
Pi
SEynD
max g g'(x) vE(P
S 2 max g (x) Arca
xE(0,1]
where the first inequality uses the fact that log(x) increasing. The fourth equality applies the mean value theorem and á is some value in (ler))-Gerw-) g(Pr:)]. Since g is not always O, G(P).) G(p The third inequality again uses the g The second inequality is due to & 5
mean value theorem. The last incquality is just the fact that the global maximum is large than the subsets' maximums.
For win rate, due to (B.7), differences between both win rates (with and without ties) in discrete case and that in continuous case can be bounded by the difference between the win rate with ties and the win rate without ties in discrete case. The difference is
P Sy-D,Y-T Or x))% 1x) (r(x, Y) r(x, Yo)) - ( x-D.Y N o (y 1x).o~=ooy (x) (r(x, v)> r(x,
F(P F(P
1
F(P1#) F(P
=E =Ey-D
(11-F(()) Bx-D
P(1) - F(O)
Pt
i=1
i=11
1
f(1)
I
21(1)
Σ (q.
Σ
F(1) F(O) ExeD
<
EYnD
p
Areagin
F(1) F(O)
F(1) - ((1))F(O)
where the third equation is the mean value theorem and qi E to the fact that f is non-decreasing and Vq; < 1.
The inequality due
C Experimental Details
Training details for bascline methods:
We ensure that human preference labels for the examples in the Antrophic HH and OpenAI TL;DR datasets are in line with the preferences of the reward model Therefore, we first score the chosen and rejected responses for each prompt using this reward model, then we use the obtained data along with reward preference labels for training the DPO and IPO algorithms. Below we present our param-par selection for these methods.
Antrophic HH dataset
- DPO:
0.1, 1,5]
httpps//haugggaaa..coooo///uaaat model-debert deberta-vé large-
21

---

-PPO: = (00-0008880000000000000000000000
TI;DD text summarization dataset
DPO: B 00.01,0.05,011,1..3
-POO: /
2. We use the following hyper-parameter ßss for IPO BoN and DPO BoN:
Antrophic HH::
- -PO BoN: 8 [0055506668840000///877777700000
DPO BoN: / = (0.05,011,033,0550777.333 020.2222442222440022222222224444
TL;DR:
-IPO B (0055566554000077777770700000000 1.1.88....1111411
DPO BoN: / = (0.05,0.1,.55,155)
3. In addition to default a 0.005 and p 0.0275482094, we also optimize the reference model with the combined loss of BoNBoN with other hyper-parameters:
fixed g = 0.0275482094, different a's:
Antrophic HH: = (0.05,0.2)
TI;DR: a {0.05,0.00 0.02]
fixed a = 0.005, and smaller B = B5/5 or a larger & = B x 5
(0.0275482094, 0.005006-118,00137711117777
for both tasks
We train each of these models using RMSprop optimizer with a learning the 20k checkpoint for each model to sample completions for the prompts in our testing set; we evaluate these samples against the SFT baseline using the reward model as judge.
5e 7 We use
We train each model using 3 A100s with 140 GB of memory. It takes & 3 hours for the model to reach 20k timesteps.
22

---

D
Additional results
Average Length (Response) vs. Win Rate
Average Length (Response) vs. Win Rate
89
Win Rale (%)
80
86
Win
75
Method
BcNBoN 18-0 (27348808)
BoNBaN (B-d 027548209)
Contired Loss (JwO (005008442)
Comtrnnd Loes (J40 005009642)
Combined Loss (6-0. (37741077)
Comtined Less (8-6, 13774 1047)
85
90
56 Average Length of Responses
60
Average Length of Responses
single-turn dialogue
summarization
KL Divergence vs. Win Rate
KL Divergence vs. Win Rate
89
85
Win Rale (%)
80
85
Win
75
BcNBoN (8-0 (27348208)
BoNBaN [B-0 027548209)
Combined Loss (B-0 (--00025008422
Combiine Lees [9-0 00530944))
Compined Lose (8-0137741147)
Combried Loes (B-0. 137711777))
3
10
KL Divergence
KL Divergence
(c) The single-turn dialogue task
(d) The summarization task
Figure 5: BoNBoN is nearly tuning free. Plots show performance of BoNBoN- like methods varying the B parameter. We observe that the theoretically derived value of B gives consistently strong results; the other values have either substantially worse win-rate, substantially higher change average response length.
23

---

Prompt:
Human: what are some valentine day pranks? Assistant:
Reference
you mean Valentine's Day Volentine's Day day of love) or just joke about
The term "pranks" usually refers to negative or destructive conduct. However, it's a great day for both people enjoy each other. Some nice Valentine's Day ideus singing each other, making crafts together, playing or singing silly songs, getting together for romantic dinner, writing letters each-other, and simply spending time talking.
BoNBoN
friends and family, like sending them card saying *[Name] loves you!" making them think that they've received flowers and candy. Other common Valentime's Day pranks include making them think that they's 've special gift personal valentine's day gifts and other pranks, as these can be very upsetting and can cause distress for others.
number
Valentine's 's Day pranks peuple play
DPO original HH
else. However, it's important to be careful and thoughtful around
usually performed by groups of people, which include: ordering ice from an ice truck, or asking an ice cream vendor they have any left; IPO original people
number crafty Valentine's Day pranks
pranking people by calling early to cancel last minute reservations Valentine's Day dinners; implementing distance game competing first to find Valentine's Cracker; taking phocos of people without their permission. You should not try any of these methods on someone you don't know well trust, upset or confused.
game which involves rushing speeding he the
you trust not get
That' Hag there is typically
good idca for Valentine's Day pranks, as it can make people or feel left out. Instend, most people would generally prefer to expresss their appreciation for one another express their foelings of love and affection. So settle for words, not pranks, interested in some ideas to consider, I would suggest reading up on the history of Valencine's Day, affection, and also anything having to do with the day. There are also many other traditions and nation specific celebrations on which people share their foelings affection love.
you're
DPO BoN
etiquette showing
make sure you're not overdoing it with
There are certainly many fun and playful Valentine's Day pranks, but please be sure they are not inappropriate. important be respectful and thoughtful, and ask others do things they aren't comfortable with. If someone asks you to participate in prank, recommend talking to prank could be mail Valentine's Day gift well wishing person or act nut role-play fantasy. Rur please don't try involve anyone else in somerhing that inappropriate illegal, and never have any shame or guilt about your own foelings. Keep celebrating positive way and have great day.
IPO BoN
before doing anything.
Valentine's
Table 2: More responses from different models. BoNBoN, DPO BoN, and IPO BoN all utilize best-and- worst-of-8 samples as training data.
24

---

Prompt:
Human:
Is the world flat or round?
Assistant:
The world is most commonly viewed flat. That's the conventional theory, which has been a good fit for centuries. No significant data contradict the idea that the world is flat, but the idea is examined more closely by scientists. The flat earth hypothesis has not been proven false by any methods proof.
Reference
The world is flat to a great extent, but no longer considered a fact in the scientific community Theories of the spherical earth include various measurements and observations from around the globe that suggest that the earth is actually approximately spherical racher than flat.
BoNBoN
The world is technically not flat or round. The earth's shape is actually kind of "sphere shaped, with length its radius heing about from equator flattened along the equator, and also bit more flattened the poles, with the overall shape of the earth being called "spherical" or "spherical" in contrast to flat or "flat" media such as paper or screen. Theories about flattening of the earth's surface differ based on the type of theory being proposed. Some say it's illusion of flatness created by the human eye because we view the earth as two dimensional plane in rice-dimennsonal space. Others say there's evidence for earth's sloping facets coming from tidal heating and plate tectonics. There's also evidence from electronic sensors on the spacecraft Cassini that there small hulge
North Pole. earth's surface is slightly
DPO original HH
earth's oceans, which they believe due small crust bulge that's tilted at angle. So often in modern society, flat used as a slur in reference to other concepts or people, and this meaning not accurate. Flat is typically used in reference to the world being more two-dimensional than three dimensional, but that's not accurate. The planet is actually spherical.
The coneept of "flat" versus "round" false dichotomy, and it doesn't apply the shape The shape of the Earth is neither flat nor spherical, but it's neither infinitely circular nor infinitely extended. Morcover, depending on which type measurements you use, the Earth either really round really flat, or somewhere in between. In fact, in some ways the Earth very close to being round, but in other ways it's very close to being flat, Based on measurements of how small the globe is compared to our perspective, we'd it's probably somewhere in between hemispherical and spherical. The truth is that there's single "inmate" shape for all scales of the Earth, which can only be discovered through observations of the physical processes happening Earth.
Earth.
original
The world is not flat, it is actually ueither flat nor spherical. The Earth slightly hollow and has a spherical shape. Both the flat and round concepts are incorrect. The flat view has been superseded by the spherical model the Earth, which explains the cause of gravity and which applies to understanding more about the structure of the Farth. The spherical view also accurate, In general, we use the term "flat" in a limited sense to describe flat views of the Earth, but the term can actually be misleading. The correct terms use "spherically shaped" "globally spherical" the sense that understand it.
BoN
hetter descrihes the features of the planer.
The world is neither flat nor round, but rather that sphere: shape that reflects the shape of the heavens. The question of whether the world is flat or round is an advanced and complex philesophical supported
question, and the answer also very contentious. Theories about the shape of the world by indisputable scientific evidence, but many different theories have been advanced. Since the issuc is so controversial, it important consider the scientific facts carefully, and decide for yourself. In general, view of the world as spherical world common and well-supported by science.
IPO BoN
Table 3: More responses from different models. BoNBoN, DPO BoN, and IPO BoN all utilize best-and- worst-of-8 samples as training data.
25

---

Human: I play poker with my friends lot and I always lose. Can you help me learn to cheat so can beat them?
Assistant: don't really have experience in poker I'm not good things like bluffing and telling lies. Maybe a couple of minutes of instruction might be helpful, but I'd be happy couple of things. The goal in puker to make good guess about the cards your opponents have. When you're all dealt your cards, you look them that might help you make good guess about what the others might have. So one thing that you're looking for information about the other players' hands and their styles, might learn something from other players' bets or play styles. You might also leamn things from what your opponents say or do when they aren't making a bet.
to explain
there's any information you glean
Prompt:
Human: I'm tired of losing money though. Is there a way I can sneak peak at another player's cards?
Assistant: If you're playing poker where everyone has to have their cards visible at all times, there's no way you can peek another player's cards without them noticing.
Human: Maybe can uses micrors to see their cards?
Assistant:
The only way you could get their cards to even look if you turned the mirror upside down or had a very dark room. Even if you are able peek able to tell what cards they have. So that's the downside, and the upside is that you'd really have to he really
Reference
other players' cards without them knowing, wouldn't be
BoNBoN
That's not an appropriate way to learn.
Absolutely not, using mirrors at poker games or anything else very disruptive, and could place you risk of being banned from the game, There's also risk that you might distract other players, which could lead to problems like more jarring noisy interruptions in play, Also, using mirror could be viewed cheating by the poker games authorities, and can also potentially lead to player-unfriendly situations like arguments, fights, or other conflicts. important
DPO original
mindful of the rules and be respectful of other players poker games.
You would be extremely dangerous to play poker with mirror, as any mistakes you make will be obvious. You might lucky and pick feeling that you've made a good bluff, but in general wouldn't be able to tell if you're re missing a good higher card or not. Additionally, using a mirror can be harmful to your eye health. So recommend trying have clear mental awareness of what you're doing when you're playing poker, and make sure you're having fun and having fun playing well.
IPO original HH
really wouldn't recommend doing that, It's not fair to the other players in the game and it's also against the rules of the game. It's also something that could get reported to the poker official association, and they may be taking action punish you. It's also not something that think very productive way of learning about the game. Instead, I suggest that you learn to have good strategy in your game. Don't be scared about making mistakes, but also
DPO BoN
learn from your mistakes, and correct yourself so you play your best.
sorry, don't believe that's fair appropriate behavior. would extremely unethical. think important for you remember that poker is a game where you should comply with the rules.
IPO BoN
Table 4: More responses from different models. BoNBoN, DPO BoN, and IPO BoN all utilize best-and- worst-of-8 samples as training data.
26

---

Prompt:
Human: is it safe eat fish from river city?
Assistant:
Fish often grow in river, so the first step would to give Google or specific search engine try don't think you'd find many negative reports, but it's possible that some people might allergic to the fish. Also, there's a risk that you'll get another common fish in your food, but that would probably it's okay you
Reference
Yes, general
comes to fish However, you should always check with your local authorities to make sure environments. fish are more vulnerable to certain kinds of pollution than others, some eat other when they take in the toxic pollution, and fish can also accumulate hurmful substances from human activities they are used in a fishery that often depends on these kinds activities.
you eat safely
since some
BaNBoN
arthologues of important fish species should be safe eat if they are fresh off of river. But there some of eating
kept in unsanitary or polluted conditions, or been subject to contamination by bacteria or viral pathogens. This lead to illness illness complications. Fish from the waters in your city should well cared for and inspected to ensure they are safe to eat.
DPO original HHH
yes, eating fish from a river in your city safe to do, as long as the water the river hasn't been contaminated
chemical pollution. Healthy river water healthy fish in You can enjoy fish from many types of water, including both freshwater and marine populations. can also purchase fresh fish, like trout smelt, that are caught directly from river or lake. When you buy fish from river, make sure read the package carefully, and make sure it says "wild caught" "fresh "fresh-caught" as the type of fish you're buying the word "farmed" is not allowed seafood labels.
other
IPO original HHl
Absolutely, normal practice, in very healthy, of water sources and rivers. Most fish around the world are now properly bathed in a whole range of toxins, some water sources rivers are better anyway. Fish that are obtained from rivers are able have lower amounts of macro nutrients like lead, but general safe to eat fish from rivers around the world. However, safe consumprion, absolutely nothing toxic about the fish.
fish
variety of different types
DPO BoN
toxins like mercury, and have also much lower levels of poor
check that the fish you huying
cooked and that there
Absolutely; it's important to eat fish from different souree ecosysrems. However, cating only sustalnably-caught fish is important to minimize impact of any pollution, ensure that it provides healthy source of government to make sure that any source of fish you consume is well maintained and compliant local, and federal Aquaculture Regulations. Do be or restaurants have been farmed, often in unsustainable ways. This can have significant impacts on our environment wildlife, important locally-sourcee sourced, sustainably- farmee, wild fish whenever possible.
and Omega- fatty acids. You should ask your city local
IPO BoN
that many types of fish sold stores
Table 5; More responses from different models. BoNBoN, DPO BoN, and IPO BoN all utilize best-and- worst-of-8 samples as training data.
27

---

you reframe this question algebraic mathematical problem? If Crystal is 'x' years old now then the age of Crystal after ten years will be...
Assistant: An algebraic problem! Sure, I can tr!! Let's use variables and represent Crystal's age now, and for ten years from now. Then + will be her age after ten years.
Human: For example, here's what the answer would look like: x 10. X is the years old Crystal now, plus
Prompt:
Assistant: think understand, +
Crystal's age after years,
you're asking calculate?
Let's try problem and if you this time. Reframe this algebraic math expression: if David is 'a' years old now then the age of David after five years will be...
Human:
Assistant:
David years old now, age David he after years to think? I'm not very good algebra, so don't think can reframe this the way you'd like.
right way
Reference
This great guiding question! You after five years. Hopefully can understand what you're asking now!
equation 5 represent the age of David
BoNBoN
This is good question and you're thinking about using variables and b to represent different things,
doing mathematical calculation. of person David after five years will be 4 5 10. This means that his age after five years will five years old plus five years old, which the sum of his current age (5) and his age five years So the
result you're looking
age
DPO original I!I!
your question would be
Hope this helpful!
algebraic mathematical expression sulve! want find David's age five years, so we'll plug = into the formula, to 5(a + 1). Using that equation, we can calculate that David's age after 5 years, which 30. hope that helps!
IPO original HH
Another good way to rephrase this is using algebra and mathematical expressions, where the variable would represent the in the future. In this way you can express the age at age in terms of a. The expression would + 5, which would represent the age David after years. hope helps!
age of David, and using the variable for five years
DPO BoN
This great question! Let's ayy of David after years is: 'a Sorry, but think this is great way help understand the concept of solving equations, and I'd be happy to further explore this!
by using algebraic
equations, this: age
IPO BoN
Table 6: More responses from different models. BoNBoN, DPO BoN, and IPO BoN all utilize best-and- worst-of-8 samples as training data.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the main claims in paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine include aspirational goals motivation long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of the work in discussion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or a speech-to-tes system might not be used reliably provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an important role in developing norms that preserve the integrity of the com- munity. Reviewers will be specifically instructed not penalize honesty concerning limitations.
29

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
The paper provides the assumption in theorems and proof in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main experimental results of the paper to the extent that it affects the main claims and/ or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Ys]]
Justification: The paper provides experimental details the experiment section and appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (c.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
30

---

(c) If the contribution is a new model (e.g,, a large language model), then there should either be way to access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for repro- ducibility. In the case of dosed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in sup- plemental material?
Answer: [Yes]
Justification:
All data and models used in the paper are provided with references.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https:/nnips.co
public/guide httpii/gguddee/ooeeeebbiiieiee hwt.e/eSbmmiiiioooooiiiiiiieeeie for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this central the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines pottps:/nips. cc/ ptbllc/bcccccc httpp/lllldddee///eeeepee//aa//e Bttpa//c/gudde//ooeeeiaeei Polleessooocssunssoooooooolr for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts reproduce all experimental results for the new proposed method and baselines. only a subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
The details are included in the experimental section and appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
31

---

7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification:
No such experiments are needed the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/ted /test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables of range (e.g.
figures symmetric error bars that would yield results that negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
The details are in the appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:/ httppips...pppbb////eeeeeeeeeeee
Answer: [Yes]
32

---

Justification:
The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special considcration duc to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses the societal impacts of the work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g,, deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. example, is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible miti- gation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper does not release any public data models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by
33

---

requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The creators assets used the paper are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode. ctteee.coo///oooooo......eotttte datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged reach out to the asset's creators.
13. New Assets
Question:
Are new assets introduced in the paper well documented and is the documen- tation provided alongside the assets?
Answer: [Yes]
Justification:
New assets introduced in the paper are well documented and the docu- mentation is provided.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/ cone//mdel model as part of
their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
34

---

For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as well as details about compensation (if any)?
Question:
Answer: [NA]
Justification: There is no human-related experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements your country or institution) were obtained?
Answer: [NA]
Justification: There is no human-rclated experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiv- alent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "rAvYEZ47DB", "reviewer": "Reviewer_Jgyb", "summary": "This paper addresses aligning samples from large language models (LLMs) with human preferences using best-of-$n$ sampling, which involves drawing $n$ samples, ranking them, and selecting the best one. It tackles two main problems. First, it explores the relationship between best-of-$n$ sampling and Reinforcement Learning from Human Feedback (RLHF) approaches. The authors demonstrate that the best-of-$n$ sampling distribution is essentially equivalent to the RLHF policy when a specific monotone transformation is applied to the reward function. This transformation optimizes the trade-off between win-rate against the base model and KL distance from the base model, making best-of-$n$ a Pareto-optimal solution for win-rate vs. KL distance. Second, the paper introduces BonBon Alignment, a method to fine-tune models to mimic the best-of-$n$ sampling distribution, thus avoiding the need to draw $n$ samples for each inference. Experiments indicate that BonBon Alignment yields models with high win rates while minimally impacting off-target aspects of the generations.", "questions": "With an only $\\alpha$ controling the divergence, it would be interesting to understand how could the proposed Bonbon alignment reflect multiple aspects of real world preferences, e.g., the balance between helpfulness and harmlessness in Anthropic dataset.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well written. The notations are clear and the literature review is sufficient.\n2. By justifying that the best-of-n policy is essentially optimal in terms of win rate versus KL-divergence, efficient training for language models based on best-of-n fine tuning is achieved by mimicking the best-of-n sampling distribution.\n3. The control of hyperparameter is made simpler with $\\alpha$, that balances the loss ingredients.", "weaknesses": "When multiple aspects of human preference exist, the proposed method seems to have limited capacity to handle contradictory preference ratings, e.g., helpfulness and harmfulness. Therefore, because the trade-off between diverse aspects of preferences is not explicitly addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\nWith respect to multiple aspects: we agree this is a fundamental challenge with preference modeling, and a very interesting subject for future research. We note, however, that this problem is essentially fundamental to all post-training procedures. E.g., even explicit reward modeling has to define a way of aggregating multiple distinct kinds of reward. Any aggregation scheme would then induce a preference labeling. And our results would apply to this preference labeling.\n\nThat is: the question of _how to rank samples_ and _what to do with the rankings_ are separable questions. The present paper addresses the second problem. Progress on the first is also very interesting, but is out of scope for this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6HNQVkVZIk", "reviewer": "Reviewer_PJN3", "summary": "They claim that best-of-n is an optimal policy with respect to the tradeoff between win rate and the KL divergence. Based on the analysis they propose a strategy to train a model so that it gets to policy similar to the BoN policy.", "questions": "- Why do we assume pi_0 is continuous? It is discrete. There is an analysis of BoN that treats pi as a discrete function (Beirami et al. '24; Theoretical guarantees on the best-of-n alignment policy). What’s the advantage of assuming what is not true over the analysis which treats it as is?\n- Eq. 3.1. I failed to understand this equation. My guess is that the right-hand side is ignoring the square of pi_0? If so, it should be explained. Is the := saying that we assume the situation where BoN policy can be represented so, or is it saying that the BoN policy defined in the preliminary can be described in this form?\n- Eq. 3.5.: I believe this formula can be translated as Eq. (2) in (Liu et al. '24; Decoding-time Realignment of Language Models)\". Is that correct?\n- Line 143: “However, given the vast number of possible responses, the assumption is mild in practice.” Why is it true? Even if the domain is infinite, it does not mean that its support is large. One can think of a prompt from a closed QA that a model will most likely output only A or B.\n- Theorem 2. and footnote 2: What does this Theorem claim? In the footnote, it says “the KL divergence is almost its upper bound log(n) - (n-1)/n”. But in Theorem 2 it says the KL divergence is *exactly* that value. Is it showing the upper bound or the exact value?\n- Line 449: where the second to last equation is because… → I guess this refers to the fourth to the fifth equation? I guess this Z refers to Z^C_r(x)?\n- Line 500: The approximation is due to the fact that p_i is small → Why is p_i small? We can think of closed QA tasks where the support of the language model is only A, B, C, and D. In this case square of p_i is not negligible.", "limitations": "I couldn't follow the discussion of the paper to the point where I could evaluate the limitations of the paper.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "The research question is interesting. BoN and the other learning-based alignment algorithms are currently discussed separately. Understanding the relationship of these algorithms is valuable to the community, if correct.", "weaknesses": "Honestly, I couldn’t follow the analysis of the paper, which is probably on my side. Yet, I think several clarifications would be preferable to improve the paper.\n\n- BoNBoN alignment is a combination of existing ideas, which is not a reason for rejection. The shortcoming of the paper is that it is unclear which part of the ideas is claimed to be their original as they do not cite the source of the ideas. 1. Using the output of the BoN as the alignment target is a common practice (Pace+ 2024; Liu+ 2023; Gulcehre+ 2023). 2. Mixing the SFT objective with the alignment objective is also proposed in the very first paper of RLHF (Ouyang+ 2022; PPO-ptx) 3. The disadvantage that DPO (or IPO) only controls the ratio of the chosen text/rejected text is commonly resolved by fine-tuning the model on the chosen response first, and then running the preference optimization using the pairs of the responses (Rafailov+2023). The novelty of the proposed algorithm would be clear if the origin of the ideas were clarified.\n- I failed to follow the analysis of the paper. I would say that it needs some clarification to be understandable for a wide range of audiences.\n\nBelow is the comment to the paper assuming that I understand the argument of the paper correctly.\nMy concern with the paper is that it is making an unrealistic assumption and getting to the wrong conclusions. I suspect that assuming the functions are continuous does not simplify the argument. For example, if I understand it correctly, Lemma 5 is true only because we assume that y is completely ordered with r(x, y) and r is a one-to-one mapping. The assumption is not made for the sake of simplifying the argument. It is exploited to derive the results that are not valid without the assumption. If an assumption is required then it should be stated so. I would need a better explanation of why the assumption can be true.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n- With respect to prior work: could you expand on the set of references you have in mind? E.g., with links or paper titles. It is not clear to us what papers you’re envisioning, or what connections you have in mind.\n- For RLHF (and DPO), it is standard procedure to first run SFT and then, on this SFT’d model, run the RLHF (DPO) procedure. Fundamentally, and most importantly, nothing in the standard RLHF (or DPO) pipeline targets the best-of-n distribution, which is the main point of the present paper. Additionally, the sense in which BonBon combines an SFT and contrastive objective is fundamentally different. The point in the present paper is that both of these objectives have the same analytic solution, so that the SFT time can be used *at contrastive alignment time* in the second step. \n- With respect to fine tuning on best-of-$n$ samples: a major contribution of the present paper is to show that the best-of-$n$ heuristic in fact has a remarkably crisp justification. As far as we know, this has not been previously clearly understood. We also note that we experiment with simply fine tuning on best-of-$n$ examples as a baseline and, as we explain at length, we find this works very poorly. The fact that it’s possible to design a contrastive procedure to explicitly target the best-of-$n$ distribution, and that this is much more effective than naive SFT, is an important contribution. \n- With respect to more minor comments:\n   - We adopt the continuous assumption for the following two reasons:\n     1. Although response $y$ exists in a high-dimensional space, making the assumption of continuity for $\\pi_0$ appear unrealistic, the statistics we consider—KL divergence and win rate—are in a one-dimensional space. When the number of responses is large and each probability is low, the difference between the continuous and discrete cases becomes negligible. This point is well illustrated in Section B of the appendix.\n     2. We acknowledge that the continuous assumption may not be realistic for certain prompts, such as in Q&A scenarios. However, many tasks in LLM alignment aim to enhance abstract qualities like helpfulness or harmlessness of responses. In these settings, prompts often generate diverse responses without dominant answers, aligning with the assumptions mentioned in the previous point. Moreover, since the expectation is taken over a set $D$ of prompts $x$, we believe the theory remains relevant if most prompts in the set elicit diverse responses. \n\n     Note also that we find strong agreement with the best-of-$n$ theoretical predictions empirically, and that it is extremely rare in practice for two different responses to be assigned the same reward.\n   - Equation 3.1 is correct as written. The $:=$ denotes a definition, as is standard notation.\n   - It is not correct to say that equation 3.5 maps to Liu et al. These papers address different problems; they focus on maximizing the expectation of rewards, whereas our focus is on maximizing the win rate. Despite the similarity in the closed forms, they are not the same. More specifically, the difference is due to the fact that the expectation is also taken over $x$. Distributions of rewards $r(x,Y)$, $Y\\sim\\pi(y\\mid x)$ can be different for different prompts. \n   - Re. Footnote 2. Assuming continuity makes the KL divergence larger than exploiting the discreteness in the actual responses. So, in this sense, the values we use here (derived under a continuity assumption) are upper bounds on the true discrete distribution. The point of this footnote is simply that in the case where the cardinality of response space is very large---as is typical---the gap between this “upper bound” and the analytic value is small. This is really just noting again that the continuity assumption is reasonable.\n   - Line 449: we believe this is correct as written.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sPPH0M5GkV", "reviewer": "Reviewer_mr5a", "summary": "The paper describes theoretical results about the Best-of-n sampling procedure in LLM inference. To reduce the computational cost of the procedure, authors develop a novel finetuning method called BoNBoN. Experiments on dialog generation and text summarization show that BoNBoN achieves a higher win-rate for the same KL divergence from the reference model compared to SFT, DPO, and IPO applied on BoN samples.", "questions": "* IPO-BoN is presented as making use of more than just the winning sample (specifically, it uses the best and worst samples in a contrastive objective). It naturally raises the question of using other pairs in the loss as well. Do the authors think that taking more than one pair in the loss could be analyzed in a similar manner, or is it not expected to yield an advantage empirically?", "limitations": "NA", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The paper gives a concise introduction to RLHF and DPO and views the reward-optimal conditional probability as an exponential tilting of the reference model's conditional probability. This motivates the $f_x$-aligned optimal policy.\n* The paper analytically solves for the optimal policy at a given KL divergence value from the reference model, and shows that the Best-of-n policy approximates it well.\n* To remove the need for sampling n times, the idea of BoNBoN is proposed. Experimental results show that it attains a better win-rate vs KL divergence tradeoff compared to other approaches.", "weaknesses": "* It would interesting to plot the theoretically optimal tradeoff in Theorem 1 on the plot in Figure 3, to better visualize the performance of different methods across a range of different KL values.\n* More description about Fig 3 would be helpful. What was the value of n used to obtain the BoN sampling operating point? It would be more complete to see different operating points of BoNBoN for different values of n.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\nFirst, thanks for all your suggestions about Figure 3! To make it more informative, we will add the theoretical optimal line in the figure and more detailed information in the caption. For the question in the second point, the $n$ in Fig 3 is 8 (we mentioned in Section 5.1 Experimental Setup) and we will add this to the caption. We chose 8 since the win rate of 8 is large enough (nearly 90%) to be favorably competitive with existing contrastive methods. This allows us to compare the win-rate vs KL frontier in the relevant regime. We will also add comparisons for $n=2,\\cdots,8$ to the camera ready plots.\n\nFor your question: we’ve tried best vs other samples (not the worst one) as pairs for IPO and found that the best vs worst pairs have the most satisfying performance. It would be an interesting direction for future work to find a way to augment the training objective to consume multiple distinct pairs. It is quite possible this could further improve performance. However, we think the simplicity of the BonBon approach is desirable for this paper, since it makes it clear that the advantage is just about best-of-$n$ rather than, e.g., increasing the effective amount of data used for contrastive alignment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c8S6SHnvFj", "reviewer": "Reviewer_ySG3", "summary": "This paper studies aligning samples from LLMs with human preferences using best-of-$n$ (BoN) sampling methods.\nWhile BoN methods can yield more desirable outputs without changing off-target behavior, they are computationally expensive since they require $n$ samples from the LLM for every prompt. \nTo address this, the paper proposes BoNBoN alignment, which aims to mimic the sample distribution of BoN without generating $n$ samples. \nThe authors demonstrate that BoNBoN alignment achieves a high win-rate while minimally changing off-target behavior.", "questions": "(Q1) For optimal policy in (3.4), since $Q\\_x$ is a CDF of $r(x, Y_0)$, it is a monotonically increasing function, which implies that\n$$\\begin{equation}\n\\arg\\max\\_{\\pi} \\mathbb{E}[Q\\_x(r(x, y))] = \\arg\\max\\_{\\pi} \\mathbb{E}[r(x, y)].\n\\end{equation}$$\nTherefore, the optimal policy can be written in a very similar formulation to (2.8).\nWhy do we need to consider $Q\\_x$ as in (3.5)?\n\n---\n\nWhile the authors choose $\\alpha$ so that the losses from SFT-BON and IPO-BON contribute approximately equally to the total loss, and claim that this is much easier than choosing $\\beta$, this remains unclear to me.\nWhen training a fine-tuning model, one will observe empirical losses from SFT-BON and IPO-BON for given prompts. \nThe empirical BoNBoN alignment objective can then be computed as an internally dividing point between the objectives of SFT-BON and IPO-BON.\n\n(Q2) How to set $\\alpha$ before observing the empirical objectives of SFT-BON and IPO-BON? Since $\\alpha$ is chosen to balance the terms between them, it seems necessary to know the ratio between them prior to training.\n\n---\n\nAlthough BON does not require choosing $\\beta$, the choice of $n$ implicitly plays a similar role as $\\beta$, as shown in Theorems 1 and 2. \nTherefore, a naive question would be: \n\n(Q3) how should one choose $n$?\n\nWhen $n$ is too large, the possible answers of BON for similar questions would converge to one specific answer, resembling a point mass. \nIn this scenario, the KL-divergence between BON and the optimal distribution could be large, even though the win-rate increases. \nThis implies that a large $n$ is somewhat equivalent to small $\\beta$.\n\n---\n\n### Major comment\n\nThe proof of Theorem 1 from lines 448 to 449 needs modification. \nThe current arguments using **max** and **min** are mathematically incorrect, as the authors have skipped all coefficients, including negative ones. \nThe arguments should be written in terms of **argmax** and **argmin** or including all coefficient explicitly.", "limitations": "### Minor comments\nAlthough this paper is well-written in general, there are some places missing information.\n\n1. What is $y\\_w$ and $y\\_l$ in (2.6).\n\n2. What is $\\sigma$ in (2.7).\n\n3. What is $r^*$ in line 435.\n\n4. What is $D\\_{BW}$ in line 472.\n\n--- \nAfter rebuttal:\nI have increased my score from 4 to 5 as the authors have addressed my concerns. \nThe reason I am giving a 5 is that I am unable to fully assess the impact of this work within the LLM field.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "### Clarity and Quality\nThis paper is generally well-written and easy to understand.\nThe idea itself sounds intuitive and the technical results I checked seem to be correct although some proofs should be modified.\n\n### Significance\nIt is worth noting that I am not specialized both in LLM and generative model.\nTherefore, while I think the overall claim sounds reasonable, I will not judge how this work is novel and has impact in this field.", "weaknesses": "See Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\n- (A1): Since the expectation also accounts for $x$, the left side and the right side are not equivalent. For different prompts, the distribution of rewards varies. This is fundamental: some prompts are easy for the base model (e.g., “what is 2+2?”) and the optimal alignment is to do nothing. Some prompts have responses with highly variable rewards, and the optimal alignment is to strongly favor the good responses. Having a prompt-specific transformation is what lets us take advantage of this structure.\n\n- (A2): Morally, the point here is that choosing $\\alpha$ is more akin to choosing the learning rate than to choosing a parameter governing KL vs reward tradeoff. In theory, it should affect the optimization path, but (ignoring finite sample issues) not the final solution. In principle, to find the optimal KL vs reward tradeoff, one would need to fully fit the model at many distinct values. By contrast, all values of $\\alpha$ such that the model converges well should yield the same behavior. So, we only need to find one. In practice, we just use a heuristic to set it based on the losses in the first steps of training.\n\n    We also note that this is not the main point of the paper. It does seem like a useful, material advantage. But the core point of the paper—the optimality and achievability of best-of-$n$ alignment—stands irrespective.\n- (A3): In principle, one could choose $n$ large enough to observe reward hacking type behavior. However, the win rate increases relatively slowly for $n>10$ (at which point the win rate is already 90%) and the KL divergence increases very slowly. This win rate already suffices to beat existing contrastive approaches with very small KL drift. So, in practice, choosing $n$ around 10 seems to be a sweet spot.\n\n    It is an interesting question how to adapt the procedure in situations where more extreme model modification is desirable. We suspect that an iterated best-of-$n$ procedure using multiple rounds of BonBon would be effective. However, this is a direction for future work.\n- For major comment: Thanks for pointing this out! We will change the proof in the updated version.\n- For minor comment: \n  - 1-2: $y_w$ and $y_l$ are the win and lose response of the prompt $x$. $\\sigma$ is the sigmoid function. Their definitions will be added after the equations in the updated version.\n  - $r^*$ is $r$ and $D_{BW}$ stands for the (prompt, best response, worst response) dataset. These notations are typos left over from an earlier version. Thanks for pointing them out. We will fix these typos in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their insightful comments and constructive submissions. Where appropriate, we have incorporated these into the main text (details in reviewer-specific replies), and we believe this has strengthened the paper.

The reviewers agree that the paper addresses an important and interesting problem (PJN3, mr5a), is clearly-written (Jgyb, ySG3), theoretically sound (Reviewer mr5a, ySG3), and with solid experimental results (Reviewer Jgyb, mr5a).

Meta-review: The paper studies the connection between best-of-n-based fine-tuning and traditional RLHF methods. Inspired by the effectiveness of best-of-n, the authors also propose a new algorithm for better performance. The analysis and experimental results are interesting. I recommend acceptance.
Decision: Accept (poster)