Variational Best-of-N Alignment
Authors: Afra Amini
Tim Vieira
Elliott Ash
Ryan Cotterell
Venue: ICLR 2025 Poster (2025)
Keywords: Alignment
RLHF
Best-of-N
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Best-of-N (BoN) is a popular and effective algorithm for aligning language models to human preferences. The algorithm works as follows: at inference time, N samples are drawn from the language model, and the sample with the highest reward, as judged by a reward model, is returned as the output. Despite its effectiveness, BoN is computationally expensive; it reduces sampling throughput by a factor of N. 
To make BoN more efficient at inference time, one strategy is to fine-tune the language model to mimic what BoN does during inference. 
To achieve this, we derive the distribution induced by the BoN algorithm. We then propose to fine-tune the language model to minimize backward KL divergence to the BoN distribution. Our approach is analogous to mean-field variational inference and, thus, we term it variational BoN (vBoN). To the extent this fine-tuning is successful and we end up with a good approximation, we have reduced the inference cost by a factor of N. Our experiments on controlled generation and summarization tasks show that BoN is the most effective alignment method, and our variational approximation to BoN achieves the closest performance to BoN and surpasses models fine-tuned using the standard KL-constrained RL objective. In the controlled generation task, vBoN appears more frequently on the Pareto frontier of reward and KL divergence compared to other alignment methods. In the summarization task, vBoN achieves high reward values across various sampling temperatures.

Published as a conference paper at ICLR 2025
VARIATIONAL BEST-OF-N ALIGNMENT
Afra Amini
Tim Vieira
Elliott Ash
Ryan Cotterell
ETH Zürich
{afra. amini, ryan. peetterell@einf.... ch tim. f. R.veiir@emaaii com ashe@ethz.ch
ABSTRACT
Best-of-N (Bo.N) is a popular and effective algorithm for aligning language models to human preferences. The algorithm works follows: at inference time, N sam- ples are drawn from the language model, and the sample with the highest reward, judged by reward model, is returned the output. Despite its effectiveness, Bo.N is computationally expensive; it reduces sampling throughput by a factor of N. To make BoN more efficient at inference time, one strategy language model to mimic what Bo does during inference. To achieve this, we derive the distribution induced by the BoN algorithm. We then propose to fine-tune the language model to minimize backward KL divergence to the Bo.V distribution. Our approach is analogous to mean-field variational inference and, thus, we term it variational BoN (vBoN)). To the extent this fine-tuning is successful and we end up with a good approximation, we have reduced the inference cost by a factor of N. Our experiments on controlled generation and summarization tasks show that BoN is the most effective alignment method, and our variational approximation BoN achieves the closest performance to BoN and surpasses models fine-tuned using the standard KL-constrained RL objective. In the controlled generation task, vBoN appears more frequently on the Pareto frontier of reward and KI, divergence compared to other alignment methods. In the summariza- tion task, vBo. achieves high reward values across various sampling temperatures.
fine-tune the
INTRODUCTION
Language models are pre-trained on large corpora to model a distribution over natural language text.' Beyond their initial pre- pre-training, they often additionally fine-tuned on domain-sseccif data through a process called supervised fine-tuning (SFT). The goal of SFT is to enable the model to better per- form various downstream tasks of interest. While the fine-tuned model, called the reference model in our paper, is indeed typically much better at performing the downstream task of interest, e.g., dialogue generation or summarization, may still generate undesirable content, e.g., harmful or offensive text. To mitigate this issue, aligning the reference model to human preferences has become a fundamental step in the development of modern large language models (Meta, 2023; OpenAL. 2023; Gemini, 2024).
The degree to which text is aligned with human preferences is typically operationalized using a real-valued reward function. Rather than constructing reward function by hand, it is typically estimated from a dataset of human preferences? 2 And, after estimation, we expect the reward function to return higher values for text that is more likely to be preferred by humans, and lower values for text that is more likely to be dispreferred. Then, given estimated reward function, an alignment algorithm further alters the reference models in a manner such that it places the highest probability on the text that is high reward under the reward model and high probability under the reference model.
Alignment algorithms can be taxonomized into two groups: (i) alignment via fine-tuning, where we change the language model's parameters to achieve alignment (Christiano al., 2017; Rafailov
Many language models are also used to model text in non-natural languages, e.g., programming languages. In some cases, the reward model is not estimated from human preference data. It is either known, e.g. code-hased exccution scores, or given by classifier, e.g., toxicity or sentiment classifiers.

---

Published as a conference paper at ICLR 2025
Best-of-(N=4)
Billy Wilder is
Billy Wilder is
a delight to watch.
credited for few fine spots, including-
LM
Reward Model
1.6
delight to watch.
LM with Variational Best-of-A fine-tuning
a delight to watch.
Jack, a troubled driver who..
bad. he was for this film..
-2.7
method: it draws N samples from the language model, ranks them according to a reward model, and outputs the best sample. Variational Best-of-N (on the right) approximates this process via fine-tuning. The goal is to ensure that sampling a single string from the fine-tuned model produces a result equivalent to applying Best-of-N. This approach allows us achieve similar performance while increasing the throughput by factor of N.
Figure 1: Best-of-N (on the left) is effective
et al., 2023), and (ii) alignment via inference (Nakano et al., 2022; Mudgal et al., 2024). A common -ii-eeet-va--ii---umme
method is reinforcement learning from human feedback (RLHF; Christiano et al., 2017; Stiennon et al., 2020; Ouyang al., 2022). RLHF typically consists of further fine-tuning the language model under KL-constrained RL objective, which is made up of two terms: a term that encourages the model to maximize the reward, and a term that discourages high KL divergence between the language model and the reference model. This objective is often maximized with an RL algorithm, e.g.. proximal policy optimization (PPO; Schulman et al., 2017). A common alignment-vin via--ifeenne method is the Best-of-N (BoN; Stiennon et al., 2020) algorithm. As such, does not require any fine-tuning of the language model. The algorithm is straightforward: One draws N samples from the reference model and returns the text that achieves the highest reward among those N samples. The BoNV algorithm has also been effectively applied in controlled decoding (Yang & Klein, 2021; Mudgal et al., 2024) and to generate a dataset for supervised fine-tuning (Meta, 2023).
Despite its simplicity, BoN has proven incredibly practical in generating high-reward text that still has a high probability under the reference model. Theoretically, Yang et al. (2024) prove that under some simplifying assumptions, the Bo.V distribution is asymptotically equivalent to the optimal distribution under the KL-constrained RL. objective. Empirically, it has been repeatedly shown (Gao et al., 2023; Rafailov al., 2023; Mudgal et al., 2024) that BoN often appears on the frontier of reward and KI. curves, surpassing the performance of models fine-tuned with RLHF. However, the main factor preventing Bo from replacing fine-tuning methods for alignment is its significant computational overhead during inference. Even when sampling is done in parallel, Bo decreases the text generation throughput by a factor of N. This drawback limits its practicality for generating text from large language models.
To speed up BoN, we devise a scheme to convert it into an alignment-viat via-fine-tuminn algorithm rather than alignment- sarpumeettviaiiffeeneee algorithm. To this end, first formally derive the probability distribution induced by the BoN algorithm. Then we approximate this distribution by minimizing the reverse KL divergence between the language model and the BoN distribution. This leads to an optimization objective that we refer to as the vBoN objective. By analyzing a lower bound of this objective, we find that it behaves similarly to the KL-regularization objective in the limit, i.e., N 1 or N + co. Importantly, the vBoN objective has unique and useful property: it is insensitive to applying any monotonically increasing function to the reward values. This distinctive feature, along with the empirical success of the BoN algorithm, suggests that the vBoN objective is promising and interesting objective to explore. Finally, we fine-tune the language model using PPO to optimize the vBoN objective. Our scheme, depicted in Fig. 1, allows to achieve performance close that of the Bo algorithm while increasing the inference throughput by a factor of N.
We experiment with vBoN on controlled generation and summarization tasks, comparing its performance to models fine-tuned using the KL-constrained RL objective. For controlled generation, results indicate that models fine-tuned with the vBo. objective more likely to fall on the Pareto frontier of the reward vs. KL curve compared to other fine- tuning- based alignment methods. This suggests that NBoN achieves better balance between maximizing reward and maintaining
2

---

Published as a conference paper at ICLR 2025
proximity the reference model. On summarization task, fine-tuning with vBoN yields higher reward values and greater win rates on average than models fine-tuned with the KL-constrained RL objective, further demonstrating its effectiveness.
BACKGROUND: REINFORCEMENT LEARNING FROM HUMAN FEEDBACK
Let 2 be an alphabet, finite, non-empty symbols." 3 The elements of > may be characters, tokens, or words; the choice lies with the modeler. A string is a finite sequence of symbols drawn from E. A language model is distribution over strings y E E", where Σ' is the of all strings over the alphabet Σ. In this paper, consider language models, e.g., those based neural networks, that are parameterized by real vector 0 E O denoted as иa. Furthermore, we restrict ourselves to language models that are differentiable functions of 0. In conditional generation tasks, e.g.. summarization or dialogue generation, it is desirable to prompt the language model with a string E" Consequently, distribution over response strings y. given a prompt string æ, as To x)). However, for notational convenience, will drop the explicit conditioning the prompt and simply write To
consider prompted language models, i..., those that give a conditional
Algorithms for RLHF fine-tune the language model to increase the expected reward of the strings sampled from it while not diverging far from the reference model. RLHF consists of three steps. First, the language model is fine-tuned on a task-specific dataset using the maximum-likelihe objective. Recall we term the language model after this step the reference model and show that with Tef. Next, a reward model r: Σ' + E is trained to capture human preferences; the reward of string high if is preferred by humans." Finally, the reference model fine-tuned maximize the KL-constrained RL objective,
(n)(()
= E y-uTT
((0)) 3 DxL. (To Tree).
where (-) is the KL divergence between two distributions, modulated by a hyperparameter B. This objective encourages the model to assign greater probability mass high-reward outputs while simultaneously penalizing excessive divergence from the reference model. Levine (201 8) shows that the policy that maximizes' this objective (Eq. (1))
(
1 Z Tiat y exp
(v))
Z
Σ (x) exp (3)(0)). DEΣ-
(2)
In simple terms, nØ is the reference model reweighted by the exponentiated reward values and normalized by the partition function Z. However, direct sampling from Tे is not feasible, as computing Z requires evaluating an infinite sum, making it intractable. However, a heuristic approach sampling from = would be to sample many strings from "ee and only keep those that have high rewards. Indeed, this heuristic is the motivation behind the BoN algorithm.
3 DERIVING THE BEST-OF-N OBJECTIVE
ofst-of-N is a simple biignment-via-infereee algorithm. The algorithm works as follows. Let YN = (N) 6
J==1 be the multi-set containing N ii.d. samples from Nee. Then, BoN returns y",
argmaxr((1) (y) EYN
We present the probability distribution induced by BoN with Toon Notably, Tbon is not the optimal distribution under Eq. (1), the KL-constrained RL objective." Despite this, the BoN algorithm often
Pleasse refer to Tab. for summary of notations used throughout the paper.
Foor example, in summarization task, preference dataset consists of document, two candidate summaries for that document, and a label indicating which summary is preferred by humans. The reward model is trained on this datasct to maximize the likelihood of correctly predicting human preference.
"This formulation implicitly assumes that there exists 0 O that achieves the unconstrained maximum. "We assume that the argmax is unique, or ties are broken in well-defined manner.
Under simplifying assumptions Toon asymptotically (in string length) equivalent Tá (Yang al., 2024).
3

---

Published as a conference paper at ICLR 2025
performs well even in comparison to RLHF-based methods. This naturally raises the question: under what optimization objective is Theu the optimal distribution? To answer this question, we first compute the probability of strings under Tbon-
Proposition 1. Suppose r: Σ" + IR is a one-to-one mapping, Then, the probability of a string y under boo is given by
N ) F(r(y)) *+ (3)'
Thon (3)
F(r(y))
dr P w'mtted
(r(y') r(y)).
(4)
Proof. See App. B.
can be understood as the strict cumulative density function of reward values under Tee. In other words, F (((())) represents the probability that random sample drawn from aee has reward value less than rly). We now describe how to fine-tune the language model to approximate Thon- Similar to variational inference, we minimize the reverse KL divergence between Te and Tuon- Concretely,
JvBON (0) DDa (To mboa)
E yuuTo
log Tun (y) log (3))
(5a)
E y~mn
log tooa (y)
+ +H(
(5b)
N log Σ 2=1
( F(r(o)) N-i Te((())] + H(TO),
E y-uo
(5c)
where H(.) is the entropy of a distribution. Thus, Eq. (5) offers an answer to the question of what objective BoN optimizes. Inspecting the objective further, we see that Eq. (5) is an entropy- regularized objective, where we use the probability of the string under the BoN distribution as the reward and discourage the model from having low entropy.
Monotonically invariant. An important property of the variational Bo.V objective is that it is invari- ant to applying a strictly monotonically increasing function to rewards. This is because the vBoN ob- jective relies on reward values solely through F, which, as defined in Eq. (4), only depends on the rank- ing between the reward values and not their exact magnitude. This suggests that the vBoN objective may be less sensitive to outliers and the scale of rewards. This property is important as RL algorithms notoriously sensitive to the scale of reward values (Henderson et al., 2018; Schaul al., 2021).
Approximating log F(.). Maximizing Eq. (5) requires us to compute log F(-) for any r(y). This, however, is computationally expensive, as we have to sum over the probabilities of all strings that have rewards less than r(y). Fortunately, we can instead maximize a lower bound of Eq. (5) using Monte Carlo estimator of F().. Concretely, we can write F(-) as an expectation,
F(r(y))
E
[1(r())) r(y)]]-
(6)
y' n
We approximate F(r(B)) using M i.i.d. samples from Tet. termed y (M) lid (m)) < r(y))- We then take the log of this Monte Carlo estimator as a biased, but consistent estimator of log F(-) in Eq. (5).* In 15.1, we empirically assess the number of samples needed for log F to accurately approximate log F.
Tet using
which we compute ((((g)) del 1 MM
N
*Using Jensen's 's inequality, we show biasedness. Concretely, note the following lower bound
M (r(e/(m))
log (r(e)) = log
E (11)) m.. Marr
< r(v)))
(7a)
M
m=1 M
E 0000 ....y (M)
log
Σ
1r (m)) < <(()))
(7b)
M
m=1
where Jensen's 's inequality is applicable because log is concave. Consistency can be shown with application of the delta method ($5.5.4; Casella & Berger, 2001).
4

---

Published as a conference paper at ICLR 2025
100
Win rate (%) over Wed
80
Average rewards
0.8
BoNBoN
BoNBoN
DPO
0.7
DPO
60
PPO
PPO
vBoN
vBoN
0.6
BoN
Bo
0
10
20
30
10
20
30
DEL (mal meer
DEL. (me [[Ttet
(a) 1% of points on Pareto front belong to BoNBoN, (b) 7% of points on Pareto from belong to BoNBoN, 1% to PPO, 12% to DPO, and 50% vBo.V.
10% DPO, 33% PPO, and 50% vBo.N.
Figure 2:
Steering generated movie reviews towards positive sentiment. Points that are not on the Pareto front of each method have lower opacity. BoN is the most effective approach in achieving high win rates and high rewards while not diverging too far from the reference model. Our variational approximation BoN gets closest to the performance of Bo compared to other fine-tuning methods, as reflected in the percentage of times it appears on the Pareto front.
COMPARING THE BON AND RL OBJECTIVES
To explore the connection between the vBoN objective and the KL-regularized RL objective, derive a lower bound for JuBoN, Through this lower bound, we hope to achieve deeper insight into how the reward function is used in the variational BoN objective, and why this objective discourages high KL divergence from the reference model.
To derive such lower bound, substitute the BoN distribution in Eq. (4) into the vBoN objective in Eq. (5). We then simplify this objective to arrive at the following theorem.
Theorem 2. We have JBBON (0) 2 L(0), where
E L(0)= def (N 1) wm=
(r()(( Dxi. (mo || ((e)..
(8)
Proof. See App. D.
Empirically, observe that models that fine-tuned maximize L L(0) perform competitively to the ones that are fine-tuned to maximize the vBoN objective; see App. G for experimental results. Interestingly, if we compare Eq. (8) to the KL-constrained RL objective, Eq. (1), see they have very similar structure. We observe that N (in the vBoN objective) acts as regularization parameter. As N + 1, the optimal distribution gets closer to Tee, which has the same effect as - oo Eq. (1). Furthermore, as N + oo, the optimal distribution only generates the string with the maximum rewards, which equivalent under the KL-regularized RL objective and the vBoN objective are equivalent.
in Eq. (1). Importantly, both limits, the optimal distribution
The main difference between the KL-constrained RL objective Eq. (1) and the derived vBoN lower bound Eq. (8) is in how the reward function is used. The KL-constrained RL objective aims to max- imize the expected reward values, whereas vBoN maximizes the cumulative probability that strings sampled from the aligned model, тe. achieve higher rewards compared to those sampled from Nea-
5 SENTIMENT CONTROL
We now employ the variational BoN objective, Eq. (5), to fine-tune language models. We perform an open-ended text generation task where the goal is generate movie reviews with positive sentiment.
5

---

Published as a conference paper at ICLR 2025
The reference model, Tee. is GPT-IMDB'. GPT-2 (Radford et al., 2019) model fine-tuned IMDB corpus (Maas et al., 2011). We use a binary sentiment classifier," denoted as p. with two classes (POS, NEG) as the reward model, and define r(y) def p(POS y). Following Rafailov et al. (2023), sample 5000 movie reviews from the training set of IMDB dataset and for each sample, we randomly choose prefix length from {2,... ..... 8) and take that prefix as the prompt. We further generate 512 prompts in the same way from the test set of IMDB that we use to evaluate our models.
We fine-tune the reference model with PPO using the vBoN objective Eq. (5). Then, compare the performance of the fine-tuned model (yBoN) to the exact BoN (BoN), i.e., applying BoN at inference time.
We implement and compare the following existing methods for language model alignment:
Perhaps the most straightforward way to approximate BoNV distribution is to fine-tune the model to maximize the likelihood of the samples taken with BoN algorithm. Unfortunately, we find that SFT is incapable of achieving a good trade-off between achieving high rewards and low KI divergence, see App. H (Fig. 7) for the experimental results.
Bo-V-SFT:
PPO: We use PPO to optimize the KL-constrained objective in Eq. (1). We use the default hyperparameters in trlx library (Havrilla et al., 2023) for fine-tuning with PPO.
DPO. Direct preference optimization (DPO; Rafailov et al., 2023) is popular alternative to RLHF that does not require training reward model. Following DPO's experimental setup, we generate reviews per prompt and use the resulting 12 pairwise comparisons per prompt to construct DPO's contrastive loss."
Concurrent work (Gui et al., 2024) explores another approach to approximate Bo.V distribution. Assuming that the reference model distribution uee is continuous, Gui et al. (Theorem 3; 2024) prove that the expected difference between the relative likelihood, i.e., Thal-) of the Best-of-N response and the Worst-of-N response is 2/3 (N (N EN- 1/k= They use this property to construct loss function similar to that of IPO (Azar et al., 2023). Furthermore, they add another term to the loss function, which simply maximizes the likelihood of the Best-of-N response. The final loss function is a convex combination of the IPO-like loss and the negative log-likelihood loss, regulated by hyperparameter
BoNBoN:
*m(())
We fine-tune models by varying the degree of regularization. For BoN approaches, that is achieved by varying N, and for DPO and PPO, we vary B. Conveniently, N in vBoN a hyperparameter, meaning that we do not need to generate more samples from nee when we increase N. However, with BoN and BoNBoN methods, we need to increase the number of samples from the reference model as increase N.
We generate movie reviews based on prompts from our test set using fine-tuned models and then measure three metries: (i) KL divergence between the fine-tuned model and the reference model; (ii) win rate, defined as the percentage of times the fine-tuned model's generations receive higher rewards compared to the reference model's generations; and (iii) average rewards obtained by the fine-tuned model's 's sampled strings.
For the BoN method, we report the empirical upper bound of log N N-1 for KL divergence (Beirami et al., 2024; Mroueh, 2024) in our plots. Furthermore, the win rate of BoN over the reference model can be computed analytically and is equal to N+I.
N
N
We visualize the win rate vs. KL curves in Fig. 2a, and Fig. 2b the average rewards of generations under To vs. the KL divergence. As expected, BoN is the most effective approach; however, this comes an extra inference cost that grows with N. We observe that among the fine-tuning methods, our variational approximation to BoN gets closest to the performance of BoN, as it appears more
"Specifically, we use https://huggingfaaae co/ htp/v/werr//ge www.rra/gpt2iimbbb...
10 "Sppcifically, we use https: htt.hhugggggfccccc wo/lwwerra/ citerraddistee disiiiiiiii weellllerrie imdb.
I "One could argue that DPO has slight advantage over other methods in this setup since it has seen 6 unique generations per prompt during training, while the others only have seen one (or 2 with BoNBoN). Nevertheless, observe that VBo.V more effective than DPO.
12 FFoloowin the authors' recommendation, we set so that hoth terms contribute cqually to the final loss. See App. F for more details regarding the regularization hyperparameters.

---

Published as a conference paper at ICLR 2025
comsenta
peonp:: Norrible. see many usez
prompt: ailly Wiider in
novie because thece
proepe: I theeoughly anjoyed this
log F F(y)
10
10
10
100
400
400
600
600
600
r(y)
r(y)
r(y)
Figure Estimates of log F-) with increasing the number of Monte Carlo samples. We test an adversarial prompt (left plot), neutral prompt (middle plot), and prompt with positive sentiment (right plot). Overall, we hardly see any difference between the estimates after taking 200 samples. For the adversarial prompt, the distribution of rewards is peaked, and we do not see any changes in our estimator after taking only 100 samples.
often on the Pareto front of the two curves compared other methods. Notably, observe that DPO performs better than PPO in terms of win rates but worse in terms of average rewards; this could be attributed the contrastive nature DPO's loss function.
5.1 ERROR IN ESTIMATING log F(-)
We empirically quantify the error when estimating log F-)) with a finite number of i.i.d samples from Tee. To get a better intuition on the eтrоr of our estimators, Fig. we visualize the estimators for different prompts: one adversarial prompt (left plot), where the prompt itself has a negative sentiment, one neutral prompt (middle plot), and one prompt with a positive sentiment (right plot). We vary the number of Monte Carlo samples from 10 to 600. We observe that for all the 3 prompts, the estimated CDF hardly changes after 200 samples. When using the adversarial prompt, the reward distribution negatively peaked, and the estimated CDF does not change after taking only 100 samples.
then quantify the change the estimator by performing two-sample Kolmogorov-Smimit test (Hodges, 1958). This test measures the closeness of two empirical cumulative distribution functions. Concretely, the test statistic
sup yEΣ*
Mi (r(y)) FMM (r(s)))
(9)
where FM, and FM, are estimated CDFs from Mj and M2 samples respectively. The statistics show the magnitude of the difference between the empirical distributions of samples. The null hypothesis is that the two distributions are identical.
In Tab. 1, for each sample size M, compare the esti- mated CDF with M samples to the estimated CDF with 600 samples. the two distributions are identical according to the test, we can reliably use the M sample to estimate the CDF. We report the number of prompts (out of 5000 prompts) for which we reject the null hypothesis, mean- ing that the distributions are not identical. Furthermore, for those prompts, report the average test statistics and p-values. In general, for very few prompts, the null hypoth-
Table I: Measuring estimation error with increasing the sample size. After 250 sam- ples. the estimated CDF is unchanged for all the prompts.
Rejection rate
Test statistics
p-value
6.14%
0.63
20
4.02%
0.33
0.03
esis is rejected. Moreover, with 250 samples, the estimated
100
1.14%
0.17
CDFs are identical to the estimated CDF with 600 samples for all prompts.
200
0.06%
0.12
0.02
250
7

---

Published as a conference paper at ICLR 2025
5.2 EFFICIENCY ANALYSIS
We break down the efficiency analysis into main parts: (i) the inference cost, (ii) the preference optimization cost, (iii) and the preprocessing cost.
Inference cost. As discussed earlier, vBoN is an alignment via via-fine fine--tttuuuue teniintting method, and along with other alignment-viat via-fine-tuniin fine-tuning methods, it is N times more efficient at inference compared to Bo.N.
Optimization cost. We compare vBoN's preference optimization cost to its closest alignment- via- weif--nntttiiit tine-tuning counterpart, PPO. the optimization loop, the main difference between PPO and vBoN is that vBoN requires computing the strict CDF function, F, using M samples. Crucially, N vBoN serves as a regularization hyperparameter, and increasing N does not incur additional computation costs. To implement vBo.N efficiently, precompute the F function before starting the optimization loop. This means the computational overhead is incurred only once, regardless of the number of optimization runs. Since the F values precomputed, empirically observe that the time needed to run the vBo. optimization loop the same as running the PPO optimization loop, and the cost of evaluating F is negligible. Therefore, the main computational overhead in vBo. comes from precomputing log F(.).
Preprocessing cost. Estimating log F(-) requires only forward passes through the LLM and reward model without the need to compute and store gradi- ents. This makes the process highly parallelizable Our experiments utilize a memory -fffccceent library for LLM inference (VI.LM; Kwon al., 2023), which allows us to perform these approximations efficiently.
Performance
0.8
0.75
We examine the impact of increasing the computa- tional cost of vBoN by varying M, which directly affects the total elapsed time and downstream performance. For this analysis, fine-tune the model using three random seeds. We report the average and standard deviation of reward values and win rates in Fig. 4 on a single A A100-40GB GPU. Our results show that increasing M generally improves the aligned model's rewards and win rates. Notably, even with MM 32 samples (laking only 10 minutes), the performance remains competitive with higher values of M. We hypothesize that the data efficiency of the simple Monte Carlo estimator can be improved by taking into account the similarity between different prompts to learn an approximation to
Average Reward
Win Rate
fix N 10 and
May
Mag
Maje Madg
Mrdt
Total Elapsed Time (Minules)
Figure 4: The average reward and win rate of the aligned models improve we increase the sample size M used for approximating the vBoN loss function.
log F function, which we plan as future work.
SUMMARIZATION
We further employ variational Bo in summarization task, where the goal is to generate summaries that align with human preferences. The reference model, Trefi is a pythia-2. 8B model fine-tuned on human-writte summaries of Reddit posts Stiennon et al. (2020). We use SFT to refer this model in the plots. We use two separate reward models for training and evaluation: a pythia-2. reward model for fine-tuning and larger pythia-6. 9B model exclusively for evaluation.
Dataset. To evaluate the generalization ability of the aligned models out-of- distribution data, fine-tune the models using only posts from the relationship and relationship advice subreddits
"This particularly advantageous since practitioners often perform the optimization multiple times to test various hyperparameter settings.
15 use https://huggingee
l' use http://hhg..nfefeeoococrrrra////
2bbbdddee sff..tldr wotpe//o/lllae/lleeeeeeeeee pythia-2 2. be-..b-deee ab-deduped reward tldr.
co.cleanrl/Eleutherrre
We use https://huggingfaac http://ggggggaaeeeooo//ara/////e EEluthhttt pythia-6 ob-deduped reward tldr.
8

---

Published as a conference paper at ICLR 2025
6.5
BoNBoN
DPO
0.7
PPO
Average Reward
5.5
vBoN
Win Rate
BoN
0.6
5
4.5
0.3
4
3.5
0.4
3-
0.25
0.5
0.75
0.25
0.5
0.75
Temperature
Temperature
(a) Comparing the win rates of alignment meth- ods against samples from the Ret. vBoN achieves closer results to Bo.V compared to other alignment-viat sitenment-via-fnn-ture fine-tuning methods.
(b) Comparing the average rewards obtained from the evalu- ator reward model. BoN outperforms other alignment meth- ods, and vBoN achieves closer results to Bo.V compared to other alignment- saanntt-viiiiisss cel--ineetuuee tine-ttning methods.
Figure Performance of different alignment methods on the summarization task. Solid traces show the performance in-distribution Reddit posts, while dashed lines demonstrate the out-of- distribution performance. Overall, Bo the most effective approach in achieving high win rates and average rewards across all sampling temperatures. Our variational approximation to Bo (vBo.) gets closest to the performance BoN while being significantly cheaper at inference time.
of the Reddit TL; (Stiennon et al., 2020) dataset. We then assess the models' performance the two types of data by dividing the test set into two equally-sized groups: in-distribution Reddit posts from the relation relationship and relationship p_adssiippiiieeeeeeee advviee subreddits, and out-of-distribution posts from the rest of the subreddits. We visualize the performance of methods on in-distribution data with solid trace and on out-o--distribution data with a dashed trace.
Experimental setup. We fine-tune the model with both the Kl--constriined RL. objective and vBo. objective for 10000 episodes. Similar to the previous experiment, we use 200 samples to estimate log F(.) values. To create smooth and continuous reward function, we further fit exponential 18 to the estimates. We set N 100 for Bo and vBoN methods and the equivalent value of B 0.05 for the KL-constrained RL objective. We closely follow Huang et al. (2024) for setting the hyperparameters details. After fine-tuning, we sample from the aligned models with different sampling temperatures t E (0.25, 0.5, 0.75, 1.], each with 3 different random seeds.
the PPO algorithm; please refer to App. F for more experimental
Win rates. In Fig. 5a, we visualize the average and standard deviation of win rates compared against the samples from the SFT model. Notably, BoNV achieves the highest win rates, which is consistent with findings from previous studies (Rafailov et al., 2023). We do observe any significant differences between BoN performance on in-distribution (solid trace) and out-of- -ittof-distribution data, 19 method. Similarly, we mostly not observe
which is expected as BoN is an deignmenn-via-infeeere significant differences between in- and out-of-distribute performance of all alignment-via-fime tuning methods, indicating that these methods can generalize effectively in this experimental setup. DPO and BoNBoN only manage perform competitively to other methods lower temperatures (0.25, 0.5), and their performance drops significantly at higher temperatures (0.75, 1). Importantly, while PPO and vBo.N perform comparably at higher temperatures, vBoN significantly outperforms PPO at lower temperatures (0.25 and 0.5).
We fit an exponential function the form f(x) = -a (xr) to data using non-linear least squares. 19e difference between the two data distributions becomes more apparent at temperature 1, potentially due to increased sample diversity in this setting.

---

Published as a conference paper at ICLR 2025
Table An example of summaries sampled temperature 0.5 and their corresponding reward obtained from the evaluator reward model.
Content
Reward
SUBREDDIT:
r/relationship advice Stuck in a rut and in need of advice/inspiration!
TITLE:
POST: My boyfriend and have been together for 3 years, and living together for 2. I'm quite the homebody, and when we first met, was very outgoing and loved partying and socialising (although he most nights we find ourselves doing the same things: watching series (luckily we enjoy the same shows). playing Minecraft or playing various board games. We's re tired alter work, and can't I bring ourselves leave the house. The weekend is much the same lots of sleep, or sitting around staring pub once every few months) and there are a few mutual friends we enjoy
student the time). We'ne re both working now, and
another. We party occasionally
(we' II head spending time with, but worry that ve've become stuck in our boring ways. I really enjoy our lifestyle, and would be quite happy to never leave the house again, but I'm starting feel guilty turning him into a year-old when he's only 24. Any ideas for shaking things up little? Bear mind that live in small lown South Africa, and neither of us has car.
3.08
SFT: I'm stuck in rut, and need shake things up if it'll work Any advice!
4.59
PPO: In need of inspiration to break out of rul and live life fully! Any ideas welcome!
6.79
VBoN: Been happily living together for 2yr+, feeling bored after work regularly, looking for ideas to spice things up!
9.18
Bo.N: My boyfriend and have been together for years, and are both working full time. We spend most of our time in the house, and have become boring. What can we do to shake things up?
Average rewards. In Fig. 5b, we measure the average rewards across different temperatures. As the temperature increases, the average reward decreases consistently across all methods. This trend is also evident in the qualitative analysis in App. I, where we show sampled summaries at different temperatures. DPO and BoNBoN suffer more from increasing the temperature, as the average rewards get close to (or even worse than) the SFT average rewards. Generally, the average reward results align with the win-rate trends, and we observe that vBoN achieves significantly higher rewards compared to PPO at lower temperatures. In Tab. 2. we show example of summaries generated from the fine-tuned models with their associated reward values.
7 CONCLUSION
Motivated by the effectiveness of the BoN algorithm, we formally derive a variational approximation the distribution induced by Bo.V algorithm via fine-tuning language models. Our analysis highlights the similarities and distinctions between the variational Bo objective and the KL-constrained RL objectives. Our empirical findings reveal that models fine-tuned using the variational approximation to BoN not only attain high reward values but also maintain proximity to the reference models. Crucially, inference on the fine-tuned models with the vBo. objective remains as cost-effective as inference on the original reference model.
ACKNOWLEDGEMENT
We thank Ahmad Beirami for the fruitful discussion in the early stages of this project. We also thank Amrit Singh Bedi for identifying a typo in a previous version of the bound derivations. Finally, we thank the anonymous reviewers for their feedback. Afra Amini is supported by the ETH AI Center doctoral fellowship.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. Computing Research Repository, arXiv:2310. 12036, 2023. URL hwtph:httt/aaeee org/abs/2310. 12036.
Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander A'mour, Jacob Eisenstein, Chirag Nagpal, and Ananda Theertha Suresh. Theoretical guarantees the best-of-n alignment policy. Computing Research Repository, Ttt:::201001877 2024. /tl.https://arxvv.or/////// 01879.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. Computing Research Repository, arXiv:2407.21788 2024. URL http://aaxxxv...r.bbbbss/e 21787.
George Casella and Roger L. Berger. Statistical Inference. Grove, CA, 2nd edition, 2001. ISBN 9781032593036. URL http::://wwwwwttt/// com/
Chapman and Hall/CRC, Pacific
Eugene Charniak and Mark Johnson. Coarse-to-fine n-best parsing and MaxEnt discriminative reranking. In Proceedings of the Annual Meeting the Association for Computational Linguistics, 2005. doi: 10.31 12t1111//294..119968289
URL http:hss/accaaanooooooooorrrggg org/P05-1022.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep rein- forcement learning from human preferences. In Advances Neural Information Processing Systems, 2017. URL
www.rreeeedinnsnniiipii.... wttppppr.cpppp.p...ceccppp... saaaper_files/eeeeeeee files/paperr/211//iike d5e2c0adad503c31 F91 4F4448800cc44449Papee pddf.
Hanze Dong, Wei Xiong. Deepanshu Goyal, Yihan Zhang, Winnie Chow. Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https: /ttps://openrevieww wwwptneumt/iiiiieeo 2tt
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Proceedings of the International Conference on Machine Learning, Proceedings Machine Learning Research, 2023. URL https: https://proceedingg mlr, (rres//202/ggo20) html.
Gemini. Gemini: A family of highly capable multimodal models. Technical report, Google, 2024. URL https://arxiv/ wttprxiiiooogp//////1112/
Lin Gui, Cristina Gårbacea, and Victor Veitch. BoNBoN alignment for large language models and the sweetness of best-of-n sampling. Computing Research Repository, arXiv:2406.00832, 2024. URL https://arxiin httpg/ddf/24665 20832.
Alexander Havrilla, Maksym Zhuravinskyi, Duy Phung, Aman Tiwari, Jonathan Tow, Stella Bi- derman, Quentin Anthony, and Louis Castricato. trlX: A framework for large scale reinforce- ment learning from human feedback. In Proceedings of the Conference on Empirical Meth- ods in Natural Language Processing, 2023. doi: 10.18653/w120223.emm conlp-mmin.530 URL .rg.
org/2023, wrgg2/23.emnn1mmmaa.. emnlp-main. 530.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pincau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the Conference Artificial Intelli- gence and Innovative Applications of Artificial Intelligence Conference and AAAI Symposium on Educational Advances in Artificial Intelligence, 2018. URL https:://dl. acm. org http:/oo////////////////////////
5555/350488 T.0.A483...5442.... 3504427.
Joseph L. Hodges. The significance probability the Smirnov two-sample test. Arkiv för Matematik, 3, 1958. URL https: /appi..ppppppp wwwpaaniischolar.rrg/rorrraaaaae

---

Published as a conference paper at ICLR 2025
Shengyi Huang, Michael Noukhovitch, Arian Hosseini, Kashif Rasul, Weixun Wang, and Lewis Tunstall. The N+ implementation details of RLHF with PPO: A case study on TL;DR summariza- tion. In Conference on Language Modeling, 2024. URL kH02ZTa8e3.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng. Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with Pttenttttentio. In Proceedings of the ACM SIGOPS Symposium on Operating Systems Principles, 2023. URL
wttpvvv....r///////3////// 02.309.06188 06180.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. Computing Research Repository, 1eXiv:1805.009000 2018. URL /ttttp://aaxvvoorrrr/ /1805. 00909.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the Anmual Meeting of the Association for Computational Linguistics: Human Language Technologies, 2011. URL httpss/aaaaaathhlooo www.ecannnoooooooog/11-----
https:
Meta.
Llama 2: Open foundation and fine-tuned chat models.
Technical re-
port, Meta, 2023.
URL https://ai. http:://ai.meta... http://aaaeeaa.commeeeeeeeee tene-ttued-chhat
Youssef Mroueh. Information theoretic guarantees for policy aligmment in large language models. Computing Research Repository, arXiv:2406.05883, 2024. URL httpp:://ariiv http:/www...gg///////a////// 05883.
Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng. Michael Collins, Trevor Strohman, Jilin Chen, Alex Beutel, and Ahmad Beirami. Controlled decoding from language models. In Proceedings of The International Conference Machine Learning, Proceedings of Machine Learning Research. PMLR, 2024. URL
https://arxixx http://aaxxvvo..g.ppe./ 2310. 17022.
Rejichiro Nakano, Jacob Hilton. Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. WebGPT: Browser-assisted question-answering with human feedback. Computing Research Repository, 1rXiv:2112.09932, 2022. URL https:://rxiv..ttt pdf / 12.12..000 09332.
OpenAl. GPT-4 technical report. Technical report, OpenAl, 2023. URL http:://///cooooaii...... magena..cm com/ papers/gpt-4.pp pdf.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddic Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022. URL http:: //ooceediioo//ooo/s/ooooo/ .ew.eeiiidiinnennnni...ee.... cc/paper. .df.en.ee..............
Filess/paper/2022/F http://222222fffeeee file/b1 o 9rdd555
Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. West-of-n: Synthetic preference generation for improved reward modeling. Computing Research Repository, /401. 12086.
arXiv:2401.12086 2024. URL
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language mod- els are unsupervised multitask learners, 2019. URL https:://4mucfppksyss cl cloudfront. net/ stw.odeese..eeuuuuurriieeeeeeddd ti task_learners. pdf.
Lentter-languagemmodee m
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Advances in Neural Information Processing Systems, 2023. URL https: http///xxvvvooooo/
www.vv.oog//ooooo wttgp/ff//33/ /2305. 18290.
pdf.
12

---

Published as a conference paper at ICLR 2025
Tom Schaul, Georg Ostrovski, Iurii Kemaev, and Diana Borsa. Return-based scaling: Yet another normalisation trick for deep RL. Computing Research Repository, arXiv:2105.05347, 2021. URL
https://arxiv.. www.://gg/////et/ 2105. 053477.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy 2017. URL https:
optimization algorithms. Computing Research Repository,
Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Nino Vieillard, Alexan- dre Ramé, Bobak Shariari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, Sertan Girgin, Pi- oir Stanczyk, Andrea Michi, Danila Sinopalnikov, Sabela Ramos, Amélie Héliou, Aliaksei Severyn, Matt Hoffman, Nikola Momchev, and Olivier Bachem. BOND: Aligning LLMs with best-of-N distillation. Computing Research Repository, arXiv:2401.1208 2024. URL https://arxiv. wwwprriiv.oo.a///.ee wrgg/bss//4/// 12086.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute opti- mally can be more effective than scaling model parameters. Computing Research Repository, S01i::2408.03311 2024. URL https://arxiv. borg/bb///ee 03314.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. In Advances in Neural Information Processing Systems, 2020. URL https: http://prooeedds..eeeeee newrips.eerppp.ccomm cc/ Paper. .df.
faper_files/
Joy Qiping Yang. Salman Salamatian, Ziteng Sun, Ananda Theertha Suresh, and Ahmad Beirami. Asymptoties of language model alignment. Computing Research Repository, arXiv:2404.01730, 2024. URL https://arxivv wwtpx/vvvoorgggpeeee htrrgpdff/2400 01730.
Kevin Yang and Dan Klein. FUDGE: Controlled text generation with future discriminators. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021. URL https: /ttp//aaaaaaalogyyooaaaa org / naacl-main, 276.
13

---

Published as a conference paper at ICLR 2025
Symbol
Type alphabet E Ex E
Explanation
Σ is set of symbols
V. y
strings in E*
a
prompt string in Σ"
E +
A real vector representing the parameters of language model
language model
A language model parameterized by 0
Tee
language model
A supervised- wwwrviieedfin--tuuee fine-tuned language model
Σ
R
A reward model
Regularization parameter for the KI. divergence term
F
R
+
R
A strict cumulative density function of reward values under rer Number samples used in BoNV algorithm
Z
M
Z+
Number of samples used in the MC estimator
Table 3: summary of the notation used in the paper
A RELATED WORK
Best-of-N. BoN is a straightforward alignment-via vianf----ii-neeeeeee algorithm to optimize the output of the language model using trained reward model (Charniak & Johnson, 2005; Stiennon al., 2020). Despite its simplicity, Bo.V performs comparably or even better than other alignment methods, such RLHF and direct preference optimization (Nakano al., 2022: Gao et al., 2023; Rafailov et al., 2023). However, as noted by Stiennon et al. (2020), BoNV is an inefficient algorithm due to the reduced throughput inference time.
Applications. BoNV has been applied successfully at various stages of the development of language models. Meta (2023); Dong et al. (2023) employ iterative supervised fine-tuning on the outputs of the BoN algorithm to clone its behavior in the model. Pace et al. (2024) leverage BoN to enhance reward modeling by training the reward model both the best and worst responses. Additionally, Brown et al. (2024); Snell et al. (2024) explore the scaling laws for biigament-via-ifeeeee methods and demonstrate how to utilize the limited inference budget to achieve the alignment.
Best-of-N as an sespeneent--iaain-ttutttee method. Two concurrent efforts to ours have also attempted to convert Bo.V mate the BoN by maximizing the likelihood of the Best-of-N response and adjusting the relative likelihood of the Best-of-N and the Worst-of-N response. Second, Sessa et al. (2024), similar to ours, uses reinforcement learning to minimize the distance between the language model and the Bo policy. Different from ours, and reduce the fine-tuning time, the authors use a crude estimation of log F' and approximate the distance to Best-of-N by iteratively distilling the Best-of-2 model as moving anchor.
wii-eenn-vvaai-e-----eee method. First, Gui al. (2024) approxi-
B PROOF OF PROP. 1
Proposition 1. Suppose r: Σ - IR. is one-to-one mapping. Then, the probability of a string under Thom is given by
N F(rls)) "er(y)) F(r(y))
Thon (y)
Σ 1=
del P y'atua
(r(w') <r())).
(4)
Proof. The proof follows Casella Berger (2001, Theorem 5.4.3). To compute Thon(y), we first
define events: (i) the event that all N samples have rewards less than equal to r(y), and (ii) the
14

---

Published as a conference paper at ICLR 2025
event that all N samples have rewards less than r(y). The probability of those events is follows:
Pi (y) def P(all samples have rewards I r(y)) (F(r(y)) + =
(10a)
(y def (all N samples have rewards r(y)) F(r(y))
(10b)
Note that for Eq. (10a) to hold, we need the assumption that the reward function one-to-one mapping." 21 Furthermore, given this assumption, Thon is the probability that at least one of the sampled strings out of N samples have the reward exactly equal to r(y) and the rest of the samples have rewards less than equal to r(y). Given how defined pi and p2. we have Thon (V)
P1 (1 (3)).
NV Tbon Tyo = (F((())) + -(e(()))-((((())))) = Σ (M) F((())) mm)())² Tu)(()))
(11)
C STRATEGIES FOR NON-INJECTIVE REWARD FUNCTIONS
the reward function is not injective, we need a tie-breaking strategy the Bo/V algorithm. We formalize this as defining total order K, on Σ* as follows: for any two strings y1 and y2. if r(y)) < r(ye) then we have Y/ 3, y2. If r(g)) = r(y2) then yı <, 1/2 only if yı < y2: where < some arbitrary but fixed total order, e.g., lexicographic order. Therefore, define F(y) as
Fly) def ( +r 9)
(12)
We then need to define the two events and their probabilities, and pa, given this total order on strings, as follows:
1, y) = (F(v) + )Te((1)) N
(1 def P(all N samples are
(13a)
p2 (2( def (all N samples are <, y) = F(y)"
(13b)
The rest of the proof is the same as with the one-to-one reward functions.
D PROOF OF THM. 2
Theorem 2. We have JUBON (0) 2 L(0), where
L(0)= (N - 1) E log F((())) - Dxa ( TTT)
(8)
The PMF of BoN also derived by Beirami et al. (Lemma I; 2024). In their notation, Pi F and p2 = F-1 2f the reward function is not one-to-one mapping, we need to devise tie-breaking strategy. See App. C. for further discussion.
15

---

Published as a conference paper at ICLR 2025
Proof.
First, prove JVBON(O) (0) 2 L(0).
(Te
Toon)
E ymma
log Te (y) - log Toon (3))
(14a)
N
N
E y--m
log (
log
F(r(U))
(14b)
8
==1
1=1
E
log (0
log
F(r(u))
merl())
(14c)
y--
log (3) log N F(r(y)) N-1 T
E
(14d)
<
E
log Ty( log F (r())) "- m
(14e)
y--
E
log Te (1) - log Tver(y) - (N - 1) log 10g (rg
(14f)
DxL. (To
Teet) (N 1)
E y-xo
log F((g))))) dd (()
(14g)
The incquality in Eq. (14c) stems from the fact that we drop positive terms in the summation and only keep the first term. Therefore, the lower bound for our objective is:
JNBON (0) --Da (To Thon 2 (N 1-11 1)
E youT
log (r(z)) DRi (TT Tret)
(15)
Another approach to deriving a lower bound is by using Jensen's inequality. By doing so, we arrive the following theorem.
Theorem 3. Let a (N+2)(N-1), B = N(N-1) and 7 N(N-1), L, (0), where we further define
Then, we have JN (0) 2
2
L (0) def ?
E. yuuT
log F((())))) - aH(To) - BDxx(To || me))-
(16)
16

---

Published as a conference paper at ICLR 2025
Proof.
DxL
Thom)
IE your
log To(y)
log Toon
(3)) (w)
(17a)
N F(r()))**
R y-ur
log (oy)
log
Σ i=1
(17b)
(t)
N F(((B)) N-1 (<)rle))))))))
E y/me
log (y)
log
(17c)
E
log (y)
log (t)
Σ log ((((z)) N-1
Σ log क re)
(17d)
N
N
N ( log F (r(v))
E youtee
log No (y))
log
E(N (N -) log Tedl y
(17e)
==1
1=1
N(N - 1)
N(N + 1)
E youtte
log (y))
log F(r()))))
log TT)
(17h)
2
2
N(N +
N(N 1)
E
log (y)
log (ee
log F(r(y))
(17g)
y-uT
2
((N + 1)
(N + +N(N 1)
N(N
DKL. (To)
Tref) +
E
log (o(
log F (r(y))
2
2
2
(17h)
N(N +
(N + 2) - 1)
N(N 1)
DEL (Ioe TTstt
+
((TT)
H
E no 2
log F (r(y))
(17i)
2
In Eq. (17c), because log(z) is convex for O, applied Jensen's 's inequality to obtain the upper bound. Abstracting away from the three multiplicative factors, naming them y, a and B, we end up with the following function
JuBON (0) -D (Te)
Toon
Y
E logF(r(v)) aH(TT) - DDx (TT Teat) (18)
which is bound for some settings of y, a and B.
Importantly, L2 a looser bound compared to We formalize this in the following theorem. Theorem 4. For every O we have L(Ø)= 2 (1))).
Proof. We prove (0) 2 -L()), meaning that L is tighter lower bound. According to Eq, (17f),
we have:
-Ia
E youo
log (v)
Σ log (((())
(19a)
E y-uT
log no(y)
log ((r()))) N m))())) Σ
(19b)
E you0
log log (
log F(r())) N-1 mer[e) -I(O).
(19c)
17

---

Published as a conference paper at ICLR 2025
Hypterparameter
Value
Episodes
10000
Optimizer
AdamW (c le
5, 1r= 3e
Scheduler
Linear
Batch Size
32
3 (Both for vBoN and KL-constrained RL objective)
0.05
Y (Discount Factor)
(for GAE)
0.95
Number of PPO Update Iteration Per Epoch
PPO's Policy Clipping Coefficient
0.2
Value Clipping Coefficient
0.2
Value Function Coefficient
0.2
Value Function Loss Clipping
True
Sampling Temperature
0.7
E vBoN PSEUDOCODE
Algorithm The vBoN algorithm
procedure VBoN T, N, E, B)
D: the prompt dataset, number of epochs, B batch size
2: Initialize my with Tiet
3: for E epochs:
4: for each batch in D:
5: (1)) ..... y (B) (-)
Sample response for each prompt in batch
6:
Compute r(y () )... ((y
7:
Compute F F(r(y" (")) )), , (()
8: Optimize with Eq. (5) (or Eq. (8) using PPO
9:
return .T0
EXPERIMENTAL DETAILS
Hyperparameter sweep in the sentiment experiment. To visualize the trade-off between the expected rewards and KL divergence, we vary the degree of the visualization using the following hyperparameters for each method:
BoN-SFT: E [00,50,00 50,90, 90, 130, 170, 210, 250, 330, 370, 410, 450, 490, 530, 570, 600] with different seeds, resulting 32 runs.
PPO: [0.005, 0.01, 0.02, 0.03, 0.04, 0.1, 0.2, 0.3, 0.4, 0.5, 1,, 2., 4., 5.] with different seeds, resulting in 32 runs.
DPO: 3 E [0.01, 0.1, 0.2, 0.3, 0.4, 0.5, 1,, 2., 3., 4., with different seeds, resulting in 33 runs.
BoNBoN and vBoN: N E [1,2,3,4, 3,4, 16, 32, 64, 128 256, 512] with 3 different seeds, resulting in 33 runs.
vBo. with L bound: 0.3, 0.4, 0.3, 1., 2., 3., 4., 5.] with 2 different seeds, resulting 32 runs. Note that comparing Eq. (5) and Eq. (1), we have N
8 E |0.005, 0.01, 0.02, 0.03, 0.04, 0.05, 0.1. 0.2,
PPO hyperparameters. In App. F, include the hyperparameters used with the PPO algorithm for the summarization experiment.
G COMPARING THE vBON OBJECTIVE AND L LOWER BOUND
We compare the performance of models fine- tuned with the vBoN objective and its lower bound (L) in Fig. 6. We observe that the performance of the models is very close each other.
18

---

Published as a conference paper at ICLR 2025
100
Win rabe (%) over Wed
Average rewards
70
VBoN-1.
V-1
vBoN
vBoN
BoN
BoN
0
10
20
30
10
20
30
DEL (mo merr
DRL. (me [|Tret))
Figure 6: Comparing models trained with the vBoN objective and its lower bound (L). We observe that the performance of the two methods is very close to each other.
100
Win rate (%) over Rrd
80
Average rewards
BoN-SFT
BoN-SFT
BoNBoN
BoNBoN
DPO
0.7
DPO
60
PPO
PPO
0.6
BoN
BoN
0
10
20
30
DeL. (wa (a))
Dxt. (To ([Tre) -
(a) 4% of points on Pareto front belong to BoNBoN, 1% to PPO, 12% to DPO, and 50% to vBoN.
(b) 7% of points on Pareto from belong to BoNBoN, 10% DPO, 33% PPO, and 50% vBoN.
Steering generated movie reviews towards positive sentiment. Points that are not on the Pareto front have lower opacity.
Figure 7:
H ADDITIONAL EXPERIMENTS WITH BoN-SFT
We further experiment with training with the maximum likelihood objective on BoN generations when varying N. The results are depicted in Fig. 7. We observe that Bo diverges too much from the reference model compared other fine-tuning methods for alignment.
QUALITATIVE RESULTS
19

---

Published as a conference paper at ICLR 2025
Table 4: An example of summaries sampled at temperature 0.25 and their corresponding reward obtained from the evaluator reward model.
Content
Reward
SUBREDDIT:
r/relationship..ee advice Stuck in rut and in need of udvice/inspiration!
TITLE:
POST: My boyfriend and have been logether for 3 years, and living together for 2. I'm quite the homebody, and when we first met, he was very outgoing and loved partying and socialising (although he most nights we find ourselves doing the same things: watching series (luckily we enjoy the same shows), playing Minecraft or playing various board games. We're tired after work, and can't I bring ourselves to leave the house. The weekend is much the same lots party occasionally of sleep, or sitting around staring at one screen another. mutual friends enjoy (we'll Il head a pub once every few months) and there a few spending time with, but worry that we' we've become stuck in our buring ways. I really enjoy our lifestyle, and would be quite happy to never leave the house again, but I' 'mm starting to feel guilty for turning him into a 50 year-old when he's only 24. Any ideas for shaking things up little? Bear mind that live in small town in South Africa, and neither of us has
student the time). We re hoth working now, and
5.83
SFT: I'm stuck a rut with my boyfriend, and I'd like to shake things up little. Any ideas?
4.88
PPO: In need of inspiration for moving past boring routines and becoming active and open minded individuals. Any ideas welcome!
7.39
vBoN: Been dating for three years and living together for two. Feeling stuck in rut after working hard all week. Looking for ideas shake things up!
7.14
BoN: Been together for 3 years, and living together for 2. We're both working now, and find ourselves doing the same things: watching series, playing games, or having sex. We're re tired after work, and can't 't bring ourselves to leave
Table 5: An example of summaries sampled at temperature 0.5 and their corresponding reward obtained from the evaluator reward model.
Content
Reward
SUBREDDIT: sfrelationship advice
TITLE:
Stuck in a rut and in need of advice/inspiration!
My boyfriend and have been logether for years, and living together for I'm quite the homebody, and when first met, he very outgoing and loved partying and socialising (although he most nights we find ourselves doing the same things: watching series (luckily we enjoy the same shows). playing Minecraft or playing various board games. We'r re tired after work, and can't I bring ourselves leave the house. The weekend is much the same lots of sleep, or sitting around staring one screen another. We do party occasionally (we')l Il head to a pub once every few months) and there are a few mutual friends we enjoy spending time with, but worry that we' ve become stuck in our boring ways. I really enjoy our lifestyle, and would be quite happy to never leave starting feel guilty turning him into a year-old when he's only 24. Any ideas for shaking things up little? Bear mind that we live in small town in South Africa, and neither of us has a car.
POST:
student at the time). We re hoth working now, and
house again, but I'm
3.08
SFT: I'm stuck in and need shake things up see if it'll work out. Any advice?
4.59
PPO: In need of inspiration to hreak out of rut and live life fully! Any ideas welcome!
6.79
VBnN: Been happily living together for 2yr+, feeling hored after work regularly, lnoking for ideas to spice things up!
9.18
BoN: My boyfriend and have been together for years, and are both working full time. We spend most of our time in the house, and have hecome boring. What can we do to shake things up?
20

---

Published as a conference paper at ICLR 2025
Table An example summaries sampled at temperature 0.75 and their corresponding reward obtained from the evaluator reward model.
Content
Reward
sfrelationship udvvce TITLE: Stuck rut and in need of advice/inspiraaion!!
SUBREDDIT:
POST: My hoyfriend and have been together for years, and living together for 2. I'm quite the homebody, and when we first met, he was very outgoing and loved partying and socialising (although he was a student at the time). We're re both working now, and most nights we find ourselves doing the same things: watching series (luckily we enjoy the same shows), playing Minecralt or playing various buard games. We're tired alter work, and can't bring ourselves to leave the house. The weekend is much the same lots of sleep, or sitting around staring at one screen or another. We do party occasionally (we'l Il head a pub once every few months) and there are a few mutual friends we enjoy spending time with, but worry that we'se 'vee become stuck in our boring ways. really enjoy our lifestyle, and would be quite happy to never leave the house again, but I' m'm starting to feel guilty for turning him into a 50 year-old when he's only 24, Any ideas for shaking things up a little? Bear in mind that we live in a small town in South Africa, and neither of us has a car.
4.41
SFT: Boyfriend and are stuck in a rut. We don't I do anything fun, and I'm starting to feel guilty for not even trying to get away from all.
4.76
PPO: In rutting boredom with boring life; any ideas for shaking things up little? Any ideas for improving relationships?
6.02
vBoN: Been happily cohabiting for three years with boyfriend of two years. Feeling guilty for turning him into an old man! Any ideas for shaking things up?
5.47
Bo.V: Boyfriend and are a little boring. and want shake things up so can go and have fun but we'r re too stuck on the idea of staying in all day.
Table An example of summaries sampled temperature 1. and their corresponding reward obtained from the evaluator reward model.
Content
Reward
SUBREDDIT: rfrelationship advice
TITLE: Stuck in a rut and in need of advice/inspiration!
POST: My boyfriend and have been together for 3 years, and living together for I'm quite the homebody, and when we met, was very outgoing and loved partying and socialising (although he most nights we find ourselves doing the same things: watching series (luckily we enjoy the same shows), playing Minecraft or playing various board games. We're tired after work, and can't 't hring ourselves to leave the house. The weekend is much the same lots of sleep, or sitting around staring one screen another. pub
student the time). We'r re both working now, and
party occasionally every few months) and there are a few mutual friends we enjoy
(we'll Il head spending time with, hut worry that we've hecome stuck in our boring ways. I really enjoy our lifestyle, and would be quite happy to never leave the house again, but I'm starting feel guilty turning him into a year-old when he's only 24. Any ideas for shaking things up little? Bear mind that we live in small town in South Africa, and neither of us has car.
5.89
SFT: My boyfriend and used to party all the time and ending up reverting to the boring old lifestyle of 24-ycar-old. How can hoth hreak out of nut and get some fun back in our relationship?
5.91
PPO: Taken classic boring suburban lifestyle and turned into boring 24yo. Looking for idcas for breaking out of rut and refreshing lifestyle. Any ideas welcome!
6.57
vBoN: ::
hoyfriend and seem to have fallen into rut of regularly doing mundane activities after working together non-stop for several months! ideas to spice things up?
6.74
BoN: in a relationship that's getting stale, looking for some inspiration make changes whim in hopes of rejuvenating
21


Reviews:
[{"review_id": "IukTcLcNME", "reviewer": "Reviewer_DQFD", "summary": "The best-of-N alignment strategy has proven very useful in generating text with high-rewards that still has a high probability under the reference model. Several studies have shown that BoN often outperforms models simply fine-tuned with RLHF. However, this improvement in BoN comes at computational overhead during inference. \n\nThis paper proposes Variational BoN (vBoN), a scheme that converts BoN from an alignment-via-inference algorithm to an alignment-via-fine-tuning algorithm. Basically, vBoN  derives the probability distribution induced by the BoN algorithm and then approximates this distribution by minimizing the reverse KL divergence between the language model and the BoN distribution. The model is optimized for the vBoN objective using the PPO algorithm. \n\nThe proposed objective is evaluated on controlled generation and summarization tasks, showing performance close to that of the BoN algorithm while being as cost effective as inference on the original reference model.", "questions": "Please see my comments in the Weakness part of the reviews.\n\nWhy BoNBoN is called with that name? Please explain.\n\n“We visualize the win rate vs. KL curves in Fig. 6a, and Fig. 6b the average rewards of generations under πθ vs. the KL divergence”. Please mention that these figures are in the appendix.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "vBoN is a novel and effective approach converting BoN from an alignment-via-inference algorithm to an alignment-via-fine-tuning algorithm. Models fine tuned with the vBoN objective achieves high reward values closer to the BoN approach, while achieving probabilities closer to the reference model. Importantly, it is as cost-effective as inference on the original reference model. In comparison , the original BoN approach is N times more expensive.\n\nProvided theoretical connections showing how vBoN is compared with other alignment objectives", "weaknesses": "Section 2 and 3 can be improved significantly by improving the notations and explanations and by bringing important details from Appendix to the main part of the paper. Currently I often find these two sections a bit confusing as well as a bit hard to appreciate some of the claims the authors have made in the paper. For example, in Eq 4, F(r(y)) will be defined as F(r(y) ) = P (r(y) < r(y) ), using Eq 5?  With this I am not sure how the vBoN objective is insensitive to applying any monotonically increasing function to the reward values?\n\nAnother important weakness of the paper is the current evals in the paper. Despite using the standard benchmarks for controlled generation and summarization, the evaluation chose not to report on standard metrics along with rewards and proximity to reference model comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xd2vkzis4K", "reviewer": "Reviewer_jVo6", "summary": "In this paper, the authors propose a novel optimization objective for reinforce ment learning in large language models: vBo𝑁. Inspired by the Bo𝑁 method during the inference phase, the authors have designed an innovative loss to aid model alignment training. Building on this, the authors have transformed the proposed loss into the easily optimizable vBo𝑁 by relaxing the optimization lower bound. The authors validated the potential of vBo𝑁 optimization on two tasks using the IMDB movie review dataset and the Reddit TLDR dataset, and compared it with other contemporary methods such as PPO and DPO, demonstrating the unique potential of the vBo𝑁 method.", "questions": "- \"It is invariant to applying any monotonically increasing function to rewards\" might not be entirely correct. Although monotonicity and insensitivity to reward scale are indeed related, the exact nature of this relationship is still unknown.\n- The extent of the loss introduced by using the Monte Carlo estimator to maximize the lower bound of Equation (6) compared to the original formula remains unclear.\n- The models used in the experiments by the authors are relatively outdated. I 'm curious if this method could achieve similar results on more recent models, such as the LLaMA-3 series.\n- The trend of vBoN changes relatively smoothly with temperature adjustments; what causes this smoother trend?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The authors conducted rigorous and thorough theoretical derivations in the paper, clarifying the process from the motivation behind the vBo𝑁 proposal to its transformation into an optimizable objective. This is highly beneficial for readers interested in optimization theory and can provide new insights for tackling more complex optimization problems.\n- The vBo𝑁 method is highly effective. Moreover, despite the substantial theoretical derivations, the illustrations used by the authors to present results and compare methods are clear and easy to read, helping readers to quickly grasp the potential of vBo𝑁 for their applications.", "weaknesses": "- Less Persuasive Experiments: While we understand that conducting RLHF is always exceedingly costly, for instance, PPO requires maintaining four sets of model parameters, the fact that the validation of the vBo𝑁 method was only focused on movie review completion and text summarization datasets makes it lacks persuasiveness. We would like to understand the potential applications of vBo𝑁 in broader and more challenging tasks, such as code generation, mathematical problem solving, and multi-step reasoning. The experimental section of the paper is not as solid as the theoretical section.\n- Lacking Efficiency Testing: One of the motivations behind the vBo𝑁 method is to address the additional overhead of the Bo𝑁 method during inference, which vBo𝑁 resolves by introducing additional training. Therefore, I believe that an analysis of the training costs of vBo𝑁 is a crucial issue that must be discussed in the paper. Based on the experimental results, I could even accept that the training efficiency of vBo𝑁 is slightly inferior to methods like PPO. Moreover, beyond absolute costs, relative costs are also worth considering, such as a comparison of the performance of vBo𝑁 and PPO under the same computational-power/time/data con straints. If the authors could address either of these two aspects, or preferably both, it would significantly enhance the completeness and the soundness of the paper.\n- Several key symbols and terms in the formulas are not defined, making it challenging for readers to understand the equations' meaning. A glossary would greatly improve readability.\n- In the \"Summarization\" section, the authors refer to the reward models with different names (\"pythia-2.8B reward model\" and \"pythia-6.9B model\"), but it’s unclear if these models differ in architecture or purpose (see Section 6).\n- The experiments focus solely on summaries from the “relationship” and “relationship advice” subreddits for training and then evaluate performance on in-distribution and out-of-distribution Reddit posts.\n- This paper provides inadequate detail when introducing the advantages and disadvantages of other alignment methods (such as RLHF and direct preference optimization). In particular, it lacks sufficient background information and details on how these methods are applied to text generation or summarization tasks , which may make it difficult for readers to understand the relative advantages of the methods proposed in this paper.\n﻿\nOthers\n-  In the paper, the authors refer to the input of the LLM as a ‘string’. Switching to the term ‘token’ would align better with most readers’ pref erence and expectations, especially there is no cross-tokenizer-aspect that requires natural language level operations in vBo𝑁.\n- For those practitioners who are more focused on practical implementa tion, including a pseudocode for the vBo𝑁 alignment would enhance the readability of the paper. Moreover, vBo𝑁 does include more complex op erations beside the usual training pattern. I’d like to suggest authors to include one in the paper, even in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XTjYQMhhH4", "reviewer": "Reviewer_N6hp", "summary": "This paper proposes a more direct preference-align method vBoN for LLMs based on the Best-of-N approach. The authors construct a new fine-tuning objective function using the scoring results from a reward model. By directly aligning the output distribution of the LLMs to the output distribution of the Best-of-N method, the authors aim to simplify the BoN inference process and achieve better alignment efficiency compared to other preference alignment methods. Experiment results are demonstrated to try to validate the claims.", "questions": "1.Refer to the weakness 1 and 3, would you please further explain the efficiency of the method?\n2.Refer to the weakness 2, would you please clarify the definition of your objective?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The motivation of this article is quite clear, and it includes several experiments to support its claims. Additionally, the overall structure of the article is fairly complete, including some theoretical derivations.", "weaknesses": "1. The effectiveness of the vBon method is questionable. Although vBon utilizes reward model scoring to depict a target distribution closer to BoN, it requires a large number of samples (controlled by N or M in the article) to generate the corresponding preference data. The efficiency of this method is not high when the sample size is large.\n2. The paper is hard to follow. Some definitions lack explanation and need clarification from the authors, such as the function F(.) used in Equations 4 and 5. \n3. There is a lack of more convincing experiments to show the significant enhancement of alignment efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8jEmSa0MtK", "reviewer": "Reviewer_Lgyc", "summary": "This paper proposes to \"distill\" the gains from the best-of-N (BoN) inference-time alignment algorithm during finetuning, thereby improving a LLM's alignment without incurring the (linear in N) inference-time costs of BoN. They do this by deriving the distribution induced by the BoN algorithm, then defining the finetuning objective (which they call vBoN) to minimize the model's reverse KL divergence to this distribution. On the theoretical front, they derive some lower bounds for this vBoN objective and show that these lower bounds resemble the standard KL-constrained RL objective. Through experiments on controlled sentiment generation and summarization, they empirically show that PPO using the vBoN objective is the most effective technique for alignment via finetuning (though inference-time BoN still outperforms all finetuning-time alignment methods).", "questions": "1. The sentence right below equation (6) (the vBoN objective) states that \"This is an entropy-regularized objective, where we...*discourage* the model from having low entropy\". And this can be seen clearly in the objective (to be maximized) as well. However, in the text below equation (9), which is a lower bound for the vBoN objective, it is mentioned that L1(\\theta) further *encourages* the model to have low entropy. Can you provide an interpretation of why this L1 lower bound encourages low entropy, while the original vBoN objective encourages high entropy?\n\n2. Legend labels, as well as axis labels and numbers, are missing from Figure 4, which is one of the more important figures in the paper. Legend labels are also missing from Figures 5 and 6.\n\n3. The abstract mentions that finetuning with the vBoN objective is \"analogous to mean-field variational inference\", but this parallel is not discussed in the main text of the paper. \n\n4. In Section 5, the BoN-SFT baseline is considered, but its performance is not competitive since it has high KL divergence from the reference model. Was a KL-constrained version of BoN-SFT considered?\n\n5. Like vBoN, both the the BoNBoN and BOND (Sessa et al, 2024) methods attempt to convert BoN to a finetuning-time alignment method. Why was BoNBoN, but not BOND, reported as a baseline in Section 5?\n\nNit: There is a typo at the end of Section 5, where the references to Fig. 6a and Fig. 6b should instead refer to Fig. 2a and Fig 2b.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clear motivation: vBoN is naturally motivated from the goal of reaping the gains of BoN without incurring the inference-time computational overhead.\n\n2. Strong theoretical grounding, and compelling comparison of vBoN lower bounds with the standard KL-regularized RL objective.\n\n3. Sound experimental setup: The win rate/average rewards vs KL curves in Figure 2 show a clean comparison against several important baselines, and the breakdown of how often each method appears on the Pareto front was also a nice statistic to report. The ablation and analysis to understand sensitivity to number of samples for the logF() approximation in the vBoN objective (Figure 3 and Table 1) were also clean and well-presented.", "weaknesses": "1. It is stated in the paper that, under some basic simplifying assumptions, the optimal distribution under the KL-regularized RL objective and the vBoN objective are asymptotically equivalent (Section 1, equation (2), Section 4). The paper also shows that lower bounds for the vBoN objective closely resemble the KL-constrained RL objective, and that models finetuned to maximize these lower bounds perform very similarly to those finetuned to maximize the vBoN objective. On the other hand, the paper shows that models finetuned to maximize the vBoN objective substantially outperform models finetuned to optimize the KL-constrained RL objective. This empirical evidence in some sense challenges the theoretical observations. Why, then (and under what conditions), is vBoN better than KL-constrained RL?\n\n2. The paper demonstrates the effectiveness of vBoN on two tasks: Sentiment control (which is somewhat of a toy task) and Summarization (which is less of a toy task). However, several important baselines (e.g., DPO and BoNBoN) are only reported for the sentiment control task, and not for the summarization task. Why weren't these baselines included for the (less toy) summarization task?\n\nSee additional questions in the \"Questions\" section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)