Evaluation of Best-of-N Sampling Strategies for Language Model Alignment
Authors: Yuki Ichihara
Yuu Jinnai
Tetsuro Morimura
Kenshi Abe
Kaito Ariu
Mitsuki Sakamoto
Eiji Uchibe
Venue: Accepted by TMLR ()

Abstract: Best-of-N (BoN) sampling with a reward model has been shown to be an effective strategy for aligning Large Language Models (LLMs) with human preferences at the time of decoding. BoN sampling is susceptible to a problem known as reward hacking. Since the reward model is an imperfect proxy for the true objective, an excessive focus on optimizing its value can lead to a compromise of its performance on the true objective. Previous work proposes Regularized BoN sampling (RBoN), a BoN sampling with regularization to the objective, and shows that it outperforms BoN sampling so that it mitigates reward hacking and empirically (Jinnai et al., 2024). However, Jinnai et al. (2024) introduce RBoN based on a heuristic and they lack the analysis of why such regularization strategy improves the performance of BoN sampling. The aim of this study is to analyze the effect of BoN sampling on regularization strategies. Using the regularization strategies corresponds to robust optimization, which maximizes the worst case over a set of possible perturbations in the proxy reward. Although the theoretical guarantees are not directly applicable to RBoN, RBoN corresponds to a practical implementation. This paper proposes an extension of the RBoN framework, called Stochastic RBoN sampling (SRBoN), which is a theoretically guaranteed approach to worst-case RBoN in proxy reward. We then perform an empirical evaluation using the AlpacaFarm and Anthropic’s hh-rlhf datasets to evaluate which factors of the regularization strategies contribute to the improvement of the true proxy reward. In addition, we also propose another simple RBoN method, the Sentence Length Regularized BoN, which has a better performance in the experiment as compared to the previous methods.

Published in Transactions on Machine Learning Research (02/2025)
Evaluation of Best-of-N Sampling Strategies for Language Model Alignment
Yuki Ichihara
ichihara yuki.uul iud (it.. naist.jt
Nara Institute of Science and Technology
Yuu Jinnai
jinnai_ www.gnnbbbrrgee..ccom co..pp
Tetsuro Morimura
morimara wwt.ssrooreeeeereeee......
Kenshi Abe
abe kenshh@@rgberrggrdd.co
Kaito Ariu
koito ariu@cyberggentttcomm co.
Mitsuki Sakamoto
sakamoto mitsubifccybenggenttcom co.jp
AgberAgenl
Eiji Uchibe
uchibe -jp
Advanced Telecommuniatiioss Research Institute International
Reviewed on OpenReview: https: // openreview. net/ forum? id- H4S4ETc8c9
Abstract
Best-of-N (BoN) sampling with reward model has been shown to be an effective strategy for aligning Large Language Models (LLMs) with human preferences the time of decoding. problem known reward hacking. Since the reward model
BoN sampling is susceptible is imperfect proxy for the true objective, excessive focus optimizing its value can lead to compromise of its performance on the true objective. Previous work proposes Reg- ularized BoN sampling (RBoN), a BoN sampling with regularization to the objective, and shows that it outperforms BoN sampling so that it mitigates reward hacking and empirically (Jinnai et al., 2024). However, Jinnai et al. (2024) introduce RBoN based on a heuristic and they lack the analysis of why such regularization strategy improves the performance of regular-
BoN sampling. The aim of this study is to analyze the effect of BoN sampling ization strategies. Using the regularization strategies corresponds to robust optimization, which maximizes the worst case over set of possible perturbations in the proxy reward. Although the theoretical guarantees are not directly applicable to RBoN, RBoN corresponds to a practical implementation. This paper proposes an extension of the RBoN framework, called Stochastic RBoN sampling (SRBoN), which is a theoretically guaranteed approach to worst-case RBoN in proxy reward. We then perform an empirical evaluation using the AlpacaFarm and Anthropic's 's hh-rlhf datasets to evaluate which factors of the regularization strategies contribute to the improvement of the true proxy reward. In addition, also propose another simple RBoN method, the Sentence Length Regularized BoN, which has better performance in the experiment as compared to the previous methods.
1 Introduction
Large Language Models (LLMs) have demonstrated remarkable capabilities in many NLP tasks related to natural language understanding and text generation (Sticmnon et al., 2020; Ouyang et al., 2022; Touvron et al., 2023; Dubey et al., 2024; OpenAl et al., 2024). Despite the strengths, LLMs are not always adept at interpreting a wide range instructions and can produce undesirable outputs, such as biased, hallucinated, or toxic responses (Bai et al., 2022; Lin et al., 2022; Touvron et al., 2023; Casper al., 2023; Guan et al., 2024). This problem underscores the challenge of language model alignment; ensuring LLMs' behaviors align with human objectives and safety considerations (Ziegler et al., 2020; Stiennon et. al., 2020; Ouyang et al.,

---

Published in Transactions on Machine Learning Research (02/2025)
2022). There is now rich set of approaches to address this problem (Stiennon et al., 2020; Ouyang et al., 2022; Rafailov et al., 2023). These papers have shown that training language models with human feedback can improve their performance. However, training language models is a computationally intensive task. In other words, improved performance comes at the cost of increased computational resources.
This paper focuses on the Best-of-N (BoN) sampling strategy, a method that involves generating N outputs from a model and selecting the most preferred output among the N samples. Despite its simplicity and the fact that it does not require an additional training phase, BoN sampling has been shown to be surprisingly effective in practice (Stiennon et al., 2020; Nakano et al., 2022). However, the BoN strategy does not scale with the number of samples N due to the reward hacking problem (Amodei et al., 2016; Ziegler et al., 2020; Stiennon et al., 2020; Skalse et al., 2022; Gao et al., 2023). Reward hacking is a behavior that satisfies the given objective without achieving the intended result. It is caused by the misspecification of the preference model used by the BoN select the most preferred output (Pan al., 2022; Lambert & Calandra, 2024).
Previous work to mitigate the reward hacking problem has proposed Regularized Best-of-N sampling (RBoN), BoN strategy with the addition of regularization term to the objective (Jimnai et al., 2024). This paper shows that the RBoN strategy is effective compared to BoN sampling in various experiments. However, why such a regularization strategy is effective against reward uncertainty is unclear in the previous work.
In this paper, we propose Stochastic RBoN sampling (SRBoN), which adds a regularization term similar to RBoN. We then draw a connection between the Reinforcement Learning (RL) problems (Sutton & Barto, 2018) and the BoN strategies: BoN sampling corresponds to solving the RL problem, and SRBoN sampling strategies correspond to solving the Regularized Reinforcement Learning (RRL. problem (Neu et al., 2017: Geist et al., 2019; Yang et al., 2019; Derman et al., 2021).
First, we exploit the knowledge of RRL. Some work has shown that regularization terms in probability distributions over outputs provide robustness to reward perturbations (Ortega & Lee, 2014; Husain et al., 2021; Derman et al., 2021; Eysenbach & Levine, 2022; Pan et al., 2022; Dorman et al., 2021). SRBoN can also apply this analysis to RRL. Its insights provide an answer to why reward hacking can be mitigated: when a regularization term is added to the BoN sampling, it also becomes an adversarial perturbation the reward.
We then evaluate the effectiveness of our approach against alternative decoder methods in a series of ex- periments, with the goal of determining the relative resilience of each method to potential exploitation by reward hacking. The results show that our proposed method outperforms many existing approaches in a variety of settings. In other words, a theoretically guaranteed, effective algorithm is proposed.
In addition, while RBoN consists of complex formula structures, we proposed a simpler RBoN, Sentence Length Regularized BoN that, despite its simple implementation, shows comparable or even better perfor- mance in experiments with the methods of previous studies.
2 Background
In this paper, we formalize the problem of decoding time alignment as Regularized Markov Decision Processes (MDPs) problem (Neu et al., 2017; Gcist et al., 2019; Yang et al., 2019; Derman et al., 2021). For brevity, we refer to reinforcement learning within Regularized MDPs as Regularized Reinforcement Learning (RRL) throughout this paper. In Section 2.1, we describe the Reinforcement Learning (RL) problem and the RRL problem. Then, we describe two sampling algorithms used for decoding time aligument, Best-of-N (BoN) sampling in Section 2.2, and the Regularized Best-of-N sampling (RBoN) in Section 2.3.
2.1 Adversarial Interpretation in Regularized Reinforcement Learning
We consider the problem of selecting an output y from a set of outputs Vret y (e.g., response text from the system) given an input x X (c.g., input prompt by a user), where the objective is to select the best output according to a reward function R: X X' + R. Let A(Y) denote the set of probability distributions over set y We define the goal of the Reinforcement. Learning (RT) problem as finding the best policy
2

---

Published in Transactions on Machine Learning Research (02/2025)
T X' A(Drer) that maximizes the expected reward for
arg max (TL(T) = arg max
Σ mly x)) (()
*ell
sell
NEDeuu
(1)
arg max (x) [R(
where II is set of all possible policies. Note that there exists deterministic policy that maximizes fRL (Sutton & Barto, 2018) and this formulation can be seen a contextual bandit problem. Specifically, the policy observes context I, chooses an output based on that context, and receives a reward R(x, v). Importantly, we do not use sequential decision operations or consider state transitions. Each decision is made independently based on the current context x.
The underlying assumption of the RL problem is that the reward model R is correctly defined and observable. That is, we consider the solution that maximizes the expected reward to be the optimal solution. However, wall-wrld applications often suffer from the reward misspecification problem - the reward model observable to the agent is only proxy for the true underlying reward of the problem (Ortega & Lee, 2014; Husain et al., 2021; Derman et al., 2021; Eysenbach & Levine, 2022; Pan et al., 2022: Derman et al., 2021). Prior work has investigated strategies to optimize under the uncertainty in the observed reward. In contrast to the RL problem, Regularized Reinforcement Learning (RRL) incorporates regularization terms to achieve a solution that robust to the reward misspccification (Neu et al., 2017: Geist et al., 2019; Yang et al., 2019; Derman et al., 2021). The objective of the RRL problem is to find the best policy π X + A(Prer) that maximizes the reward with an additional regularization function Ω(x)) A(Vret) + IR U {+0o}. Let JRRI (T) ;= (-)> ( y)],,,)]=1 ((x) be the objective function of RRL problem with Ω. Then, define the following as the optimal solution to the RRL problem:
e arg max fiia (T)
arg max E E [R(z, y)] - [((T).
(2)
sell
#ell
Note that, unlike the RL problem, there may not exist an optimal policy that is deterministic for the RRL problem (Geist et al., 2019).
Brekelmans et al. (2022) uses Legendre Fenchel transformation (Touchette, 2005) to show that the RRL problem can be viewed as a variant of the RL problem with an adversarial agent adding perturbations to the reward AR: : X Vref + R if the regularization term Ω is convex and lower semi-continnous function (Boyd & Vandenberghe, 2004):
arg max /RR (T) arg max min [a>x(1a)][((. (R(,,)) v) - AR(r, v) (AR), Hell ARERA
(3)
HEll
where Ω E Xx is the conjugate function of Ω (Boyd & Vandenberghe, 2004), and RA = { (A E Xx)rr Fa(AR) & 0). where Fn is a function or operator dependent to Ω that imposcs a con- straint or condition on the values of AR.
Eq. (3) shows that the problem of maximizing fRRL can be reformulated as the max-min problem and the regularizer Ω is effectively an adversarial reward perturbation that forces us to optimize the worst case performance (Derman et al., 2021).
2.2 Best-of-N (BoN) Sampling
BoN sampling has emerged as effective method for preference optimization in LLMs (Stiennon et al., 2020; Nakano et al., 2022). BoN sampling has several advantages over preference learning methods. First, it is straightforward and does not require additional training in the language model. Although learning-basce alignment methods require rel retraining the LLMs whenever human preferences are updated, BoN sampling can be applied immediately, requiring only an update the reward model. This is particularly advantageous since training L.MSs is the most resourco-intensive process. Second, BoN sampling an effective strategy in its own right, with numerous studies demonstrating that can outperform learning- based adaptation methods (Gao et al., 2023). Recent literature has expanded our understanding of BoN sampling. In particular, Beirami
3

---

Published in Transactions on Machine Learning Research (02/2025)
al. (2024) conducted analysis comparing the policies selected by BoN sampling with the base policies used for sample generation. In addition, Gui et al. (2024) showed that BoN sampling achieves an optimal balance between win rate and KL divergence when aligning large language models to human preferences.
BoN sampling has similarities to the objective function used in RL (e.g.: the response with the highest reward score, determined by a proxy reward model R(x, y), is selected). The objective function BoN is given by:
(x) arg max R(x, VEVret
This reward model is used as mcasure of the quality of the text when making an assignment. The reward model we consider in this paper is open access as described in Appendix I. We also mention that the objective
function of BoN sampling is equal to the objective function of the (unregularized) RL problem (Eq. (1)):
YBoN (x) : arg max RI y)
arg max max Eyony [R(2, v)l
(4)
vE TyEllde
arg max max (RL( yEVeet 8y
where Ilatt is set of deterministic policies and ny is deterministic policy that selects y given with probability of 1.
2.3 Regularized Best-of-N Sampling (RBoN)
Although BoN sampling is shown to be effective, it is prone to the reward hacking problem (Amodei et al., 2016; Ziegler et al., 2020; Stiemnon et al., 2020; Skalse et al., 2022; Gao et al., 2023). The reward hacking problem is phenomenon where the decision to optimize the proxy reward is made without considering its potential misspecification, resulting in worse performance on the actual reward objective. Dubois et al. (2023) showed that with 25% label noise, which the amount of disagreement observed in real-world preference annotations (Stiennon et al., 2020; Ouyang al., 2022), BoN sampling degrades performance with N greater than 16 (Figures 12 and 13 in Dubois et al. 2023).
Regularized Best-of-N sampling (RBoN) is proposed to mitigate the reward hacking problem for BoN sam- pling (Jinnai et al., 2024). Jinnai et al. (2024) presented two variants of RBoN: using the KL divergence as a regularizer (RBoNKLL: Section 2.3.1) and using the Wasserstein Distance as a regularizer (RBoNwd: Sec- tion 2.3.2). In the following, we describe the two variants RBoN and draw its connection to the objective lunction of RRL (Eq. (2)).
2.3.1 KL divergence Regularized BoN Sampling (RBoNRL.)
The objective function of RBoNKL (KL divergence Regularized BoN Sampling) is given by:
yKLBBN arg max R(x, u) -BDLL BEyeer
| [T]]r [x]],
arg max max KL BE F, Ellaet
(Ty).
where 3 is regularization parameter, reference policy Rreff X A(Uret), and DKL denotes the KL divergence. The reference policy here takes an input and returns meaningful output y. The regularization described in this paper is like a penalty to stay away from the reference policy and the reference policy is the language model.
By incorporating the KL divergence as a regularization term in the objective function, RBoNKL encourages the learned policy be close to the reference policy Tref- A higher value 3 emphasizes the regularization term, encouraging the learned policy to be closer to the reference policy, while a lower value of / prioritizes maximizing the reward function.
4

---

Published in Transactions on Machine Learning Research (02/2025)
2.3.2 Wasserstein Distance Regularized BoN Sampling (RBoNww)
The objective function of RBoNwD (Wasserstein Distance Regularized BoN Sampling) defined as follows:
VWDBON (x)
arg max R(z,y) 3WD (m | [)]l ( (11=)),
NEV/et
arg max max WD Mearet TyEllde
(Ty).
where WD denotes 1-Wasserstein Distance.
The WD (Wang, 2012) is defined as:
WD|pllu] inf Σ Yi Cijy
(5)
TET(Mm) (I.)ENXN
the total number of samples, consisting of the set {yr U2....... UNJ. U E A(N): probability mcasure on the aforementioned sets (Vi, 14 refer to the probability value v(y)), C: N R cost function measuring the distance between two outputs (e.g. Cij refers to the amount to be transported from place I to palace Yj), and I' ['v, μ) denotes the set of all joint distributions y whose marginals are and μ. The constraints on y are given by:
where N:
Ens Th 14,
Vie
JEN
Σ EEN
Tij Ujj
EjEn,
Mi 2 0, En E
The WD, also known as the Earth Mover's Distance (EMD), is a metric used to quantify the dissimilarity between two probability distributions. Intuitively, it measures the minimum cost required to transform one distribution into the other. This cost conceptualized as the amount probability mass that must moved multiplied by the distance that would be moved. The concept has been used in NLP mcasure the dissmmilarity of texts (Kusner et al., 2015; Zhao et al., 2019).
The exact computation of WD [- x) Tret (. (+]=)] is intractable due to the enormous size of the output space. To address this computational challenge, prior work (Jinnai et al., 2024) has employed samplo-basod approximation techniques. Let free represent the empirical distribution computed using a set samples (yy=
Pref- This distribution is defined as: Arer(y .) = n Ev'erwr I = y') where N is the total number of
samples in Vref. The objective function can then be approximated as follows:
yWDBON (x) arg max R(x,,) 3WD (. x) [) (:1=)). yErier
For practical implementation aspects, the WD term for Jinnai et al. (2024) is computed as follows:
WD [T [)]l xx]] = Σ N c(.
(6)
3
The cosine distance is used as C' to measure the distance between the outputs (Reimers & Gurevych, 2019).
C (v.v') - cos (emb(y), emb eembb (y/)),
where emb(y) and emb (y') represent the embeddings of output y and y', respectively.
3 Stochastic RBoN (SRBoN)
We propose the stochastic version of RBoN, Stochastic RBoNKL (Section 3.1) and Stochastic RBoNwD (Section 3.2). These novel algorithms, while similar to the original RBoN (deterministic version), allow for the optimal policy π to a probabilistic output distribution. By relaxing the deterministic constraint, we can apply thcoretical tools that were previously inaccessible. Our approach focuses on the analysis of this stochastic version, aiming provide theoretical results that shed light on the underlying mechanisms of RBoN's effectiveness.
5

---

Published in Transactions on Machine Learning Research (02/2025)
3.1
Stochastic RBoNKL (SRBoNkL)
First, consider a stochastic version of RBONKL. The policy of SRBoNkI. is given by:
TSRBoNKL (I) = arg max E ( (R(,, y)] 3Dxl [=( x) x)]
sell
(8)
KLL (T).
arg max
*ell
We define SRBoNKL as a method to sample a response y that follows the probability distribution of TTRBBNKI.
USRBoNKL (ar) TSRBoNKL (ar).
(9)
In Section 4 we evaluate the performance of this stochastic text generation algorithm defined by Eq. (9).
3.1.1 Theoretical Guarantee of SRBoNkt.
By relaxing the deterministic policy constraint of RBBNKL, SRBoNkL. follows the formulation of the RRL with adversarial perturbations studied by Brekelmans et al. (2022). As such, the computation of SRBoNKL can be transformed into max-min problem using Legendre-Fenched transformation (Touchette, 2005) as in Eq. (3). In this way, SRBoNKL has the following theoretical guarantee proven by Brekelmans et al. (2022): Theorem 3.1. (Brekelmans et al. (2022), Proposition 1) The problem of mazimizing AKI fRRI (T) can be
interpreted as a robust optimization problem with an adversarial perturbation AR:
arg max JRRL (T) arg max Well
min
Ey v) AR(x,v),,
(10)
Tlll
A ARERA
where the feosible set of reward perturbations Rs available to the adversary is bounded:
|AR E 18 xx)oor
Tror(y ARl(E, v) }
RA
Σ Vrar
(11)
The theorem shows that SRBoNKL is an algorithm that optimizes the worst-case performance under the assumption that the error between the true reward and the given proxy reward model is guaranteed to be within Rs (Eq. (11)).
Let R' be set of possible reward models under the reward perturbations: R' = -R-(R----- AR Ra).
Let JKL,(T R) be the objective of the policy given (proxy) reward model R. Then,
arg max fRRL fell
(T; R) = arg max
min HEll ARERA
(R( v) AR(x,))
arg max
min L [x)n [1'(2,))) v)]
well
R'RR
arg max min
JRRL (T; R').
(12)
*EN FER
Thus, SRBoNKI. is robust optimization of the policy for set of possible reward models in R'. In other words, it assumes that the true payoff model is in RA and optimizes for the worst case.
The theorem derived by translating the proposition proved by Brekelmans et al. (2022) for the generic RRL problems to the text generation scenario. The contribution of our work to show the relation of their theoretical result to the RBoN sampling algorithm in LLMs alignment.
3.2 Stochastic RBoNwD (SRBoNwD)
We now consider an optimization problem over space of probability functions, to derive an optimal proba- bilistic policy x with the Wasserstein distance as the regularization term. The objective function of RBoNgwD is the following:

---

Published in Transactions on Machine Learning Research (02/2025)
TSRRONWD (x) = arg max E By [(x, - BWD [Trat() ())l [x)]
=ell
(13)
arg max (RRL (IT). sell
WD
3.2.1 Theoretical Guarantee of SRBoNwD
Similar to SRBoNKL: SRBoNwD also be reformulated max-min problem, and thus we can show that it optimizes the worst-case performance under certain constrain:
Theorem 3.2. The problem of marimizing WWD (T) can be interpreted robust oplimization problem with adversarial perturbation AR:
arg FEII
WWD (T) arg max HURE
min E (1)) - 3AR(2,0)+ 3
E
Trerly x))R(z,
(14)
well
ARERA
DEVIN
where the feasible set of reward perturbations Rs available to the adversary is bounded:
RA = E X ||RR(z,)) - AR v11 S (v.3)) Vy.y' E Neet).
(15)
The proof is provided in Appendix A.
This expression represents an optimization problem with strategies and perturbation AR. The goal is to find the optimal strategy x² under the modified reward I'' (= R BAR..
The intuition behind the second term Eyeyret Trer(y x)) x)) v) can be understood by examining AR constraints (Eq. (15)). While this feasible set does not explicitly constrain AR to avoid large values, the second term, MinAR [R(,, y) - BAR(2, w)], helps to avoid such huge values. Additionally, it reveals mechanism that inherently limits the magnitude of perturbations for actions that have high probability under Teef and this is consistent with the WD distance intuition.
We have analyzed the role of the regularization term for BoN sampling in the previous section 3.1.1 and section 3.2.1. Since the previous study (Jinnai et al., 2024) imposed deterministic constraints, the results are not exactly the same, but we consider that the analysis performed here helps to explain why the previous study performed better.
Note The fcasible set reward perturbations R& is bounded to be a Lipschitz continuous function with respect to a cost function C, which generally takes a non-negative value in applications. The perturbation behavior corresponds to the Lipschitz continuity condition, which has traditionally been well-treated in the RL community. For example, previous studies such as Rachelson & Lagoudakis (2010); Pirotta et al. (2015) considered continuous state and action spaces in RL and derived Lipschitz continuity for reward functions to aid their analysis.
4 Experimental Evaluation
We evaluate the performance of SRBoN compared to other text generation approaches. The datasets and models used in the experiments are all publicly available (Appendix I).
Datasets. We conduct experiments using two datasets: the AlpacaFarm dataset (Dubois et al., 2023) and Anthropic's hh-rlhf (HH) datasct, which we use the Harmlessness and Helpfulness subsots (Bai et al., 2022). For the AlpacaFarm datasel, we use the first. 1000 entries of the train split (alpaca human preference) as the development set and the 805 entries the evaluation split (alpaca farm evaluation) for evaluation. For Anthropic's is datasots, we separately conduct experiments on the helpful-base (Helpfulness) and harmless-base (Harmlessness). For each dataset, we use the first 1000 entries of the train split as the development set and the first 1000 entries of the evaluation split for evaluation.

---

Published in Transactions on Machine Learning Research (02/2025)
Table 1: Description of the text generation algorithms evaluated in the experiments. A checkmark (V) indicates that the method uses the specified function, while blank space mcans that it does not.
Reward Function
Similarity Function
Method
Description
Use an output that is randomly sampled from the reference model.
Random sampling
Best-of-N (BoN) (Stiennon et al., 2020)
Generate N outputs, evaluate with reward function, select the best.
Generate outputs, evaluate with expected utility function, select the best. (Details in section 4)
MBR (Fikema & Aziz, 2022)
Maximize the mixture of the reward function and KT. divergence with a constraint that the resulting policy is deterministic.
RBoNKL (Jinnai et al., 2024)
Maximize the mixture of the reward function and WD distance with a constraint that the resulting policy is deterministic.
RBoNwD (Jinnai et al., 2024)
V
SRBoNki.. (Section 3.1)
Maximize the mixture of the reward function and KL divergence. Maximize the mixture of the reward function and WD distance.
SRBoN WD (Section 3.2)
Consider both the reward function and the to- ken length of the sentence. (Details in sec- tion 4 and Appendix G)
RBoNL
(Section 4)
Language Model, Reward Model, and Embedding Model. We employ Mistral 7B SFT 3 (Jiang et al., 2023a) as the language models. We set the maximum entry length and the maximum output length to be 256 tokens. We sample response texts using nucleus sampling (Holtzman et al., 2020) with temperature set to 1.0 and top-p set to 0.9. For each entry, in the AlpacaFarm dataset and Anthropic's datasets, 128 responses are generated using Mistral 7B SFT B.
To evaluate the performance of the algorithms under different preferences, we use OASST (reward-model deberrta-v-large--2)) SHP-Large (SteanSHIP-fan--5-llae Mistral-7B and Eurus-RM-7b RM-7b (Köpf et al., 2023; Ethayarajh et al., 2022; Jiang et al., 2023b; Dong et al., 2023; Yuan et al., 2024) as reward models. For the text embedding model we use all-mpnet-base-v2 (Song et al., 2020), a sentence transformer model (Reimers & Gurevych, 2019) shown to be effective in various sentence embedding and semantic search tasks.
SHP-XL (SteamSHIP-llnn465-x)) PairRM, RM-
Baselines. The list of text generation methods we evaluate is present in Table 1. The baseline methods include random sampling (nucleus sampling; Holtzman et al. 2020), Best-of-N (BoN) sampling, Minimum Bayes Risk (MBR) decoding, and RBoN,, which we describe in the following.
Minimum Bayes Risk (MBR) decoding (Kumar & Byrne, 2002; 2004; Eikema & Aziz, 2022) text generation strategy that selects an output from N outputs that maximizes the expected utility (Berger, 1985). Let utility function u(h, y) quantify the benefit of choosing h Drer if y is the correct output. Then, MBR. decoding is defined as follows:
yMBR (x) arg max
Σ N (h,
(16)
hEY
We include MBR decoding as one of the basclines because it has been shown be effective in a variety of text generation tasks (Suzgun et al., 2023; Bertsch et al., 2023; Li et al., 2024; Heineman et al., 2024). We
8

---

Published in Transactions on Machine Learning Research (02/2025)
follow the implementation of Jinnai et al. (2024) and use the cosine similarity of the sentence embedding as the utility function. We use the same embedding model as the RBoNwD, all-mpnet-base-v2. Note that MBR corresponds to RBoNwD with u(h,z) y) = 1 - C(h, y) with no reward function or B + to0 (Eqq. (6)) (Jinnai et al., 2024).
As an additional evaluation method, we propose Sentence Length Regularized BoN (RBoNI.), a simple baseline that adjusts the output token length the target reward model. In RBoNKL and SRBoNKL, Fref used for regularization. However, we have observed bias in language models with respect to sentence length, namely that these models tend to produce shorter sentences with higher probability (Appendix B). To this end, we propose a simple implementation of RBoN that regularizes the generation probability of the sequence token length instead of the generation probability of each sequence. The objective function of RBoNL is given by:
B
yRBoN, (I)) = arg max R(x, v) yENbrr
(17)
lyl*
where 3 is regularization parameter and ly| denotes the sequence length (i.c., the number of tokens).
The rationale for this particular form of the regularization term and the experimental details of this approach are described in Appendix G.
4.1 Evaluation of the Algorithms
Setup. We compare the 7 methods using win rates BoN sampling the evaluation splits of the datasets. Since the RBoN method has a hyperparameter 3, we first find the optimal 3 on the train splits. Hyperparameter 3 range is {1.0 x 10-1, 2.0 x 10-1, 5.0 x 10-1,1.0 x 10-3 ..... 2.0 x 101}. We first find the optimal beta value 3* in the train split, then we use the optimal values the development split for the evaluation split. In this experiment, we use OASST, SHP-Large, SHP-XL, PairRM, and RM-Mistral-7B as proxy reward models. As as the gold reward model, we use Eurus-RM- RM-7B evaluate the performance of the algorithms. We evaluate the performance of the algorithms as the win rate against BoN sampling according to the reward score of the gold reward model (we count ties as 0.5 wins). We use RM-u as the gold reward model because it is reproducible as it is open source and has been shown to have a high correlation with human preference in RewardBench (Lambert et al., 2024b).
Results. Table 2 reveals several noteworthy results for the AlpacaFarm, Harmlessness, and Helpfulness datasots and the optimal beta f is Table 4. The win rate result shows that higher Spearman rank correlation values (Table 3) correspond to better BoN sampling accuracy. This observation is intuitive.
Table 2 shows that the win rate of SRBoNKL is inferior to the deterministic version RBoNKL. While SRBoNkI. is proposed as theoretically robust algorithm (section 3.2.1), its performance in our experiments did not fully meet expectations. One possible factor contributing to this discrepancy could be related to the perturbation range of AR. In our experimental setup, it is plausible that the actual perturbations of AR may have exceeded the assumed theoretical limits.
Other reasons for suboptimal performance, applicable to both deterministic and stochastic versions, concern the relationship between the reference policy reef and the reward model. If the correlation between Tref and the reward model is weak, the regularization effect may not contribute positively the performance of the algorithm (Appendix B).
SRBoNwd shows superior performance across several settings and achieves comparable performance to RBoNwD- This robust performance is remarkable given the low positive correlation between the reference policy Fref and the reward model.
A plausible explanation for this effectiveness, especially contrast to SRBoNKL, the constraint on the reward perturbation AR in SRBoNwD. Unlike SRBoNKL, the constraint on AR in SRBoNwp is independent of Tref; which mitigates low performance when there is no correlation between the reward model and Tref-
9

---

Published in Transactions on Machine Learning Research (02/2025)
Table 2: The win rate of various methods against BoN sampling.
Method
OASST
SHP-Large
SHP-XL
PairRM
R.M-Mistral-7B
A AlpacaFarm
BoN
50.0
50.0
50.0
50.0
50.0
MBR
36.0
40.8
39.1
13.0
Random
20.5
30.3
29.4
27.1
3.0
wD
50.6
50.2
49.0
50.7
49.9
RBoNKL
47.7
26.4
26.2
50.0
48.6
RBoN1.
52.0
50.3
50.2
50.1
50.8
SRBoNwD
50.1
50.6
49.5
50.0
50.1
SRBoNKL
12.6
20.9
18.7
28.0
4.7
OASST
SHP-Large SHP-XL
PairRM
RM-Mistral-7B
Harmlessness
BoN
50.0
50.0
50.0
50.0
50.0
MBR
40.8
57.4
50.7
42.7
14.8
Random
26.7
52.7
46.3
28.0
7.1
RBoNwD
52.1
62.2
57.1
50.0
49.9
RBoNKL
48.2
46.9
40.4
50.0
47.4
RBoN1.
52.2
54.8
54.2
50.0
51.6
SRBoNwd
49.7
49.8
50.0
49.9
SRBoNKL
20.5
12.3
37.1
30.4
5.5
OASST
SHP-Large SHP-XL
PairRM
R.M-Mistral-IB
Helpfulness
BoN
50.0
50.0
50.0
50.0
50.0
MBR
41.4
39.2
33.2
40.0
6.1
Random
23.6
23.7
15.1
23.3
0.8
RBoNwD
52.5
52.4
50.1
50.1
49.9
RBoNKI.
44.9
19.9
13.9
50.0
50.0
RBoN1.
52.7
49.9
50.8
50.0
50.2
SRBoNwd
50.4
49.5
19.6
50.0
50.0
SRBoNKL
13.4
18.5
11.8
24.3
1.4
Table 3: Spearman's rank correlation between Eurus-RM-7B and each proxy reward. The comprehensive Spearman's rank correlation results for all the aforementioned analyses are presented in Appendix D.
Datasel
OASST
SHP-Large
SHP-XL
PairRM
RM-Mistral--B
Aara
0.39
0.29
0.35
0.33
0.62
Harmlessness
0.37
0.09
0.14
0.36
0.60
Helpfulness
0.39
0.38
0.50
0.34
0.75
Despite its simple implementation, RBoN,, consistently outperformed BoN sampling, achieving higher win rate almost all tasks and models with no instances of underperformance. A detailed discussion of RBoN, is presented in Appendix G.
4.2 RBoN Sensitiveness of Parameters
Setup. In this section, we evaluate the generalization performance the model using 3. values {1.0x 10 2.0 10-4, 5.0 x 10-4, 1,0 10-3 ...., ...., 2.0 x 10') the evaluation splits. We also use several models as proxy reward models, including OASST, SHP-Large, SHP-XL, PairRM, and RM-Mistral-7B As a gold reward model, Furus-RM-7B evaluate the performance of the proxy models. The results visualized
10

---

Published in Transactions on Machine Learning Research (02/2025)
Table 4: Optimal beta 8 in the train split
Method
OASST
SHP-Large
SHP-XL
PairRM
R.M-Mistral-BB
AlpacaFarm
RBoNwD
20
0.5
0.5
20
0.1
RBoNKL
0.0001
0.0001
0.0001
0.0001
0.0001
RBoN1,
20
0.5
0.2
20
15.0
wRBoNwD
0.5
0.0002
0.0001
0.0001
1.0
SRBoNKL
20
0.05
0.05
20
20
Harmlessness
wD
20
1.0
1.0
0.0001
5.0
RBoNKL.
0.0001
0.0001
0.0001
0.0001
0.0001
RBoN,
20
5.0
0.0001
20
SRBoNww
0.05
0.0001
0.0001
0.0001
0.02
SRBoNKL
20
0.05
20
20
20
Helpfulness
RBoNwD
15.0
0.05
0.1
20
0.5
RBoNKI.
0.0001
0.0001
0.0001
0.0001
0.0001
RBoN1.
20
0.02
0.2
5.0
20
SRBoNwD
0.5
0.001
0.005
5.0
0.0002
SRBoNKL
20
0..5
20
20
20
plot showing the win rates of each method compared to BoN sampling on the evaluation splits. We assign point for win and 0.5 points for tic.
Results The performance result of RBoN method in AlpacaParm is illustrated in Figures This result reveals that the optimal parameters for the RBoNwD and SRBoNwD method vary between different models and reveals the performance of SRBoNwd across various problem settings, as the value of the regularization parameter & increases, we observe a degradation performance. Intuitively, upon examining the adversarial formulation of SRBoNwd. we can infer that as the regularization parameter 3 increases, the magnit ude of potential perturbations AR also increases. Furthermore, as evidenced in Table 4, the optimal value for SRBoNwd is typically smaller than that for RBoNwD-
This result shows that SRBoNK1. consistently underperforms within the 3 range examined in our experiments. In particular, as shown in Table 4, the optimal regularization parameter 8 for SRBoNKL is often found to be 8 = 20 across different problem settings. This obscrvation leads to an intriguing hypothesis, that the performance of SRBoNKL could potentially improve with higher values of B.
The performance result of RBoN, demonstrates superior performance across a wide range of 3 values, exhibiting performance characteristics comparable to RBoNWD. Notably, this robust performance across varying values indicates that RBoN, exhibits low sensitivity changes in the regularization parameter.
The results for Harmlessness and Helpfulness datascts presented in Appendix C.
5 Related Work
Robust MDPs Several studies have investigated RL considering the worst-case scenario for rewards. Ortega & Lee (2014) considers only a single-step analysis for the reward robust problem. Husain et al. (2021) proposes a deep RL algorithu related to Q learning for the reward robust problem. Derman el al. (2021) considers both a reward function and the transition probability as unknown. The policy regularization is considered a perturbation of the rewards, while the transition probability perturbations address the worst- case scenario with respect the associated set of value functions. They define specific uncertainty sets and conduct thorough experiments. Eysenbach & Levine (2022) shows that incorporating the policy's Shannon
11

---

Published in Transactions on Machine Learning Research (02/2025)
SHi-LLage
SHP-XL
Win Rate (%)
Win Rate (%)
Oparat indehiel: 2:30
Optirel belu mndeheel
Obtan thea 198 rotuRGat 1a
OptraiBen (.tehal c.e0
Optirall Ben (Ktohaa) c.00er
Optiral Ben (SBs poo1
Optrattees (Heolle) 20.0
Optirec bes: (SRoohe)) oue
Outer thea csi derMd hic
OperaiDem SRaoNao! 28.0
Oparal (Sibelivo
Bra (SRib 1.B
15
13
PairRM
RM: RM-Mistral-79
45
Vitn Rate (%)
thioohe
SREehup
Optinall Bata R: 28.0
Optiral bo 8:1
Viln
Operal Eate Sibbrlgg) 20.0
Opanal Delu Stiblivee Bdf
Optrall Bea Kiek. 223
10
Optnoll bet (Kiek))) 'sa
16
12"
10"
10"
10" Beta
16
Beta
Figure Exaluation of RBoN sensitiveness on the AlpacaFarm dataset with varying parameter B. We use proxy reward models, OASST, SHP-Large, SHP-XL, PairRM, and RM-Mistral-7BB As the gold reward model, we utilize Burus-RM-77
entropy into the reinforcement learning objective function represents the worst-case scenario for a given uncertainty set of rewards.
Alignment Strategies Two notable adaptation strategies have recently gained attention: Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) (Stiennon et al., 2020; Rafailov el al., 2023). RLHF incorporates human feedback into the reinforcement learning process to align the agent's 's behavior with human preferences. By using human feedback reward signal, RLHF aims to optimize it. This approach has been successfully used in LLM (Ouyang et al., 2022). On the other hand, DPO uses the same objective function as RLHF without an explicit reward function. However, it still suffers more than RLHF from overoptimization when dealing with out-of-distribution data (Xu et al., 2024). Beyond the methods discussod, there rescarch in robust optimization that addresscs the development algorithms for scenarios with unstable preference information (Wu al., 2024). In partieular, Chowdhury et al. (2024) have introduced robust DPO approach that achieves robustness without explicitly employing robust optimization techniques. Another technique (Mudgal et al., 2024) is to train token-level scoring value function module to select the optimal output. Khanov et al. (2024) is a novel decoding method that does not require additional learning and uses both the language model and the reward model knowledge. There a parameter that determines which is more important, and depending on its value, it can a conventional method.
robust
6 Conclusions
This paper introduces three novel BoN sampling methods: SRBoNKL, SRBoNww, and RBoN To rigorously evaluate the effectiveness of these proposed hods, we conducted extensive experiments using two datasets: AlpacaFarm and Anthropic's hh-rlhf.
The SRBoNkL and SRBoNwd methods extend the previous RBoNKL and RBoNwD methods, respectively. In particular, SRBoNgL, and SRBoNwd produce stochastic optimal policy that differs from their deter-
12

---

Published in Transactions on Machine Learning Research (02/2025)
ministic counterparts. The thcoretical guarantees of their robustness increase the reliability of the methods
in different scenarios.
The RBBNN, method is a contribution to the field of RBoN sampling, providing a simple yet effective approach. Despite its simplicity, our experiments show that RBoN, achieves performance comparable to the more complex RBoNwD- This finding highlights the potential of RBoNr, as a computationally efficient alternative to more complicated methods, making it particularly attractive for applications with limited resources or stringent performance requirements.
In conclusion, this paper presents three innovative BoN sampling methods that significantly contribute to the field. The experimental results and theoretical guarantees underscore the effectiveness and reliability of these methods. Our work lays the foundation for further research and applications of robust BoN sampling techniques in a wide range of domains.
7 Limitations
While our proposed method demonstrates promising results, there are several limitations to note. The proposed method requires no fine-tuning of the LLMs but incvitably increases computational overhead at inference. In contrast, fine-tuning approaches incur a one-time cost during training while eliminating overhead at. inference. Another concern is that the proposed method considers a max-min problem, so if, for example, the correlation between the proxy reward and the gold reward is strong, performance is reduced duc conservative output selection.
Our study lacks analysis of whether the reward perturbations satisfy the conditions outlined in Theo- rems 3.1 and 3.2. Evalusting the error of the reward and utility function in experiments remains area for future work. Additionally, the selection of the parameter 8 requires a validation set in the current setting, and developing an automated method to determine 3 is promising direction for further research.
Furthermore, our approach relies on a specific utility function, which is a prerequisite for applying the pro- posed method, and the method does not account for process reward models, which may limit its applicability in some scenarios. It also worth noting that the experiments conducted in this study were limited three English datasets, leaving open the question of its generalizability to other languages domains.
In addition, the proposed method is based on probabilistic framework, which, while effective for uncertainty, may not align with real-world applications where deterministic versions (RBoN) are often preferred for their prodictability and safety. Based on the analysis in this paper, the analysis of the deterministic RBoN is a possible direction for future work.
Finally, while the current formulation is specific, the proposed method has the potential be extended other divergence measures, such f-divergences, offering an exciting avenue for future investigation.
Acknowledgments
We sincerely thank the Action Editor, Pascal Poupart, and the anonymous reviewers for their insightful comments and suggestions. Kaito Ariu's research is supported by JSPS KAKENHI Grant No. 23K19986.
References
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete Problems in AI Safety. arXiv preprint orXiv:1606. 06565, 2016.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2rXi::2204005863 2022.
Ahmad Beirami, Alckh Agarwal, Jonathan Bcrant, Alexander D'Amour, Jacob Eisenstein, Chirag Nagpal, and Ananda. Theertha Suresh. Thcoretical guarantees on the best-of-n alignment policy. arXiv preprint 011:02001.01879, 2024.
13

---

Published in Transactions on Machine Learning Research (02/2025)
James O Berger. Statistical decision theory and Bayesion analysis; 2nd ed. Springer series in statis- tics. Springer, New York, 1985. doi: 10.1007/976-1-47777-42888 URL https://cds cern. chttrccrr///////// 1327974.
Amanda Bertsch, Alex Xie, Graham Neubig, and Matthew Gormley. It's MBR all the way down: Modern generation techniques through the lens of minimum Bayes risk. In Yanai Elazar, Allyson Ettinger, Nora Kassner, Sebastian Ruder, and Noah A. Smith (eds.), Proceedings of the Big Piclure Workshop, pp. 108 122, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.
bigpicture-1.9, 1.9. CRL attps://aclanntholee org/2023. bigpicture-
Stephen Boyd and Lieven Vandenberghe. Conver optimization. Cambridge university press, 2004.
Rob Brekelmans, Tim Genewein, Jordi Gran-Moya, Gregoire Detetang, Markus Kunesch, Shane Legg, and Pedro A Ortega. Your Policy Regularizer Secretly an Adversary. Transactions on Machine Learn- ing Research, 2022. ISSN 2835-8856. Certification.
URL
httpa//owr....eoo/ooooooooooeerr Expert
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Tong Wang, Samuel Marks, Charbel-Raphacl Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mchul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Biyik, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan HadficldMMenel.. Open Problems and Fundamental Limitations of Reinforce- ment Learning from Human Feedback. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL
Survey Certification.
Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably Robust DPO: Aligning Language Models with Noisy Feedback. In Forty-first International Conference on Machine Learning, 2024. URL
https://openreviiet wwtprrewwwi.eeeouunn/eee wtwForrmmnidyypDsss77y.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe reinforcement learning from human feedback. In The Twelfth International Conference net nett//oruunldd-yEPPPKKK wwtpperrrviee.ee/feroreeeeeeeeee
on Learning Representations, 2024. URL
Esther Derman, Matthieu Geist, and Shie Mannor. Twice regularized MDPs and the equivalence bet ween robustness and regularization. In M. Ranzato, A. Beygelzimer, Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Newral Information Processing Systems, volume 34, pp. 22274-22287. Cur- ran Associates, Inc., 2021. URL https://proce: http::::///oceeeinggeee wtw.oeeeiigg..nuurp.......p cwturips.ccppppprr.. wwtpapper.flele/ppppreeeeeeeeeee fflles//appe///22//eeee bb1448oc:31d/299bf77337777 7858ce111817111111 1attsBeeaa1e1---111
Shizhe Diao, Rui Pan, Hanze Dong, KaShun Shum, Jipeng Zhang, Wei Xiong, and Tong Zhang. LMFlow: An Extensible Toolkit for Finctuning and Inference of Large Foundation Models. In Kai- Wei Chang, Annie Lee, and Nazneen Rajani (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies (Volume 3: System Demonstrutions), pp. 116-127, Mexico City, Mexico, June 2024. Association for Computational Linguistics. org/2024. 2024. naacl 12.
doi: 10.18653/v1 /2024 h1t/:///222a222l2dooooooo naacl- URL
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnic Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang, RAFT: Reward rAnked Finetuning for Generative Foundation Model Alignment. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https:
//openreview met:/toruumiidmmm7757771.1
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788, 2024.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A Simulation Framework for Methods that
14

---

Published in Transactions on Machine Learning Research (02/2025)
Learn from Human Feedback. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advonces in Newral Information Processing Systems, volume 36, pp. 30039 -0069. Cur- ran Associates, Inc., 2023. URL https: inguuruepig... REC778000830087714aaaaaaaaa33333 Conerr-ooear-e-eeeeeeeeooooo-C
neurips. wwwr.pp.cccpppprr..r. Conf cww.neereee..... pdf.
Bryan Eikema and Wilker Aziz. Sampling-Basd Based Approximations to Minimum Bayes Risk Decoding for
Neural Machine Translation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 10978 10993, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1 2022.emmlp-main.7555 754. URL. https::/aallantoology
emnlp-main. 754.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding Dataset Difficulty with V- usable Information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th International Conference on Machine Learning, vol- ume 162 of Proceedings of Machine Learning Research, pp. 5988 6008. PMTR, 17-23 Jul 2022. URL attps://proceedinn wwwggg............ mwr. wtppes///11d//tttaaaaaaaa222 html.
Benjamin Eysenbach and Sergey Levine. MaximumEntropy RL (provably) Solves Some Robust RL Problems. In International Conference on Learning Represenlations, 2022. URL id=PtSAD3caaA2.
Leo Gao, John Schulman, and Jacob Hilton. Scaling Laws for Reward Model Overoptimization, In Andreas Krauso, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.); Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings PMLR, 23 29 Jul 2023. URL https://procedding
of Machine Learning Research, pp. mlr. perrppress/v22/// http///22////aa/// gao23h. html.
Matthicu Gcist, Bruno Scherrer, and Olivier Pictquin. A Theory of Regularized Markov Decision Processes. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2160-2169) PMLR, 09-15 Jun 2019. URL bttpss//proceedingg maleeeddiinnmm...i....m.m..aeeem www.ar.rress////eeee /ttgggesttttateeee
/ html.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbench: An Advanced Diag- nostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision- Vision-Language Models. arXiv preprint ar Xrx:::2310.1 14566, 2024.
Lin Gui, Cristina Gârbacea, and Victor Veitch. BoNBoN Aligument for Large Language Models and the Sweetness of Best-of-n Sampling. arXiv preprint or Xiv:224006 00832, 2024.
David Heineman, Yao Dou, and Wei Xu. Improving Minimum Bayes Risk Decoding with Multi-Prompt. arXiv preprint XrXix:2407. 15343, 2024.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The Curious Case of Neural Text De- generation. In International Conference on Learning Representations, 2020. URL https://openrevies: bet//forum?id=ryyd@yee
Hisham Husain, Kamil Ciosek, and Ryota Tomioka. Regularized policies are reward robust. In International Conference Artificial Intelligence and Statistics, pp. 64 72. PMLR, 2021.
Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A Smith, Yejin Choi, and Hannaneh Hajishirzi. Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. arXiv preprint arXio:2406.0022 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B. Xiv preprint ar Xio:2310..00 068225, 2023a.
15

---

Published in Transactions on Machine Learning Research (02/2025)
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14163-14178, Toronto, Canada, July 2023b. Association for Computational Linguistics. doi: 10.18653/v wtt.//22.cc-oooo./.. URL attps://aclaanthology .rggg..rg///// aa2.2....lloooo. 792.
Yuu Jinnai, Tetsuro Morimura, Kaito Ariu, and Kenshi Abe. Regularized best-of-n sampling to mitigate reward hacking for language model alignment. In ICML 2024 Workshop on Models of Human Feedback for Al Alignment, 2024. URL http:: http://wwww...eetoooomomccccaaaa
Maxim Khanov, Jirayu Burapacheep, and Yixnan Li. ARGS: Alignment as Heward-Guider Search. In The Twelfth International Conference on Learning Represenlations, 2024. URL attps://openrevieet net/ forum? ddsummddddsgggeed.w...
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdul- lah Barhoum, Duc Nguyen, Oliver Stanley, Richard Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Ar- Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. Openassis- tant conversations democratizing large language model alignment. In A. Oh, T. Naumann, A. Globerson, K. Sacnko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 47649-4644488 47681. Curran Associates, Inc., 2023. URL httttss:/pproceediinng neur wwwpriis.ccpppppppae files/ http:///////1///00ffoo//2/22/11/
PaapeeeeeeeeeeeeeP----eeeeeeee Paper- _an___dd___ Band_BBnchmarks pd
Shankar Kumar and William Byrne. Minimum Bayes-risk word alignments of bilingual texts. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002), pp. 140 147. Association for Computational Linguisties, July 2002. doi:
URL https:
/ //aclanthology org/W02-1019.
Shankar Kumar and William Byrne. Minimum Bayes-risk decoding for statistical machine translation. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Compulational Linguislics: HLTT-AACL, 2004, pp. 169-176, Boston, Massachusetts, USA, May - May 7 2004. Association for Computational Linguistics. URL https://cclanthology org/ NO4-1022
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. From word embeddings to document dis- tances. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 957-966, Lille, France, 07 09 Jul 2015. PMLR. URL /ptp://///occcccs.....o. mlr. www../rress//v// /ttp///uuseb..../ html.
Nathan Lambert and Roberto Calandra. The Alignment Coiling: Objective Mismatch in Reinforcement Learning from Human Feedback. arXiv preprinl ar XrXi::2311.0010 2024.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannanch Hajishirzi. Reward- bench: Evaluating reward models for language modeling, reward-bench, 2024a.
htt.ggfac..c..aaeeeeeeeeeeeeeeee sspaces/allenaa
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, IJJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. RewardBench: Evaluating Reward Models for Language Modeling. arXiv preprint 2xX.::2408.18888 2024b.
Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye. More Agents Is All You Need. arXiv preprinl arXiv:2402.05122 2024.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human false- hoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214-3252, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.18653/v1 //05s//1/022aaclloo 229. URL https: httpp::/aclaatthology www.://rrrr222222/. acl 1o1g-lon..220 229..
16

---

Published in Transactions on Machine Learning Research (02/2025)
Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, Jilin Chen, Alex Beutel, and Ahmad Beirami. Con- trolled Decoding from Language Models. In Forty-first International Conference on Machine Learning, 2024. URL https: httpa::///perreeee.eeeee mett/forrmmiddbiic22070
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet. Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answering with human fcodback. arXiv preprint arXiv:2112.09933 2022.
Gergely Neu, Anders Jonsson, and Vicenç Gómez. A unified view of entropy-rogullriized markov decision processes. arxiv preprint 175500755,7798 2017.
OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad,, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cumminns, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Elcti, Tyna Eloundou, David Farhi, Liam Fodus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Montijo-Lopess Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Tukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Ben- jamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, 1lya Sutskever, Jie Tang, Nikolas Tezak, Madcleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizaboth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wicthoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang,
17

---

Published in Transactions on Machine Learning Research (02/2025)
Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. GPT-4 technical report. arXiv preprinl arXim:2333.8777 087744 2024.
Pedro Ortega and Daniel Lee. An adversarial interpretation of informttion-theoretic bounded rationality, Tn Proceedings of the AAAT Conference Artificial Intelligence, volume 28, 2014.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddic Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 27730 27744. Curran Associates, Inc., 2022. URL https: httpp::/ neurips cc wtwpepppprffeeeeeeeeeeeeeeeeeeee erence. pdf.
Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The Effects of Reward Misspocifiiation: Mapping and Mitigating Misaligned Models. In International Conference Learning Represenlations, 2022. URL https:://openreview met/ffouumidd--Yyaaanil
Matteo Pirotta, Marcello Restelli, and Luca Bascetta. Policy gradient Lipschitz Markov Decision Processes. Machine Learning, 100:255 283, 2015. URL https://app...emmatiicccc ottp://tt.....t...aoooooooot/aaa :25474414
Emmanuel Rachelson and Michail G. Lagoudakis. On the locality of action domination in sequential decision making. In International Symposium on Artificial Intelligence and Mathematics, 2010. URL https:
seapp.ssmmnticccholaat www.oolaar.rg/oopssst... 14029770..
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stelano Ermon, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly Re- ward Model. A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 53728-53748 Curran Associates, Inc., 2023. URL. https://proceedans..eem neurips. Shaapp_ffllspppreeeeeeaeeeeeeeee filles/paper22233/11e2 pdf.
Nils Reimers and Iryna Gurevych. Sentence-BERI: Sentence embeddings using Siamese BERT-networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the Ith International Joint Conference on Natural Language Processing (EMNI.--ICNI.P), pp. 3982 3392, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 00.18633/v1/DDD 1410. orr/D19-1///01110
URL https::/accaaas
Joar Skalsc, Nikolans Howe, Dmitrii Krasheninnikov, and David Kruegor. Defining and Charac- terizing Reward Gaming, In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advonces in Neural Information Processing Systems, volume 35, pp. 9460-9471. Cur- Associates, Inc., 2022. URL https://proceeddnng www.:/ceee.dnnnnn............... newp.ps.cccpppprrrrrrrr httee/ppeefffeee//eeeeeeeeeeeeee /t1////pppp//222///tll pdf.
3d7 2
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. Mpnet: Masked and Permuted Pre- training for Language Understanding. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 16857-16888 Cur- ran Associates, Inc., 2020. URL https: http::///poeeeegs.eeeeeeeee wwwpuripsccpppperrrree. filles/paperr2220//ttl .df.
Charles Spearman. The Proof and Measurement of Association between Two Things. The American Journal of Psychology, 15(1), 1904.
Nisan Stiennon, Long Ouyang, JefTrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. In H. Larochelle, M. Ran- zato, R. Hadsoll, M.F. Balcan, and H. Lin (ods.), Advances in Newval Information Processing Systems, volume 33, pp. 3008-3021. Curran Associates, Inc., 2020. URL httpps:/prroceedingg http:///ew............../../eear http:poooeeiiin...uuiirppppp neurips.cc. pdf..
18

---

htt/fforummdddxftf1 Published in Transactions on Machine Learning Research (02/2025)
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Mirac Suzgun, Luke KyrisiKyriazi,, and Dan Jurafsky. Follow the wisdom of the crowd: Effective text generation via minimum Bayes risk decoding. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 4265 4293, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18633/v1/2022 /tt/://///22..ndingaal.... URL https://aclanthooogy wrg. fontosfffnninggs findings-acl.22
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https://gwenln..giinn io/ //eg.//
Hugo Touchette. Legendre-Fenchel transforms in a nutshell. URL http://www. maths. qraul. ac. uk/" st//archive/ftthe pdf, 2005.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Fer- rer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Tnan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Tsabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylor, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subra- manian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Tine-Tund Chat Models. aixiv preprint or arXin:2307.0022 092888 2023.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprial arXi::2310. 16944, 2023.
R.J. Vanderbei. Linear Programming: Foundations and Falensions. International Series in Operations
Rescarch & Management Seience. Springer International Publishing, 2020. ISBN 9783030394158. URL co. co. wttpee....pp..oooop.eee..o.oooo. htpppookssdd-byjfbbbeee https://books. www.bbowwo.o.oo...oooooooooo...o google.co
Feng-Yu Yu Wang. Coupling and applications. In Stochastic Analysis and Applications to Finance: Essays in
Honour of Jia-An Yan, pp. 411-424 World Scientific, 2012.
Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jiawei Chen, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization. arXiv preprint arXiv:2407.078: 2024.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is DPO superior to PPO for LLM Alignment? A Comprehensive Study. In Forty- first International Conference on Machine Learning, 2024. URL
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXie:2407. 10671, 2024.
Wenhao Yang, Xiang Li, and Zhihua Zhang. A Regularized Approach to Sparse Optimal Pol- icy in Reinforcement Learning. In H. Wallach, H. Larochelle, A. Beygekzimer, F. d'Alché-Bue,
19

---

Published in Transactions on Machine Learning Research (02/2025)
E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL. https://proceeding www.iiinnnnnnuus...m seuripp.cc/panee htt.://..........pppppppp////ppp filles//ppper/2119//fee 3f#86eebbaF7caa000003000446e11
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. Ad- vancing LLM Reasoning Generalists with Preference Trees. In AI for Molh Workshop @ ICML 2024, 2024.
URL https:/ wttp:/viwiieeeefttteeeeeee nett/forumn1dd22iiicot
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 563-578, Hong Kong, URL
China, November 2019. Association for Computational Linguistics. doi: 10.18653v11/D19-120 https: https://aclanee hgtp::///ccaanhooooyyyoooo hatpa/ttooooooooo//t-/00000 org/D19-1053.
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.085%0 2020.
20

---

Published in Transactions on Machine Learning Research (02/2025)
A Detailed proof of Theorem 3.2
Definition A.1 (Similarity-basee based Lipschitz Continuity). A function f is said to have Simmlarityy-br based Lip- schitz Continuity if. for any 3 E ), the following holds:
where
C(y.v') cos (emb(y), emb(y'))
We first explain how the objective function is reformulated to max-min problem. Let us focus the regularization term, 1-WD term rewrite related to π, Tref
The following analysis is done in the framework of finite probability spaces. To simplify the following proof, we introduce the following notation. Let 22, .* on n places and consider the function f, where fo refers to the value fxx)).
WD[vlla] min (ET(P.N)
Σ
Ciyys
(iJ)Exx
(18)
min XYRYXY'
Σ
Camrs + W(7),
(KJ)Exy
where 7 is
coupling of the probability
measure
and 14, [(,,))
{TERYXY' Ejey Mr 44 Eiey My My Try 0 for all t.1). Y and Y (= }) is sample space re- spectively, corresponding to outcomes and j respectively and ((()) = 0 if I(E, p)), too otherwise.
Constraint terms, a coupling of the probability measure needs to satisfy:
Σ Ths u ViEy
Σ 7hs = P1 Vjey
(19)
Ni 20 Vi,j e
This constraint can be expressed in Ar = b, indicating its linear nature. Specifically, and are defined as A () (*) In this formulation, and I, denote identity matrices dimension y y, while 1; and 1j represent column vectors of dimension y with all components equal to The symbol @ denotes the Kronecker product.
Lemma A.2. Eq. (18) is reformulated as max problem from a min problem.
max Σ - Σ faps
(20)
[f f15
Proof. Taking into account the constraints specified in Eq. (19), we proceed with the application of the Lagrange multiplier method:
WD(elln)= min CiST 1.9
max ESin Egiu EV++
"ERYXY" XY 1.j
i
For a more intuitive understanding, f and g can be considered analogous to Lagrange multipliers. Except for the first term, all subsequent entries refer to constraints on 7.
WD
min max TERY XY f.g
[ - -gg) + >EE fin+ + Eglls
21

---

Published in Transactions on Machine Learning Research (02/2025)
can be seen from Eq. (19), these constraints lincar. From Theorem 5.2 (Vanderbei, 2020), in linear programming, there is never gap between the primal and the dual optimal objective values. Under the strong duality theorem (e.g., ming max, f (z, y) max, ming f(z, v)), we can exchange the min max term.
WD = max fig
min VERVRY
E(o - fo- - ft 9;) + Ean + Σ gil4
If Ctj - 2 0 for all i, the optimal value of min, (Ci( f: gy) is O, otherwise co. This observation allows us to derive the inequality constraint for the first item. We can include this as a constraint in the equation:
(Cuj
WD(ullm] = max f.g
Eses Σ 9ill
SCCs
Our next goal is to express the above function, currently represented by ʃ and g. exclusively in terms of the function f. From the given constraints, we have established that f. + S Cij for all and j. We can express this as follows:
95 S min {Ci fi}
(21)
To fix i= since min; picks the minimum value. The index i" gives this minimum, and fixing i turns the inequality in Eq. (21) into the equality in Eq. (22).
93 (C
(22)
Eq. (22) gives us a function which callod the e-transform of fy and often denoted by ff.
1 = = (Cs,, - -se}
We can now rewrite WD with ff as
WD[NllA] max E Σ
Efins fjps
(23)
If f is similarity-base Lipschitz, fo is also similarity-basee Lipschitz, for all i and j we have
fj ff s Cjj
= C. S ff f S Co.
= ff s Ci - ff
= = - S min
{Ciz - 19}
3
Upper bound of miny (Ci - f8} is choosing j +
min {Cis f5} < f j
It can be shown that f=== = fi miny {Ci fÿ}. This means that -g -fe = Substituting ff -
into Eq. 23, we get
max 1--b
E Saps
Σ
(24)
which is the dual form of 1-Wasscrstcin distance.
22

---

Published in Transactions on Machine Learning Research (02/2025)
Finally, by substituting AR for f. get:
TSRBoNWD (2) = max E y (z) [[ w)] (2(TT)
mell
max E ( [R y)
max
B
Σ AR(z, (r
Σ AR( ( x)
ARERA
Yeet
Uoer
max E ) (R(,) v)1
min ARERA
B
Σ
V) (v))) + EAR(E,))
TEll
Dror
Voor
where (2T))= = BWD[ Fre(( x)|[T( x)]
TSRBoNwD (x) = max aell
min ARERS By-a(11>) (R(e, v) - 3AR(2, v) +
Σ
Teer(y x))
VEVret
where
[AR(,,)) - (R( ) Vy.u E Vret) RA = E
E XX
23

---

Published in Transactions on Machine Learning Research (02/2025)
B Relationship Between Teef and the Proxy Reward Model
Despite the thcoretical robustness of SRBoNKL demonstrated in the analyses presented in section 3.1.1, the experimental results (section 4.1 and section 4.2) did not show comparable robustness. This section aims to explain the reasons for this discrepancy. Recall the objective function of SRBoNkL.:
TSRBoNKL (2) = max (1)x(1 [R(x, v)] 12(T)
Tl x)) log x ())
= max E. By [[(, y)
Σ Dear
Tref (y x:)
This implies that ideally, Tref and the reward function It should have some form of relationship (e.g. positive correlation) that facilitates learning. However, Tref is influenced by complex factors such as length bias.
To verify this hypothesis, we examine two aspects: (1) the correlation between the Eurus-RM-7B reward values, which were used as the gold reward model in our experiments, and the probabilities assigned by Trefi (2) the relationship between the length of the outpuls generated by Fref and the generation probabilities of those outputs.
Table The correlation between the Eurus-RM-7B reward values and the probabilities assigned by Tref
AlpacaFard Farm
Harmlessness 0.088
Helpfulness
0..24
-0.097
Table 6: The relationship between the length of the outputs generated by Tref and the generation probabilities of these outputs.
AlpacaFarm
Harmlessness -0.924
Helpfulness -0.854
-0.877
As can be seen from Table 5, there is negligible correlation between Tref and Eurus-RM-7B (gold reward model) in terms of Harmlessness and Helpfulness. In addition, the domain the AlpacaFarm dataset tends to be negatively correlated.
These results explain the performance degradation observed when this relationship is included in the reg- ularization term. Table 6 shows that Tref has a bias towards shorter sentences, with output probabilities increasing as sentence length decreases.
24

---

Published in Transactions on Machine Learning Research (02/2025)
C Supplemently Results
Figures 2 and 3 show evaluation of RBoN sensitivity on the Harmlessnoss subset and Helpfulness of the hh-rlhf dataset. These results were similar to those seen in AlpacaParm using section 4.2. This means that each method is not necessarily dependent on the dataset.
Figures to 6 compare RBoNwD and SRBoNwp and Figures compare RBoNKL. KL and SRBoNKL- These results show that SRBoN not superior R.Bo.. This is for reasons also discussed in section 4
QASST
SHP-Large
SHP-XL
66
575
50
550
NBoliw
RBoMwa
Win Rate (%)
Win Rate (%)
60
Win Rale
475
650
28
Outinal (M 20-2
Dea
480
Octinal Deta (1Bo).:: 5.0
10
10
12"
10"
10
10"
12"
10
10
Beta
Beta
Bela
PairRM
RM-Mistral-7B
50
Rãoo
SAOtNN
Win Rate
Win Rata
Rho%
Optinal Sata Ritellest 8.001
Celvrl (RZohhusl 5.
Oponel Sela (Sbolly 1-00001
Seel (kSols, J: 8200
Optonal Seta (5R Ra): 2000
Opeite (SRD)Na)
Oproval Rett [58 Merk
OptinaRaah (SRBeNun))
(50wk))
Optinalttos Sekii 30
10
10
10"
Beta
Beta
Figure 2: Evaluation of RBoN sensitiveness on the Harmlessness subset of the hh-rlhf datasot with varying parameter B. We use proxy reward models, OASST, SHP-Large, SHP-XI, PairRM, and RM-Mistral-7B. As the gold reward model, we utilize Eurus-RM-7B.
25

---

Published in Transactions on Machine Learning Research (02/2025)
OASST
SHP-Large
SHP-XL
NBohww
RS0
R5-heu
Roo No
RB:Nu,
Win Rale (%6
SAB/IN life
Skdolles
Noo
AB0%
Win Rata
Win Rale
RBoM,
(ftofficgl laze
Ovtimal diboliue
Operell (hoofh..):
Ovtimal (eeeli
30
Openal (5t15- (u)) in
Ostiral (SRB6Na))
bool (Stib
(SAB)Nac)
20
2>
13
10
10
10"
10
10"
"0"
10"
10
10
10
Beta
Bela
Bela
PairRM
RM-Mistral-7B
45
Win Rale (%6
Win Raie (%)
30
R.o
SanoNNo
Ootiral (kZehivsl: 0J
Optonal deta (ndethy (0.801
cpérallssu (Hben,, 0:201
Optonal Seta (SA Nas): 2003
Opeinal Both (S86sNa) 203
Oponall Rata (58 (SR Mapo 2012
Opennal Rees (SBneMur) 0.
18
Oponal Rata (SBek)) 5.0
10
Openai neth (ele 20
15
Figure 3: Evaluation of RBoN sensitivencss the Helpfulness subset of the hh-rlhf dataset with varying parameter B. We use proxy reward models, OASST, SHP-Large, SHP-XL, PairRM, and RM-Mistral-7B. As the gold reward model, we utilize Eurus-RM-7B.
QASST
SHP.Large
SHP-XL
Win Rate (56)
Win Rate (%6
45
Win Rate (%))
Tata [iitelles! 28.0
(5Ribo Mus)
20
Oytival Beth (Bilehiusd: R.
Optiral Dets (SRDoNue))
Optirol Dets (Riletiue: 6.5
Optiral Dats (SRD 85
10"
10"
10
10"
10" Beta
12?
10
10"
10
10" Beta
10
Beta
PairRM
RM-Mietral-7B
Win Ratl (%6
hoo
Oyérvel Bott (kSihissll d.1
Oelival Bota (SRBiNues)
Figure 4: Evaluation of RBoNwD and SRBoNNwD sensitiveness the AlpacaFarm dataset with varying parameter B. We use proxy reward models, OASST, SHP-Large, SHP-XL, PairRM, and RM-Mistral-7B. As the gold reward model, we utilize Eurus-RM-78
26

---

Published in Transactions on Machine Learning Research (02/2025)
OASST
SHige
SHP-XL
54
R30%
075
SRD-NNa
Gydiral Dets (Klehiasl
Opéiral Deth (SRDsNup
525
Win Rate (%6
16
0000
Win Rale
52.6
Win Rate
475
425
41.5
480
(SAioNNus)
Optiral Rots (SRBoNad)
10
42.6
10
10"
10"
10"
92
10"
12²
10'
10
10"
10
10
Beta
Bela
Bela
PairRM
RM-Mistral-7B
60
Win Rale (%G
Wi Rats (%6
60
1d0lve
1:5oNoe
Opanal Sew /hbofleut 80001
Cyérell Dou RiBoheell 5J
Opannl Sela (ShdoNes) 00Rst
Oyereel Duue (SHss/Mu)) od
Figure 5: Evaluation of RBoNwD and SRBoNwD sensitivencss the Harmlessness subset of the hh-rlhf dataset with varying parameter B. We use proxy reward models, OASST, SHP-Large, SHP-XT, PairRM, and RM-Mistral-7B. As the gold reward model, we utilize Earus-RM-7B.
QASST
SHP-Large
SHP-XL
52
50
50
Win Rate (56)
Win Rate (%6)
Win Rate
Tata [Sitelles! 15.0
30
Optival Seth (Kitehiusd: nat
Optinol Dets (Riletuue: 6.1
(SR Mep?
Optiral Dets (SRDoNuee) 0.55
Optiral Dats (SaD 81
10
10"
10"
10
107
10"
12?
10
10"
10
10" Bela
10
Beta
Beta
PairRM
RM-Mietral-B
00
20.0
40.6
Win Rets (%6)
40
29
Noofive Sk5u Noo
hoo
Oponnl bela (SBollea) 2810
Oydivel Bott (kSihissl do
Optoral beta (aa) 2013
Gelival Beta (SRBiNue))
Beta
Figure 6: Evaluation of RBoNwD and SRBoNwd sensitiveness the Helpfulness subset of the hh-rlhf dataset with varying parameter 3. We use proxy reward models, OASST, SHP-Large, SHP-XI, PairRM, and RM-Mistral-7B. As the gold reward model, we utilize Eurus-RM-7B.
27

---

Published in Transactions on Machine Learning Research (02/2025)
OASST
SHP-Large
SHP-XL
Rooka
RBoNe
RBcNe
SRacKs
Optiall Ben (8to%k)) 0:001
Opnmall Bera (Rittee 0:000
23
Optina Det (Bcck)))))) 0 0011
Optiaal Beta (5RDoNg1 20
Opennal Dara (SRDcNg) 865
Optine (e5, D06
Win Rate (%)
Win Rata
224
Win Rate
20
21
10"
10
10"2
10
10"
10"
12
10"
12'
10'
10"
10"
10-2
10"
10²
Beta
Bela
Bela
PairRM
RM-Mistral-78
00
RB-Me.
46
(kto%ls): 0:031
Booa
Winn Rale (%6
40
Win Rate (%)
15
10
-00
20
06
Beta
Figure 7: Exaluation of RBoNKL and SRBoNKL sensitiveness the AlpacaFarm dataset with varying parameter B. We use proxy reward models, OASST, SHP-Targe, SHP-XL, PairRM, and RM-Mistral-7B. As the gold reward model, we utilize Eurus-RM-7B.
OASST
SHP-Large
SHP-XL
RB-Ne
48h
RB:No.
Optiaal Beta (R30%kd: 00001
S100ke
SRSeNG
Openaal Bea iRDoN 0:00
480
StDeN
Optinal Data (BBckg) 0 8001
Win Rate (56)
Optiral Beta (5R5oNg1122 20
Win Rate (%)
Optnaal Dra (SPBcN)) 805
30
Optinal (sta 200
12
Win Rate
38
30
20
370
10
10"
10"
10
10"
12
10"
12²
10'
10"
10"
10
10" Bela
10¹
10'
Beta
Beta
PairRM
RM-Mietrsl-TB
50
45
Win Ralle (%6
Win Rate (%)
30
28
20
10
10"
10'
Beta
Figure 8: Evaluation of RBoNKL and SRBoNKL sensitiveness the Harmlessness subset of the hh-rlhf dataset with varying parameter 3. We use proxy reward models, OASST, SHP-Large, SHP-XI, PairRM, and RM-Mistral-7B. As the gold reward model, we utilize Eurus-RM-7B.
28

---

Published in Transactions on Machine Learning Research (02/2025)
OASST
LaPee
SHP-XL
Rookk
RDoNG
140
RBcNc.
5RSche
Optimal (h5olldd-00 00001
Opond Bra insoN 0.001
135
Optimal Bota (1Bclk)) 8821
(585-Na1:22 2010
Bia 1318chh)))
Optimal (ata (2t00 280
Win Rate (%5
Win Rate (%)
13.0
-0
Win Rate
19.00
125
18:75
20
18.00
16
1925
11.5
10"
10"
10
10"
12
10"
10"
10
12"
10"
10" Bela
10
Beta
Beta
PairRM
RM Mistral-7F
50
RiooN
Optimaal (hdolld) 02021
15ttbMuu 2010
Buaa (SRBCNa)
Win Rale (%5)
Win Rate (%)
30
20
20
10
Beta
Beta
Figure 9: Evaluation of RBBOKLL and SRBoNKL sensitiveness on the Helpfulness subset of the hh-rlhf dataset with varying parameter B. We use proxy reward modcls, OASST, SHP-Large, SHP-XL, PairRM, and RM- Mistral-7B. As the gold reward model, we utilize Eurus-RM-7B.
QASST
SHP.Large
SHP.XL
49
45
43
SRIeha
Win Rate (%)
Win Rate (%)
a5
S3ch,
40
VER
MSH
18/14
33
VER
35
cardon
Optratibet 200
Optrainee 8 Kathhol cree-
OptralDea GnBoNo)) 0.0
Dea (ShideMe)): 8.0
Opteotivnn (SRalie) 20.0
Optrocctes SShellee 250
25
Chrmeithe (k.50)) 11.
Outtroitee (lin 6.5.
Ouriaal thes (SA beMky! 1.
fhrr (AH)))
Osaral J.dah.
20
12
10
16?
12
10
10
16
12
10"
10
107
Beta
Beta
Beta
PairRM
RM-Mistral-B
53
R.t.ha
N.
Win Rate (%)
Operaal Datu Jibohugl:: 28.8
Bea (RdcHog) :.11
Deftrol Pets Rithbll.t Ein0e
25
Optmmal Dalu Sinborla)
2010
Opatra crtelMe) 28.0
19
16'
10
Beta
Bata
Figure 10: Evaluation of the decoder method on the AlpacaFarm datasct with varying parameter 3. We use proxy reward models, OASST, SHP-Large, SHP-XL, PaiRRM, and RM-Mistral-7B. As the gold reward model, we utilize Eurus-RM-7B.
29

---

Published in Transactions on Machine Learning Research (02/2025)
QASST
SHP-Large
SHP-XL
515
93
86.0
AasNa
R25
Shboohe
Wn Rate (1)
VER
300
Candaa
We Ras
41.
35
VER
Oparal 3olohee 20
Opatreiiie (SRBONa) 6.6
Optina Deza (1BoM 8001
Optrattee (ksehle cree
Ovtiral bora rioliMag! 1: 200
Oparai (noole) 260
OpareiBea (18cN)1-3)
Optina Bea Ro 1.5.
280
Operea fora pil An
12"
10"
10"
12"
10'
10"
10
10
13"
18
12"
10
10
19"
10
Bela
Bela
Bela
RM-Miltral778
45
R.thhi
Boliee
Win Rate (%)
Wiln
Optirel Boto ! Eiren
Optral Bake R.bhha
Optiaal Bots Reak.
Figure 11: Evaluation of the decoder method on the Harmlessness dataset with varying parameter 9. We use proxy reward models, OASST, SHP-Large, SHP-XL, PairRM, and RM-Mistral-7BB As the gold reward model, we utilize Murus-RM-7B.
QASST
SHP-Large
SHP-XL
100
53
S35NG
45
St3chuu
Wh Rate (N)
18/M
SRtohve
Wh Rate
NBR
VER
Oprteeitee (Sasteee) as
thes (aastay
Opanal 3idehval: 15.8
Oparai (1.cNNic peat
chhiclals 28
Outrall Bwr Bekuel "bbt.
Oparattehi prevhee) 20.0
Optraiteak cittllael Eice
Optire Bea (185)) 0.2
Optiaaleen SRBeliel
Oyste (Aboh))
15
10
10
10
Bata
Bala
Bata
RM-Mistral-7B
50
R.tche
40
1.5cN.
Win Rate (96)
cnndion
Bats Rihehucl
Optiral Sehuel
Gptmre Enke mbchhe
Opsimal (hteMaal 8.0011
Openall Bats :i
Oyainal Ribeh 200
Figure 12: Evaluation of the decoder method on the Helpfulness dataset with varying parameter 8. We use proxy reward models, OASST, SHP-Large, SHP-XI,, PairRM, and RM-Mistral-7BB. As the gold reward model, we utilize RM-TBB
30

---

Published in Transactions on Machine Learning Research (02/2025)
D Spearman's Rank Correlation (Spearman, 1904)
Figure 13, Figure 14, and Figure 15 show the average Spearman's rank correlation cocfficient (p) between pairs of reward models (Spearman, 1904). These results suggest that pairs of reward models with higher correlation values are more similar, indicating a preference for greedy methods in such cases.
Reward Correlation
1.0
OASST
1.00
0.42
0.50
0.39
0.29
0.32
0.9
3M-Mistral
0.42
1.00
0.42
0.62
0.38
0.44
0.8
0.7
PairRM
0.50
0.42
1.00
0.33
0.20
0.25
0.6
Eurus
0.39
0.62
0.33
1.00
0.29
0.35
0.5
SHP-Large
0.29
0.38
0.20
0.29
1.00
0.66
0.4
0.3
SHP-XL
0.32
0.44
0.25
0.35
0.66
1.00
OASST RM-Mistral PairRM
Eurus
SHP-Large SHP-XL
Figure 13:
The average Spearman's rank correlation coefficient (p) between pairs reward models in the Alra dataset.
31

---

Published in Transactions on Machine Learning Research (02/2025)
Reward Correlation
1.0
OASST
1.00
0.40
0.48
0.37
0.07
0.13
0.8
RM-Mistral
0.40
1.00
0.36
0.60
-0.08
-0.02
0.6
PairRM
0.48
0.36
1.00
0.36
0.04
0.11
0.4
Eurus
0.37
0.60
0.36
1.00
SHP-Large
0.07
-0.08
0.04
0.09
1.00
0.74
0.2
SHP-XL
0.13
-0.02
0.11
0.14
0.74
1.00
0.0
OASST RM-Mistral PairRM
Eurus
SHP-Large SHP-XL
Figure 14: The average Spearman's rank correlation coefficient (p) between pairs reward models in the Harmlessness dataset.
Reward Correlation
1.0
OASST
1.00
0.43
0.51
0.39
0.28
0.32
0.9
RM 3M-Mistral
0.43
1.00
0.40
0.75
0.45
0.54
0.8
0.7
PairRM
0.51
0.40
1.00
0.34
0.21
0.26
0.6
Eurus
0.39
0.75
0.34
1.00
0.38
0.50
0.5
SHP-Large
0.28
0.45
0.21
0.38
1.00
0.65
0.4
0.3
SHP-XL
0.32
0.54
0.26
0.50
0.65
1.00
OASST RM Mistral PairRM
Eurus
Large SHP-XL
Figure 15: The average Spearman's 's rank correlation coefficient (p) between pairs of reward models in the Helpfulness dataset.
32

---

Published in Transactions on Machine Learning Research (02/2025)
E
Supplementary Result on Lat---laaaa----Isstttte (Dubey et al., 2024)
We compared the average Spearman's rank correlation coefficient of the reward model and the performance of RBoNwD on the evaluation split using the Llama (Meta--lamma-3----mmsttaa language model. The purpose of this analysis to verify the performance of RBoNwD, even when applied to samples generated by state-of-the-art language models.
Reward Correlation
1.0
OASST 1.00
0.40
0.35
0.32
0.18
0.17
0.9
XM-Mistral
0.40
1.00
0.33
0.46
0.21
0.29
0.7
PairRM
0.35
0.33
1.00
0.21
0.13
0.18
0.6
Eurus
0.32
0.46
0.21
1.00
0.12
0.15
0.5
0.4
SHP-Large
0.18
0.21
0.13
0.12
1.00
0.51
0.3
SHP-XL
0.17
0.29
0.18
0.15
0.51
1.00
0.2
OASST RM-Mistra PairRM Eurus
SargLarge SHP-XL
Figure 16:
The average Spearman's rank correlation coefficient (p) between pairs of reward models in the AlpacaFarm dataset, using Llama as the language model.
33

---

Published in Transactions on Machine Learning Research (02/2025)
Reward Correlation
1100
OASST
1.00
0.34
0.16
0.33
0.23
0.14
0.8
RM-Mistral
0.34
1.00
0.24
0.36
0.31
0.23
PairRM
0.16
0.24
1.00
0.10
0.11
0.13
0.6
Eurus
0.33
0.36
0.10
1.00
0.4
SHP-Large
0.23
0.31
0.11
0.21
1.00
0.60
0.2
SHP-XL
0.14
0.23
0.13
0.14
0.60
1.00
OASST RM-Mistral PairRM
Eurus
SHP-Large SHP-XL
Figure 17: The average Spearman's rank correlation coefficient (p) between pairs reward models in the Harmlessness dataset, using Llama as the language model.
Reward Correlation
1.0
OASST
1.00
0.38
0.43
0.31
0.22
0.18
0.9
RM 3M-Mistral
0.38
1.00
0.36
0.43
0.27
0.36
0.8
0.7
PairRM
0.43
0.36
1.00
0.25
0.18
0.23
0.6
Eurus
0.31
0.43
0.25
1.00
0.17
0.23
0.5
0.4
SHP-Large
0.22
0.27
0.18
0.17
1.00
0.49
0.3
SHP-XL
0.18
0.36
0.23
0.23
0.49
1.00
0.2
OASST RM MM-Mistral PairRM
Eurus
Large SHP-XL
Figure 18: The average Spearman's 's rank correlation coefficient (p) between pairs of reward models in the Helpfulness dataset, using Llama as the language model.
34

---

Published in Transactions on Machine Learning Research (02/2025)
OASST
SargLarge
50
45
RBoNup (Lisms)
Win Rate 19
RBoNwa (Liama)
RBoNeo (mistral)
Win Rale
RBoN (mistral)
48
40
MBR (Liama)
MBR (Liama)
MBR (mietral)
46
(Lleme)
MBR (mistral)
RBoNun (mistral)
MBR (Llama)
42
MBR (mistral)
30
10
10
10
10"
10
10
10'
10
10"
10"
10"
10
Bela
Bela
Bela
PairRM
RM-Mistral-78
48
RBoNec (Liama)
Win Rate (%)
40
RBoNwe (Liama)
RBoNwo (mrsstol)
RBoNwo (mietral)
(Llama)
(Llama)
Win
44
MBR (mistra)
MBR (mistral)
42
20
40
10"
10"
10'
10"
10"
10"
10'
Figure 19: Evaluation the RBoN method on the AlpacaFarm dataset with varying parameter We use proxy reward models, OASST, SHP-Large, SHP-XT, PairRM, and RM-Mistral-7BB. As the gold reward model, we utilize Eurus-RM-7B, and Llama the language model.
35

---

Published in Transactions on Machine Learning Research (02/2025)
OASST
SHrgLarge
XL
57.5
RBoNwa (Liam
60
66.0
RBoNwa (mistral)
45
Win Rate (%)
56
RBoNoe (Liama)
Win Rate 18
MBR (mistral)
40
RBoNço (misttal)
50.0
MBR (Llama)
35
RBoNoo (Liare)
MBR (mistral)
47.5
RBoNoo (mistral)
45.0
30
MBR (Liama)
MBR (mistral)
42.
25
10"
10
10"
10
10
10
10
10
10
10"
10"
10"
10
Bela
Bela
Bela
PairRM
RM-Mistral-78
45
48
40
RBoNwe (Liama)
Win Rate
Win Rate
36
RBoNwo (mietral)
46
(Llama)
RBoNoo (Liama)
MBR (mistral)
RBoNoo (mistral)
MBR (LIsma)
MBR (mistral)
16
10
10" Beta
10"
10"
10"
10"
10'
Figure 20: Evaluation of the RBoN method on the Harmlessness dataset with varying parameter 3. We use proxy reward models, OASST, SHP-Large, SHP-XT, PaiRRM, and RM-Mistral-7BB. As the gold reward model, we utilize Eurus-RM-7B, and Llama the language model.
36

---

Published in Transactions on Machine Learning Research (02/2025)
OASST
SHP-Large
SHP-XL
50
Win (%)
45
2
RBoNwa (Llama)
Win Rale
RBoNyp (mistral)
40
12
MBR (Liama)
ABoNoo (Laama)
(Llame)
40
MBR (mistral)
RBoNoo (mistral)
RBoNun (mistral)
35
MBR (Liama)
MBR (Llama)
MBR (mistral)
40
MBR (mistral)
36
30
10"
10
10
10
10"
10
10
10'
10"
10"
10"
10
Bela
Bela
Bela
PairRM
RM-Mistral-78
48
40
RBoNec (Liama)
Win Rate (%)
RBoNwe (Liama)
46
RBoNwo (misstol)
30
RBoNwo (mietral)
(Llama)
(Llama)
Win
44
MBR (mistra) w))
MBR (mistral)
42
10"
10"
10'
10"
10
10"
10'
Figure 21: Evaluation of the RBoN method on the Helpfulness dataset with varying parameter We use proxy reward models, OASST, SHP-Large, SHP-XT, PairRM, and RM-Mistral-7BB. As the gold reward model, we utilize Eurus-RM-7B, and Llama the language model.
37

---

Published in Transactions on Machine Learning Research (02/2025)
F Robustness of RBoN Under Suboptimal Reward Models
We evaluate the performance of suboptimal reward models, Beaver (beaver-7b-vl..0-rewwee (Dai et al., 2024), Open Llama (tb-rihffrm-oope--llaaaa 3b) (Diao et al., 2024), and Tulu (lulu-v2.5113b-uf1) (Ivison et al., 2024) selected from Lambert et al. (2024a), which underperforms compared to other reward models in some cases. We set these models as proxy models, set Strus-RM--T (Eurus) as the gold model, and also show the reward corrclation of these models.
50
RBoNue
8.0
MBR
RBoNk
WnRae
60
RBoNww
35
RBON,
R.oNWw
MBR
SRBoNwo
MBR
RBoNa.
SRBON
45
RBoN
ABoN
35
SR
40
SRBoNwe
SRBoNa
SRBoN
20
Bota
Figure 22: Evaluation of RBoN sensitiveness on the AlpacaFarm dataset with varying parameter 3. We use proxy reward models, Beaver, Open Llama, and Tulu. As the gold reward model, utilize Eurus.
Reward Correlation
-1.0
Beaver
1.00
-0,03
0.07
0.03
0.8
Tulu
-0.03
1.00
-0.02
0.05
0.6
0.4
Eurus
0.07
-0.02
1.00
0.30
cen Llama
0.03
0.05
0.30
1.00
0.0
Beaver
Eurus
Open
Figure 23: The average Spearman's rank correlation coefficient (p) between pairs reward models in the AlpacaFarm datasct.
38

---

Published in Transactions on Machine Learning Research (02/2025)
Beaver
Open Liana
Tulu
70
60
65
55
RBolue
RBoNwo
50
RBoNk
RBoNi.
RBoNwe
RBON,
RBoN,
44
MBR
SRBoNwo
SRBoNwo
RBoNa
SRBoN
SRBoNN
40
SRBoNao
50
SRBoNa
15
10
10"
10
10
10
10
10
10
10
10
10
10
10
10
Bota
Bota
Bata
Figure 24: Evaluation of RBoN sensitiveness the Helpfulness dataset with varying parameter 3. We use proxy reward models, Beaver, Open Llama, and Tulu. As the gold reward model, we utilize Eurus.
Reward Correlation
-1.0
Beaver 1.00
0.09
0.8
Tulu
0.08
1.00
-0.09
0.02
0.6
0.4
Eurus
0.05
-0.09
1.00
0.33
0.2
Llama
0.09
0.02
0.33
1.00
0.0
Beaver
Tulu
Eurus
Open Llama
Figure 25: The average Spearman's 's rank correlation cocfficient (p) betwoen pairs of reward models in the Helpfulness dataset.
Basoer
Open Liama
52.5
48
50.0
45.00
47.5
MDR
RBoNAn
46
MBR
RBoNwo
RBoN.:
64
RBoN.
RBoNac
42.5
RBoN,
46.0
42.5
MBR
RBoNu
40.0
SRBoNww
SRDoNec
SADoNAa
SRBaNun
RBoN
37.5
SRBoN
38
37.5
38
10
10¹
10
10
Bata
Bata
Bata
Figure 26: Evaluation of RBoN sensitiveness the Harmlessness dataset with varying parameter B. We proxy reward models, Beaver, Open Llama, and Tulu. As the gold reward model, utilize Eurus.
39

---

Published in Transactions on Machine Learning Research (02/2025)
Reward Correlation
1.0
Beaver
1.00
0.06
0.01
0.03
0.8
Tulu
0.06
1.00
0.17
0.35
0.6
Eurus
0.01
0.17
0.11
0.4
0.2
sen Llama 0.03
0.35
0.11
1.00
Beaver
Tulu
Eurus
Open Llama
Figure 27: The average Spearman's rank correlation coefficient (p) between pairs of reward models in the Harmlessness dataset.
G Sentence Length Regularized BoN (RBoNL)
The objective function of RBoN, (Sentence Length Regularized BoN) given by:
3
ULBoN (x) arg max R(z, y)
DeDece
1yl
where 3 a regularization parameter, yy] denotes the token length of the sentence y
This approach aims to address the inherent bias toward shorter outputs often observed in large language model used in experiments. We now explain the rationale behind the specific form of the regularization term in R Let μ represent probability that is inversely proportional to the token length of the text y.
For example, we could define p(ylz) = 1/lel (e.g. p()(()) 1/10/1 (x) 1/bb"(..., where |y/ represents
the token length of output y.
Definition G.1. We define newly normalized distribution y²:
1/lyl
p' 1 x) = Eivr M(:
(())))) (x)
1/1y/
where
Ex(-1z) Z
Noer
Proposition G.2. The objective function of RBoN, is derived by considering the TV distance between the oulput, probability 1 ( x) and pl(. x) regularization term.
40

---

Published in Transactions on Machine Learning Research (02/2025)
Proof. Let us examine how the objective function of RBoNL is derived using definition G.1.
VL)oo(())
arg max R(x,y) + BTV (- a m))lk(( [x),,,, veloor
arg R(z,y) + B Σ VED'ar
x)) -
yED'or
B
arg max R(x,y) + vEN'or arg max R(x, v) + 3 yE)
Z yl
Z
Zly"
Z Z
1
Z
B
arg max R(x,y) yE
Z
B'
8 Z
arg max R(x,8) ()
B'
lyl
where 3' is a regularization parameter and TV denotes TV distance.
The purpose of this normalization is to counteract the effect of SRBONKL, which tends to favor shorter outputs. This formulation provides a theoretical basis for understanding how RBoN,, achieves its length- aware bchavior, and offers insight into its potential advantages over other decoding methods that may inadvertently bias toward shorter outputs.
Our methodological approach to assessing the divergence of output distributions from the length distribution p' involves comparative analysis of BoN sampling and RBoN,, For each output y selected, we construct the corresponding (- æ) distribution. We then measure the TV distance between L- x) and μl(: x).
The results of this comparative analysis are visualized Figure 28, Figure 29, and Figure 30.
41

---

Published in Transactions on Machine Learning Research (02/2025)
OASST
SHP-Large
SHP-XL
RBON.
RBON
RBoN,
0.921 10
10
9.820 10
9.844 10
10
9.944 -0
10"
9.00 10
9.940 1000x100
10"
10
10
10"
10'
10 Bela
10
Bela
Bela
PairRM
RM Mistral-tll
RBoN.
10-
RBOM.
BoN
BoN
10
TVL
9.922 10-1
9.922 10
9.934 x 10
10
9.034 10
10
10
10'
Bela
Beta
Figure 28: BoN sampling and RBoN, methods by measuring the TV distance between their output dis- tributions and sentence length distribution j' in AlpacaFarm. This allows us to evaluate how closely each method's outputs align with the desired distribution, with a smaller TV distance indicating a preference for shorter sentences.
OASST
SHP-Large
SHP-XL
9.904 10-1
RBON.
9.992 10-1
RDoN,
9.962 10
RBoN,
0.023 10-4
9.360 10
3.980 10
0.968 10-
9.922
9.058 10
TV
9.921 10-1
9.908 10
9.804 10-1
9.900 10-1
9.854 10
10-1
9.952
9.800 10
9.948 10-1
10
10
10
10
10
10" Bota
10
Beta
Bela
PairRM
9.821 10
RM-Mistral-78
1.040 100
RBoNL BoN
RBoN. BaN
8 10-1
1.020 100
9.019 10-1
TV (
1.000 100
8.918 10
9,800 10
8.917 10
9,800 10
3.916x100
8.015x10-1
9.400 10
10
10"
Beta
Bata
Figure 29: BoN sampling and RBoN, methods by mcasuring the TV distance between their output distri- butions and sentence length distribution p in Harmlessness. This allows us to evaluate how closely each method's outputs align with the desired distribution, with a smaller TV distance indicating a preference for shorter sentences.
42

---

Published in Transactions on Machine Learning Research (02/2025)
OASST
9.948 10-3
SHP-Large
SHP-XL
RBoN.
RBoN, BuN
RBoNL
9.922 x 10-1
BoN
BoN
9.948
9.846 -10
9,921 10-1
19644
9.920 10-1
9.942 10
9.940 10
9.919 10
10"
10
10'
PaiRRM
RM-Mistral-7B
1.040 x 10"
RBONL
3.941x10
RBONL
BoN
1.020 x 10"
1.000 x 10"
TVC
9,800 x 10-1
9.941 -0
9,600 x 10-1
9.400 10-1
10
10
10"
10"
10
Beta
Bata
Figure 30: BoN sampling and RBoN, methods by measuring the TV distance between their output dis- tributions and sentence length distribution p' in Helpfulness. This allows us to evaluate how closely each method's outputs align with the desired distribution, with smaller TV distance indicating preference for shorter sentences.
Our analysis shows that the output probability of RBBNN deviates more from jí' than the output probability of BoN sampling. Table 7 illustrates the correlation between the length of the sequence and the values of gold reference reward (Eurus-Ra RM--nB, focusing on subsets of sentences comprising the top 5, 10, 15 based on the proxy reward values. The strength of this correlation is an indication of the offectiveness of RBoN1: stronger correlation indicates greater effectiveness of the method.
In Table 7, have highlighted in bold the instances of high correlation compared to all samples used correlation, which corresponds to superior performance as shown in Table 2. In contrast, areas with lower correlation tend to show lower performance. This patlern shows a consistent. relationship bet ween correla- tion strength and method effectiveness. We also explored an alternative view PairRM that had a high correlation but did not produce correspondingly strong results in Table
We hypothesized that this discrepancy might be due the range of the regularization parameter B. To investigate this hypothesis and to demonstrate the potential of RBoN,, performed extensive analysis by varying 3 over wide range, from 10 to 5000 Figure 31.
H Experiment with 2wwen25--B-Inssrucct
As an ablation study, evaluate the methods using the Qwen Saren2.5-7BIInttrue the language model. Overall, we observe the same results as with Mistral-7B-SFT,, where RBoNwD outperforms the baseline algorithms (Figure 32).
43

---

Published in Transactions on Machine Learning Research (02/2025)
NoocaFam
Hormlessness
Hostuiness
10r 180%
R.DeNus R.50%..
Beta
Beta
Figure 31: Performance analysis of RBoN,, with varying 3 (10 to 5000) across AlpacaFarm, Harmlessness, and Helpfulness datasets. PairRM and Eurus-RM, RM-7B 7B are used as proxy and gold reward models, respectively,
OASST
SHP-Large
SHP-XL
RBoNwo
RBoNk.
45
MBR
Win (86)
RBoNoo
Win Rale 38 46
RBoNun
Win Rale (96)
RBoNgwa
RBoNk.
RBoNk.
40
RBoNsa-
RBoNswa
RBoNywo
RBoNgcc
RBoNor.
35
30
10
10
10
10
10
10
10
10
10'
10
10
10
Beta
Beta
Beta
PairRM
RM-Mistral-7B
50
45
Win Rata (%)
40
RBoNwa
RBoN,,
40
36
MBR
RBoNws
30
RBoNswn
RBoNa
RBoNça
RBoNowa
RBoNga
30
10
10"
10"
10"
10'
10
10
10
10
10
10'
Beta
Beta
Figure 32: Evaluation of the RBoN method the AlpacaFarm Farm dataset with varying parameter We use proxy reward models, OASST, SHP-Large, SHP-XL, PairRM, and RM-Mistral-7BB As the gold reward model, we utilize Eurus-RM-7B, and Qwen as the language model.
44

---

Published in Transactions on Machine Learning Research (02/2025)
Table The correlation between sequence length and gold reference reward (Burus-RM-7B) values, focusing on a subset of sentences that include the top 5, 10, 15 based on proxy reward values.
Top N
OASST
SHP-Large
SHP-XL
PairRM
RM-Mistral-7B
AlpacaFarm
All
0.11 (0.33)
0.11 (0.33) 0.11 (0.33)
0.11 (0.33)
0.11 (0.33)
5
0.27 (0.55)
-0.04 (0.56) 0.05 (0.55)
0.15 (0.59)
0.10 (0.56)
10
0.24 (0.44)
-0.02 (0.44) 0.06 (0.41)
0.17 (0.48)
0.09 (0.56)
20
0.21 (0.39)
-0.02 (0.37) 0.06 (0.36)
0.16 (0.41)
0.08 (0.44)
Harmlessness
All
0.08 (0.45)
0.08 (0.45) 0.08 (0.45)
0.08 (0.45)
0.08 (0.45)
5
0.24 (0.58)
0.10 (0.57)
0.13 (0.58)
0.20 (0.62)
0.37 (0.51)
10
0.25 (0.50)
0.11 (0.46)
0.12 (0.47)
0.19 (0.54)
0.36 (0.41)
20
0.22 (0.47)
0.11 (0.41) 0.11 (0.43)
0.21 (0.49)
0.34 (0.39)
Helpfulness
All
0.07 (0.40)
0.07 (0.40) 0.07 (0.40)
0.07 (0.40)
0.07 (0.40)
5
0.28 (0.56)
-0.04 (0.58)
0.11 (0.54)
0.14 (0.62)
0.06 (0.54)
10
0.27 (0.47)
-0.05 (0.45)
0.11 (0.42)
0.15 (0.52)
0.06 (0.40)
20
0.24 (0.43)
-0.06 (0.40) 0.10 (0.37)
0.17 (0.46)
0.03 (0.36)
Reproducibility Statement
All datasets and models used in the experiments publicly available (Table 8). Our code will be available open source upon acceptance.
45

---

Published in Transactions on Machine Learning Research (02/2025)
Table 8: List of datasets and models used in the experiments.
Name
Reference
AlpacaFarm
httgss//augtungaaec.///aaasss tatsu-lab/alpe hat.-llaaalpaca_faaea farm
Dubois ct al. (2023)
Anthropic's hh-rlhf
Bai et al. http:://wwgggggg.....oo///tttttt
hh-rlhf
mistral-7b-sft-beta (Mistral)
Jiang et al. (2023a); Tunstall et al. (2023) https://huggiingfac..
c//
Meta--llama-388-IInstruu (Llama)
Dubey al. (2024) http:://buggingaaae..oo//aa----
Meta-Llama-3-983 Instrucct
Qwenn..--7Bllustruuc (Qwen)
Yang et al. (2024); Team (2024) https:/hhuggnngffccccc///aa
5ran2.5-7B-nntzeee
SHP-Large
Ethayarajh et al. (2022) httpes//hunginggaaee.oo//ttttaap
SteamSiP--llan-66-large
SHP-XT.
httpp://wwngingaaaee...t/ttttttt SteamSHP-flan--5-zzl
Ethayarajh et al. (2022)
OASST
Köpf et al. (2023)
http:://hugiiggaeeeccc//ppsssss/
PairRM
Jiang et al. (2023b)
PairRM
RM-Mistral-7B
Dong el al. (2023)
http:://hgginnfaae..cceveesssss
RM-Mistral-7B
Yuan et al. (2024)
http:://luggingffceecc//ppptttp
Burus-RM-7b
Beaver
Dai et al. beaver-7b-vl -0-b-vvi----earddd
Tulu
Ivison et al. (2024) http:://wngiiigfeeeeee////ll/l/a 5 1-ppo-13b-uf -f
70b-uf-rm
Open Llama
Diao et al. (2024)
rlhf ______oppel 1lama_30
MPNet
Song et. al. (2020)
https//huggingface.co/
sentence-tren hra.enceettaaaffooresslllll haapiierr/almmmmpte------s---e
46


Reviews:
[{"review_id": "NR6JaTkcvo", "reviewer": "Reviewer_GPCT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n(1) The paper is well-motivated and well-written.\n\n(2) The paper's analysis is comprehensive and the results are strong, demonstrating a good performance across several benchmarks.\n\n(3) It is great to see the proposed method has a good theoretical foundation.\n\n\nWeaknesses:\n\n(1) The model backbone used in the experiments is weak. Mistral 7B SFT is outdated and whether the proposed method could be applied to the latest model is a problem. I would suggest adding experiments on Llama-3.1 and Qwen-2.5. In addition, I am curious about whether the proposed method could be scaled up with larger models. It would be better to add the results with various model sizes.\n\n(2) A minor question, there are many recent discussions on process reward models. I understand this paper mainly focuses on the outcome reward model and therefore only includes outcome reward models as baselines. I would appreciate to see discussion about the improvement compared with process reward models. \n\n(3) Why not include DPO, PPO as baselines? It is too limited to compare with best-of-n methods only. \n\nTypo: Section 4 Datasets - hh-rlhf datasets (HH) dataset, repeated \"dataset\".", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores the optimization methods and limitations of \"Best N Sampling\" (BoN) used in the alignment process of LLMs. They proposed  an extension of RBoN framwork, SRBoN and the sentence Length Regularzied BoN. The paper analyzes how regularization techniques can alleviate the reward hacking problem. Through theoretical analysis and experimental evaluation, the study shows that SRBoN achieves a good performance.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Add more experiments on different model series.\n\nAdd more baselines compared with DPO and PPO.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7Lx26rmkLh", "reviewer": "Reviewer_uLRn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The structure of this paper is good. The optimization is based on theoretical finding \n\n2. The proposed solutions achieve SOTA solution.\n\n3. The considered problem is new and can attract many audiences. \n\n\nWeakness:\n\n1. Some technical points are not clearly articulated. \n     * Section 2.3..1, what is the reference policy here. Is it the original model output given x? How to calculate the KL distance between pi_y and pi_ref?  \n      * What does it mean by reward hacking? Does it mean that the reward model cannot accurately reflect the real value of the model's output? What cause this inaccuracy? Is that beacause of the data quality for training a reward model? \n\n2. Some settings are not clear. For example, what is the reward model? Is it assumed to be known by this setting? How can we obtain such a reward model. I know this is not the main focus of the paper, but the authors should at least explicit it in an obvious position of the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper first systematically sutdy the existing BoN solutionss with regularization, and finds tht the regularization strategies in essence maximizes the worst case over a set of possible perturbation in the proxy reward. Inspired by this finding, this paper proposes stochastic RBoN, which has theoretically gurantee with the worst-case performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Explicitly mention what is the regularization and what is the reference policy. \n\n2. Discuss what is the reward model and how we can obtain it. Is it a reward model trained on the preference data like the reward model for RLHF?\n\nI don't have a strong backgroud on BoN and reinforcement learning but the paper indeed seems to be a solid submission. All the requested change can be done with a minor revision.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nRz0U01VEg", "reviewer": "Reviewer_FU9k", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength:**\n\nThis paper is well organized. The English writing is proficient, all sections are explained logically in good clarity, and proofs of the proposed theorems are explained well.\n\nThe proposed methods show empirical improvements over previous RBoN methods, together with theoretical bound, which improved the validity of the proposed method.\n\n**Weakness:**\n\nSome claims could be stated in a more moderate way, e.g., \"The RBoNL method is a significant contribution to the field of RBoN sampling\" in conclusion, better to avoid \"significant\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper improves the best-of-N samples for LLMs at test time, by introducing randomness into the previously proposed Regularized BoN algorithm. Specifically, it proves that the proposed stochastic methods are bounded theoretically, and leveraged two scalable open-source datasets to evaluate the method. It helps to improve LLM inference with minimum re-training efforts and potentially reduce bias on the reward model.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness section above.", "broader_impact_concerns": "I see no concerns regarding broader impact for this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are satisfied with the answers to their questions and the revisions made to the paper.  They unanimously recommend acceptance of the paper.  Recent work has emerged that should be cited. The authors are asked to add a discussion of the following works in the related work section.  The following two papers also optimize expected rewards subject to a KL divergence, but instead of reformulating the optimization as a min-max robustness optimization problem, they directly exploit the fact that there exists a closed form solution.  Hence, the optimization is simpler.

Mudgal, Sidharth, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen et al. "Controlled Decoding from Language Models." In Forty-first International Conference on Machine Learning, 2024.

Khanov, M., Burapacheep, J., & Li, Y. ARGS: Alignment as Reward-Guided Search. In The Twelfth International Conference on Learning Representations, 2024.

The paper also lacks a limitation section.  The authors are asked to add a limitation section.  This section should at the very least discuss the cost of the optimization at inference time and acknowledge that there is a tradeoff in comparison to techniques that fine tune the LLM.  The proposed techniques do not fine tune the LLM, but inference time is increased while techniques that fine tune the LLM incur a one-time cost at training time while avoiding any overhead at inference time.
Decision: Accept (poster)