Transfer Q-star : Principled Decoding for LLM Alignment
Authors: Souradip Chakraborty
Soumya Suvra Ghosal
Ming Yin
Dinesh Manocha
Mengdi Wang
Amrit Bedi
Furong Huang
Venue: NeurIPS 2024 poster (2024)
Keywords: RLHF
AI Alignment
Decoding
LLM
Transfer Decoding
Primary area: safety_in_machine_learning
TL;DR: Efficient alignment of Large Language models without updating the model parameters via Transfer decoding

Abstract: Aligning foundation models is essential for their safe and trustworthy deployment. However, traditional fine-tuning methods are computationally intensive and require updating billions of model parameters. A promising alternative, alignment via decoding, adjusts the response distribution directly without model updates to maximize a target reward $r$, thus providing a lightweight and adaptable framework for alignment. However, principled decoding methods rely on oracle access to an optimal Q-function ($Q^*$), which is often unavailable in practice. Hence, prior SoTA methods either approximate this $Q^*$ using $Q^{\pi_{\text{sft}}}$ (derived from the reference $\texttt{SFT}$ model) or rely on short-term rewards, resulting in sub-optimal decoding performance. In this work, we propose $\texttt{Transfer Q}^*$, which implicitly estimates the optimal value function for a target reward $r$ through a baseline model $\rho_{\texttt{BL}}$  aligned with a baseline reward $r_{\texttt{BL}}$ (which can be different from the target reward $r$). Theoretical analyses of $\texttt{Transfer Q}^*$ provide a rigorous characterization of its optimality, deriving an upper bound on the sub-optimality gap and identifying a hyperparameter to control the deviation from the pre-trained reference $\texttt{SFT}$ model based on user needs. Our approach significantly reduces the sub-optimality gap observed in prior SoTA methods and demonstrates superior empirical performance across key metrics such as coherence, diversity, and quality in extensive tests on several synthetic and real datasets.

Transfer Q*: Principled Decoding for LLM Alignment
Souradip Chakraborty' Soumya Suvra Ghosal Ming Yin?
Dinesh Manocha
Mengdi Wang² Amrit Singh Bedi
Furong Huang
UUnieesitty of Beryland-College Park; Princeton University; University of Central Florida
Abstract
Aligning foundation models is essential for their safe and trustworthy deployment. However, traditional fine-tuning methods computationally intensive and require updating billions of model parameters. A promising alternative, alignment via de- coding, adjusts the response distribution directly without model updates maximize a target reward r, thus providing a lightweight and adaptable framework for align- ment. However, principled decoding methods rely on oracle access to optimal Q O-function (Q"), ), which is often unavailable in practice. Hence, prior SoTA meth- ods either approximate this Q" using TT (derived from the reference SFT model) or rely on short-term rewards, resulting in sub-optimal decoding performance. In this work, we propose Transfer Q", which implicitly estimates the optimal value function for target reward r through a baseline model Pat. aligned with a baseline reward Fez (which can be different from the target reward r). Theoretical analyses of Transfer Q* provide rigorous characterization of its optimality, deriving upper bound on the sub-optimality gap and identifying hyperparameter to control the deviation from the pre-trained reference SFT model based on user needs. Our ap- proach si significantly reduces the sub-optimality gap observed prior SoTA methods and demonstrates superior empirical performance across key metrics such coher- ence, diversity, and quality in extensive tests on several synthetic and real datasets. The code is available at https ////ttttu//// cow.om/////co//. mommmmdduu /auudumddduunggggeeee LaabTrrasffeee
Introduction
As artificial intelligence (AI) systems continue to demonstrate super-human performance across various tasks, it is becoming increasingly critical to ensure that such Al systems align well with human preferences, goals, and ethical standards. Alignment via fine-tuning with reinforcement learning from human feedback (RLHF) [38, 34, 9], has proven highly effective [36, 44). However, aligning LLMs by fine-tuning the model parameters requires gradient updates across several billion parameters (size of LLMs) which require vast computational resources [31, 25] and have a significant environmental impact [15]. Additionally, many SoTA models not fully open-sourced [44, 36), offering limited access only to certain components like logits, making fine-tuning impossible.
As an alternative to fine-tuning, decoding for alignment has recently emerged a potential solution [33, 26]. Decoding aims to alter the LLM's response distribution at the token level to align with target reward without updating the parameters of the LLM. Decoding facilitates alignment by accessing the token-level optimal value function, Q", which corresponds to the target reward r. Tokens are then sampled based on Q". This approach ensures rapid, efficient, and cost-effective alignment. A detailed discussion of related work is provided in Appendix E.
denotes equal contribution denotes equal advising
38th Conference on Neural Information Processing Systems (NeurTPS 2024).

---

A fundamental challenge. Decoding effectively hinges on accessing an oracle to the optimal value function, Q", or a token-level optimal policy, which are typically not available in practical scenarios. To address this fundamental challenge, recent studies [33] have adopted a proxy, Q for Q" However, this approach results suboptimal decoding due to a distribution shift inherent in approximating the true, unknown Q-function, Q" as illustrated in Figure 1. This raises a critical question: Is it possible to devise a more efficient and accurate estimate of the optimal value function Q" for decoding purposes? ? In this work, we affirmatively address this query.
Baseline Model
1.0
Oracle
Average Reward
0.8
Transfer
Decoding
0.6
0.4
Gap
0.2
SET Model
Model
Target
0.0
SOTA (CD)
TO* (Ours)
(Pro trined)
Model
Figure I: Left. This figure highlights the conceptual idea of proposed transfer decoding in this work. It clearly shows that the current SoTA method [33] exhibits suboptimality with respect alignment with the target reward denoted by the dotted red arrow. On the other hand. the proposed transfer decoding method utilizes an immediately available aligned language model called the baseline, which is aligned with some bascline reward raz to bridge the gap between the SoTA method and the target model. Right. This figure provides empirical evidence of the performance gap of the current SoTA decoding strategy [33] with respect Oracle (best of N sampling). Our proposed Transfer Q* (TQ") reduces the gap and provides new decoding method.
Our approach leverages crucial observation regarding the key challenges identified: effective decoding requires access to a language model already aligned with the target reward r for trajectory-
level response generation. Notably, recent advances in DPO-based methods [38, 32] have facilitated the development of fine-tuned language models-referred refeereed to as baseline models- that are capable of
generating aligned trajectories [27]. Our first key idea involves utilizing these publicly available trajectory- models to estimate Q' and subsequently derive the optimal token-level language model for decoding. We term this approach direct transfer decoding.
Moreover, it is possible that these publicly available baseline models are aligned with a different baseline reward FBL. rather than the intended target reward Our second key idea addresses this challenge by proposing a novel indirect transfer decoding method. We introduce our proposed tech- nique as Transfer Q (TQ"), which facilitates efficient and on-the-fly alignment through decoding. We note that Transfer Q* proves effective even when there are substantial discrepancies between the target and baseline rewards.
We summarize our major contributions as follows.
(1) A novel concept of transfer decoding (TQ"). We introduce novel concept of transfer decoding in principled manner by leveraging already-available baseline language models aligned either with the target reward (direct transfer) or with some other (significantly) different baseline reward (indirect transfer). Our proposed approach TQ" efficiently reduces the sub-optimality gap inherent in previous SoTA decoding methods, as highlighted in Figure 1.
(2) Theoretical characterization of TQ'. We provide a rigorous theoretical characterization of the optimality of Transfer Q* Specifically,
derive upper bound on the gap between the optimal LLM policy and the I.I decoding policy resulting from TQ* (see Theorem 1). Additionally, we present a principled approach control the deviation of the resulting policy from the pre- pre-trained SFT language model Tsft (see Theorem 1, statement 2). This identifies hyperparameters that allow users to specify the desired amount of deviation or improvement based on their specific needs.
(3) Experimental evaluations. We provide detailed empirical evaluation of TQ" in various decoding tasks in both real and simulated settings, comparing against SoTA baselines, including DPO [38], ARGS [26]. and CD [33]. Empirical results demonstrate consistent superiority over the baselines. Notably, TQ* surpasses the current SoTA decoding strategy CD [33], achieving an improvement of up to 1,45x in average reward and 67. ..% in GPT-4 based win-tie rate. We further evaluate and compare several attributes of the text generated by our algorithm (such as coherence, diversity, and quality) against baselines, demonstrating the superiority of our algorithm in all these aspects.

---

2 Problem Formulation: Alignment via Controlled Decoding
2.1 Token-level Markov Decision Process
Since the control decoding procedure operates at the token level, before formulating the problem mathematically, start by defining token-level Markov decision process (MDP) in the context of LLMs. Let us consider a token-level MDP M ;= {S,A, A, P, R} with the state-space S consisting of the concatenated sequence of tokens and the action space A representing the space of the next token which is essentially the vocabulary V.
Next-token generator: the token-level policy Given state y E S, which is sequence of tokens containing the prompt/query x == {Ii, I2, xx appended with the tokens generated so far, LLM is token-level decoding policy that generates
y-cr := (Vo, "/.... -it-1) the action (i.e.. the next token) ag = ys via sampling yt ~ TT- st). The transition P to the next state S:+1 is deterministic: S [x, y <t: yol, the concatenation of the current state and action.
Response generator: the trajectory-level policy p. We denote the trajectory level probability by Pa (y|x)) - II क (y) y<r). Some commonly used sampling techniques in the literature include Beam Search [17], Top-p sampling [20], and Top-k Sampling [16].
From trajectory-level reward token-level reward R. Successful decoding depends on sampling from a token-level policy that yields high rewards, reflecting the inherently token-level nature of the decoding process. However, detailed Appendix F, we obtain a reward model r(x, y), only at the trajectory-level rather than the desired token-level, by fitting feedback on human preferences. To close the gap, similar to existing literature [33], we define the token-level reward R(x, y1) from the trajectory-level reward model r rxx,) y) as follows:
[0.
1/+ + EOS
R(x. yb)
(r(x, Y/+ EOS,
(1)
where EOS E V represents the end of sequence token. The token-level reward in (1) implies that we only receive a reward once we have the full sequence/response, otherwise, no reward.
Action-value function Q* for π and optimal Q' for optimal π.. From the token-level reward R(x, Vt), we can define the action-value function associated with the reward
Q"((s,oo ae) = (x y<l 91)
E Σ
ye T( (-1(5(+))
where St+i = [st, 20. 1 and expectation is over the randomness due to the sampling from token level language model æ. The optimal Q Q-function from the definition equation (2) is given by
Q" (st, a4) max Q"((s." oz).
The optimization problem in (3) denotes an unconstrained objective as in standard reinforcement learning; however, in the context of alignment for LLMs, we also need to consider the distance of optimal aligned policy to the pre-trained unaligned token-level policy Teh [37, 41, 38].
2.2 Principled Decoding for LLM Alignment
In this section, we will formulate the problem of aligning LLMs during deployment via a controlled decoding procedure as initially discussed in [33, 26].
Decoding process. We start by defining what decoding means in the context of LLMs. We consider access a pre-trained unaligned language model Van which takes in prompt x as input and generates a response y = [yo. 3/1, EOS) token by token by sampling Ye ~ T [x. y<.)) for all t. This token-by token-by-tokor by-token generation of response is called decoding in LLMs. Hence, the natural next question arises if we can control the decoding process and generate responses that are aligned with respect to target reward function r(x, y). The quest to answer this question has given rise an interesting research problem of utilizing decoding for LLM alignment [33].
LLM alignment via decoding. The problem of LLM alignment via decoding can be formally defined as solving for the optimal decoding policy T under the token level MDP M as
Tdce (-1s,) : arg max Ez-n((la,) IQ" (St. 2)] aDxr. [ (((]s.)) | (-ss))) xe
(4)
3

---

where s = Y<c and Q" (st, 2) denotes the optimal state-action value function for the token-level MDP M defined in (3). We remark that the KL regularization equation (4) ensures that the optimal decoding policy Tee remains in the close neighborhood of the pre-trained model Tsft which contains other important properties. In (4), a > denotes the alignment hyperparameter which controls the trade-off between the objective of maximizing the target reward r return and the closeness to Tsst- We can write the closed-form solution of the problem in (4) as
ddec (2 (s:) = (=)s,) exp (2 (.
(5)
C.
where C===E,
(z(ss) exp (aQ* z)) is the normalizing constant for state s,, Although the close form expression in (5) poses an interesting form, it is difficult to implement it in practice due to the various challenges we will discuss next.
Challenges of implementing (5): oracle access to the optimal Q", A major challenge in imple- menting the aligned decoding policy in (5), lies in the requirement of access to the optimal Q* =) in (5) for each state-action pair (8), which is unavailable in practice. To emphasize that, first note that (s,, 2) in (3) can be written using the trajectory level reward r as
(St,
Q" ( 2) ( [r[[x,, y<t 2),
(6)
where 7 denotes the trajectory T := {=1, 22, ... ,2,, and pa (.1s,, 2) = =IT x (2/(())d)) repre- sents the distribution over the trajectory level response induced by the optimal policy x (+(ss (cf. (3)). From (6), we note that the optimal Q" (s,, 2) relies on the trajectory/respont
generated by the optimal p' which is unknown. This creates a bottleneck in efficiently deploying the decoding policy in (5) for alignment with target reward r. Next, we discuss how some existing approaches in the literature deal with this fundamental issue and what the limitations are.
Limitations in existing approaches. An interesting approach called controlled decoding (CD) is proposed in recent literature by Mudgal et al. [33], and it approximates Q* 2) by QT 2). which is tractable and easily computable due to availability of ksft- However, this approximation results in significant suboptimal performance, described in Figure 1 (right). Given this limitation of existing approaches and the above-mentioned fundamental challenge of decoding, we pose the question: Is it possible to provide better estimate of Q' for decoding? We answer this affirmatively in the following section by introducing a novel concept of transfer decoding.
3 Proposed Approaches: Alignment via Transfer Q*
Our key ideas of transfer decoding. Our proposed approach hinges on an interesting observation that recent advancements in alignment, particularly through direct preference optimization (DPO)-based approaches [38], have led to the development of open source freely available fine-tuned language models [27, 53]. We call such aligned models as baseline models, that generate trajectory responses in an aligned manner. Our first key idea is to utilize these baseline models aligned with target reward if available, to estimate Q* and subsequently derive the optimal token-level language model for decoding. We remark that baseline models are aligned the trajectory level (see Appendix F for details), while decoding requires the optimal models at the token level. We term this approach direct transfer decoding. However, is possible that the available aligned baseline model is aligned with a different baseline reward rar. instead of the target reward r. Hence, our second key idea addresses this issue by proposing a novel method called indirect transfer decoding. We discuss both the ideas and detailed algorithms in detail next.
3.1 Direct Transfer Decoding
For the direct transfer, we start by considering that we are given a target reward model r and an unaligned pre-trained SFT language model given by T&t: the corresponding trajectory-level response distribution is given by Pert- We are operating under the assumption that we have a baseline model (x)x) with target reward r. We note that the closed-form expression for the RLHF- optimal model Pau can be written as (see Appendix F, Equation (25)) follows:
( Pau (y|x) (x Patt (ylx) exp (f)(x.7)).

---

where Z, (x) is the normalizing constant and [ > is the trajectory-level alignment parameter. We note that the trajectory level optimal policy in (7) is usually obtained in the literature via the fine-tuning stage and efficient algorithms such as DPO [38].
Estimating Q" for direct transfer. Now we get back to the fundamental bottleneck of optimal decoding, which lies in estimating the token-level optimal Q* (st, We propose providing a solution the problem with the help of PEL (y (x). As defined in (6), we begin by considering
(a, 2) = (-18..) [r( (st =]]
where p² (-|st, z) ;= arg maxp B7--p--s,,.)) [r(s,, T)]. However, we know that the available baseline model Pau (y)x) in (7) is the solution of following optimization problem:
Pac Pac(-1x (-1x) == arg max (x=p(()x) [r(x, r)] - _9DK.. [p(-1 x) |l Patt(-) x)]]
which constraints the drift of the optimal distribution Pat. from the trajectory level reference policy Pett with the KI divergence term. We propose approximating the optimal Q² (se, 2) for decoding by
TQ (St: 2) E (+ls..)) [r[[st: a].
(10)
With the definition in (10), we propose obtaining our token-level decoding policy mäag(-1s) for the token-level MDP
Tigg (-|sc) == arg max (- [TQ =)] allKk.. (-|ss))] (a (-(s:]] FEll
(11)
where Tat (-1se) is the token-level language model, which induces the trajectory level model Pat. Due to the strong convexity of the objective in (11) owing to the additional KI. regularization term, we get the closed-form solution of mrig(-Ise) as
(z)(2 exp (
TQQ (8t,,))))
Aig (2|sp) = (st)
(12)
where C. (s,) is the normalizing constant. We summarize the proposed approach in Algorithm 1.
3.2 Indirect Transfer Decoding
We remark that, for the direct transfer decoding, we started with the assumption that the available baseline model Peu is aligned with the target model only. In practice, however, it is possible that the freely available baseline model Pst. is actually aligned with some other reward function we call baseline TBL rather than the target reward r. We argue that this condition is even more easily satisfied under the ongoing active research scenario alignment because we have easy access to open-source, well-aligned LLMs trained on various reward functions [27, 52, 13, 38]. Under this setting of baseline reward TsL. and language model Pat, we define our novel indirect transfer decoding process as follows.
The Transfer Process. The baseline language model Par. is also an RLHF aligned model corre- sponding to reward function rst. holds that:
Pat (y
FBu (x, log
Pott (y)x) Blog (x).
(13)
where ZEL (x) is the corresponding partition function. From the closed-form expression in (7), holds for the target reward r that
p+(y(x) + +ol (x).
r(x. y) Blgg Pesc (y)))
(14)
Using equations (14) and (13), we can obtain the trajectory-level optimal policy p-(y))x (x) for the target reward r(x, y) as
1 (r(x,t) - (xL,
Zxc(x)
(r(x)) = Pau (yl exp
(15)
3
Zx(x)
(xbx))
5

---

Algorithm Transfer Q": LLM Alignment via Transfer Decoding
1: Input: Trajectory level baseline model Par (y x) aligned with baseline reward TeL., target reward
r, token-level baseline policy TEL, number of tokens sampled k; decoding alignment parameter a, vocabulary set V
2: for .,....T'' do
3: Current state 5e = [x, where is prompt and y<r [Mo, y1, 33t-1]
4: Sample top-k tokens using token-level baseline policy TEL and store as V {zz
TEL (-1s))
5:
for do
6:
if TBL =r then (Direct transfer)
7:
Evaluate (s; 2) r([s 2). r), where PBL PBL(-1[se. [[]s))=)] =l)
8:
else (Indirect transfer)
9:
Evaluate (s, 2) w r((s,, =}, where Pau (-1ss,, z)),
pm((7 (e,)a)))
10:
Compute decoding score for token 2: Os LTQ log aa (z)ss))
11:
Next token = arg maxxxe2
12:
Next state 1 4 [Sr, Wrl
13:
Return y
Wo. ................................ yrl
where we note that Pr (ylx) is the unnormalized probability with the normalization factory ((x) == (x(x)
(x) We show in the Appendix H that (x) is the normalization constant for (y|x).. We emphasize that calculating the trajectory-level optimal language model for the target reward function in (15) is the erucial step in estimating the optimal Q" for the token-level MDP M for our decoding.
Estimating Q" for indirect transfer. Similar to the direct transfer setting, we propose approximating the optimal ("((s: z) for decoding by using pr (y|x) in (15) as
TQ (st, 2) = (--pp((Ise.a))
[r([sr, =]]
(16)
Pr (y(y) r([se, 21. 2).
= -E (188,22)
PBL (ylx)
where we use the importance sampling trick and then utilizing equation (15), we can get the Q* for indirect transfer. Now, following (11) and (12), we can write
(-se() arg max (2 [TQ* (st, 2)] aD [T<[ss
(17)
where n, (-1ss)))) is the token level probability derived from the trajectory level policy (-(()) in (15). Following (11) and (12), we can obtain our optimal decoding policy for the indirect transfer well. We summarize the proposed approach in A Algorithm 1.
3.3 Theoretical Results and Insights
This subsection provides the theoretical analysis of Transfer Q* algorithm under direct transfer setup. Existing works [33, 18] leverage the reward-Ke KL tradeoff curve to measure the reward gain versus deviation from reference policy. Ideally, a good algorithm should achieve high rewards while staying close to the reference policy (be KL-efficient). We follow these two aspects and consider two performance metries: (1) suboptimality gap and (2) KL divergence between our algorithm's 's policy and the SFT policy. Specifically, we borrow the notion of suboptimality gap from offline RL literature [1] and define in terms of value function difference for any prompt x as
Sub-Gap(x) == Vx) - Vxag(x).
(18)
where V*(x)= = max, (xxp(()x) (r(x,T) Vxng(x)= = Byreii (- (r(xxT)) and Plag represents the distribution over the trajectories induced by the token level policy (-1g(+1x) ) in (17). The KL divergence between our algorithm and the reference policy denoted by ITx (Pia (-1 x), Pase(-) x)). We present main theorem follows, and the full proof is deferred Appendix G.
Theorem 1. For the proposed Transfer Ø* Algorithm 1, the following results hold.

---

(1) Suboptimality gap for all x is upper bounded as
Sub Gab--ap(()) ) I (3l (p" (-) x); Pafe( x)) (x)(x))
(19)
where B is defined in (9) for baseline policy, and a is defined in (17) for decoding process. Here ho (x) 2 and its formula is defined in Appendix
(2) Assume reward satisfies O Sr r. then the divergence to SFT policy is given by
1 TT)
Tex (1)) x)) =
r Teaz-
(20)
Remark 1 (*Double Robustness" of Transfer Q*). Theorem indicates the suboptimality is bounded by (0Dx (p", Patt), and this guarantees our algorithm will achieve high accuracy in two cases. 1. The penalty parameter /3 is small, 2. The SFT policy Patt is close to p'. In addition, our decoding design (17) is also effective for improving the performance. This is due -ah, I 0. Our decoding coefficient a could further reduce the suboptimality gap when it is appropriately tuned (e.g. choose a = arg min, ahaa).
Remark 2 (KL-Efficiency of Transfer Q*). Via Theorem 1, the deviation from our algorithm to the reference policy jointly controlled by parameter / and decoding parameter a. When both parameters are set large, our algorithm is more conservative and hence more KL-efficient. When the parameters are small, the KL deviation becomes larger.
Experimental Evaluations
We present a comprehensive empirical analysis of both direct and indirect Transfer Q", tested across various open-source datasets and state-of-the-art models [27]. Our findings demonstrate Tq effectiveness in aligning language model outputs with specific target rewards. For implementation, we set the number of tokens sampled k = and the decoding alignment parameter a = 1. We report ablations in Appendix J.3. Reproducibility is ensured through the use of publicly available resources.
Table Summary of datasets and model architectures used for experimental evaluations in Section 4.
Dataset
Model Architectures
Reward Preference
SFT
Reward
Evaluation-
Ultaaeeddhack [12]
Mustral-1B 7B-a [24]
Zephyr:7B-o [46]
Mistral-BB-o [24]
Relevant, Helpfal, and Ethical responses.
Evaluation 2
HH RLHF [5]
Pythia 9B [7]
Pythia-6.90 9B [7]
Pythia 6. 9B [7]
Helpful and Harmless responses.
Evaluation-3
Berkeley Nectar [53]
OpenChat 3.5 3.5-7B [47]
Swiling-7B: Balinng-B-a [53]
Mistral-B-a [24]
Accuratic Helpful, and Harmless
UttaFeedhack
Llama-2: [45]
Tulu-v2-7B [23)
Mistral-BB-. [24]
Relevant, Helpful,
Ethical responses. Relevant, Helpfal, and Ethical responses.
Evaluation
UltaFeedback [12]
Mistrl- 7B-a [24]
Zephyr-7B-o [46]
Gemma- 7B [6]
Evaluation-6
UtraFeedhack (12)
Mistral- 7B-a [24)
Zephyr-7B-or [46)
Gemma-2B (6]
Relevant, Helpfal, and Ethical responses.
3.00
2.0
1-0
KL Divergence
2.
0.6
Sax
8.2
DAO
ARES CD
T0
0.0
0F0
MADE
06
0.0
CRO
ARSS
C0
101
(Dum)
Methods
[SFT)
(Ours)
Methods
Methods
Evaluation
(b) KL Divergence
(c) Evaluation
(d) Evaluation
Figure 2: In plots (a). (c), and (d) we present the normalized average reward values obtained using the correspond- ing setup outlined in Table 1. ARGS (SFT) and ARGS (DPO) refer to the reward modeling approach described [26] to the SFT and DPO model respectively. Our analysis reveals that across all setups, TQ consistently outperforms other competitive baselines summarized in Table 1, demonstrating its superior efficacy. We report results on other evaluation setups in Appendix J. In (b), we compare (for Evaluation- I setup) the trajectory-level KI. Divergence between different decoding policies and the base model pare show the effectiveness of the proposed approach compared the state-of-the-an
Evaluation Methodology. For evaluation, we compare the performance of the response generated by the language model corresponding to each prompt in the test dataset. Following [26), we limit the maximum length of the prompt and generated continuation to 128 and 2048 tokens, respectively. For all baselines, we utilize a greedy- sampling method. The quality of the generated responses assessed based on multiple attributes (including reward achieved, win-tie rate, coherence, diversity,
7

---

etc.) using the following evaluation metrics [26]: Average Reward: We report the mean of the rewards for generations corresponding to all prompts the test set. A higher mean reward score signifies that the model's outputs are better aligned with the attributes represented in the reward model. Diversity: This metric measures the ability to generate texts with wide range of vocabulary. Specifically, it calculates the frequency of repeated n-grams in text. Coherence: We assess the semantic closeness between each prompt and its generated response using the cosine similarity of their
[43] embeddings.
4.1 Transfer Q': Evaluations with Direct Transfer Decoding
Experiment Details. For the direct transfer experiments, we consider our baseline model as a DPO [38] fine-tuned model aligned with the target reward. To evaluate the performance of Transfer Q' (denoted as TQ in figures), we conduct experiments across multiple datasets and model architec- tures, encompassing 6 distinct tasks. Our experimentation is primarily based on the Ultrafeedback [12), Berkeley Nectar [53], and HH-RLHF (Helpful and Harmless) [5] datasets. For each task, we utilize the DPO [38] fine-tuned model an aligned policy, with configurations listed in Table 1. This comprehensive approach allows us to gauge the adaptability and efficacy of TQ in various contexts, providing robust measure of its capabilities.
Results Discussion. erage rewards for the first three
Figure 2.
we setups detailed
present the normalized av- in Table 1.
We report the results for other setups in Appendix J.1. We compare our proposed method TQ* with competitive existing ap- proaches such ARGS [26], CD- { and DPO [38]. To provide clearer compar- ison of results, we normalize the average rewards (further details of normalization in Appendix I.1). We observe that across all setups, TQ" consistently outperforms the existing approaches by a large margin, high- lighting its efficacy in learning token-level optimal policy during inference. Further, in Figure 3, we report that TQ* not only pro- duces responses with high rewards but also outperforms other decoding strategies in terms of diversity and coherence.
0.8
C..
Coherence
9.6
0.5
0.5
0.4
ARGS
DPO ARCS
CO
10"
0.3
AR3s
AROS
DPO
ARGS CO
TO"
(SFT)
(DPO]
(Ours)
(SFT)
(DPO)
(Ours)
Methods
Methods
(b)
Figure 3: Diversity and Coherence analysis of generated responses. We observe that the responses generated using TQ* obtain the highest coherence and diversity. These results based on the prompts from Berkeley Nectar datasct.
GPT-4 Evaluation. To further understand the quality of the responses generated, we employ GPT-4-based based evaluation framework. Specifically, we use GPT-4 as a surrogate for human assessment.
In this method, we prompt GPT-4 to assess and rate two responses on the same prompt on a scale from to 10, focusing on criteria such as rel- evance, accuracy, and insightfulness. For this, we randomly sample 300 prompts from the test set and compare the response between TQ" and other competitive decoding methods. We present the GPT-4 evaluation results in Table 2, measured by the percent- age of win-ties of our method over the alternative decoding strategies. A higher percentage indicates that our proposed method more proficient in generating responses that exhibit better alignment with human preferences. Our analysis in Table shows that TQ' consistently has higher win-tie percentage compared to other decoding approaches, reaffirming its efficacy.
Table 2: GPT-4 Based Evaluation. We prompt GPT-4 to rate re- sponses from various decoding strategies relevance, accuracy, and insightfulness, scoring them from 10. A higher win-tie percentage indicates our method's effectiveness in generating con- textually relevant and accurate responses.
Win-Tie (%)
Ours
vs. Methods
Evaluation-1
Evaluation-2
Evaluation-3
TQ' TQ' TO"
ARGS-SPT
86.67
72.00
75.34
DPO
70.67
77.34
70.00
ARGS-DPO
68.00
71.33
74.00
TQ'
CD
66.67
65.34
67.34
3 Duee to unavailability of code base, we compare using approximate version of CD [33] in which we do not train adapter module.

---

4.2 Transfer Q": Evaluations with Indirect Transfer Decoding
Synthetic experiments. In the synthetic experiments, we simulate four scenarios to examine shifts in the reward distribution between source and target reward models. These scenarios are instrumental in elucidating the advantages of our proposed method. The shift in reward distribution is achieved through model intervention techniques, such as inducing sparsity in the final linear layer. Our analysis utilizes the UltraFeedback [12] and Berkeley Nectar [53] datasets. The specifics of the models utilized are detailed in Table 4 in Appendix 1.2. For each dataset, we design two transfer tasks by modulating the reward distribution. We visualize the shift in reward distribution on the Uttraeedbbcck dataset in Figure (a) and (c) respectively. Given that this is a synthetic setup, no DPO-aligned policies exist for the newly generated reward distribution, which underscores the significance of addressing the transfer problem. We present the results for this analysis the Ultrafeedback [12] dataset in Figure 4, Due to space constraints, we report the results on the Berkeley Nectar [53] in Appendix J.2. We make the following key observations: (1) Our proposed decoding framework consistently attains the highest average reward and outperforms other competitive strategies. (2) The variant of our decoding strategy obtained by direct transfer to the target reward, i.e., DT has subpar performance.
0.200
0.175
150
Target
Source
2.0
0.8
0.200
Target
Average Reward
0.8
1.0
0.125
0.8
0.6
0.100
0..4
80.4
0.075
0.080
0.2
0.025
0.000
15 Reward Value
0.0
ARGS AAGS CD
(SFT) (DPO)
OT
(Ours)
0.002
0.0
ARGS ARGS CD
(SFT) (DPO)
(Ours)
Y0+
Methods
Reward Value
Methods
(c)
(d)
Figure 4: Evaluation for Synthetic Indirect Transfer Tasks. We plot the distribution of the reward values for the source and two transfer tasks on the Ultrafeedback in (a) and (c). The reward model architecture is Mistral-7B-o [24]. In (b) and (d), we compare the normalized average reward scores for competitive decoding strategies. We represent variant of our decoding strategy with direct transfer as DT. We observe that consistently outperforms other baselines. Results on other datasets are reported in Appendix J.2.
Real transfer experiments. To further evaluate our proposed approach real reward transfer, we consider two transfer setups as outlined in Table 5 in Appendix I.3. We illustrate the distribution shift in reward values UltraFeedback [12] and HH-RLHF [5] datasets in Figure 5 (a) and (c) respectively. We compare the normalized average reward values diffcrent decoding strategies in Figure 5. Consistent with findings from the synthetic experiments, consistently outperforms other decoding strategies, underscoring its effectiveness in real transfer tasks well. Notably, observe that even when there is significant shift in the source and target reward values, as evident in Figure 5 (c), TQ* outperforms all competitive decoding approaches in terms of average reward.
proposed method
0.40
1.0
0.40
Average Rewara
0.36
Source
0.8
0.35
Source
0.8
0.30
Target
0.30
Target
0.25
0.6
0.25
0.6
0.20
30.4
0.20
0..4
0.15
0.15
0.10
0.2
0:10
0.2
0.00
0.05
15 -10 -5
(sm (DPO)
TQ* (Ours)
0.05
15
-5
0.0
(SFT)
TQ
Reward Value
Methods
Reward Value
Methods
(a)
(b)
(c)
(d)
Figure 5: Evaluation for Real Indirect Transfer Tasks. In (a) and (c), we visualize the distribution shift in reward values between the source and target for Sctup-1 and Sctup-2, respectively, as outlined in Table 5. In (b) and (d), we report the normalized average reward scores of different decoding strategies corresponding to Setup-1 and Setup-2, respectively.
Conclusions
In this paper, we introduce Transfer Q" novel decoding strategy for Al alignment. Our method effectively addresses the prior gaps in alignment via decoding by leveraging an estimate of optimal Q" for target reward through an existing aligned policy PEL- Our theoretical analysis provides

---

rigorous characterization of the sub-optimality gap with respect to the token-level value function and the KL divergence the reference policy. Experimentally. we demonstrate the consistent and significant superiority of the proposed approach. Hence, this work provides a principled solution to efficient decoding for Al alignment.
10

---

References
[1] Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, 32:96, 2019.
[2] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as laboratory for alignment. arXiv preprint arXiv:211200088 2021.
[3] Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences, 2023.
[4] Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosseini, Pushmeet Kohli, and Edward Grefenstette. Learning to understand goal specifications by modelling reward. arXiv preprint arXiv:1806.01944 2018.
[5] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 200.22004.8822 2022.
[6] Tris Warkentin Jeanine Banks and Tris Warkentin. Gemma: Introducing new state-of-the-art open models, 2024.
[7] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' D'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling, In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
[8] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4):324-345 1952.
[9] Souradip Chakraborty, Amrit Bedi, Alec Koppel, Huazheng Wang, Dinesh Manocha. Mengdi Wang, and Furong Huang. Parl: A unified framework for policy alignment in reinforcement learning. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
[10] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.0133. 2024.
[11] Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[12] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:23 10.01377, 2023.
[13] Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback, 2023.
[14] Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint wxXiv:2304.06760 2023.
[15] Ahmad Faiz, Sotaro Kaneda, Ruhan Wang, Rita Osi, Prateek Sharma, Fan Chen, and Lei Jiang. LImcarbon: Modeling the end-to-end carbon footprint of large language models, 2024.
[16] Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation, 2018.
[17] Markus Freitag and Yaser Al-Onaizan. Beam search strategies for neural machine translation.
arXiv preprint :ttt::1702.008000 2017.
[18] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning. pp. 10835-10866. PMLR, 2023.
[19] Amelia Glaese, Nat McAleese, Maja Trębac7, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint 2XX::22009.1373 2022.
11

---

[20] Ari Holtzman. Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv: 10X.25790.00757, 2019.
[21] James Y Huang, Sailik Sengupta, Daniele Bonadiman, Yi-an Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchoff, and Dan Roth. Deal: Decoding- time aligmment for large language models. arXiv preprint arXiv:2402.06 147, 2024.
[22] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
[23] Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in a changing climate: Enhancing Im adaptation with tulu 2. arXiv preprint arXiv:231 10702, 2023.
[24] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 2023.
[25] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
[26] Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment as reward search, 2024.
[27] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannanch Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024.
[28] Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. Chain of hindsight aligns language models with feedback. arXiv preprint arXiv:2302.022 02676, 2023.
[29]
Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares, Jessica Hoffmmann, Lucas Dixon, Michal Valko, and Mathieu Blondel. Decoding-ting time realignment of language models, arXiv preprint arXiv:2402.0090
2024.
[30] James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, Guan Wang, David L Roberts, Matthew E Taylor, and Michael I. Littman. Interactive learning from policy-dependent human feedback. In International conference on machine learning, pp. 2285-2294. PMLR, 2017.
[31] Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey, 2024.
[32] Tetsuro Morimura, Mitsuki Sakamoto, Yuu Jinnai, Kenshi Abe, and Kaito Ariu. Filtered direct preference optimization, 2024.
[33] Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng. Michael Collins, Trevor Strohman, Jilin Chen, Alex Beutel, and Ahmad Beirami. Controlled decoding from language models, 2024.
[34] Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel J. Mankowitz, Doina Precup, and Bilal Piot. Nash learning from human feedback, 2023.
[35] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju. William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint XrXi:2112.009332 2021.
[36] OpenAI, Josh Achiam, Steven Adler. Sandhini Agarwal, Lama Ahmad. Iige Akkaya, Floren- cia Lconi Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji. Valerie Balcom, Paul Baltescu, Haiming Bao, Moham- mad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Beemadett-Shapingg Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brock- man, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila
12

---

Dunning, Adrien Ecoffet. Atty Eleti, Tyna Eloundou. David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Graf- stein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang. Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim. Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz I Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniy Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Ra- jeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang. Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano. Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong. Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Sel- sam, Kyla Sheppard, Toki Sherbakov, Jessica Shich, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song. Natalie Staudacher, Felipe Petroski Such, Natalic Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergyya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang. Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024.
[37] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin. Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
[38] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2023.
[39] Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q learning. arXiv preprint arXiv:2206. /1711, 2022.
[40] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
[41] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022.
[42] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022.
13

---

[43] Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A contrastive framework for neural text generation. In Alice H. Oh, Alekh Agarwal. Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
[44] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- rean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul R. Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, Jack Krawczyk, Cosmo Du, Ed Chi. Heng-Tze Cheng, Eric Ni, Purvi Shah, Patrick Kane. Betty Chan, Manaal Faruqui, Aliaksei Severyn, Hanzhao Lin, YaGuang Li, Yong Cheng, Abe Ittycheriah, Mahdis Mahdieh, Mia Chen, Pei Sun, Dustin Tran, Sumit Bagri, Balaji Lakshminarayanan, Jeremiah Liu, Andras Orban, Fabian Güra, Hao Zhou, Xinying Song, Aurelien Boffy, Harish Ganapathy, Steven Zheng, HyunJeong Choe, Ágoston Weisz, Tao Zhu, Yifeng Lu, Siddharth Gopal, Jarrod Kahn, Maciej Kula, Jeff Pitman, Rushin Shah, Emanuel Taropa, Majd AI Merey, Martin Baeuml, Zhifeng Chen, Laurent El Shafey, Yujing Zhang, Olcan Sercinoglu, George Tucker, Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, Alexandre Frechette, Charlotte Smith, Laura Culp. Lev Proleev. Yi Luan, Xi Chen, James Lottes, Nathan Schucher, Federico Lebron, Alban Rrustemi, Natalic Clay, Phil Crone. Tomas Kocisky, Jeffrey Zhao, Bartek Perz, Dian Yu, Heidi Howard, Adam Bloniarz, Jack W. Rae, Han Lu, Laurent Sifre, Marcello Mag- gioni, Fred Alcober, Dan Garrette, Megan Barnes, Shantanu Thakoor, Jacob Austin, Gabriel Barth-Maron, William Wong, Rishabh Joshi, Rahma Chaabouni, Deeni Fatiha, Arun Ahuja, Gaurav Singh Tomar, Evan Senter, Martin Chadwick, Ilya Kornakov, Nithya Attaluri, Iñaki Iturrate, Ruibo Liu, Yunxuan Li, Sarah Cogan, Jeremy Chen, Chao Jia, Chenjie Gu, Qiao Zhang, Jordan Grimstad, Ale Jakse Hartman, Xavier Garcia, Thanumalayan Sankaranarayana Pillai, Jacob Devlin, Michael Laskin, Diego de Las Casas, Dasha Valter, Connie Tao, Lorenzo Blanco, Adrià Puigdomènech Badia, David Reitter, Mianna Chen, Jenny Brennan, Clara Rivera, Sergey Brin, Shariq lqbal, Gabriela Surita, Jane Labanowski, Abhi Rao, Stephanie Winkler, Emilio Parisotto, Yiming Gu, Kate Olszewska, Ravi Addanki, Antoine Miech, Annie Louis, Denis Teplyashin, Geoff Brown, Elliot Catt, Jan Balaguer, Jackie Xiang, Pidong Wang, Zoe Ashwood, Anton Briukhov, Albert Webson, Sanjay Ganapathy, Smit Sanghavi, Ajay Kannan, Ming-Wei Chang, Axel Stjerngren, Josip Djolonga, Yuting Sun, Ankur Bapna, Matthew Aitchison, Pedram Pejman, Henryk Michalewski, Tianhe Yu, Cindy Wang. Juliette Love, Junwhan Ahn. Dawn Bloxwich, Kehang Han, Peter Humphreys, Thibault Sellam, James Bradbury, Varun Godbole, Sina Samangooei, Bogdan Damoc, Alex Kaskasoli, Sébastien M. R. Arnold, Vijay Vasude- van, Shubham Agrawal, Jason Riesa, Dmitry Lepikhin, Richard Tanburn, Srivatsan Srinivasan, Hyeontack Lim, Sarah Hodkinson, Pranav Shyam, Johan Ferret, Steven Hand, Ankush Garg, Tom Le Painc. Jian Li, Yujia Li, Minh Giang, Alexander Neitz, Zaheer Abbas, Sarah York, Machel Reid, Elizabeth Cole, Aakanksha Chowdhery, Dipanjan Das. Dominika Rogoziríska, Vitaliy Nikolaev, Pablo Sprechmann, Zachary Nado, Lukas Zilka, Flavien Prost, Luheng He, Marianne Monteiro, Gaurav Mishra, Chris Welty, Josh Newlan, Dawei Jia, Miltiadis Allamanis, Clara Huiyi Hu, Raoul de Liedekerke, Justin Gilmer, Carl Saroufim, Shruti Rijhwani, Shaobo Hou, Disha Shrivastava, Anirudh Baddepudi, Alex Goldin, Adnan Ozturel, Albin Cassirer, Yunhan Xu, Daniel Sohn, Devendra Sachan, Reinald Kim Amplayo, Craig Swanson, Dessie Petrova, Shashi Narayan, Arthur Guez, Siddhartha Brahma, Jessica Landon, Miteyan Patel, Ruizhe Zhao, Kevin Villela, Luyu Wang, Wenhao Jia, Matthew Rahtz, Mai Giménez, Legg Yeung, James Keeling, Petko Georgiev, Diana Mincu, Boxi Wu, Salem Haykal, Rachel Sapu- tro, Kiran Vodrahalli, James Qin, Zeynep Cankara, Abhanshu Sharma, Nick Fernando, Will Hawkins, Behnam Neyshabur, Solomon Kim, Adrian Hutter, Priyanka Agrawal, Alex Castro- Ros, George van den Driessche, Tao Wang, Fan Yang, Shuo yiin Chang, Paul Komarek, Ross Mcllroy, Mario Lučié, Guodong Zhang, Wael Farhan, Michael Sharman, Paul Natsev, Paul Michel, Yamini Bansal, Siyuan Qiao. Kris Cao, Siamak Shakeri, Christina Butterfield, Justin Chung, Paul Kishan Rubenstein, Shivani Agrawal, Arthur Mensch, Kedar Soparkar, Karel Timothy Chung, Aedan Pope, Loren Maggiore, Jackie Kay, Priya Jhakra, Shibo Wang, Joshua Maynez, Mary Phuong, Taylor Tobin, Andrea Tacchetti, Maja Trebacz, Kevin Robinson, Yash Katariya, Sebastian Riedel, Paige Bailey, Kefan Xiao, Nimesh Ghelani, Lora Aroyo, Ambrose
Lene,
14

---

Slone, Neil Houlsby, Xuehan Xiong, Zhen Yang, Elena Gribovskaya, Jonas Adler, Mateo Wirth, Lisa Lee. Music Li, Thais Kagohara, Jay Pavagadhi, Sophie Bridgers, Anna Bortsova, Sanjay Ghemawat, Zafarali Ahmed, Tianqi Liu, Richard Powell, Vijay Bolina, Mariko linuma, Polina Zablotskaia, James Besley, Da-Woon Chung, Timothy Dozat, Ramona Comanescu, Xiance Si, Jeremy Greer, Guolong Su, Martin Polacek, Raphaël Lopez Kaufman, Simon Tokumine, Hexiang Hu, Elena Buchatskaya, Yingjie Miao, Mohamed Elhawaty, Aditya Siddhant, Nenad Tomasev, Jinwei Xing, Christina Greer, Helen Miller, Shereen Ashraf, Aurko Roy, Zizhao Zhang, Ada Ma, Angelos Filos, Milos Besta, Rory Blevins, Ted Klimenko, Chih--Kan Ych, Soravit Changpinyo, Jiaqi Mu, Oscar Chang. Mantas Pajarskas, Carrie Muir, Vered Cohen, Charline Le Lan, Krishna Haridasan, Amit Marathe, Steven Hansen, Sholto Douglas, Rajkumar Samuel, Minggiu Wang. Sophia Austin. Chang Lan, Jiepu Jiang, Justin Chiu, Jaime Alonso Lorenzo, Lars Lowe Sjösund, Sébastien Cevey, Zach Gleicher, Thi Avrahami, Anudhyan Boral, Hansa Srinivasan, Vittorio Selo, Rhys May. Konstantinos Aisopos, Léonard Hussenot, Livio Baldíni Soares, Kate Baumli, Michael B. Chang, Adrià Recasens, Ben Caine, Alexander Pritzel, Filip Pavetíc, Fabio Pardo, Anita Gergely, Justin Frye, Vinay Ramasesh, Dan Horgan, Kartikeya Badola, Nora Kassner, Subhrajit Roy, Ethan Dyer, Víctor Campos Campos, Alex Tomala, Yun- hao Tang. Dalia El Badawy, Elspeth White, Basil Mustafa, Oran Lang, Abhishek Jindal, Sharad Vikram, Zhitao Gong, Sergi Caelles, Ross Hemsley, Gregory Thornton, Fangxiaoyu Feng, Woj- ciech Stokowiec, Ce Zheng. Phoebe Thacker, Çaglar Unlüi Zhishuai Zhang, Mohammad Saleh, James Svensson, Max Bileschi, Piyush Patil, Ankesh Anand, Roman Ring, Katerina Tsihlas, Arpi Vezer, Marco Selvi, Toby Shevlane, Mikel Rodriguez. Tom Kwiatkowski. Samira Daruki, Keran Rong, Allan Dafee Nicholas FitzGerald, Keren Gu-Lemberg, Mina Khan, Lisa Anne Hendricks, Marie Pellat, Vladimir Feinberg, James Cobon- Tara Sainath, Maribeth Rauh, Sayed Hadi Hashemi, Richard Ives, Yana Hasson, Eric Noland, Yuan Cao, Nathan Byrd, Le Hou, Qingze Wang. Thibault Sottiaux, Michela Paganini, Jean- -aan-Baptiste Lespiau, Alexandre Mou- farek, Samer Hassan, Kaushik Shivakumar, Joost van Amersfoort, Amol Mandhane, Pratik Joshi, Anirudh Goyal, Matthew Tung, Andrew Brock, Hannah Sheahan, Vedant Misra, Cheng Li, Nemanja Rakićević, Mostafa Dehghani, Fangyu Liu, Sid Mittal, Junhyuk Oh, Seb Noury, Eren Sezener, Fantine Huot, Matthew Lamm, Nicola De Cao, Charlic Chen, Sidharth Mudgal, Romina Stella, Kevin Brooks, Gautam Vasudevan, Chenxi Liu, Mainak Chain, Nivedita Melink- cri, Aaron Cohen, Venus Wang, Kristic Seymore, Sergey Zubkov, Rahul Goel, Summer Yuc, Sai Krishnakumaran, Brian Albert, Nate Hurley, Motoki Sano, Anhad Mohananey, Jonah Joughin, Egor Filonov, Tomasz Kepa, Yomna Eldawy, Jiawern Lim, Rahul Rishi, Shirin Badiezadegan, Taylor Bos, Jerry Chang, Sanil Jain, Sri Gayatri Sundara Padmanabban, Subha Puttagunta, Kalpesh Krishna, Leslie Baker, Norbert Kalb, Vamsi Bedapudi, Adam Kurzrok, Shuntong Lei, Anthony Yu, Oren Litvin, Xiang Zhou, Zhichun Wu, Sam Sobell, Andrea Siciliano, Alan Papir, Robby Neale, Jonas Bragagnolo, Tej Toor, Tina Chen, Valentin Anklin, Feiran Wang. Richie Feng, Milad Gholami, Kevin Ling, Lijuan Liu, Jules Walter, Hamid Moghaddam, Arun Kishore, Jakub Adamek, Tyler Mercado, Jonathan Mallinson, Siddhinita Wandekar, Stephen Cagle, Eran Ofek, Guillermo Garrido. Clemens Lombriser, Maksim Mukha, Botu Sun, Hafeezul Rahman Mohammad, Josip Matak, Yadi Qian. Vikas Peswani, Pawel Janus, Quan Yuan. Leif Schelin, Oana David, Ankur Garg, Yifan He, Oleksii Duzhyi, Anton Älgmyr, Timothée Lottaz Qi Li, Vikas Yadav, Luyao Xu, Alex Chinien, Rakesh Shivanna, Aleksandr Chuklin, Josie Li, Carrie Spadine, Travis Wolfe, Kareem Mohamed, Subhabrata Das, Zihang Dai, Kyle He, Daniel von Dincklage. Shyam Upadhyay, Akanksha Maurya, Luyan Chi, Sebastian Krause, Khalid Salama, Pam G Rabinovitch, Pavan Kumar Reddy M, Aarush Selvan, Mikhail Dektiarev, Golnaz Ghiasi, Erdem Guven, Himanshu Gupta, Boyi Liu, Deepak Sharma, Idan Heimlich Shtacher, Shachi Paul, Oscar Akerlund, François-Xavii Xavier Aubet, Terry Huang, Chen Zhu, Eric Zhu, Elico Teix- cira, Matthew Fritze. Francesco Bertolini, Eiana-Eleonora Marinescu, Martin Bölle, Dominik Paulus, Khyatti Gupta, Tejasi Latkar, Max Chang. Jason Sanders, Roopa Wilson, Xuewei Wu, Yi-Xuan Tan, Lam Nguyen Thict, Tulsee Doshi, Sid Lall, Swaroop Mishra. Wanming Chen, Thang Luong, Seth Benjamin, Jasmine Lee, Ewa Andrejczuk, Dominik Rabiej, Vipul Ranjan, Krzysztof Styre, Pengcheng Yin, Jon Simon, Malcolm Rose Harriott, Mudit Bansal, Alexei Robsky, Geoff Bacon, David Greene, Daniil Mirylenka, Chen Zhou, Obaid Sarvana, Abhi- manyu Goyal, Samuel Andermatt, Patrick Siegler. Ben Horn, Assaf Israel, Francesco Pongetti, Chih-Wei "Louis" Chen, Marco Selvatici, Pedro Silva, Kathie Wang, Jackson Tolins, Kelvin Guu, Roey Yogev, Xiaochen Cai, Alessandro Agostini, Maulik Shah, Hung Nguyen, Noah Ó Donnaile, Sébastien Pereira, Linda Friso, Adam Stambler, Adam Kurzrok, Chenkai Kuang, Yan Romanikhin, Mark Geller, ZJ Yan, Kane Jang, Cheng-Chun Lee, Wojeiech Fica, Eric Malmi,
15

---

Qijun Tan, Dan Banica, Daniel Balle, Ryan Pham, Yanping Huang, Diana Avram, Hongzhi Shi, Jasjot Singh, Chris Hidey, Niharika Ahuja, Pranab Saxena, Dan Dooley, Srividya Pranavi Potharaju, Fileen O'Neill, Anand Gokulchandran, Ryan Foley, Kai Zhao, Mike Dusenberry, Yuan Liu, Pulkit Mehta, Ragha Kotikalapudi, Chalence Safranek-Shrader, Andrew Goodman, Joshua Kessinger, Eran Globen, Prateek Kolhar, Chris Gorgolewski, Ali Ibrahim, Yang Song, Ali Eichenbaum, Thomas Brovelli, Sahitya Potluri, Preethi Lahoti, Cip Baetu, Ali Ghorbani, Charles Chen, Andy Crawford, Shalini Pal, Mukund Sridhar, Petru Gurita, Asier Mujika, Igor Petrovski, Pierre-Louis Cedoz, Chenmei Li, Shiyuan Chen, Niccolò Dal Santo, Siddharth Goyal, Jitesh Punjabi, Karthik Kappaganthu, Chester Kwak, Pallavi LV, Sarmishta Velury. Himadri Choudhury, Jamie Hall, Premal Shah, Ricardo Figueira, Matt Thomas, Minjie Lu, Ting Zhou, Chintu Kumar, Thomas Jurdi, Sharat Chikkerur, Yenai Ma. Adams Yu, Soo Kwak, Victor Åhdel, Sujeevan Rajayogam, Travis Choma, Fei Liu, Aditya Barua, Colin Ji, Ji Ho Park, Vincent Hel- lendoorn, Alex Bailey, Taylan Bilal, Huanjie Zhou, Mehrdad Khatir, Charles Sutton, Wojciech Rzadkowski, Fiona Macintosh, Konstantin Shagin, Paul Medina, Chen Liang, Jinjing Zhou, Pararth Shah, Yingying Bi, Attila Dankovics, Shipra Banga, Sabine Lehmann, Marissa Bredesen, Zifan Lin, John Eric Hoffmmann Jonathan Lai, Raynald Chung, Kai Yang. Nihal Balani, Arthur Bražinskas, Andrei Sozanschi, Matthew Hayes, Héctor Fernández Alcalde, Peter Makarov, Will Chen, Antonio Stella, Liselotte Snijders, Michael Mandl, Ante Kärrman, Pawel Nowak, Xinyi Wu, Alex Dyck, Krishnan Vaidyanathan, Raghavender R. Jessica Mallet, Mitch Rudominer, Eric Johnston, Sushil Mittal, Akhil Udathu, Janara Christensen, Vishal Verma, Zach Irving, Andreas Santucci. Gamaleldin Elsayed, Elnaz Davoodi, Marin Georgiev, Ian Tenney, Nan Hua, Geof- frey Cideron, Edouard Leurent, Mahmoud A Alnahlawi, Ionut Georgescu, Nan Wei, Ivy Zheng, Dylan Scandinaro, Heinrich Jiang, Jasper Snoek, Mukund Sundararajan, Xuezhi Wang. Zack Ontiveros, Itay Karo, Jeremy Cole, Vinu Rajashekhar, Lara Tumeh, Eyal Ben-David Rishub Jain, Jonathan Uesato, Romina Datta, Oskar Bunyan, Shimu Wu, John Zhang, Piotr Stanczyk, Ye Zhang, David Steiner, Subhajit Naskar, Michael Azzam, Matthew Johnson, Adam Paszke, Chung-Cheng Chiu, Jaume Sanchez Elias, Afroz Mohiuddin, Faizan Muhammad, Jin Miao, Andrew Lee. Nino Vieillard, Jane Park, Jiageng Zhang, Jeff Stanway, Drew Garmon, Abhijit Karmarkar, Zhe Dong, Jong Lee, Aviral Kumar, Luowei Zhou, Jonathan Evens, William Isaac, Geoffrey Irving, Edward Loper, Michael Fink, Isha Arkatkar, Nanxin Chen, Izhak Shafran, Ivan Petrychenko, Zhe Chen, Johnson Jia, Anselm Levskaya, Zhenkai Zhu, Peter Grabowski, Yu Mao, Alberto Magni, Kaisheng Yao, Javier Snaider, Norman Casagrande, Evan Palmer, Paul Sugan- than, Alfonso Castaño, Irene Giannoumis, Wooyeol Kim, Mikotaj Rybiński, Ashwin Sreevatsa, Jennifer Prendki, David Soergel, Adrian Goedeckemeyer, Willi Gierke, Mohsen Jafari, Meenu Gaba, Jeremy Wiesner, Diana Gage Wright, Yawen Wei, Harsha Vashisht, Yana Kulizhskaya, Jay Hoover, Maigo Le, Lu Li, Chimezie Iwuanyanwu, Lu Liu, Kevin Ramirez, Andrey Khorlin, Albert Cui, Tian LIN, Marcus Wu, Ricardo Aguilar, Keith Pallo, Abhishek Chakladar, Ginger Perng, Elena Allica Abellan, Mingyang Zhang, Ishita Dasgupta, Nate Kushman, Ivo Penchev, Alena Repina, Xihui Wu, Tom van der Weide, Priya Ponnapalli, Caroline Kaplan, Jiri Simsa, Shuangfeng Li, Olivier Dousse, Fan Yang. Jeff Piper. Nathan Ie, Rama Pasumarthi, Nathan Lintz, Anitha Vijayakumar, Daniel Andor, Pedro Valenzuela, Minnic Lui, Cosmin Paduraru, Daiyi Peng. Katherine Lee, Shuyuan Zhang, Somer Greene, Due Dung Nguyen, Paula Kurylowicz, Cassidy Hardin, Lucas Dixon, Lili Janzer, Kiam Choo, Ziqiang Feng, Biao Zhang, Achintya Singhal, Dayou Du, Dan McKinnon, Natasha Antropova, Tolga Bolukbasi, Orgad Keller, David Reid, Daniel Finchelstein, Maria Abi Raad, Remi Crocker, Peter Hawkins, Robert Dadashi, Colin Gaffney, Ken Franko, Anna Bulanova, Rémi Leblond, Shirley Chung, Harry Askham, Luis C. Cobo, Kelvin Xu, Felix Fischer, Jun Xu, Christina Sorokin, Chris Alberti, Chu-Cheng Lin, Colin Evans, Alek Dimitriev, Hannah Forbes, Dylan Banarse, Zora Tung, Mark Omernick, Colton Bishop, Rachel Sterneck, Rohan Jain, Jiawei Xia, Ehsan Amid, Francesco Piccinno, Xingyu Wang, Praseem Banzal, Daniel J. Mankowitz, Alex Polozov, Victoria Krakovna, Sasha Brown, MohammadHossein Bateni, Dennis Duan, Vlad Firoiu, Meghana Thotakuri, Tom Natan, Matthieu Geist, Ser tan Girgin, Hui Li, Jiayu Ye, Ofir Roval, Reiko Tojo, Michael Kwong, James Lee-Thorp, Christopher Yew, Danila Sinopalnikov, Sabela Ramos, John Mellor, Abhishek Sharma, Kathy Wu, David Miller, Nicolas Sonnerat, Denis Vnukov, Rory Greig, Jennifer Beattie, Emily Caveness, Libin Bai, Julian Eisenschlos, Alex Korchemniy, Tomy Tsai, Mimi Jasarevic, Weize Kong, Phuong Dao, Zeyu Zheng, Frederick Liu, Fan Yang. Rui Zhu, Tian Huey Teh, Jason Sanmiya, Evgeny Gladchenko, Neje Trdin, Daniel Toyama, Evan Rosen, Sasan Tavakkol, Linting Xue, Chen Elkind, Oliver Woodman, John Carpenter, George Papamakarios, Rupert Kemp, Sushant Kafle, Tanya Grunina, Rishika Sinha, Alice Talbert, Diane Wu, Denese Owusu-ARe Owusu-Afriyie.
16

---

Cosmo Du, Chloe Thortton, Jordi Pont-Tuset, Pradyumna Narayana, Jing Li, Saaber Fatehi, John Wieting, Omar Ajmeri. Benigno Uria, Yeongil Ko, Laura Knight, Amélic Héliou, Ning Niu, Shane Gu, Chenxi Pang, Yeqing I.i, Nir Levine, Ariel Stolovich, Reheca Santamaria- Sonam Goenka, Wenny Yustalim, Robin Strudel, Ali Elqursh, Charlie Deck, Hyo Lee, Zonglin Li, Kyle Levin, Raphael Hoffmann, Dan Holtmann-Rice, Olivier Bachem, Sho Arora, Christy Koh, Soheil Hassas Yeganch, Siim Pöder, Mukarram Tariq, Yanhua Sun, Lucian Ionita, Mojtaba Seyedhosseini, Pouya Tafti, Zhiyu Liu, Anmol Gulati, Jasmine Liu, Xinyu Ye, Bart Chrzaszez, Lily Wang. Nikhil Sethi, Tianrun Li, Ben Brown, Shreya Singh, Wei Fan, Aaron Parisi, Joe Stanton, Vinod Koverkathu, Christopher A. Choquette- -hho, Yunjie Li, TJ Lu, Abe Ittycheriah, Prakash Shroff, Mani Varadarajan, Sanaz Bahargam, Rob Willoughby, David Gaddy, Guillaume Desjardins, Marco Cornero, Brona Robenek, Bhavishya Mittal, Ben Albrecht, Ashish Shenoy, Fedor Moiseev, Henrik Jacobsson, Alireza Ghaffarkhah, Morgane Rívière Alanna Walton, Clément Crepy, Alicia Parrish, Zongwei Zhou, Clement Farabet, Carey Radebaugh, Praveen Srinivasan, Claudia van der Salm, Andreas Fidjeland. Salvatore Scellato, Eri Latorre-Chimoto, Hanna Klimczak-Plucifska, David Bridson, Dario de Cesare, Tom Hudson, Piermaria Mendolic- chio, Lexi Walker, Alex Morris, Matthew Mauger, Alexey Guseynov, Alison Reid, Seth Odoom, Lucia Loher, Victor Cotruta, Madhavi Yenugula, Dominik Grewe, Anastasia Petrushkina, Tom Duerig, Antonio Sanchez, Steve Yadlowsky, Amy Shen, Amir Globerson, Lynette Webb, Sahil Dua, Dong Li, Surya Bhupatiraju, Dan Hurt, Haroon Qureshi, Ananth Agarwal, Tomer Shani, Matan Eyal, Anuj Khare, Shreyas Rammohan Belle, Lei Wang, Chetan Tekur, Mihir Sanjay Kale, Jinliang Wei, Ruoxin Sang, Brennan Sacta, Tyler Liechty, Yi Sun, Yao Zhao, Stephan Lee, Pandu Nayak, Doug Fritz, Manish Reddy Vuyyuru, John Aslanides, Nidhi Vyas, Martin Wicke, Xiao Ma, Evgenii Eltyshev, Nina Martin, Hardie Cate, James Manyika, Keyvan Amiri, Yelin Kim, Xi Xiong, Kai Kang, Florian Luisier, Nilesh Tripuraneni, David Madras, Mandy Guo, Austin Waters, Oliver Wang, Joshua Ainslie, Jason Baldridge, Han Zhang, Garima Pruthi, Jakob Bauer, Feng Yang. Riham Mansour, Jason Gelman, Yang Xu, George Polovets, Ji Liu, Honglong Cai, Warren Chen, XiangHai Sheng. Emily Xue, Sherjil Ozair, Christof Angermueller, Xiaowei Li, Anoop Sinha, Weiren Wang. Julia Wiesinger, Emmanouil Koukoumidis, Yuan Tian, Anand Iyer, Madhu Gurumurthy, Mark Goldenson, Parashar Shah, MK Blake, Hongkun Yu, Anthony Urbanowicz, Jennimaria Palomaki, Chrisantha Fernando. Ken Durden, Harsh Mehta, Nikola Momchev, Elahe Rahimtoroghi, Maria Georgaki, Amit Raul, Sebastian Ruder, Morgan Redshaw, Jinhyuk Lee, Denny Zhou, Komal Jalan, Dinghua Li, Blake Hechtman, Parker Schuh, Milad Nasr, Kieran Milan, Vladimir Mikulik, Juliana Franco, Tim Green, Nam Nguyen, Joe Kelley, Aroma Mahendru, Andrea Hu, Joshua Howland, Ben Vargas, Jeffrey Hui, Kshitij Bansal, Vikram Rao, Rakesh Ghiya, Emma Wang, Ke Ye, Jean Michel Sarr, Melanie Moranski Preston, Madeleine Elish, Steve Li, Aakash Kaku, Jigar Gupta, Ice Pasupat, Da-Cheng Juan, Milan Someswar, Tejvi M., Xinyun Chen, Aida Amini, Alex Fabrikant, Eric Chu, Xuanyi Dong, Amruta Muthal, Senaka Buthpitiya, Sarthak Jauhari, Nan Hua, Urvashi Khandelwal, Ayal Hitron, Jie Ren, Larissa Rinaldi, Shahar Drath, Avigail Dabush, Nan-Jiang Jiang, Harshal Godhia, Uli Sachs, Anthony Chen, Yicheng Fan. Hagai Taitelbaum, Hila Noga, Zhuyun Dai. James Wang, Chen Liang, Jenny Hamer, Chun-Sung Ferng, Chenel Elkind, Aviel Atias, Paulina Lee, Vít Listik, Mathias Carlen, Jan van de Kerkhof, Marcin Pikus, Krunoslav Zaher, Paul Müller, Sasha Zykova, Richard Stefanec, Vitaly Gatsko, Christoph Hirnschall, Ashwin Sethi, Xingyu Federico Xu, Chetan Ahuja, Beth Tsai, Anca Stefanoiu, Bo Feng, Keshav Dhandha- nia, Manish Katyal, Akshay Gupta, Atharva Parulekar, Divya Pitta, Jing Zhao, Vivaan Bhatia, Yashodha Bhavnani, Omar Alhadlaq, Xiaolin Li, Peter Danenberg, Dennis Tu, Alex Pine, Vera Filippova, Abhipso Ghosh, Ben Limonchik, Bhargava Urala, Chaitanya Krishna Lanka, Derik Clive, Yi Sun, Edward Li, Hao Wu, Kevin Hongtongsak, Ianna Li, Kalind Thakkar, Kuanysh Omarov, Kushal Majmundar, Michael Alverson, Michael Kucharski, Mohak Patel. Mudit Jain, Maksim Zabelin, Paolo Pelagatti, Rohan Kohli, Saurabh Kumar, Joseph Kim, Swetha Sankar, Vincet Shah, Lakshmi Ramachandruni, Xiangkai Zeng. Ben Bariach, Laura Weidinger, Amar Subramanya, Sissie Hsiao, Demis Hassabis, Koray Kavukcuoglu, Adam Sadovsky, Quoe Le, Trevor Strohman, Yonghui Wu, Slav Petrov, Jeffrey Dean, and Oriol Vinyals. Gemini: A family of highly capable multimodal models, 2024.
[45] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Ilama: Open and efficient foundation language models, 2023.
17

---

[46] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv: 10. 16944, 2023.
[47] Guan Wang. Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. Openchat: Ad- vancing open-source language models with mixed-quality data. arXiv preprint 1rX1, 2023.
[48] Tengyang Xie, Ch Cheng, Nan Jiang, Paul Mineiro, and Alckh Agarwal. Bellman- consistent pessimism for offline reinforeement learning. Advances in neural information pro- cessing systems, 34:6683-6699 2021.
[49] Ming Yin and Yu-Xiang Wang. Towards instance-optimal offline reinforcement learning with pessimism. Advances in neural information processing systems, 34:4065 -0788, 2021.
[50] Ming Yin, Yu Bai, and Yu-Xiang Wang. Near-optimal provable uniform convergence in of- fline policy evaluation for reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pp. 1567-1575. PMLR, 2021.
[51] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint wrXi:2230.055302, 2023.
[52] Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling- 7b: Improving IIm helpfulness and harmlessness with rlaif, November 2023.
[53]
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b 7b: Improving Ilm helpfulness and harmlessness with rlaif, 2023.
[54] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 1909:: 2019.
18

---

Contents
Introduction
Problem Formulation: Alignment via Controlled Decoding
2.1
Token-level Markov Decision Process
2.2
Principled Decoding for LLM Alignment
3
Proposed Approaches: Alignment via Transfer q
3.1 Direct Transfer Decoding
3.2 Indirect Transfer Decoding
3.3 Theoretical Results and Insights
Experimental Evaluations
4.1 Transfer Q": Evaluations with Direct Transfer Decoding
8
4.2 Transfer q": Evaluations with Indirect Transfer Decoding
................................
9
5 Conclusions
A Limitations
21
Broader Impact
Software and Hardware
21
D Notations
21
E Detailed Description of Related Works
21
Discussion of RLHF Pipeline
22
Analysis Details in Section 3.3
23
G.1 Proof for the Suboptimality gap
23
G.2 Proof for the KL divergence
25
H Partition Function for Implicit Transfer
27
Additional Details of the Experiments
27
L.1 Reward normalization
27
1.2 Synthetic Indirect Transfer Setup
27
I.3 Real Indirect Transfer Setup
27
Additional Experimental Evaluations
28
J.1
Additional Evaluations of Direct Transfer
28
J.2 Additional Evaluations of Indirect Transfer
28
19

---

J.3 Ablations
28
K Examples of the Generated text of our Algorithm compared Baselines
28
20

---

Appendix
A Limitations
In this paper, we introduce a novel decoding strategy for Al alignment. While we demonstrate the effectiveness of approach over baseline decoding strategies, TQ" suffers from increased time complexity. Specifically, our approach generates trajectory responses for each token. Consequently, compared other baseline using heuristic approaches such ARGS [26], TQ" incurs greater time complexity. However, this limitation can be mitigated by training small value function adapter as discussed in Mudgal et al. [33].
B Broader Impact
Our algorithm Transfer Q* focuses on the broader goal of aligning Al systems with human values and preferences, which is crucial Al technologies become integrated into daily life. By avoiding extensive retraining of large models, Transfer Q can potentially reduce the environmental footprint associated with AI development. We do not anticipate any potentially harmful consequences of our work.
C Software and Hardware
We run all experiments with Python 3.7.4 and PyTorch 1.9.0. For all experimentation, we use two Nvidia RTX A6000 GPUs.
D Notations
In this section, we summarize the notations used in this work for quick reference.
Table Notations. This table presents the notations we used this work.
Implication
Notation
Vocabulary
V
Prompt
x
Response
y
Token in response
Yt
End of Sentence token
EOS
a concatenation b
[a, bj
Response till token / Trajectory level reward
Ye
[V] V2,
; -/-
r r(x, y)
Token level reward
R(x, Ut)
Token level state
= Xx, Yell
Token level action
ag
Trajectory level LLM policy
P x)
Token level LLM policy
. 8t)
Action value function corresponding to R under policy
QF
Value function corresponding R under policy
VF
E Detailed Description of Related Works
In this section, we summarize the related works in the literature.
Reinforcement Learning from Human Feedback. Recent advancements in large language models (LLMSs) have increasingly leveraged reinforcement learning from human feedback (RLHF) to enhance model performance and alignment with human preferences. Foundational works to demonstrate RLHF includes Christiano et al. [11], Bahdanau et al. [4], Ibarz et al. [22], MacGlashan et al. [30].
21

---

Specifically, Christiano et al. proposed an approach where reward model is trained based on human preferences between pairs of model outputs, guiding the reinforcement learning process to produce more desirable outcomes. Stiennon et al. [40], Ziegler et al. [54], Snell et al. [39], Nakano et al. [35] extended the concepts of RI.FF to train a language model. Notably, Stiennon et al. [40] established that training LLMs using RLHF approaches significantly improved task performance compared to supervised fine-tuning. In this section, we broadly classify RLHF techniques into two distinct categories, namely Alignment via Fine-tuming (training time) and Alignment via Decoding (inference time), discussed in detail as follows.
Alignment via Fine-tuning, A prevalent approach involves employing an RLHF framework wherein a reward model is trained on human feedback, and proximal policy optimization (PPO) is used to derive the aligned policy model [5. 37, 2, 19]. Despite its effectiveness, the PPO training paradigm is often criticized for its instability and considerable resource requirements. Addressing these concerns, recent contributions have explored alternative supervised fine-tuning methods. Notably, Rafailov et al. [38] leveraged the Bradley- Terry [8] model to parameterize the reward policy, thereby obtaining the aligned model by optimizing simple classification loss. Liu et al. [28] proposed the Chain of Hindsight approach which allows the model to learn from any form of feedback and does not require specific hand-picked model generations for alignment. Yuan et al. [51] introduced framework that aligns model probabilities of multiple responses with human preferences through a ranking loss. Dong al. [14] advocated for applying supervised fine-tuning on the highest reward samples. Lastly, Chen et al. [10] proposed a self-play based fine-tuning mechanism that refines capabilities by having the model engage with instances of itself, thus obviating the need for an explicit human-annotated preference dataset. While these fine-tuning-basee
methods have proven effective for aligning LLMs, they also pose significant computational demands and assume white-box access to model parameters to update them, which may not always be available. Notably, TQ" diverges from these training tuining-based approaches, by providing new decoding- time framework align language models without requiring expensive fine-tuning.
Alignment via Decoding. The work by Khanov et al. [26] is one of the first study the integration of alignment procedure directly into the decoding process. At each decoding step, Khanov et al. [26] proposed to adjust the generation probabilities of the model based on feedback from reward model. Huang et al. [21] re-conceptualized the eext-generation process as a search problem with L.MSs as search agents. The state space of the search problem is defined as the sequences of tokens and the action set consists of the vocabulary of tokens. Specifically, given prompt, Huang et al. employs a heuristic-guided search to generate responses. Liu al. [29] proposed a technique for multiplicatively reweighing the probability of each potential generation with an importance ratio derived from the aligned and reference models. A related approach is controlled decoding (CD) [33]. a framework that approximates the decoding problem by collecting samples from the reference based language model.
F
Discussion of RLHF Pipeline
First, we discuss the standard RLHF with Reward Learning and Policy Optimization pipeline as used in Ziegler et al., which has also been adopted in subsequent work [42, 5, 37]. It usually consists of three phases: (1) supervised fine-tuning (SFT); (2) preference sampling and reward learning, and (3) reinforcement-larninng optimization.
(1) SFT phase:
RLHF typically begins with a generic pre-trained LM, which is fine-tuned with supervised learning (maximum likelihood) on a high-quality dataset for the downstream task(s) of interest, such as dialogue, instruction following, summarization, etc., to obtain a model Part.
(2) Reward Modelling Phase: In the second phase, the SFT model is prompted with prompts x E P to produce pairs of answers (2) ~ Pete x) induced by toekn level Tyfe (y x). These are then presented to human labelers who express preferences for one answer, denoted as y > Y: x where yn and yı denotes the preferred and dispreferred completion amongst (y Y2) respectively. The preferences are assumed to be generated by some latent reward model r" r'(y,, x), which we do not have access to. There are a number of approaches used to model preferences, the Bradley-Terry (BT) [8] model being popular choice. The BT model stipulates that the human preference distribution p' can
22

---

be written as:
exp (r" (y1,
pd [y1 Y2
x)
(21)
exp (r* (y1. exp (r (y2 x))
Assuming access to static dataset of comparisons D = fx(1) are sampled from Tist and the preference feedback is samples from true pd (constitute the human feedback), we can estimate r via maximum likelihood. Framing the problem as binary classification we have the negative log-likelihood loss:
[xx00 yw :
(2)
where the trajectories
[rr, D) (x [log a (r x) r(y),)))]
(22)
where a is the logistic function. We can write the population version of the loss function in (22) as
((r) = - Exx Px-Peurrrpes x))))=>>>))>p [log (r(y x) ())]
(23)
We want to highlight the dependence of the loss function of on the language model Pare x).
(3) Fine-Tuning with RL Phase: During the RL phase, feedback to the language model. In particular, we formulate the following optimization problem
the learned reward function to provide
max Ex-P Eyyet- x)) r(y, x)]
31DxL [p( x) || Parc( [x]]]
(24)
where Patt is induced by (y x). In the above formulation, / is a parameter controlling the deviation from the SFT model Patt. As derived in recent work by Rafailov et al. [38], closed-form expression of the optimal solution of problem in (24) is given by
(ylx) Z(x) Patt (ylx) exp (3r
(25)
where (xx) is the normalizing constant for x. It is important to note that the optimal policy in (25) is defined at the trajectory level (y the generated trajectory for given x) and not the token level for each which we denote by
G Analysis Details in Section 3.3
In this section, we provide the proof of Theorem 1. Theorem (Restatement of Theorem 1). For the proposed Implicit Q* Algorithm 1, the following results hold.
(1) Suboptimality gap for all is upper bounded as
Sub Bub-Gapfse Gap(xx) S [9Dxu [p" [-] x-)x) ll Pafe(-) x)] aha
(26)
where 3 is defined in (9) for implicit policy, and a is defined in (17),or the proposed decoding process.
In (26), (x(x) = -T'1 Ez Eg (Pxi"pi,(((x)) (1xx) [Dxr.] [=igg(->x, x² (T =*)]] 2 0.
(2) Assume reward satisfies 0 < r S Twax then the Divergence to reference- policy given by
Der (ping (x)) x)) S
1 + T
3
Tean-
(27)
G.1 Proof for the Suboptimality gap
We first provide the proof for the suboptimality gap. For notation convenience, in this proof we rename suboptimality-ee gap defined in (18) follows:
( = V (x) Vxig(x).
(28)
Recall the definition of V and Vang. we have A can be equivalently expressed as:
A (xx Exznr (-1x) [[xpp [r(xx =]. 7)] E Eg [rrois (+ (x..) [r([x,=), +)]],
(29)
where p² represents the distribution over the trajectories induced by the optimal token level policy n², and Ping denotes the distribution over the trajectories induced by the proposed token level policy
23

---

(-x). To proceed next. we add and subtract the term Ez (1xx [Br-pul (xx,) 2), the right hand side of (29) to obtain
in
A (x) Egont (1/x)) [rr (xx
((ss) (1xx,)
r]]] - Ba (x) (
(30)
=x)
E () Be (-bx,)) (r( 2), 7)]] Rewnang (Brx (|, =),
=:A2(x)
In order derive an upper bound on the suboptimality gap (x), will develop upper bounds for
Ai (x) and (x(x) separately below.
Upper bound on A (x)): Let us consider the term A (x) in (30) as
A1 (x) = Eg--- (+1x) (rxpp (Ix,,))/(((X x), []]] - E E (-bx) (Br [r[ 2). r)]]
(31)
= Er (-1x) (xx 7)] EEpppt x) [x((x T)},
where PBL. is the trajectory-level policy that maximizes the constrained objective (9) (with 8, replaced by x). This directly implies
Bynp' (+bx) [r(x, r)] -3D [p" x) || Part [-])
SB, Paa (x1x) [r(x, r)] (8K.. [paa (+)xx |I Pate(1l x)]
< <Br~pe (1)) [r(x,T]].
where the last step uses the non-negatinn non-ncgativity of the KL-divergence. This equivalent to
ETap' (-1x) [r 7)] (a [r(x,)]] s 3DkL(o" (- || Pare(-) x)].
Now plug the above into (31), obtain
A1 S PD x) P-rc(-) x)]
(32)
Upper bound on A2(x): We expand the terms A2 (x) as follows
(x = E Ezwxal xc) [Br (-Ix.)) [r[[, =|, T]] Exwnit (ob0) E (xx.)) (r.)))/(((( +]]
(x.:)
(33)
41
+ Bzwning(())) [
BB [r(xx, 2), r)]]
Ag
where we add and subtract the term Eawniy (-1x) Pra px, [r[[x,][]]]]] 2/, r)]]
Next, note
A (e By By
(xxx)) (r(()))(((() 2), r)] - E Br (r(xx +],])]
(34)
where the inner term inside the first expectation is denoted by Ag. Note that Ag represents the similar structure as Ag in equation (30) conditioned on (x, 2), where is the next token. We proceed similarly as before and for simplicity of notations represent x' : [x, 2] and
write
Aa(x,21 2) Egg (-lx') [Ernpetix (
7]]] E Egg [Eropau (r((xl
4
Berrwing((- [By +)]
Egt Beg x²) ( ///lr///(///////1/(l21 =11,,)]]
(35)
24

---

Therefore, we can keep the iteration and drive at (I' is the termination step)
42(x) = + E - Ax, =)]
(x) + ( A 5 (x, z) A 2)]
-x(xx) + - (x) | z)] + Ez ag ( A 2)]
=A)(x) + -
[x, =)] E xx Bi [ (x'x22))
=x](x) + E2, B xx)
x, =)] + Eg kig (- (xe) B (1xx) (x(xx
+B o (1xx")
B2 Lin ( | (x" ,²")]
+ E.." ~o (1xx)) B Be [ ,²"]
Σ E a I (1, 2(t=xx,
where Piig (-x) is the algorithm trajectory distribution and 2:11 empty, 222 2, 223 (2, and so on. Here (note z represents single token and denotes the completion trajectory)
A ( = _221-11 1) -Eg (x,22
[Eroe (-1x, [r(x, , 22 b)]]] EEstnnng (-x,, Berpp [r(,,r((___ (xx,22 ,)]].
Denote x" = [x, 2,, then above is equivalent to
(x(x11 =E, =E ( (xt-1)
[Erpput [r(x',
;oxx(,,))]] [)]] - Ey Bu
Tie (1xxt (-1xt-1) () (rrpput (xe)) (r(")[[('''1)
-E, Eet (xt-1) [E [r(x, r]]] - a DKi. [TLL (xxxx=1)) TBL (-1x(-1)) [x{-")]
Bethaaiee (-bet bet--y ( (xe) [r(x', []]] - a lKK Tiig ((xx-11) TaL (-1x [xx]_)]]]
-a DxL Mhag((1xx >) TEL (-1x" ')] -a DxL ( (-1x² 1l (-1x-=1)) ')]
where the last inequality uses Thig the optimizer for objective (17). Hence, we have
T
(x
Σ E.at- Hop
2: 2it- -a
B -pp x)) [DKr. (magg( (-1x, = =2:1-111 [|/TTLL [//TTa (-1x, z 2: 22:1-1]]
1
Combining above with (32), and denote z² = 22101 we obtain
T-1
((x) < 3DLL Ip" (-) x) Parc(-] x)] - a
E Eginpi (11x)) [Kx (iag (-1x, 1T/ x*]]].
This completes the proof for suboptimality.
Remark 1. We do mention that in the practical implementation of the Algorithm I is sampled-based. but our analysis considers the population level performance. Indeed, adapting our analysis the data driven perspective and obtaining finite sample guarantee is standard, since the suboptimality gap Sub Gap we defined is identical the existing offline reinforcement learning literature such as [48, 50, 49].
G.2 Proof for the KL divergence
To derive upper-bound on the KL divergence of the distribution over the responses generated by proposed algorithm with the reference policy given the prompt, we first expand upon the definition of DxL. (Pas x), Parc(-] x))
Ping ( x)
DeL. (Piag(1) x), (- x)) Erren, (-1 x) log
(36)
Pare (T) x)
Next, we first expand upon the trajectory distribution induced by our algorithm
Pkag (Tl x) = Tbng (V1 ( (y2 v1) Ting (Url V/, VT -
(37)
25

---

where the trajectory [y1, U2, ,3r given the prompt x and corresponding token level algo- rithm's policy is given by Trig (g..) We know from the definition of our algorithm's policy equation (12) that Tiig(2) (2|x) = [x(x) (=|xx exp ( 1-0 z)). Now, using that we expand the
equation (37)
PAig (Tl x) = T TAg (vn ( ( (922 2z v1) Fig (UT Vr, 9r-11
(38)
TEL (M x) exp 1 (x, v1)) Ca(x, 31) TBL (2/2 x. V1) exp ( 1Q" (x, y1, v1.
C. (x)
C. (x, yr, y2 1/T-1)) (ur xx, y/, ". -'--1 exp ! IQ" (x, y1, V2 yr-1))
where we expanded our algorithm's policy from definition in equation (37) and C. are normalizing factors. Next,
Piig(T| x) = (v1 Tat. (1/ x, v1) ... TaL (UT |x, y1, 1....-..--- 9T)
(39)
exp _ (xx + IQ" (x, 8/. 12)) + IQ" (x, 01.02 .. --.r--)-1)))
(x) (x, 31) y1 (x, V1. U2 yr-1))
where we simplified the expression by re-arranging the product into similar terms over the trajectory. Next, the definition of the trajectory distribution (r)(T from the token level policy TEL (-lar) to get
(rag(F| x) = PEL(TT x)
(40)
exp - (1Q"(x,))) + IQ" (x. +... ....IQ((( (x, 11.11:22 "..:0/--- v1--)))
Cx(x) (x, C1 V1, /2 yr-1))
Furthermore, we expand the definition of Pau (T(ar) from equation (25) to obtain
(Tig(T) x) = Pere (T| x)
(41)
1 x expl r((x r) +
1 (1Q" (x, TQ,, (x, + ... IQ" U1, .. "..."'/..77 yrr-))))
x C(x) 21 C. U1, v1,32 VT-1))
Püg(rl x) Now, compute the ratio log (are xj
from equation (41) as
log Piig (T x) 1 3 r) + _ (
IQ"
(2"x(xx (x, M1, v2)) + ... ......1Q" v1. v... -Vr-))))
Part (T] x)
(42)
log Z(z) - log (x) - log Ca (x. ... log (x, V/. 3r-1))
Since reward 0 < < Teax. have Z(x) Eyrper (exp( (x, T))] 2 1. and this implics
- log ((s) < 0, similarly log (x)) < so we arrive at
log Phig(T (Tl x) (rre(T] x)
< B max + Tranax
(43)
which implies
DK (Pāgg() (.)))(),)ttttt Porce x)) < nax + - Traax
(44)
26

---

H
Partition Function for Implicit Transfer
Here, we establish that the ratio
(x(x)) Zrz (x)
is partition function. From (15), we note that
(x((x) = PeL (y|x) exp
(r(x,)) - (x)(
(45)
(yrc(y)x) exp ( (T exp - -ra((xx)
Zre (x)
Σ Pate(y)x) exp ())(x.)))
(46)
Zrn (x)
(x)
(47)
(1))
Hence proved.
I Additional Details of the Experiments
I.1 Reward normalization
To provide clearer comparison of results, normalize the average rewards. For example: let IDPo represent the average reward achieved by the DPO model across all generated responses to the test prompts. The normalized reward, îDPO, calculated as: ADPP now-rrsrr ensuring that the results are scaled relative to existing methods.
Tro" -Tant
1.2 Synthetic Indirect Transfer Setup
In Table 4, we summarize the different datasets and model architectures used in analysis of synthetic indirect transfer setups. We presented the results for the Ultrafeedback [12] dataset in Section 4.2. We compare the normalized average reward of different decoding policies on the Berkeley Nectar [53] dataset in Appendix J.1.
Table 4: Synthetic Transfer Setup. Summarizing of the datasets and model architectures used for experimental evaluations in synthetic indirect task.
Source Model Architectures
Dataset
SFT
Reward
Synthetic Setup-1
UltraFeedback Berkeley Nectar
Mistral- -Barral-BB-rr
Zephyr-7Bi 7B-a Starling-7B-o
Mistral-7B-o
Synthetic Sctup-2
OpenChat 3.5-7B
Mistral-7B-o
1.3 Real Indirect Transfer Setup
In Table 5, we summarize the different datasets and model architectures used in our analysis of real indirect transfer.
Table 5: Real Transfer Setup. Summarization of the datascts and model architectures used for experimental evaluations in real indirect task.
Source Models
Target Model
Dataset
SFT
DPO
Reward
Reward
Real Sctup-1
UltraFoedback
Mistral-7B-a Pythia-6.9B
Zephyr-7B-o Pythia-6.95
Mistral-7B-o Pythia-6.9B
Gemma-7B
Real Semp-2
HH-RLHF
Llama-3B
27

---

Additional Experimental Evaluations
J.1 Additional Evaluations of Direct Transfer
In Figure 2, we report the average reward obtained by different decoding strategies on three evaluation setups outlined in Table 1. We present the results for the additional setups mentioned in Figure 6. Con- sistent with our original findings, TQ* outperforms all the competitive baselines across all evaluation setups.
Average Reward
Average Reward
1.0
I
Average Reward
0.8
0.8
0.8
0.6
0.6
0.6
0.2
0.2
0.0
ARGS (SFT)
DPO
ARGS
CD
TO" (Qurs)
0.0
ARGS (SFT)
DPO
ARGS
CD
TO" (Ours)
0.0
ARGS (SFT)
DPO
ARGS
CD
TO (Ours)
(DPO)
(DPO)
(DPO)
Methods
Methods
Methods
(a) Evaluation
(b) Evaluation 5
(c) Evaluation
Figure 6: We present the normalized average reward values obtained using the setups outlined in Table 1. ARGS (SFT) and ARGS (DPO) refer the reward modeling approach described in [26] the SFT and DPO model respectively. observe that across all setups, TQ consistently outperforms other competitive baselines summarized in Table 1, demonstrating its superior efficacy.
J.2 Additional Evaluations of Indirect Transfer
Figure 7, we report the average reward obtained by different decoding strategies on the synthetic transfer setup the Berkeley Nectar [53] dataset. Consistent with our findings on Ultrafeedback [12] dataset Section 4.2, TQ* outperforms all the competitive baselines.
0.14
Average Reward
1.00
0:175
1.0
Source
0.8
0.150
Source
0.8
0.10
Target
0.6
0 12125
Target
0.6
0.08
0.44
0.075
0.4
0.04
0.050
0.2
0.00
-20
10
0 10 20
ARGS ARGS (SFT) (DPO)
TQ (Ours)
0.000
-20
10
0 10
20
0.0
ARGS ARGS
(SFT) (DPO)
(Ours)
TQ
Reward Value
Methods
Reward Value
Methods
(a)
(b)
(c)
(d)
Figure 7: Evaluation Synthetic Indirect Transfer Tasks. plot the distribution of reward values for the source and two transfer tasks on Berkeley Nectar in (a) and (c). In plots (b) and (d), we compare the normalized average reward scores for competitive decoding strategies. We represent the variant of our decoding
strategy with direct transfer as DT. We observe that TQ" consistently outperforms the other baselines.
JJ3 Ablations
In Section 4.1 and Section 4.2, we demonstrated the superior efficacy of TQ" compared to the baselines through comprehensive evaluation. In this section, we present ablation study different hyperparameters, such as the number tokens sampled (k), and the decoding alignment parameter (a) defined in Algorithm We report the ablation results A and a in Figure 8. Specifically, measure the diversity and coherence of the generated responses based on different combinations of the hyperparameters k and a. This analysis based on the Evaluation- setup defined in Table Our observations indicate that using = 10 and a 1.0 produces responses with the highest diversity and coherence.
K Examples of the Generated text of our Algorithm compared to Baselines
In this section, we show some representative examples of generated responses of different decoding strategies. The prompts are randomly selected from the test of the Ultrafeedback dataset [12].
28

---

0.9
0.9
0.8
0.8
Diversity
0077
Coherence
0.7
0.6
0.6
0.5
0.5
0.4
0.4
0.3
0.5
1.0
1.5
2.0
2.5
0.3
0.5
1.0
1.5
2.0
2.5
k=
10
k=
50
We measure diversity and coherence for responses generated using different combinations of hyper- parameters and
Figure 8:
Example-1
Can you determine the missing number in the sequence: 6, 14, 30, 62, _ chal- lenge you to figure out the pattern used to generate each number in the sequence. Use your mathematical reasoning skills to predict the next number.
Prompt:
ARGS:
Good luck! The pattern used to generate each number in the sequence is the sum of the two preceding numbers. The first two numbers are 2 and 6, and the sum of these two numbers is 8. The next number in the sequence is 14, which the sum of and 8. The sum of 14 and is 22, and the number in the sequence 30, which is the sum of 14 and 22.
CD-": The sequence is example of geometric sequence, where each term is found by multiplying the previous term by a fixed number called the common ratio. In this case, the common ratio we can use the formula for the nth term of geometric sequence: a(1 r") p²))(1 r), where the first term, r the common ratio, and n the position of the term we want to find.
since each term is twice the previous term. To find the missing number,
TQ (Ours): The sequence
geometric series, where each term found by multiplying the previous term by 3. The first term is 2. The formula for geometric series is: a r where 'a' first term, 'r' the common ratio (the factor by which each term is multiplied), and 'n' is the position of the term in the sequence. In this case:
3 (the common ratio).
29

---

Example-2
You are given set rectangular tiles of varying sizes. Your task is to arrange them in specific pattern to create rectangle with the largest possible area. The catch is that each tile can only used once and must used its entirety (i.e., cannot be cut or modified). Additionally, you cannot overlap tiles or leave any gaps between them. Can you figure out the optimal arrangement tiles to create the largest rectangle? Good luck!
Prompt:
ARGS: solve this problem, we can use dynamic programming to find the optimal arrange- ment of tiles. We can define 2D array dp[i][j] where dp[i][j] represents the maximum of rectangle that can be formed using tiles of size x j. The base case when i= 0 or j= = O, in which case dp[il[i] O. For each tile size (i. j), can iterate over all possible positions (x, y) in the rectangle.
CD The optimal arrangement of tiles to create the largest rectangle would depend on the specific sizes of the tiles given. However, general strategy to maximize the area of the rectangle would be to start with the largest tile and then arrange the smaller tiles around it in way that covers much of the remaining space as possible. This would ensure that the largest tile is used to its full potential and that the smaller tiles are used efficiently to fill in any gaps.
TQ" (Ours): To find the optimal arrangement of tiles to create the largest rectangle, we need way that maximizes the number of smaller tiles
first find the largest tile and arrange that can be placed around it. Let's s say the largest tile has dimensions a b. We can place it the top left corner of our rectangle, and then try to fit as many smaller tiles around it as possible. To do this, we can start by placing the next largest tile below the largest tile, aligned with its left edge.
Example-3
Prompt: Explain why this statement is hyperbole: I'm so hungry I could eat horse.
Explain why this statement is hyperbole. I'm so cold I could freeze to death. Explain why this statement is hyperbole. I'm so tired I could sleep for a week. Explain why this statement is hyperbole. I'm so happy could burst. Explain why this statement is hyperbole. I'm so angry could kill someone. Explain why this statement is hyperbole. I'm so in love I could die. Explain why this statement is hyperbole. I'm bored could scream.
ARGS:
CD This statement is hyperbole because it exaggerates the speaker's level of hunger. While it possible for someone to be extremely hungry, is not physically possible to eat an entire horse. The statement meant emphasize the speaker's hunger and convey the idea that they are extremely hungry, but it is not meant to be taken literally.
TQ (Ours): This statement hyperbole because no one can physically eat an entire horse. The statement is an exaggeration to emphasize how extremely hungry the person is. It is unlikely that the person is actually that hungry, but the statement is used to convey the intensity of their hunger. The use hyperbole in this statement also adds humorous or lighthearted tone to the statement, making it more relatable and understandable to the listener reader.
30

---

Example-4
Prompt: Describe the phenomenon of carbon dioxide absorption the atmosphere.
ARGS: Carbon dioxide absorption in the atmosphere is natural process that occurs when car- bon dioxide (CO2) is absorbed by the Earth's 's surface and vegetation. This process is essential for maintaining the balance of CO2 the atmosphere and preventing it from accumulating to dangerous levels. Carbon dioxide absorption in the atmosphere occurs through a process called photosynthesis. During photosynthesis, plants and other organisms use energy from the sun to convert carbon dioxide and water into glucose and oxygen. This process releases oxygen into the atmosphere and removes carbon dioxide.
CD Carrbon dioxide absorption in the atmosphere refers to the process by which carbon dioxide (CO2) is taken up by the atmosphere from the Earth's surface. This process is essential for maintaining the balance of greenhouse gases the atmosphere, which is critical for regulating the Earth's 's temperature. The primary sources of carbon dioxide in the atmosphere are the burning of fossil fuels, deforestation, and other human activities that release large amounts of CO2 into the atmosphere. meant to be taken literally.
TQ (Ours): Carbon dioxide (CO2) is greenhouse gas that plays crucial role in the Earth's climate system. It is naturally present the atmosphere, but human activities such as burning fossil fuels, deforestation, and industrial processes have led to significant increase in CO2 concentrations. When CO2 is emitted into the atmosphere, it absorbs some of the infrared radiation (heat) that is emitted by the Earth's surface and re-emits it back towards the Earth. This process is known as the greenhouse effect.
Example-5
Prompt:
Classify the following incident as breach of protocol. Output 1 for breach, and O for breach: Using a school laptop for personal use.
ARGS:
Taking school laptop off the school premises. Taking school laptop off the school premises without permission. Taking school laptop off the school premises without permission and using it for personal use. Taking a school laptop off the school premises without permission and using for personal and downloading personal files onto the laptop. Taking a school laptop off the school premises without permission and using it for personal and downloading personal files onto the laptop and deleting school files from the laptop.
Using school laptop to access non-school websites. Using a school laptop to access social media sites. Using school laptop access email accounts associated with the school. Using school laptop to access personal files. Using a school laptop to access personal files stored on a USB drive. Using school laptop
CD
personal files stored cloud service. Using school laptop to access personal files stored on personal computer. Using school laptop access personal files stored on personal mobile device. Using school laptop to access personal files stored on a personal tablet.
TQ* (Ours): A student using a school laptop for personal use is a breach of protocol because it violates the school's policy on acceptable use of technology. The school laptop is provided for educational purposes, and using it for personal use compromise the security and integrity of the device, as well as consume valuable resources that could be used for educational purposes. This can also result in potential legal and financial consequences for both the student and the school, as well as damage to the school's reputation. Therefore, using a school laptop for personal use is breach.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly reflect claims and contributions in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in Appendix A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assump- tions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g.. if the approach was only tested on few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[Yes]
Justification:
We discuss the assumptions and complete proof for each theorem provided.
Guidelines:
32

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We report the detailed evaluation setups in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
the contribution
While NeurlPS does not require releasing code, the conference does require all submissions to pro- vide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-soure models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We submit the code with the submission.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines
guidde//ooeeuubmisssioo0000.... for more details.
33

---

While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https: bttppp://nissscc///e/e guidess/CodeSubmiistt asloiiiccy)) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We report all hyperparameters and evaluation details in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
We discuss statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, least for the experiments that support the main claims the paper.
The factors of variability that the error bars are capturing should be clearly stated (for exam- ple, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK report 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
asymmetric distributions, the authors should careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer re- sources (type of compute workers, memory, time of cxecution) needed to reproduce the experiments?
34

---

Answer: [Yes]
Justification:
We report the required software and hardware details in Section C.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, every respect, with the NeurIPS Sabllic/Ethissaiidee
Code of Ethics https://neuriii
Answer: [Yes]
Justification:
Our paper conforms NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impact in Section B.
Guidelines:
The answer NA means that there no societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g., dis- information, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point out. For example, it is legitimate to point out that an improvement the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g,, gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
35

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators. or scraped datasets)?
Question:
Answer: [NA]
Justification:
[NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite and properly credit for all the used assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and. if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. dataattassss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online. the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemmodel as part of their submissions via structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used. At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
36

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution. For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "Cd9npWVXS6", "reviewer": "Reviewer_ypg9", "summary": "The paper proposes a new estimation of Q function for controlled decoding. Instead of using a Q function derived from SFT model, the paper propose to use Q function derived from separate aligned models. Evaluation shows the proposed method can achieve higher reward in benchmarks.", "questions": "1. Did the authors compare decoding speed for different approaches? It would be useful to report even thought the proposed approach is slower.\n2. For gpt4 based evaluation, why the authors only report Win-Tie instead of splitting win and ties? In particular I am curious what is the percentage of ties for each comparison.", "limitations": "Limitations are discussed in the appendix.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of using aligned models to estimate Q is quite intuitive as these models closer to optimal policy than SFT model, also the paper proposed mathematical derivations and explanations on how and why aligned models are better for estimating Q.\n2. The benchmarks showed the proposed method tend to achieve higher reward, especially it outperforms the controlled decoding baseline along with other alignment methods.", "weaknesses": "1. The proposed approach seems computationally expensive to do decoding.\n2. The approach require accessing an already aligned model, this adds more requirements for using the method and limits it use cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Reviewer Summary:**  We sincerely thank the reviewer for the thoughtful review of our work, highlighting the strength and novelty in both our formulation and experimental design in leveraging available aligned models for principled decoding. We address all other comments one by one as follows. \n\n\n\n>**Weakness 1:** The proposed approach seems computationally expensive to do decoding.\n\n**Response to Weakness 1:**  We thank the reviewer for this important point. We agree with the reviewer that performing the proposed principled decoding would indeed add an additional computational overhead at the inference time. But we would like to emphasize that ***our main focus and contribution*** in this work is to provide principled method to perform LLM alignment via decoding and provide theoretical gaurantees. \n\nMoreover, our decoding method has a comparable inference time to the existing state-of-the-art Controlled Decoding (CD) [A]. For the practical implementation, similar to CD method in [A],  one can train a small Q-function adaptor offline, which would allows for faster inference time.  This significantly reduces time complexity, similar to ARGS [B], which only introduces a constant factor of k (top-k tokens) over classical decoding methods. \n\nAnother important point to highlight is that there are several critical scenarios—such as  reasoning, mathematical puzzles, chess, and coding where accuracy is more crucial than inference time. In these situations, principled decoding can provide substantial benefits.\n\n>**Weakness 1:** The approach require accessing an already aligned model, this adds more requirements for using the method and limits it use cases.\n\n\n**Response to Weakness 2:** Thank you for raising this critical point. We remark that our proposed decoding method does not require access to a model specifically aligned to a particular reward function. Instead, interstingly, we can leverage any existing aligned model (BL), even if it is aligned to a different reward (Indirect transfer) and thus is not inherently restrictive. Therefore, utilizing already existing aligned model (such as on Hugging face) for improved decoding is the strength of our approach. We show this via extensive experiments in our paper. \n\n\n>**Question 1 :** Did the authors compare decoding speed for different approaches? It would be useful to report even thought the proposed approach is slower.\n\n**Response to Question 1:**  Thank you for pointing this out. We have now included a comparison of the inference times for all baseline decoding algorithms in the table below. Specifically, we report the inference time for decoding a single prompt, based on the system configuration detailed in Appendix C.\n\n| Algorithm              | Inference Time | Avg Reward |\n|------------------------|----------------|------------|\n| Naive Decoding         | 3s             | 0.13        |\n| ARGS                   | 7s             | 0.29        |\n| $\\text{CD}^{--}$                   | 40s            | 0.71       |\n| $\\texttt{TQ}^{\\star}$ (Ours)    | 41s            | 1.0        |\n\n\n\nIt is evident that $\\texttt{TQ}^{\\star}$ perform comparably with CD in terms of the time complexity, whereas it is slower than ARGS which suffers from low reward. However, with an offline trained adpator we can scale the inference time to be same as ARGS.\n\n\n>**Question 2 :** For gpt4 based evaluation, why the authors only report Win-Tie instead of splitting win and ties? In particular I am curious what is the percentage of ties for each comparison.\n\n**Response to Question 2:**  \nWe follow the same evaluation protocal as followed in prior decoding approaches like ARGS [B], thereby reporting the Win-tie in Table 2 for uniformity in comparison. However, as mentioned by the reviewer, we also provide the win-rate and tie-rate separately for Evaluation-1 and Evaluation-3 Setup below. \n\n| Ours                   | Methods          | Win-Rate | Tie-Rate | Lose-rate |\n|------------------------|------------------|----------|----------|-----------|\n| $\\texttt{TQ}^{\\star}$ | ARGS-SFT         |    85.33 |     1.33 |     13.33 |\n| $\\texttt{TQ}^{\\star}$ | $\\text{CD}^{--}$ |    60.67 |     6.00 |     33.33 |\n| $\\texttt{TQ}^{\\star}$ | DPO              |    64.00 |     6.67 |     29.33 |\n| $\\texttt{TQ}^{\\star}$ | ARGS-DPO         |    62.67 |     5.33 |     32.00 |\n\n**Table:** Win, Tie and Lose-rate for Evaluation Setup-1\n\n| Ours                   | Methods          | Win-Rate | Tie-Rate | Lose-rate |\n|------------------------|------------------|----------|----------|-----------|\n| $\\texttt{TQ}^{\\star}$ | ARGS-SFT         |    68.67 |     6.67 |     24.67 |\n| $\\texttt{TQ}^{\\star}$ | $\\text{CD}^{--}$ |    63.33 |      4.0 |     32.67 |\n| $\\texttt{TQ}^{\\star}$ | DPO              |    62.67 |     7.33 |     30.00 |\n|$\\texttt{TQ}^{\\star}$ | ARGS-DPO         |    68.00 |     6.00 |     26.00 |\n\n**Table:** Win, Tie, and Lose-rate for Evaluation Setup-2\n\n\n[A]. Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, Jilin Chen, Alex Beutel, and Ahmad Beirami. Controlled decoding from language models, 2024\n\n[B]. Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment as reward-guided search,2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jdQyiEHrka", "reviewer": "Reviewer_hjEu", "summary": "The paper Transfer Q⋆: Principled Decoding for LLM Alignment proposes a novel approach to aligning large language models by leveraging a principled decoding strategy. The authors propose a method to estimate the optimal Q-function for decoding using an existing aligned policy, addressing limitations in previous approaches like Controlled Decoding (CD). The paper presents an indirect transfer method, allowing for alignment even when the baseline model is trained on a different reward function. The authors provide theoretical analysis characterizing the sub-optimality gap and KL divergence to the reference policy. Extensive experiments across multiple datasets and model architectures demonstrate the effectiveness of TQ⋆ compared to existing methods.", "questions": "1. Have you investigated the stability of the alignment achieved by TQ⋆ over extended generation sequences? Does the alignment quality degrade for longer outputs, and if so, how does this compare to other methods?\n\n2. How sensitive is TQ⋆ to the choice of baseline model? If multiple baseline models are available, each aligned with different rewards, how might one optimally select or combine them to estimate Q⋆ for a given target reward?", "limitations": "The authors acknowledge some limitations of their work, such as the potential for hallucination in responses to very obscure queries. However, they could improve their discussion of limitations by addressing:\n\n1. The reliance on existing aligned baseline models, which may not always be available or suitable for all target rewards.\n2. The potential for errors or biases in the GPT-4 based evaluation, which is used as a proxy for human assessment.\n3. The computational overhead of TQ⋆ compared to standard decoding methods, which may impact real-time applications.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "**Novelty**: The paper introduces an original approach to LLM alignment via decoding, leveraging existing aligned policies to estimate the optimal Q-function.\n\n**Theoretical foundation**: The authors provide a rigorous theoretical analysis of their method, including bounds on the sub-optimality gap and KL divergence. This adds credibility to the approach and helps explain its effectiveness.\n\n**Comprehensive evaluation**: The experimental section is thorough, covering multiple datasets, model architectures, and evaluation metrics. The inclusion of both synthetic and real transfer tasks demonstrates the method's robustness.\n\n**Practical relevance**: TQ⋆ addresses a significant challenge in LLM alignment, offering a computationally efficient alternative to fine-tuning approaches. This has potential implications for improving the deployment of aligned LLMs.", "weaknesses": "**Comparison with Baseline**s: The comparisons with existing baselines like DPO are insightful, but additional baselines, especially those focusing on inference-time control, could strengthen the evaluation.\n\n**Hyperparameter sensitivity**: The paper does not thoroughly explore the sensitivity of TQ⋆ to its hyperparameters, particularly the decoding alignment parameter α. A more detailed analysis of this aspect would strengthen the work.\n\n**Scalability**: While the method is tested on 7B parameter models, it's unclear how well it scales to larger models that are increasingly common in practical applications.\n\n**Evaluation metrics**: While the paper uses several evaluation metrics, including GPT-4 based assessment, it lacks human evaluation studies. Given the subjective nature of language quality and alignment, human evaluation would provide valuable validation of the method's effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Reviewer Summary:**   We thank the reviewer for the encouraging remarks and recommending acceptance of our work. We provided detailed responses to other comments one by one as follows. \n\n>**Weakness 1:** Comparison with Baselines: The comparisons with existing baselines like DPO are insightful, but additional baselines, especially those focusing on inference-time control, could strengthen the evaluation.\n\n\n\n**Response to Weakness 1:**  Thank you for your comment. We have now included a comparison with additional inference time control baselines (aka decoding methods) as well in the table below. Specifically, we report the time taken to decode a single prompt, using the system configuration described in Appendix C in the main paper.\n\n| Algorithm              | Inference Time | Avg Reward |\n|------------------------|----------------|------------|\n| Naive Decoding         | 3s             | 0.13        |\n| ARGS                   | 7s             | 0.29        |\n| $\\text{CD}^{--}$                   | 40s            | 0.71       |\n| $\\texttt{TQ}^{\\star}$ (Ours)    | 41s            | 1.0        |\n\n>**Weakness 2:** Hyperparameter sensitivity: The paper does not thoroughly explore the sensitivity of TQ* to its hyperparameters, particularly the decoding alignment parameter α. A more detailed analysis of this aspect would strengthen the work.\n\n\n**Response to Weakness 2:** We agree with the reviewer that doing more ablations with respect to hyperparameters would strengthen the work. To this end, as suggested by the reviewer, we did additional ablations experiments to understand the effect of varying the decoding alignment parameter $\\alpha$ on the quality of the generated text. Specifically, we performed decoding by varying $\\alpha$ such that $\\frac{1}{\\alpha} \\in [0.1, 0.25, 0.5, 0.75, 1, 2, 5, 7.5]$. We report the results in Figure-3 in the [rebuttal PDF](https://openreview.net/attachment?id=6Il3qOI0FO&name=pdf). To be specific, we compare the tradeoff between the win-rate and the KL divergence to the base reference SFT policy for different values of decoding alignment parameters. \n\n>**Weakness 3:** Scalability: While the method is tested on 7B parameter models, it's unclear how well it scales to larger models that are increasingly common in practical applications.\n\n**Response to Weakness 3:**  Thank you for raising this concern. As the reviewer suggested, we performed an evaluation on two additional setups using larger models, as detailed in table below. \n|              | Dataset                             | SFT Model  | DPO Model  | Reward Model |\n|--------------|-------------------------------------|------------|------------|--------------|\n| Evaluation-7 | HH-RLHF                             | LLAMA2-13B | LLAMA2-13B | LLAMA2-13B   |\n| Evaluation-8 | OpenAssistant Conversations Dataset | Pythia-12B | Pythia-12B | Pythia-6.9B  |\n\nWe report the results in Figure-2 in the [rebuttal PDF](https://openreview.net/attachment?id=6Il3qOI0FO&name=pdf). \n\n>**Weakness 4:** Evaluation metrics: While the paper uses several evaluation metrics, including GPT-4 based assessment, it lacks human evaluation studies. Given the subjective nature of language quality and alignment, human evaluation would provide valuable validation of the method's effectiveness.\n\n\n**Response to Weakness 4:**  Our current comparison includes win rate, reward model performance, coherence, and diversity, which are designed to approximate human preferences. However, we agree that incorporating direct human evaluation would provide more robust validation of our method's effectiveness. We plan to include human evaluation in the final version and are currently in the process of obtaining the necessary permissions.\n\n\n\n>Question 1 : Have you investigated the stability of the alignment achieved by TQ⋆ over extended generation sequences? Does the alignment quality degrade for longer outputs, and if so, how does this compare to other methods?\n\n\n**Response to Question 1:**  Thank you for your suggestion. We performed additional evaluation by varying the length of generated text. We report the results in Figure-1 in the [rebuttal PDF](https://openreview.net/attachment?id=6Il3qOI0FO&name=pdf). We observed that irrespective of the length of generated text, $\\texttt{TQ}^{\\star}$ consistently outperforms all the compared baselines.\n\n\n>Question 2 :  How sensitive is TQ⋆ to the choice of baseline model? If multiple baseline models are available, each aligned with different rewards, how might one optimally select or combine them to estimate Q⋆ for a given target reward?\n\n\n**Response to Question 2:** This is a great question and is a valid scope of future research work. We can try by selecting argmax Q not only over the action token by as well as models itself. We are working as a future research.\n\n\n>Limitations: The authors acknowledge ..... impact real-time applications.\n\n**Response to Limitations:** Thank you for your insightful comments. We will expand our discussion of limitations in the final version to include: (1)\tThe reliance on existing aligned baseline models, noting their availability and suitability challenges for different target rewards. (2)\tPotential errors or biases in the GPT -4-based evaluation, addressing its limitations as a proxy for human assessment. (3)\tThe computational overhead of TQ⋆ compared to standard decoding methods, considering its impact on real-time applications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ieKJ37JpFv", "reviewer": "Reviewer_1x8S", "summary": "This paper addresses decoding for aligning large language models, which is a process of inference-time, token-level optimisation without updating the parameters of the LLM. Two scenarios are considered: 1. where a baseline policy is given and is aligned with the target trajectory-level reward. 2. where the baseline policy is given and aligned with a different trajectory-reward than the target one. Two approaches are provided in accordance: one named direct transfer, which as the name suggests directly derive a token-level policy from the baseline LLM, and the other named indirect transfer, whose key lies in an importance sampling trick to reweigh using the ratio between the target and baseline trajectory policies. The authors also provide theoretical analysis characterising the sub optimality gap and the divergence from the supervise-fine-tuned model. Empirical results demonstrates superior performance in several key metrics.", "questions": "equation (2) - This seems like a typo since for i > 0, the first argument of R should also take in the so-far generated sequence z_0 ... z_{i-1}.\n\nequation (15) - how is $Z_{BL}$ and $Z_r$ computed? This is needed later in the importance ratio, if I understood correctly, but I don't see how to compute these quantities which are usually intractable.", "limitations": "Aside from the limitations I mentioned in Weaknesses, the authors also acknowledges that TQ* suffers from increased time complexity. They suggest that this can be mitigated by training a small value function adapter as discussed in prior work.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper is very well written, poses a reasonable problem, namely how to more efficiently decode an existing trajectory level policy into a token level policy, and provides an elegant solution. The proposed approach draws on latest developments in the field (DPO) and enjoys a rigorous theoretical characterisation. The gap between the trajectory level learnt policy and the sft-policy is interpretably characterised in terms of the regularisation coefficients in front of the KL terms. The authors finished with a nice empirical analysis showing the efficacy of their method.", "weaknesses": "Morally speaking, it is not clear to me why should decoding this way works much better than directly taking $\\rho_{BL}$- at least in the case of direct decoding. After all, TQ* is just the 1-step-action-value function of $\\rho_{BL}$, so I think the readers would benefit from an explicit explanation of why this process of *obtaining $\\rho_{BL}$* --> *estimating its (1-step-action) value function* --> *compute the closed form of the optimal policy of this value function with a KL term* should do better than just directly using the trajectory level policy which is already the (regularised) optimal of the reward function $r$. More concretely, it would be nice to have this shown as a theoretical result - a theorem that shows that the suboptimality gap of the policy proposed by the authors is smaller than $\\rho_{BL}$. I appreciate that the authors have shown this empirically, though.\n\nFor the indirect case, the method proposed by the authors require access to the ground truth trajectory-level reward function. But if we had this in practice, would we not have just directly used it to compute $\\rho_{BL}$? Thus, I think here it makes sense to add some explanation regarding why this is useful.\n\nFinally, I find it a bit of a misleading juxtaposition with DPO - the point of DPO is that we no longer need to model the reward function, but in this work we rely on both the trajectory level optimal policy learned by DPO as well as the reward function which also needs to be learned. Both of these suffer from statistical/optimisation instability in their own right. So it is strong to assume you have both of these.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. \n\n>**Weakness 1:** Morally speaking, it is not  ...... empirically, though.\n\n**Response to Weakness 1:**  We believe the reviewer wants to understand the advantage of directly generating the response using $\\rho_{\\text{BL}}$ over TQ*. We explain in detail below.\n\n***Our proposed method enables optimal action at the Token Level.*** In our algorithm (TQ*), for every state ($s_t$) we compute the action value function $Q^*(s_t,z)$ using $\\rho_{\\text{BL}}$  and select the optimal action by maximizing the action value function $Q^*(s_t,z)$, ensuring the optimal performance for any token as highlighted in Theorem 1. Intuitively, it ensures the possibility of performing better credit assignment with token level MDP while decoding, when compared to $\\rho_{\\text{BL}}$ which is trained as a contextual bandit for trajectory responses. This aspect has also been highlighted in the very recent work [C]. As a result, the majority of the prior decoding methods demonstrate improvement over directly using the DPO policy\n\n***Theoretical Justification:*** From Theorem 1 in the paper, we can extract the theoretical justification of the improvement of our proposed algorithm over $\\rho\\_{\\text{BL}}$. The sub-optimality gap of our algorithm, defined in Equation 18, can be decomposed into two terms ($\\Delta \\leq \\Delta_1 - \\Delta_2$) as detailed in Appendix G.1 (Equations 28 and 29). \n\nThe first term, $\\Delta_1$, is a function of parameter $\\beta$ and represents the sub-optimality from the optimal value function due to $\\rho\\_{\\text{BL}}$, with its upper bound $\\Delta\\_1\\leq \\beta\\mathbb{D}_{\\text{KL}}[\\rho^*(\\cdot|\\mathbf{x}) \\mid \\mid \\rho\\_{\\text{sft}}(\\cdot|\\mathbf{x})]$. This indicates the sub-optimality incurred at the token level when using $\\rho\\_{\\text{BL}}$ alone. \n\nHowever, the second term, $\\Delta_2 = \\alpha h_{\\alpha}$ , is always non-negative, illustrating that our algorithm provides an additional benefit of improving the suboptimality gap. By tuning the value of $\\alpha$, we can improve the performance beyond what is possible with $\\rho_{\\text{BL}}$ alone. This improvement is consistently demonstrated across all our experimental results, as also highlighted by the reviewer.\n\n>**Weakness 2:** For the indirect case, .... is useful.\n\n**Response to Weakness 2:**  We believe what the reviewer is asking is if we have access to the ground truth reward, why can't we directly have an aligned $\\rho_{\\text{BL}}$ for that ground truth reward? We expand on this in detail as follows. \n\n***Obtaining aligned $\\rho_{\\text{BL}}$ to ground truth would require fine tuning.*** We note that even if we have access to ground truth reward, obtaining an aligned model $\\rho_{\\text{BL}}$ using standard DPO would require fine-tuning of parameters which is not the focus of this work. Our motivation is in tuning-free alignment settings, where we don't update the parameters of the language model. \n\n***Regarding our approach and decoding methods.*** We emphasize that any decoding method would require access to the target reward function (trajectory level). However, for indirect transfer, we leverage any open-sourced baseline model $\\rho_{\\text{BL}}$ aligned to an arbitrary reward function $r_{\\text{BL}}$ which is different from the target reward. Thus, for principled decoding even though we have $r_{\\text{target}}$, still we can't directly use $\\rho_{\\text{BL}}$ to estimate action value function and decode, which will result in sub-optimal decoding due to the distribution shift. Thus, we need to estimate $\\rho_{\\text{target}}$ using Equation 16. \n\n\n>**Weakness 3:**  Finally, I find it a bit of a ....... So it is strong to assume you have both of these.\n\n\n**Response to Weakness 3:**  We remark that any decoding method requires the access to target reward function for alignment. As reviewer mentioned, if the target reward function is learned through preferences, there will be statistical error due to coverage and suboptimality of optimization methods. Furthermore, the DPO policy could also have it's own instability issues which might affect the final performance of our decoding method. We agree with the reviewer and will highlight this limitation in the revised final version of our work. \n\n**Examples of Access to Ground truth reward:** We note that there can be true rewards/scores that don't need to be always trained from data, for example - coding, and mathematical tasks where one might have fixed ground truth avoiding above mentioned statistical errors. \n\n\n>**Question 1:** equation (2) - This seems like a typo ....sequence z_0 ... z_{i-1}.\n\n**Response to Question 1:**  Thanks for pointing out the typo, we will fix it in the final version of the paper. The first argument should be $s_{t+i}$ to make sure it take in so far generated sequence.\n\n\n>**Question 2:**  equation (15) - how is $Z_{\\text{BL}}$ and $Z_r$ computed.... intractable.\n\n\n**Response to Question 2:** This is a good question. We note that for the theoretical analysis in this work, we assume access to the ratio mentioned in Equation (15). For the experimental purposes, we utilized unbiased estimates. For instance, we can estimate the partition from collecting samples from $\\rho_{\\text{SFT}}$ and evaluating the empirical estimate of $\\mathbb{E}\\_{y \\sim \\rho\\_{\\text{SFT}}}[\\exp(\\frac{1}{\\beta}r(x,y))]$. Regarding this ratio explicitly, we had an interesting observation that for several realistic scenarios of transfer, as observed in our empirical experiments, when either the reward difference $r_1(x,y) - r_{\\text{BL}}(x,y)$ is small or the reference policy is not aligned to any specific reward $r(x,y)$, the ratio of the partition functions will be 1. This would effectively relax the computational bottleneck for the implementations. \n\n\n[B] Sidharth Mudgal et al., Controlled decoding from language models, 2024\n[C]. Rafailov, R et al., From r to q*: Your language model is secretly a q-function. arXiv preprint arXiv:2404.12358.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CmwkTpOCqv", "reviewer": "Reviewer_xz5J", "summary": "This paper introduces 'Transfer Q*', which is a decoding strategy to align language models with target reward models without any fine-tuning. The main contribution of this paper is the estimation of the optimal Q* function, which is often unavailable in practice and is necessary for approximating the RL optimal policy, through DPO (Direct Preference Optimization). The paper first introduces an algorithm for direct transfer decoding using DPO-trained models as optimal baselines, and then presents indirect transfer decoding that uses these as conservative baselines. The authors describe the theoretical characteristics of this methodology and experimentally demonstrate that it achieves state-of-the-art performance on target reward models.", "questions": "1. In Figure 2 (b), when measuring the KL divergence for TQ*, which model was used to compare the decoding results? For a fair comparison, it should be measured against $pi_sft$, like the other baselines.\n2. I'm curious about the inference cost. How do the memory requirements and time complexity compare to naive greedy decoding?", "limitations": "This paper has done truly promising work, but it has the following limitations:\n\n1. Although DPO is used as a baseline, it is an offline trainer and there is a theoretical gap with the optimal Q*. This issue is addressed through the indirect transfer decoding part, but there is still room for improvement, and various discussions are likely to follow.\n2. Due to the use of the DPO model as the initial policy, the theoretical justification is weakened, and a clear interpretation of the experiments is not possible.\n3. In the process of training the DPO model, over-optimization towards the reward model is also controlled by the KL penalty. It would be interesting to add analysis and experiments on this aspect.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Aligning to specific values through decoding algorithms without fine-tuning is highly useful. In particular, estimating the token-level Q* function in this process is very challenging, and existing works typically train external networks to estimate these values. This paper uses DPO-trained models as baselines to estimate the token-level Q* function, as DPO learns the Q* function in an offline manner. Given recent reports that DPO-trained models predict preferences better than classification-based reward models, this approach is very interesting and can be considered a valuable contribution. Additionally, since DPO-trained models are offline trainers and thus may be distant from the optimal Q* function, the introduction of an indirect transfer decoding algorithm that treats them as suboptimal, along with the presentation of its theoretical properties, is also a significant contribution.", "weaknesses": "1. Equation 5 on line 121 is introduced roughly. Its derivation should be described in the Appendix, or references discussing it should be introduced.\n2. The introduction of Controlled Decoding (CD) is incorrect. It states that CD uses $Q^{\\pi_{sft}}$ as a proxy for Q*, but in reality, it uses an energy-based model of $Q^{\\pi_{sft}}$ and a value function through an external network as Q*. The difference between this work and CD is that CD uses FUDGE and Q function as a baseline, while the proposed TQ* uses DPO as a baseline reward model. Therefore, Figure 1 and its explanation are incorrect. Considering that the indirect method, which conservatively estimates the target model, is the contribution, Figure 1 should be drawn in reverse.\n3. Equation 12 on line 175 is believed to be incorrect. Theoretically, the $\\pi(z|s)$ used in this equation should be the reference model $\\pi_{ref}$ before exploration, not the aligned model $\\pi_{BL}$. The equation development started with the premise that the optimal policy can be expressed as an energy-based model of the reference model and reward model, but in equation 11, it roughly transitions from the reference model to the aligned DPO, losing theoretical justification. In this case, it's expected that the original distribution will collapse significantly and over-optimize to the reward model.\n4. A crucial experiment on the trade-off between KL divergence and the reward model is missing. In RL, there's a strong trade-off in reward model optimization depending on how much KL penalty is given, so it's not sufficient to only check the performance against the reward model as in this paper's experiments. While Figure 2(b) touches on this, it's not enough. An experiment showing whether it demonstrates pareto-frontier performance in the trade-off relationship according to KL penalty is also needed. This experiment is particularly important given the use of the DPO-trained model as the reference model, as mentioned in weakness 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback on our paper. \n\n>**Weakness 1:** Equation 5.... introduced.\n\n\n**Response to Weakness 1:**  We will add the derivation for the closed-form solution in Equation 4 (also shown in CD [B]) as a Lemma in the appendix for better clarity. \n\n\n>**Weakness 2:** The introduction of CD..... reverse.\n\n\n**Response to Weakness 2:** We thank the reviewer for this point but there seems to be a confusion. We apologize for any oversight during our explanation of CD in our paper. We take this opportunity to expand on this point more below.\n\nWe note that the objective of Controlled Decoding which is a KL regularized RL problem (also highlighted in CD) is given by\n\n\\begin{align}\n    \\pi^*(\\cdot|s\\_t) := \\arg \\max\\_{\\pi} \\mathbb{E}\\_{z \\sim \\pi(\\cdot|s\\_t)}[Q^*(s\\_t, z)] - \\alpha KL (\\pi(\\cdot|s\\_t), \\pi\\_{\\text{SFT}}(\\cdot|s_t)) \\tag{4}.\n\\end{align}\n\nHowever, it is extremely crucial to note that in order to obtain the optimal policy in (4), the Q-function in the equation should be $Q^*(s\\_t, z)$ (the optimal action-value function i.e the reward return calculated with optimal policy) and not any arbitrary $Q$ function from any policy. However, the approach in CD indicates that the $Q$ estimation (either directly or through an external network) is done through the data generated using $\\pi\\_{sft}$ policy (refer to Equation 1, Equation 4 in [B]), leading to sub-optimality as demonstrated in our Figure 1. Our main contribution is to show that with an access to an aligned model, we can estimate  $Q^{*}(s_t, z)$ in a much more efficient way than CD as demonstrated in all our experimental settings. \n\n***Possible source of confusion:*** However, we believe the source of confusion is that in the CD paper, the definition of action value function used in (4) above (which is equation (1) in CD paper [B]) utilized $\\pi_{sft}$ to sample but denotes the value function with $V^*$ which is usually reserved for the optimal value function.\n\n>**Weakness 3:**  Equation 12 on line 175 .... the reward model.\n\n\n**Response to Weakness 3:** We thank the reviewer for providing us with this opportunity to clarify further.\n\n***Solution in (12) is theoretically correct and justified.*** We have stated our proposed decoding optimization problem in Equation (11), and solving that leads to the choice of having the reference policy as $\\pi_{\\text{BL}}$ in Equation 12 on line 175. This is a specific choice of our algorithmic solution design which we have also implemented in our work. However, we want to emphasize that our theoretical results in Sec. 3.3 (specifically the KL divergence upper bound in Theorem 1, statement (2)) derives the divergence of proposed algorithm's policy to the original reference policy $\\rho_{\\text{SFT}}$. The upper bound of Theorem 1 [statement 2] is given by \n\\begin{align}\n\\mathbb{D}\\_{\\text{KL}}(\\rho^*\\_{Alg}(\\cdot|\\mathbf{x}),\\rho\\_{SFT}(\\cdot|\\mathbf{x}) )\\leq(\\frac{1}{\\beta}+\\frac{1}{\\alpha}T)r\\_{\\max}.\n\\end{align}\n\nBy controlling the value of $\\alpha$ and $\\beta$, we can control the distance to the original reference model. Moreover, in our experimental ablation Figure 2(b) we show the KL divergence to the original SFT reference model which proves that the proposed method does comparable to other baselines and is not diverging from the SFT policy. \n\n>**Weakness 4:** A crucial experiment on t...... in weakness 3.\n\n\n**Response to Weakness 4:**  As suggested by the reviewer, we performed additional experiments to generate the pareto-frontier results. To be specific, in Figure-3 in the [rebuttal PDF](https://openreview.net/attachment?id=6Il3qOI0FO&name=pdf), we compare the tradeoff between the win-rate and the KL divergence to the base reference SFT policy. TQ* outperforms existing baselines. We will add this to the final version. \n\n\n>**Question 1:** In Figure 2 (b), when measuring the KL divergence for TQ*, which model was used to compare the decoding results? For a fair comparison, it should be measured against $\\pi_{\\text{SFT}}$\n\n\n**Response to Question 1:**  Yes, you are right.  We have indeed used the reference model $\\pi_{\\text{SFT}}$ for all the algorithms while measuring the KL divergence for TQ*. \n\n>**Question 2:**  I'm curious about the inference cost. How do the memory requirements and time complexity compare to naive greedy decoding?\n\n\n**Response to Question 2:**  We report the inference time-complexity of all the existing deocding algorithms. Ours is comparable to state-of-the-art decoding methods. \n\n| Algorithm              | Inference Time | Avg Reward |\n|------------------------|----------------|------------|\n| Naive Decoding         | 3s             | 0.13        |\n| ARGS                   | 7s             | 0.29        |\n| $\\text{CD}^{--}$                   | 40s            | 0.71       |\n| $\\texttt{TQ}^{\\star}$ (Ours)    | 41s            | 1.0        |\n\n\n**Reference:**\n\n[B] Sidharth Mudgal et al., Controlled decoding from language models, 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## General Response

We want to thank all the reviewers for their time and effort in providing detailed comments to our work. We are encouraged that the reviewers found our proposed approach

- ***novel*** (Reviewer ypg9, Reviewer xz5J) & ***theoretically rigorous*** (Reviewer hjEu, Reviewer 1x8S),
-  our experimental evaluation to be **comprehensive** (Reviewer 1x8S), 
-  and our paper to be **very well-written** (Reviewer 1x8S) & **truly promising** (Reviewer xz5J). 

We have addressed the reviewer's comments and concerns in individual responses to each reviewer. As requested by the reviewers, we have performed  new experimental evaluations as folows. Please find the attached rebuttal pdf for the experimental results. 



**New Experimental Results:**


1. ***Pareto Front for KL Divergence to True Reference Policy**:* We perform experiments to understand the tradeoff between the win-rate and the KL divergence to the base reference SFT policy. Our findings, reported in Figure 3 of the rebuttal PDF, show that our proposed method, $\texttt{TQ}^{\star}$, outperforms existing baselines.

2. ***Stability to Generation Length:*** We compared $\texttt{TQ}^{\star}$ against all baselines by varying the length of the output responses. We report our findings in Figure 1 of the rebuttal PDF. Our observations indicate that $\texttt{TQ}^{\star}$ consistently outperforms all baselines, demonstrating its stability across different output lengths.

3. ***Generality to Larger Model Size:*** We conducted evaluations on two additional setups using larger models, as detailed in the table below. The results, presented in Figure 2 of the rebuttal PDF, show that for both setups, $\texttt{TQ}^{\star}$ significantly outperforms competitive baselines. This demonstrates the scalability and generality of our approach to different model sizes.

|              | Dataset                             | SFT Model  | DPO Model  | Reward Model |
|--------------|-------------------------------------|------------|------------|--------------|
| Evaluation-7 | HH-RLHF                             | LLAMA2-13B | LLAMA2-13B | LLAMA2-13B   |
| Evaluation-8 | OpenAssistant Conversations Dataset | Pythia-12B | Pythia-12B | Pythia-6.9B  |



4. ***Inference Time for $\texttt{TQ}^{\star}$:*** We report the time taken to decode a single prompt for different decoding strategies, using the system configuration described in Appendix C in the main paper.


| Algorithm              | Inference Time | Avg Reward |
|------------------------|----------------|------------|
| Naive Decoding         | 3s             | 0.13        |
| ARGS                   | 7s             | 0.29        |
| $\text{CD}^{--}$                   | 40s            | 0.71       |
| $\texttt{TQ}^{\star}$ (Ours)    | 41s            | 1.0        |

It is evident that $\texttt{TQ}^{\star}$ perform comparably with CD in terms of the time complexity, whereas it is slower than ARGS which suffers from low reward. However, with an offline trained adpator we can scale the inference time to be same as ARGS.

Meta-review: The paper proposes a decoding-based approach for aligning LMs. The key idea is to derive an equivalent token sampling distribution that mimics the behavior of an existing LM that has been aligned, or a suitable approximation thereof. This approach leads to sizable gains over existing approaches like Controlled Decoding.

Reviewers were unanimously in favor of acceptance: they consistently found the paper to be well-motivated and presented, with interesting insights and results. One reviewer noted that there is scope for the paper to be more careful in its discussion and framing of prior work. We encourage the authors to incorporate this feedback in the final version of the paper.
Decision: Accept (poster)