Transfer Q-star : Principled Decoding for LLM Alignment

Authors: souradip  chakraborty
soumya  suvra ghosal
ming  yin
dinesh  manocha
mengdi  wang
amrit  singh bedi
furong  huang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 19:48:05

Abstract: Aligning foundation models is essential for their safe and trustworthy deployment. However, traditional fine-tuning methods are computationally intensive and require updating billions of model parameters. A promising alternative, alignment via decoding, adjusts the response distribution directly without model updates to maximize a target reward $r$, thus providing a lightweight and adaptable framework for alignment. However, principled decoding methods rely on oracle access to an optimal Q-function ($Q^*$), which is often unavailable in practice. Hence, prior SoTA methods either approximate this $Q^*$ using $Q^{\pi_{\text{sft}}}$ (derived from the reference $\texttt{SFT}$ model) or rely on short-term rewards, resulting in sub-optimal decoding performance. In this work, we propose $\texttt{Transfer Q}^*$, which implicitly estimates the optimal value function for a target reward $r$ through a baseline model $\rho_{\texttt{BL}}$  aligned with a baseline reward $r_{\texttt{BL}}$ (which can be different from the target reward $r$). Theoretical analyses of $\texttt{Transfer Q}^*$ provide a rigorous characterization of its optimality, deriving an upper bound on the sub-optimality gap and identifying a hyperparameter to control the deviation from the pre-trained reference $\texttt{SFT}$ model based on user needs. Our approach significantly reduces the sub-optimality gap observed in prior SoTA methods and demonstrates superior empirical performance across key metrics such as coherence, diversity, and quality in extensive tests on several synthetic and real datasets.

Main paper: 
 introduction: 
 As artificial intelligence (AI) systems continue to demonstrate super-human performance across various tasks, it is becoming increasingly critical to ensure that such AI systems align well with human preferences, goals, and ethical standards. Alignment via fine-tuning with reinforcement learning from human feedback (RLHF) [38, 34, 9] , has proven highly effective [36, 44] . However, aligning LLMs by fine-tuning the model parameters requires gradient updates across several billion parameters (size of LLMs) which require vast computational resources [31, 25] and have a significant environmental impact [15] . Additionally, many SoTA models are not fully open-sourced [44, 36] , offering limited access only to certain components like logits, making fine-tuning impossible.As an alternative to fine-tuning, decoding for alignment has recently emerged as a potential solution [33, 26] . Decoding aims to alter the LLM's response distribution at the token level to align with target reward r without updating the parameters of the LLM. Decoding facilitates alignment by accessing the token-level optimal value function, Q * , which corresponds to the target reward r. Tokens are then sampled based on Q * . This approach ensures rapid, efficient, and cost-effective alignment. A detailed discussion of related work is provided in Appendix E.A fundamental challenge. Decoding effectively hinges on accessing an oracle to the optimal value function, Q * , or a token-level optimal policy, which are typically not available in practical scenarios. To address this fundamental challenge, recent studies [33] have adopted a proxy, Q πsft , for Q * . However, this approach results in suboptimal decoding due to a distribution shift inherent in approximating the true, unknown Q-function, Q * , as illustrated in Figure 1 . This raises a critical question: Is it possible to devise a more efficient and accurate estimate of the optimal value function Q * for decoding purposes? In this work, we affirmatively address this query.Figure 1 : Left. This figure highlights the conceptual idea of proposed transfer decoding in this work. It clearly shows that the current SoTA method [33] exhibits suboptimality with respect to alignment with the target reward denoted by the dotted red arrow. On the other hand, the proposed transfer decoding method utilizes an immediately available aligned language model called the baseline, which is aligned with some baseline reward rBL to bridge the gap between the SoTA method and the target model. Right. This figure provides empirical evidence of the performance gap of the current SoTA decoding strategy [33] with respect to Oracle (best of N sampling). Our proposed Transfer Q ⋆ (TQ ⋆ ) reduces the gap and provides a new decoding method.Our approach leverages a crucial observation regarding the key challenges identified: effective decoding requires access to a language model already aligned with the target reward r for trajectorylevel response generation. Notably, recent advances in DPO-based methods [38, 3, 32] have facilitated the development of fine-tuned language models-referred to as baseline models-that are capable of generating aligned trajectories [27] . Our first key idea involves utilizing these publicly available trajectory-level models to estimate Q * and subsequently derive the optimal token-level language model for decoding. We term this approach direct transfer decoding. Moreover, it is possible that these publicly available baseline models are aligned with a different baseline reward r BL , rather than the intended target reward r. Our second key idea addresses this challenge by proposing a novel indirect transfer decoding method. We introduce our proposed technique as Transfer Q ⋆ (TQ ⋆ ), which facilitates efficient and on-the-fly alignment through decoding. We note that Transfer Q ⋆ proves effective even when there are substantial discrepancies between the target and baseline rewards.We summarize our major contributions as follows.(1) A novel concept of transfer decoding (TQ ⋆ ). We introduce a novel concept of transfer decoding in a principled manner by leveraging already-available baseline language models aligned either with the target reward (direct transfer) or with some other (significantly) different baseline reward (indirect transfer). Our proposed approach TQ ⋆ efficiently reduces the sub-optimality gap inherent in previous SoTA decoding methods, as highlighted in Figure 1 .(2) Theoretical characterization of TQ ⋆ . We provide a rigorous theoretical characterization of the optimality of Transfer Q ⋆ . Specifically, we derive an upper bound on the gap between the optimal LLM policy and the LLM decoding policy resulting from TQ ⋆ (see Theorem 1) . Additionally, we present a principled approach to control the deviation of the resulting policy from the pre-trained SFT language model π sft (see Theorem 1, statement 2) . This identifies hyperparameters that allow users to specify the desired amount of deviation or improvement based on their specific needs.(3) Experimental evaluations. We provide a detailed empirical evaluation of TQ ⋆ in various decoding tasks in both real and simulated settings, comparing against SoTA baselines, including DPO [38] , ARGS [26] , and CD [33] . Empirical results demonstrate consistent superiority over the baselines. Notably, TQ ⋆ surpasses the current SoTA decoding strategy CD [33] , achieving an improvement of up to 1.45x in average reward and 67.34% in GPT-4 based win-tie rate. We further evaluate and compare several attributes of the text generated by our algorithm (such as coherence, diversity, and quality) against baselines, demonstrating the superiority of our algorithm in all these aspects. 
problem formulation: alignment via controlled decoding 2.1 token-level markov decision process: 
 Since the control decoding procedure operates at the token level, before formulating the problem mathematically, we start by defining a token-level Markov decision process (MDP) in the context of LLMs. Let us consider a token-level MDP M := {S, A, P, R} with the state-space S consisting of the concatenated sequence of tokens and the action space A representing the space of the next token which is essentially the vocabulary V.Next-token generator: the token-level policy π. Given a state s t = [x, y <t ] ∈ S, which is a sequence of tokens containing the prompt/query x := {x 1 , x 2 , • • • , x N } appended with the t tokens y <t := {y 0 , y 1 , • • • , y t-1 } generated so far, an LLM is a token-level decoding policy π that generates the action (i.e., the next token) a t = y t via sampling y t ∼ π(• | s t ). The transition P to the next state s t+1 is deterministic: s t+1 = [x, y <t , y t ], the concatenation of the current state and action.Response generator: the trajectory-level policy ρ. We denote the trajectory level probability by ρ π (y|x) = T t=1 π(y t |[x, y <t ]). Some commonly used sampling techniques in the literature include Beam Search [17] , Top-p sampling [20] , and Top-k Sampling [16] .From trajectory-level reward r to token-level reward R. Successful decoding depends on sampling from a token-level policy that yields high rewards, reflecting the inherently token-level nature of the decoding process. However, as detailed in Appendix F, we obtain a reward model r(x, y), only at the trajectory-level rather than the desired token-level, by fitting feedback on human preferences. To close the gap, similar to existing literature [33] , we define the token-level reward R(x, y t ) from the trajectory-level reward model r(x, y) as follows: R(x, y t ) := 0,EQUATIONwhere EOS ∈ V represents the end of sequence token. The token-level reward in (1) implies that we only receive a reward once we have the full sequence/response, otherwise, no reward.Action-value function Q π for π and optimal Q * for optimal π * . From the token-level reward R(x, y t ), we can define the action-value function associated with the reward asEQUATIONwheres t+i := [s t , z 0 , z 1 , • • • , z i-1 ]and expectation is over the randomness due to the sampling from token level language model π. The optimal Q-function from the definition in equation ( 2) is given byEQUATIONThe optimization problem in (3) denotes an unconstrained objective as in standard reinforcement learning; however, in the context of alignment for LLMs, we also need to consider the distance of optimal aligned policy to the pre-trained unaligned token-level policy π sft [37, 41, 38] .In this section, we will formulate the problem of aligning LLMs during deployment via a controlled decoding procedure as initially discussed in [33, 26] .Decoding process. We start by defining what decoding means in the context of LLMs. We consider access to a pre-trained unaligned language model π sft which takes in prompt x as an input and generates a response y = [y 0 , y The quest to answer this question has given rise to an interesting research problem of utilizing decoding for LLM alignment [33] .LLM alignment via decoding. The problem of LLM alignment via decoding can be formally defined as solving for the optimal decoding policy π * dec under the token level MDP M asEQUATIONwhere s t = [x, y <t ] and Q * (s t , z) denotes the optimal state-action value function for the token-level MDP M defined in (3) . We remark that the KL regularization in equation ( 4) ensures that the optimal decoding policy π * dec remains in the close neighborhood of the pre-trained model π sft which contains other important properties. In (4), α > 0 denotes the alignment hyperparameter which controls the trade-off between the objective of maximizing the target reward r return and the closeness to π sft . We can write the closed-form solution of the problem in (4) asEQUATIONwhere C α := z π sft (z|s t ) exp (αQ * (s t , z)) is the normalizing constant for state s t . Although the close form expression in (5) poses an interesting form, it is difficult to implement it in practice due to the various challenges we will discuss next.Challenges of implementing (5): oracle access to the optimal Q * . A major challenge in implementing the aligned decoding policy in (5) , lies in the requirement of access to the optimal Q * (s t , z) in ( 5) for each state-action pair (s t , z), which is unavailable in practice. To emphasize that, first we note that Q * (s t , z) in ( 3) can be written using the trajectory level reward r asEQUATIONwhere τ denotes the trajectory τ:= {z 1 , z 2 , • • • , z T }, and ρ * (•|s t , z) = T i=1 π * (z i |[s t+i ]) represents the distribution over the trajectory level response induced by the optimal policy π * (•|s t ) (cf.(3)). From ( 6), we note that the optimal Q * (s t , z) relies on the trajectory/response generated by the optimal ρ * which is unknown. This creates a bottleneck in efficiently deploying the decoding policy in (5) for alignment with target reward r. Next, we discuss how some existing approaches in the literature deal with this fundamental issue and what the limitations are.Limitations in existing approaches. An interesting approach called controlled decoding (CD) is proposed in recent literature by Mudgal et al. [33] , and it approximates Q * (s t , z) by Q π sft (s t , z), which is tractable and easily computable due to availability of π sft . However, this approximation results in significant suboptimal performance, as described in Figure 1 (right). Given this limitation of existing approaches and the above-mentioned fundamental challenge of decoding, we pose the question: Is it possible to provide a better estimate of Q * for decoding? We answer this affirmatively in the following section by introducing a novel concept of transfer decoding.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss limitations in Appendix A. Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
proposed approaches: alignment via transfer q ⋆: 
 Our key ideas of transfer decoding. Our proposed approach hinges on an interesting observation that recent advancements in alignment, particularly through direct preference optimization (DPO)-based approaches [38] , have led to the development of open source freely available fine-tuned language models [27, 53] . We call such aligned models as baseline models, that generate trajectory responses in an aligned manner. Our first key idea is to utilize these baseline models aligned with target reward r, if available, to estimate Q * and subsequently derive the optimal token-level language model for decoding. We remark that baseline models are aligned at the trajectory level (see Appendix F for details), while decoding requires the optimal models at the token level. We term this approach direct transfer decoding. However, it is possible that the available aligned baseline model is aligned with a different baseline reward r BL instead of the target reward r. Hence, our second key idea addresses this issue by proposing a novel method called indirect transfer decoding. We discuss both the ideas and detailed algorithms in detail next.For the direct transfer, we start by considering that we are given a target reward model r and an unaligned pre-trained SFT language model given by π sft ; the corresponding trajectory-level response distribution is given by ρ sft . We are operating under the assumption that we have a baseline model ρ BL (y|x) with target reward r. We note that the closed-form expression for the RLHF-aligned optimal model ρ BL can be written as (see Appendix F, Equation ( 25)) follows:EQUATIONwhere Z r (x) is the normalizing constant and β > 0 is the trajectory-level alignment parameter. We note that the trajectory level optimal policy in ( 7) is usually obtained in the literature via the fine-tuning stage and efficient algorithms such as DPO [38] .Estimating Q * for direct transfer. Now we get back to the fundamental bottleneck of optimal decoding, which lies in estimating the token-level optimal Q * (s t , z). We propose providing a solution to the problem with the help of ρ BL (y|x). As defined in (6) , we begin by consideringEQUATIONwhere ρ * (•|s t , z) := arg max ρ E τ ∼ρ(•|st,z) [r(s t , τ )]. However, we know that the available baseline model ρ BL (y|x) in ( 7) is the solution of following optimization problem:EQUATIONwhich constraints the drift of the optimal distribution ρ BL from the trajectory level reference policy ρ sft with the Kl divergence term. We propose approximating the optimal Q * (s t , z) for decoding byEQUATIONWith the definition in (10) , we propose obtaining our token-level decoding policy π * Alg (•|s t ) for the token-level MDP asEQUATIONwhere π BL (•|s t ) is the token-level language model, which induces the trajectory level model ρ BL . Due to the strong convexity of the objective in (11) owing to the additional KL regularization term, we get the closed-form solution of π * Alg (•|s t ) asEQUATIONwhere C α (s t ) is the normalizing constant. We summarize the proposed approach in Algorithm 1.We remark that, for the direct transfer decoding, we started with the assumption that the available baseline model ρ BL is aligned with the target model only. In practice, however, it is possible that the freely available baseline model ρ BL is actually aligned with some other reward function we call baseline r BL rather than the target reward r. We argue that this condition is even more easily satisfied under the ongoing active research scenario in alignment because we have easy access to open-source, well-aligned LLMs trained on various reward functions [27, 52, 13, 38] . Under this setting of baseline reward r BL and language model ρ BL , we define our novel indirect transfer decoding process as follows.The Transfer Process. The baseline language model ρ BL is also an RLHF aligned model corresponding to reward function r BL . It holds that:EQUATION)where Z BL (x) is the corresponding partition function. From the closed-form expression in (7) , it holds for the target reward r that r(x, y) = β log ρ r (y|x)EQUATIONUsing equations ( 14) and ( 13), we can obtain the trajectory-level optimal policy ρ r (y|x) for the target reward r(x, y) as : EQUATIONy <t = [y 0 , y 1 , • • • , y t-1 ] 4:Sample top-k tokens using token-level baseline policy π BL and store as V = {z i :z i ∼ π BL (•|s t )} k i=1 5: for z ∈ V do 6: if r BL = r then (Direct transfer) 7: Evaluate TQ ⋆ (s t , z) = r([s t , z], τ ), where τ ∼ ρ BL (•|[s t , z]) 8:else (Indirect transfer) 9:Evaluate TQ ⋆ (s t , z) = w • r([s t , z], τ ) where τ ∼ ρ BL (•|[s t , z]), w = ρr(τ |[st,z]) ρBL(τ |[st,z])) 10:Compute decoding score for token z: g z = 1α TQ ⋆ (s t , z) + log π BL (z|s t )11:Next token y t = arg max z∈ V g z 12:Next state s t+1 ← [s t , y t ] 13: Return y = [y 0 , . . . , y T ] where we note that ρ r (y|x) is the unnormalized probability with the normalization factory Z(x) := Zr(x) ZBL(x) . We show in the Appendix H that Z(x) is the normalization constant for ρ r (y|x). We emphasize that calculating the trajectory-level optimal language model for the target reward function r in ( 15) is the crucial step in estimating the optimal Q * for the token-level MDP M for our decoding.Estimating Q * for indirect transfer. Similar to the direct transfer setting, we propose approximating the optimal Q * (s t , z) for decoding by using ρ r (y|x) in (15) asEQUATION= E τ ∼ρBL(•|st,z) ρ r (y|x) ρ BL (y|x) r([s t , z], τ )where we use the importance sampling trick and then utilizing equation (15) , we can get the Q * for indirect transfer. Now, following (11) and ( 12), we can writeEQUATIONwhere π r (•|s t ) is the token level probability derived from the trajectory level policy ρ r (•|s t ) in (15) . Following (11) and ( 12), we can obtain our optimal decoding policy for the indirect transfer as well.We summarize the proposed approach in Algorithm 1.This subsection provides the theoretical analysis of our Transfer Q ⋆ algorithm under direct transfer setup. Existing works [33, 18] leverage the reward-KL tradeoff curve to measure the reward gain versus deviation from reference policy. Ideally, a good algorithm should achieve high rewards while staying close to the reference policy (be KL-efficient). We follow these two aspects and consider two performance metrics: (1) suboptimality gap and (2) KL divergence between our algorithm's policy and the SFT policy. Specifically, we borrow the notion of a suboptimality-gap from offline RL literature [1] and define it in terms of value function difference for any prompt x asEQUATION)whereV * (x) = max ρ E τ ∼ρ(•|x) [r(x, τ )], V Alg (x) = E τ ∼ρ * Alg (•|x) [r(x, τ )],and ρ * Alg represents the distribution over the trajectories induced by the token level policy π * Alg (•|x) in (17) . The KL divergence between our algorithm and the reference policy is denoted byD KL ρ * Alg (•| x), ρ sft (•| x). We present our main theorem as follows, and the full proof is deferred to Appendix G. Theorem 1. For the proposed Transfer Q ⋆ Algorithm 1, the following results hold.(1) Suboptimality gap for all x is upper bounded asEQUATION)where β is defined in (9) for baseline policy, and α is defined in (17) for decoding process. Here h α (x) ≥ 0 and its formula is defined in Appendix G.(2) Assume reward satisfies 0 ≤ r ≤ r max .then the divergence to SFT policy is given byEQUATION)Remark 1 ("Double Robustness" of Transfer Q ⋆ ). Theorem 1 indicates the suboptimality is bounded by βD KL (ρ * , ρ sft ), and this guarantees our algorithm will achieve high accuracy in two cases. 1. The penalty parameter β is small, 2. The SFT policy ρ sft is close to ρ ⋆ . In addition, our decoding design ( 17) is also effective for improving the performance. This is due to -αh α ≤ 0. Our decoding coefficient α could further reduce the suboptimality gap when it is appropriately tuned (e.g. choose α * = arg min α αh α ).Remark 2 (KL-Efficiency of Transfer Q ⋆ ). Via Theorem 1, the deviation from our algorithm to the reference policy is jointly controlled by parameter β and decoding parameter α. When both parameters are set large, our algorithm is more conservative and hence more KL-efficient. When the parameters are small, the KL deviation becomes larger.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: We discuss the assumptions and complete proof for each theorem provided. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experimental evaluations: 
 We present a comprehensive empirical analysis of both direct and indirect Transfer Q ⋆ , tested across various open-source datasets and state-of-the-art models [27] . Our findings demonstrate TQ ⋆ 's effectiveness in aligning language model outputs with specific target rewards. For implementation, we set the number of tokens sampled k = 10 and the decoding alignment parameter α = 1. We report ablations in Appendix J.3. Reproducibility is ensured through the use of publicly available resources. 1 . ARGS (SFT) and ARGS (DPO) refer to the reward modeling approach described in [26] to the SFT and DPO model respectively. Our analysis reveals that across all setups, TQ ⋆ consistently outperforms other competitive baselines summarized in Table 1 , demonstrating its superior efficacy. We report results on other evaluation setups in Appendix J. In (b), we compare (for Evaluation-1 setup) the trajectory-level KL Divergence between different decoding policies and the base model ρsft to show the effectiveness of the proposed approach compared to the state-of-the-art.Evaluation Methodology. For evaluation, we compare the performance of the response generated by the language model corresponding to each prompt in the test dataset. Following [26] , we limit the maximum length of the prompt and generated continuation to 128 and 2048 tokens, respectively. For all baselines, we utilize a greedy-based sampling method. The quality of the generated responses is assessed based on multiple attributes (including reward achieved, win-tie rate, coherence, diversity, etc.) using the following evaluation metrics [26] : Average Reward: We report the mean of the rewards for generations corresponding to all prompts in the test set. A higher mean reward score signifies that the model's outputs are better aligned with the attributes represented in the reward model. Diversity: This metric measures the ability to generate texts with a wide range of vocabulary. Specifically, it calculates the frequency of repeated n-grams in text. Coherence: We assess the semantic closeness between each prompt and its generated response using the cosine similarity of their SimCSE-generated [43] embeddings.Transfer Q ⋆ : Evaluations with Direct Transfer Decoding Experiment Details. For the direct transfer experiments, we consider our baseline model as a DPO [38] fine-tuned model aligned with the target reward. To evaluate the performance of Transfer Q ⋆ (denoted as TQ ⋆ in figures), we conduct experiments across multiple datasets and model architectures, encompassing 6 distinct tasks. Our experimentation is primarily based on the Ultrafeedback [12] , Berkeley Nectar [53] , and HH-RLHF (Helpful and Harmless) [5] datasets. For each task, we utilize the DPO [38] fine-tuned model as an aligned policy, with configurations listed in Table 1 . This comprehensive approach allows us to gauge the adaptability and efficacy of TQ ⋆ in various contexts, providing a robust measure of its capabilities.Discussion.In Figure 2 , we present the normalized average rewards for the first three setups detailed in Table 1 .(a) (b)Figure 3 : Diversity and Coherence analysis of generated responses. We observe that the responses generated using TQ ⋆ obtain the highest coherence and diversity. These results are based on the prompts from the Berkeley Nectar dataset.We report the results for other setups in Appendix J.1. We compare our proposed method TQ ⋆ with competitive existing approaches such as ARGS [26] , CD --[33]3 , and DPO [38] . To provide a clearer comparison of results, we normalize the average rewards (further details of normalization in Appendix I.1). We observe that across all setups, TQ ⋆ consistently outperforms the existing approaches by a large margin, highlighting its efficacy in learning token-level optimal policy during inference. Further, in Figure 3 , we report that TQ ⋆ not only produces responses with high rewards but also outperforms other decoding strategies in terms of diversity and coherence.GPT-4 Evaluation. To further understand the quality of the responses generated, we employ a GPT-4-based evaluation framework. Specifically, we use GPT-4 as a surrogate for human assessment. In this method, we prompt GPT-4 to assess and rate two responses on the same prompt on a scale from 1 to 10, focusing on criteria such as relevance, accuracy, and insightfulness. For this, we randomly sample 300 prompts from the test set and compare the response between TQ ⋆ and other competitive decoding methods. We present the GPT-4 evaluation results in Table 2 , measured by the percentage of win-ties of our method over the alternative decoding strategies. A higher percentage indicates that our proposed method is more proficient in generating responses that exhibit better alignment with human preferences. Our analysis in Table 2 shows that TQ ⋆ consistently has a higher win-tie percentage compared to other decoding approaches, reaffirming its efficacy.Synthetic experiments. In the synthetic experiments, we simulate four scenarios to examine shifts in the reward distribution between source and target reward models. These scenarios are instrumental in elucidating the advantages of our proposed method. The shift in reward distribution is achieved through model intervention techniques, such as inducing sparsity in the final linear layer. Our analysis utilizes the UltraFeedback [12] and Berkeley Nectar [53] datasets. The specifics of the models utilized are detailed in Table 4 in Appendix I.2. For each dataset, we design two transfer tasks by modulating the reward distribution. We visualize the shift in reward distribution on the Ultrafeedback dataset in Figure 4 (a) and (c) respectively. Given that this is a synthetic setup, no DPO-aligned policies exist for the newly generated reward distribution, which underscores the significance of addressing the transfer problem. We present the results for this analysis on the Ultrafeedback [12] dataset in Figure 4 .Due to space constraints, we report the results on the Berkeley Nectar [53] in Appendix J.2. We make the following key observations: (1) Our proposed decoding framework consistently attains the highest average reward and outperforms other competitive strategies.(2) The variant of our decoding strategy obtained by direct transfer to the target reward, i.e., DT has subpar performance. Real transfer experiments. To further evaluate our proposed approach on real reward transfer, we consider two transfer setups as outlined in Table 5 in Appendix I.3. We illustrate the distribution shift in reward values on UltraFeedback [12] and HH-RLHF [5] datasets in Figure 5 (a) and (c) respectively. We compare the normalized average reward values of different decoding strategies in Figure 5 . Consistent with our findings from the synthetic experiments, our proposed method consistently outperforms other decoding strategies, underscoring its effectiveness in real transfer tasks as well. Notably, we observe that even when there is a significant shift in the source and target reward values, as evident in Figure 5 (c), TQ ⋆ outperforms all competitive decoding approaches in terms of average reward. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We report the detailed evaluation setups in Section 4.Guidelines: • The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
conclusions: 
 In this paper, we introduce Transfer Q ⋆ , a novel decoding strategy for AI alignment. Our method effectively addresses the prior gaps in alignment via decoding by leveraging an estimate of optimal Q * for a target reward through an existing aligned policy ρ BL . Our theoretical analysis provides a rigorous characterization of the sub-optimality gap with respect to the token-level value function and the KL divergence to the reference policy. Experimentally, we demonstrate the consistent and significant superiority of the proposed approach. Hence, this work provides a principled solution to efficient decoding for AI alignment. Alignment via Fine-tuning. A prevalent approach involves employing an RLHF framework wherein a reward model is trained on human feedback, and proximal policy optimization (PPO) is used to derive the aligned policy model [5, 37, 2, 19] . Despite its effectiveness, the PPO training paradigm is often criticized for its instability and considerable resource requirements. Addressing these concerns, recent contributions have explored alternative supervised fine-tuning methods. Notably, Rafailov et al. [38] leveraged the Bradley-Terry [8] model to parameterize the reward policy, thereby obtaining the aligned model by optimizing a simple classification loss. Liu et al. [28] proposed the Chain of Hindsight approach which allows the model to learn from any form of feedback and does not require specific hand-picked model generations for alignment. Yuan et al. [51] introduced a framework that aligns model probabilities of multiple responses with human preferences through a ranking loss. Dong et al. [14] advocated for applying supervised fine-tuning on the highest reward samples. Lastly, Chen et al. [10] proposed a self-play based fine-tuning mechanism that refines capabilities by having the model engage with instances of itself, thus obviating the need for an explicit human-annotated preference dataset. While these fine-tuning-based methods have proven effective for aligning LLMs, they also pose significant computational demands and assume white-box access to model parameters to update them, which may not always be available. Notably, TQ ⋆ diverges from these training-based approaches, by providing a new decoding-time framework to align language models without requiring expensive fine-tuning.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We submit the code with the submission. Guidelines: • The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We report all hyperparameters and evaluation details in Section 4.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We discuss statistical significance. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We cite and properly credit for all the used assets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used.• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]  
 
 Appendix: Alignment via Decoding. The work by Khanov et al. [26] is one of the first to study the integration of alignment procedure directly into the decoding process. At each decoding step, Khanov et al. [26] proposed to adjust the generation probabilities of the model based on feedback from a reward model. Huang et al. [21] re-conceptualized the text-generation process as a search problem with LLMs as search agents. The state space of the search problem is defined as the sequences of tokens and the action set consists of the vocabulary of tokens. Specifically, given a prompt, Huang et al. employs a heuristic-guided search to generate responses. Liu et al. [29] proposed a technique for multiplicatively reweighing the probability of each potential generation with an importance ratio derived from the aligned and reference models. A related approach is controlled decoding (CD) [33] , a framework that approximates the decoding problem by collecting samples from the reference-based language model.First, we discuss the standard RLHF with Reward Learning and Policy Optimization pipeline as used in Ziegler et al., which has also been adopted in subsequent work [42, 5, 37] . It usually consists of three phases: (1) supervised fine-tuning (SFT); (2) preference sampling and reward learning, and (3) reinforcement-learning optimization.(1) SFT phase: RLHF typically begins with a generic pre-trained LM, which is fine-tuned with supervised learning (maximum likelihood) on a high-quality dataset for the downstream task(s) of interest, such as dialogue, instruction following, summarization, etc., to obtain a model ρ sft .(2) Reward Modelling Phase: In the second phase, the SFT model is prompted with prompts x ∈ P to produce pairs of answers (y 1 , y 2 ) ∼ ρ sft (• | x) induced by toekn level π sft (y | x). These are then presented to human labelers who express preferences for one answer, denoted as y w ≻ y l | x where y w and y l denotes the preferred and dispreferred completion amongst (y 1 , y 2 ) respectively. The preferences are assumed to be generated by some latent reward model r * (y, x), which we do not have access to. There are a number of approaches used to model preferences, the Bradley-Terry (BT) [8] model being a popular choice. The BT model stipulates that the human preference distribution p * can be written as:EQUATIONAssuming access to a static dataset of comparisons D = x (i) , yw , y(i) l N i=1where the trajectories are sampled from π sft and the preference feedback is samples from true p * (constitute the human feedback), we can estimate r via maximum likelihood. Framing the problem as a binary classification we have the negative log-likelihood loss:EQUATION)where σ is the logistic function. We can write the population version of the loss function in (22) asEQUATIONWe want to highlight the dependence of the loss function of on the language model ρ sft (• | x).(3) Fine-Tuning with RL Phase: During the RL phase, we use the learned reward function to provide feedback to the language model. In particular, we formulate the following optimization problemEQUATIONwhere ρ sft is induced by π sft (y | x). In the above formulation, β is a parameter controlling the deviation from the SFT model ρ sft . As derived in recent work by Rafailov et al. [38] , a closed-form expression of the optimal solution of problem in ( 24) is given byEQUATIONwhere Z(x) is the normalizing constant for x. It is important to note that the optimal policy in ( 25) is defined at the trajectory level (y is the generated trajectory for given x) and not at the token level for each y t which we denote by π.In this section, we provide the proof of Theorem 1. Theorem 2 (Restatement of Theorem 1). For the proposed Implicit Q * Algorithm 1, the following results hold.(1) Suboptimality gap for all x is upper bounded asEQUATION)where β is defined in (9) for implicit policy, and α is defined in (17) for the proposed decoding process. In (26) ,h α (x) = T -1 t=1 E z t ∼ρ * Alg (•|x) [D KL π * Alg (•|x, z t )||π BL (•|x, z t ) ] ≥ 0. (2) Assume reward satisfies 0 ≤ r ≤ r max ,EQUATIONWe first provide the proof for the suboptimality gap. For notation convenience, in this proof we rename suboptimality-gap defined in (18) as follows:EQUATIONRecall the definition of V * and V Alg , we have ∆ can be equivalently expressed as:EQUATIONwhere ρ * represents the distribution over the trajectories induced by the optimal token level policy π * , and ρ * Alg denotes the distribution over the trajectories induced by the proposed token level policy π * Alg (•|x). To proceed next, we add and subtract the term E z∼πBL(•|x) E τ ∼ρBL(•|x,z) [r([x, z], τ )] in the right hand side of (29) to obtain∆(x) = E z∼π * (•|x) E τ ∼ρ * (•|x,z) [r([x, z], τ )] -E z∼πBL(•|x) E τ ∼ρBL(•|x,z) [r([x, z], τ )] =:∆1(x) (30) + E z∼πBL(•|x) E τ ∼ρBL(•|x,z) [r([x, z], τ )] -E z∼π * Alg (•|x) E τ ∼ρ * Alg (•|x,z) [r([x, z], τ )] =:∆2(x).In order to derive an upper bound on the suboptimality gap ∆(x), we will develop upper bounds for ∆ 1 (x) and ∆ 2 (x) separately below.Upper bound on ∆ 1 (x): Let us consider the term ∆ 1 (x) in (30) asEQUATION= E τ ∼ρ * (•|x) [r(x, τ )] -E τ ∼ρBL(•|x) [r(x, τ )],where ρ BL is the trajectory-level policy that maximizes the constrained objective (9) (with s t replaced by x). This directly impliesE τ ∼ρ * (•|x) [r(x, τ )] -βD KL ρ * (•| x) || ρ sft (•| x) ≤E τ ∼ρBL(•|x) [r(x, τ )] -βD KL ρ BL (•| x) || ρ sft (•| x) ≤E τ ∼ρBL(•|x) [r(x, τ )] .where the last step uses the non-negativity of the KL-divergence. This is equivalent toE τ ∼ρ * (•|x) [r(x, τ )] -E τ ∼ρBL(•|x) [r(x, τ )] ≤ βD KL ρ * (•| x) || ρ sft (•| x) .Now plug the above into (31), we obtainEQUATIONUpper bound on ∆ 2 (x): We expand the terms ∆ 2 (x) as followsEQUATION+ E z∼π * Alg (•|x) E τ ∼ρBL(•|x,z) [r([x, z], τ )] -E z∼π * Alg (•|x) E τ ∼ρ * Alg (•|x,z) [r([x, z], τ )] ∆ 2 2 ,where we add and subtract the termEQUATION)where the inner term inside the first expectation is denoted by ∆ 3 . Note that ∆ 3 represents the similar structure as ∆ 2 in equation ( 30) conditioned on (x, z), where z is the next token. We proceed similarly as before and for simplicity of notations we represent x ′ := [x, z] and we can writeEQUATIONTherefore, we can keep the iteration and drive at (T is the termination step)∆ 2 (x) =∆ 1 2 (x) + E z∼π * Alg (•|x) [∆ 3 (x, z)] =∆ 1 2 (x) + E z∼π * Alg (•|x) [∆ 1 3 (x, z) + ∆ 2 3 (x, z)] =∆ 1 2 (x) + E z∼π * Alg (•|x) [∆ 1 3 (x, z)] + E z∼π * Alg (•|x) [∆ 2 3 (x, z)] =∆ 1 2 (x) + E z∼π * Alg (•|x) [∆ 1 3 (x, z)] + E z ′ ∼π * Alg (•|x ′ ) E z∼π * Alg (•|x) [∆ 4 (x ′ , z ′ )] =∆ 1 2 (x) + E z∼π * Alg (•|x) [∆ 1 3 (x, z)] + E z ′ ∼π * Alg (•|x ′ ) E z∼π * Alg (•|x) [∆ 1 4 (x ′ , z ′ )] +E z ′′ ∼π * Alg (•|x ′′ ) E z ′ ∼π * Alg (•|x ′ ) E z∼π * Alg (•|x) [∆ 1 5 (x ′′ , z ′′ )] +E z ′′ ∼π * Alg (•|x ′′ ) E z ′ ∼π * Alg (•|x ′ ) E z∼π * Alg (•|x) [∆ 2 5 (x ′′ , z ′′ )] = . . . = T t=2 E z 2:t-1 ∼ρ * Alg (•|x) [∆ 1 t ([x, z 2:t-1 ])],where ρ * Alg (•|x) is the algorithm trajectory distribution and z 2:1 is empty, z 2:2 = z, z 2:3 = {z, z ′ } and so on. Here (note z represents a single token and τ denotes the completion trajectory)∆ 1 t ([x, z 2:t-1 ]) =E z t ∼πBL(•|x,z 2:t-1 ) E τ ∼ρBL(•|x,z 2:t ) [r(x, z 2:t , τ )] -E z t ∼π * Alg (•|x,z 2:t-1 ) E τ ∼ρBL(•|x,z 2:t ) [r(x, z 2:t , τ )] . Denote x t = [x, z 2:t ], then above is equivalent to ∆ 1 t (x t-1 ) =E z t ∼πBL(•|x t-1 ) E τ ∼ρBL(•|x t ) [r(x t , τ )] -E z t ∼π * Alg (•|x t-1 ) E τ ∼ρBL(•|x t ) [r(x t , τ )] =E z t ∼πBL(•|x t-1 ) E τ ∼ρBL(•|x t ) [r(x t , τ )] -α • D KL π BL (•|x t-1 )||π BL (•|x t-1 ) -E z t ∼π * Alg (•|x t-1 ) E τ ∼ρBL(•|x t ) [r(x t , τ )] -α • D KL π * Alg (•|x t-1 )||π BL (•|x t-1 ) -α • D KL π * Alg (•|x t-1 )||π BL (•|x t-1 ) ≤ -α • D KL π * Alg (•|x t-1 )||π BL (•|x t-1) where the last inequality uses π * Alg is the optimizer for objective (17) . Hence, we have∆ 2 (x) = T t=2 E z 2:t-1 ∼ρ * Alg (•|x) [∆ 1 t ([x, z 2:t-1 ])] ≤ -α T t=2 E z 2:t-1 ∼ρ * Alg (•|x) [D KL π * Alg (•|x, z 2:t-1 )||π BL (•|x, z 2:t-1 ) ]Combining above with (32) , and denote z t := z 2:t-1 , we obtain∆(x) ≤ βD KL ρ * (•| x) || ρ sft (•| x) -α T -1 t=1 E z t ∼ρ * Alg (•|x) [D KL π * Alg (•|x, z t )||π BL (•|x, z t ) ].This completes the proof for suboptimality. Remark 1. We do mention that in the practical implementation of the Algorithm 1 is sampled-based, but our analysis considers the population level performance. Indeed, adapting our analysis to the data driven perspective and obtaining finite sample guarantee is standard, since the suboptimality gap Sub-Gap we defined is identical to the existing offline reinforcement learning literature such as [48, 50, 49] .To derive an upper-bound on the KL divergence of the distribution over the responses generated by our proposed algorithm with the reference policy given the prompt, we first expand upon the definition ofEQUATIONNext, we first expand upon the trajectory distribution induced by our algorithm asEQUATIONwhere the trajectory τ = [y 1 , y 2 , • • • , y T ] given the prompt x and corresponding token level algorithm's policy is given by π * Alg (•|x, y ≤t ). We know from the definition of our algorithm's policy equation ( 12) that π * Alg (z|x) = 1 Cα(x) π BL (z|x) exp ( 1 α • IQ * (x, z)). Now, using that we expand the equation (37) asρ * Alg (τ | x) = π * Alg (y 1 |x)π * Alg (y 2 |x, y 1 ) • • • π * Alg (y T |x, y 1 , •y T -1 ) (38) = 1 Cα (x) π BL (y 1 |x) exp ( 1 α • IQ * (x, y 1 )) • 1 Cα (x, y 1 ) π BL (y 2 |x, y 1 ) exp ( 1 α • IQ * (x, y 1 , y 2 )) • • • 1 Cα (x, y 1 , y 2 • y T -1 )) π BL (y T |x, y 1 , y 2 • • • y T -1 )) exp ( 1 α • IQ * (x, y 1 , y 2 • y T -1 ))where we expanded our algorithm's policy from definition in equation ( 37) and Cα 's are normalizing factors. Next,ρ * Alg (τ | x) = π BL (y 1 |x)π BL (y 2 |x, y 1 ) • • • π BL (y T |x, y 1 , y 2 • • • y T -1 )) (39) × exp ( 1 α IQ * (x, y 1 ) + IQ * (x, y 1 , y 2 )) + • • • IQ * (x, y 1 , y 2 • • • y T -1 ) ) × 1 Cα (x) Cα (x, y 1 ) Cα (x, y 1 , y 2 • y T -1 ))where we simplified the expression by re-arranging the product into similar terms over the trajectory. Next, we use the definition of the trajectory distribution ρ BL (τ |x) from the token level policy π BL (•|x) to getEQUATION× exp ( 1 α IQ * (x, y 1 ) + IQ * (x, y 1 , y 2 )) + • • • IQ * (x, y 1 , y 2 • • • y T -1 ) ) × 1 Cα (x) Cα (x, y 1 ) Cα (x, y 1 , y 2 • y T -1 ))Furthermore, we expand the definition of ρ BL (τ |x) from equation ( 25) to obtainρ * Alg (τ | x) = ρ sft (τ | x) (41) × exp ( 1 β r(x, τ ) + ( 1 α IQ * (x, y 1 ) + IQ * (x, y 1 , y 2 )) + • • • IQ * (x, y 1 , y 2 • • • y T -1 ) ) × 1 Z(x) Cα (x) Cα (x, y 1 ) Cα (x, y 1 , y 2 • y T -1 ))Now, we compute the ratio logEQUATION-log Z(x) -log Cα (x) -log Cα (x, y 1 ) • • • -log Cα (x, y 1 , y 2 • y T -1 )) Since reward 0 ≤ r ≤ r max , we have Z(x) = E τ ∼ρsft [exp( 1 β r(x, τ))] ≥ 1, and this implies -log Z(s) ≤ 0, similarly -log Cα (x) ≤ 0 so we arrive atEQUATIONwhich impliesEQUATIONHere, we establish that the ratio Zr(x) Zr BL (x) is a partition function. From (15), we note thatEQUATIONEQUATIONHence proved.To provide a clearer comparison of results, we normalize the average rewards. For example: let r DPO represent the average reward achieved by the DPO model across all generated responses to the test prompts. The normalized reward, rDPO , is calculated as: rDPO = rDPO-rSFT r TQ ⋆ -rSFT , ensuring that the results are scaled relative to existing methods.In Table 4 , we summarize the different datasets and model architectures used in our analysis of synthetic indirect transfer setups. We presented the results for the Ultrafeedback [12] dataset in Section 4.2. We compare the normalized average reward of different decoding policies on the Berkeley Nectar [53] dataset in Appendix J.1. Synthetic Setup-1 UltraFeedback Mistral-7B-α Zephyr-7B-α Mistral-7B-α Synthetic Setup-2 Berkeley Nectar OpenChat 3.5-7B Starling-7B-α Mistral-7B-αIn Table 5 , we summarize the different datasets and model architectures used in our analysis of real indirect transfer. Prompt: Can you determine the missing number in the sequence: 2, 6, 14, 30, 62, _? I challenge you to figure out the pattern used to generate each number in the sequence. Use your mathematical reasoning skills to predict the next number.ARGS: Good luck! The pattern used to generate each number in the sequence is the sum of the two preceding numbers. The first two numbers are 2 and 6, and the sum of these two numbers is 8. The next number in the sequence is 14, which is the sum of 6 and 8. The sum of 14 and 8 is 22, and the next number in the sequence is 30, which is the sum of 14 and 22.CD --: The sequence is an example of a geometric sequence, where each term is found by multiplying the previous term by a fixed number called the common ratio. In this case, the common ratio is 2, since each term is twice the previous term. To find the missing number, we can use the formula for the nth term of a geometric sequence: a(1 -r n )/(1 -r), where a is the first term, r is the common ratio, and n is the position of the term we want to find.The sequence is a geometric series, where each term is found by multiplying the previous term by 3. The first term is 2. The formula for a geometric series is: a * r n where 'a' is the first term, 'r' is the common ratio (the factor by which each term is multiplied), and 'n' is the position of the term in the sequence. In this case: a = 2, r = 3 (the common ratio).Prompt: You are given a set of rectangular tiles of varying sizes. Your task is to arrange them in a specific pattern to create a rectangle with the largest possible area. The catch is that each tile can only be used once and must be used in its entirety (i.e., it cannot be cut or modified). Additionally, you cannot overlap tiles or leave any gaps between them. Can you figure out the optimal arrangement of tiles to create the largest rectangle? Good luck!To solve this problem, we can use dynamic programming to find the optimal arrangement of tiles. We can define a 2D array dp[i][j] where dp[i][j] represents the maximum area of a rectangle that can be formed using tiles of size i x j. The base case is when i = 0 or j = 0, in which case dp[i][j] = 0. For each tile size (i, j), we can iterate over all possible positions (x, y) in the rectangle.CD --: The optimal arrangement of tiles to create the largest rectangle would depend on the specific sizes of the tiles given. However, a general strategy to maximize the area of the rectangle would be to start with the largest tile and then arrange the smaller tiles around it in a way that covers as much of the remaining space as possible. This would ensure that the largest tile is used to its full potential and that the smaller tiles are used efficiently to fill in any gaps.TQ ⋆ (Ours): To find the optimal arrangement of tiles to create the largest rectangle, we need to first find the largest tile and arrange it in a way that maximizes the number of smaller tiles that can be placed around it. Let's say the largest tile has dimensions a x b. We can place it in the top left corner of our rectangle, and then try to fit as many smaller tiles around it as possible. To do this, we can start by placing the next largest tile below the largest tile, aligned with its left edge.Prompt: Explain why this statement is hyperbole: I'm so hungry I could eat a horse.Explain why this statement is hyperbole. I'm so cold I could freeze to death. Explain why this statement is hyperbole. I'm so tired I could sleep for a week. Explain why this statement is hyperbole. I'm so happy I could burst. Explain why this statement is hyperbole. I'm so angry I could kill someone. Explain why this statement is hyperbole. I'm so in love I could die. Explain why this statement is hyperbole. I'm so bored I could scream.CD --: This statement is hyperbole because it exaggerates the speaker's level of hunger. While it is possible for someone to be extremely hungry, it is not physically possible to eat an entire horse. The statement is meant to emphasize the speaker's hunger and convey the idea that they are extremely hungry, but it is not meant to be taken literally.TQ ⋆ (Ours): This statement is hyperbole because no one can physically eat an entire horse.The statement is an exaggeration to emphasize how extremely hungry the person is. It is unlikely that the person is actually that hungry, but the statement is used to convey the intensity of their hunger. The use of hyperbole in this statement also adds a humorous or lighthearted tone to the statement, making it more relatable and understandable to the listener or reader.Prompt: Describe the phenomenon of carbon dioxide absorption in the atmosphere.Carbon dioxide absorption in the atmosphere is a natural process that occurs when carbon dioxide (CO2) is absorbed by the Earth's surface and vegetation. This process is essential for maintaining the balance of CO2 in the atmosphere and preventing it from accumulating to dangerous levels. Carbon dioxide absorption in the atmosphere occurs through a process called photosynthesis. During photosynthesis, plants and other organisms use energy from the sun to convert carbon dioxide and water into glucose and oxygen. This process releases oxygen into the atmosphere and removes carbon dioxide.CD --: Carbon dioxide absorption in the atmosphere refers to the process by which carbon dioxide (CO2) is taken up by the atmosphere from the Earth's surface. This process is essential for maintaining the balance of greenhouse gases in the atmosphere, which is critical for regulating the Earth's temperature. The primary sources of carbon dioxide in the atmosphere are the burning of fossil fuels, deforestation, and other human activities that release large amounts of CO2 into the atmosphere. meant to be taken literally.TQ ⋆ (Ours): Carbon dioxide (CO2) is a greenhouse gas that plays a crucial role in the Earth's climate system. It is naturally present in the atmosphere, but human activities such as burning fossil fuels, deforestation, and industrial processes have led to a significant increase in CO2 concentrations. When CO2 is emitted into the atmosphere, it absorbs some of the infrared radiation (heat) that is emitted by the Earth's surface and re-emits it back towards the Earth. This process is known as the greenhouse effect.Prompt: Classify the following incident as a breach of protocol. Output 1 for breach, and 0 for no breach: Using a school laptop for personal use.Taking a school laptop off the school premises. Taking a school laptop off the school premises without permission. Taking a school laptop off the school premises without permission and using it for personal use. Taking a school laptop off the school premises without permission and using it for personal use and downloading personal files onto the laptop. Taking a school laptop off the school premises without permission and using it for personal use and downloading personal files onto the laptop and deleting school files from the laptop.CD --: Using a school laptop to access non-school websites. Using a school laptop to access social media sites. Using a school laptop to access email accounts not associated with the school. Using a school laptop to access personal files. Using a school laptop to access personal files stored on a USB drive. Using a school laptop to access personal files stored on a cloud service. Using a school laptop to access personal files stored on a personal computer. Using a school laptop to access personal files stored on a personal mobile device. Using a school laptop to access personal files stored on a personal tablet.TQ ⋆ (Ours): A student using a school laptop for personal use is a breach of protocol because it violates the school's policy on acceptable use of technology. The school laptop is provided for educational purposes, and using it for personal use can compromise the security and integrity of the device, as well as consume valuable resources that could be used for educational purposes. This can also result in potential legal and financial consequences for both the student and the school, as well as damage to the school's reputation. Therefore, using a school laptop for personal use is a breach.1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: We clearly reflect our claims and contributions in the abstract and introduction. Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.Due to unavailability of code base, we compare using an approximate version of CD[33] in which we do not train an adapter module. 


Decision: The paper proposes a decoding-based approach for aligning LMs. The key idea is to derive an equivalent token sampling distribution that mimics the behavior of an existing LM that has been aligned, or a suitable approximation thereof. This approach leads to sizable gains over existing approaches like Controlled Decoding.

Reviewers were unanimously in favor of acceptance: they consistently found the paper to be well-motivated and presented, with interesting insights and results. One reviewer noted that there is scope for the paper to be more careful in its discussion and framing of prior work. We encourage the authors to incorporate this feedback in the final version of the paper.

Reviews: review: paper_summary: paper_summary: The paper proposes a new estimation of Q function for controlled decoding. Instead of using a Q function derived from SFT model, the paper propose to use Q function derived from separate aligned models. Evaluation shows the proposed method can achieve higher reward in benchmarks.
strength_weakness: strength_weakness: 1. The proposed approach seems computationally expensive to do decoding.
2. The approach require accessing an already aligned model, this adds more requirements for using the method and limits it use cases.
questions: questions: 1. Did the authors compare decoding speed for different approaches? It would be useful to report even thought the proposed approach is slower.
2. For gpt4 based evaluation, why the authors only report Win-Tie instead of splitting win and ties? In particular I am curious what is the percentage of ties for each comparison.
limitations: limitations: Limitations are discussed in the appendix.
paper_summary: paper_summary: The paper Transfer Q⋆: Principled Decoding for LLM Alignment proposes a novel approach to aligning large language models by leveraging a principled decoding strategy. The authors propose a method to estimate the optimal Q-function for decoding using an existing aligned policy, addressing limitations in previous approaches like Controlled Decoding (CD). The paper presents an indirect transfer method, allowing for alignment even when the baseline model is trained on a different reward function. The authors provide theoretical analysis characterizing the sub-optimality gap and KL divergence to the reference policy. Extensive experiments across multiple datasets and model architectures demonstrate the effectiveness of TQ⋆ compared to existing methods.
strength_weakness: strength_weakness: **Comparison with Baseline**s: The comparisons with existing baselines like DPO are insightful, but additional baselines, especially those focusing on inference-time control, could strengthen the evaluation.

**Hyperparameter sensitivity**: The paper does not thoroughly explore the sensitivity of TQ⋆ to its hyperparameters, particularly the decoding alignment parameter α. A more detailed analysis of this aspect would strengthen the work.

**Scalability**: While the method is tested on 7B parameter models, it's unclear how well it scales to larger models that are increasingly common in practical applications.

**Evaluation metrics**: While the paper uses several evaluation metrics, including GPT-4 based assessment, it lacks human evaluation studies. Given the subjective nature of language quality and alignment, human evaluation would provide valuable validation of the method's effectiveness.
questions: questions: 1. Have you investigated the stability of the alignment achieved by TQ⋆ over extended generation sequences? Does the alignment quality degrade for longer outputs, and if so, how does this compare to other methods?

2. How sensitive is TQ⋆ to the choice of baseline model? If multiple baseline models are available, each aligned with different rewards, how might one optimally select or combine them to estimate Q⋆ for a given target reward?
limitations: limitations: The authors acknowledge some limitations of their work, such as the potential for hallucination in responses to very obscure queries. However, they could improve their discussion of limitations by addressing:

1. The reliance on existing aligned baseline models, which may not always be available or suitable for all target rewards.
2. The potential for errors or biases in the GPT-4 based evaluation, which is used as a proxy for human assessment.
3. The computational overhead of TQ⋆ compared to standard decoding methods, which may impact real-time applications.
paper_summary: paper_summary: This paper addresses decoding for aligning large language models, which is a process of inference-time, token-level optimisation without updating the parameters of the LLM. Two scenarios are considered: 1. where a baseline policy is given and is aligned with the target trajectory-level reward. 2. where the baseline policy is given and aligned with a different trajectory-reward than the target one. Two approaches are provided in accordance: one named direct transfer, which as the name suggests directly derive a token-level policy from the baseline LLM, and the other named indirect transfer, whose key lies in an importance sampling trick to reweigh using the ratio between the target and baseline trajectory policies. The authors also provide theoretical analysis characterising the sub optimality gap and the divergence from the supervise-fine-tuned model. Empirical results demonstrates superior performance in several key metrics.
strength_weakness: strength_weakness: Morally speaking, it is not clear to me why should decoding this way works much better than directly taking $\rho_{BL}$- at least in the case of direct decoding. After all, TQ* is just the 1-step-action-value function of $\rho_{BL}$, so I think the readers would benefit from an explicit explanation of why this process of *obtaining $\rho_{BL}$* --> *estimating its (1-step-action) value function* --> *compute the closed form of the optimal policy of this value function with a KL term* should do better than just directly using the trajectory level policy which is already the (regularised) optimal of the reward function $r$. More concretely, it would be nice to have this shown as a theoretical result - a theorem that shows that the suboptimality gap of the policy proposed by the authors is smaller than $\rho_{BL}$. I appreciate that the authors have shown this empirically, though.

For the indirect case, the method proposed by the authors require access to the ground truth trajectory-level reward function. But if we had this in practice, would we not have just directly used it to compute $\rho_{BL}$? Thus, I think here it makes sense to add some explanation regarding why this is useful.

Finally, I find it a bit of a misleading juxtaposition with DPO - the point of DPO is that we no longer need to model the reward function, but in this work we rely on both the trajectory level optimal policy learned by DPO as well as the reward function which also needs to be learned. Both of these suffer from statistical/optimisation instability in their own right. So it is strong to assume you have both of these.
questions: questions: equation (2) - This seems like a typo since for i > 0, the first argument of R should also take in the so-far generated sequence z_0 ... z_{i-1}.

equation (15) - how is $Z_{BL}$ and $Z_r$ computed? This is needed later in the importance ratio, if I understood correctly, but I don't see how to compute these quantities which are usually intractable.
limitations: limitations: Aside from the limitations I mentioned in Weaknesses, the authors also acknowledges that TQ* suffers from increased time complexity. They suggest that this can be mitigated by training a small value function adapter as discussed in prior work.
paper_summary: paper_summary: This paper introduces 'Transfer Q*', which is a decoding strategy to align language models with target reward models without any fine-tuning. The main contribution of this paper is the estimation of the optimal Q* function, which is often unavailable in practice and is necessary for approximating the RL optimal policy, through DPO (Direct Preference Optimization). The paper first introduces an algorithm for direct transfer decoding using DPO-trained models as optimal baselines, and then presents indirect transfer decoding that uses these as conservative baselines. The authors describe the theoretical characteristics of this methodology and experimentally demonstrate that it achieves state-of-the-art performance on target reward models.
strength_weakness: strength_weakness: 1. Equation 5 on line 121 is introduced roughly. Its derivation should be described in the Appendix, or references discussing it should be introduced.
2. The introduction of Controlled Decoding (CD) is incorrect. It states that CD uses $Q^{\pi_{sft}}$ as a proxy for Q*, but in reality, it uses an energy-based model of $Q^{\pi_{sft}}$ and a value function through an external network as Q*. The difference between this work and CD is that CD uses FUDGE and Q function as a baseline, while the proposed TQ* uses DPO as a baseline reward model. Therefore, Figure 1 and its explanation are incorrect. Considering that the indirect method, which conservatively estimates the target model, is the contribution, Figure 1 should be drawn in reverse.
3. Equation 12 on line 175 is believed to be incorrect. Theoretically, the $\pi(z|s)$ used in this equation should be the reference model $\pi_{ref}$ before exploration, not the aligned model $\pi_{BL}$. The equation development started with the premise that the optimal policy can be expressed as an energy-based model of the reference model and reward model, but in equation 11, it roughly transitions from the reference model to the aligned DPO, losing theoretical justification. In this case, it's expected that the original distribution will collapse significantly and over-optimize to the reward model.
4. A crucial experiment on the trade-off between KL divergence and the reward model is missing. In RL, there's a strong trade-off in reward model optimization depending on how much KL penalty is given, so it's not sufficient to only check the performance against the reward model as in this paper's experiments. While Figure 2(b) touches on this, it's not enough. An experiment showing whether it demonstrates pareto-frontier performance in the trade-off relationship according to KL penalty is also needed. This experiment is particularly important given the use of the DPO-trained model as the reference model, as mentioned in weakness 3.
questions: questions: 1. In Figure 2 (b), when measuring the KL divergence for TQ*, which model was used to compare the decoding results? For a fair comparison, it should be measured against $pi_sft$, like the other baselines.
2. I'm curious about the inference cost. How do the memory requirements and time complexity compare to naive greedy decoding?
limitations: limitations: This paper has done truly promising work, but it has the following limitations:

1. Although DPO is used as a baseline, it is an offline trainer and there is a theoretical gap with the optimal Q*. This issue is addressed through the indirect transfer decoding part, but there is still room for improvement, and various discussions are likely to follow.
2. Due to the use of the DPO model as the initial policy, the theoretical justification is weakened, and a clear interpretation of the experiments is not possible.
3. In the process of training the DPO model, over-optimization towards the reward model is also controlled by the KL penalty. It would be interesting to add analysis and experiments on this aspect.

Comments: title: Additional New Experimental Evaluations
Additional Pareto Front Results for Two more Evaluation Setups
Thank you for the response.
Thanks for the response
Clarifications regarding core technical contributions and new experiments [Pareto Front plot] in rebuttal pdf
comment: Thank you for addressing my comments. I'm satisfied with maintaining my original score.
I appreciate the authors providing the new results and the response, I am satisfied with the response and raised my score by one.
Before providing my response, I would like to express my sincere gratitude for the numerous experiments, efforts, and time dedicated to addressing the review comments. You have demonstrated a wide range of attempts, which I believe will greatly contribute to the further development of the paper. I have attempted to provide my opinions in as much detail as possible below.

However, I still find it difficult to change my position. Fundamentally, the core insight that CD's value is estimated based on the SFT policy, thus addressing the gap with the optimal policy's value, is very meaningful. However, the key novelty of CD lies in its ability to perform decoding-time alignment solely by accessing next token probabilities without editing parameters. The improvement in Q* estimation through DPO in this paper certainly has merit. However, learning a model through DPO and then performing distributional shift based on DPO raises two major concerns, as I mentioned in my initial review.

Firstly, as discussed, there is a significant divergence from the original distribution. While you have demonstrated Pareto-optimality through experiments and provided a theoretical upper bound, when comparing overall 'average reward', you should adjust the hyperparameters $\beta$ that the baselines can commonly control, analyze the trade-off with divergence from multiple angles, and discuss this thoroughly. This is why I still feel that the experiments added in the rebuttal are not sufficient.

Secondly, following the paper's logic, if we abandon the motivation of 'no control over the pre-trained representations', CD could also learn a DPO model and find Q* based on the DPO model. This aspect felt unfair in my initial review. However, I believe this issue could be resolved by experimentally demonstrating and discussing the mismatch that would occur when applying a Q-function learned based on DPO to the base model in future work.

Therefore, I feel that a major revision is necessary regarding the overall tone of the current draft and rebuttal. However, I agree that the extent of revision has decreased based on the various arguments and discussions you have provided, and I will slightly increase my score accordingly.
Dear Reviewers and ACs, 

Thank you for your time and efforts in reviewing our paper and rebuttal discussions. 

***Additional New Experimental Results:*** To further strengthen our empirical evaluations, we ran experiments and obtained Pareto front results for two more evaluation setups: Evaluation 2 and 3 (details in the paper Table 1). We present the results in the form of tables here. We are committed to adding them for all the evaluations in the final version of our paper. 

-  ***For Evaluation 2 Setup*** (detailed in Paper Table 1): This table shows the value of KL and the corresponding win rate and shows that our proposed method outperforms the existing methods. 

| Method                       |          |       |       |       |       |       |       |       |       |
|------------------------------|----------|-------|-------|-------|-------|-------|-------|-------|-------|
| ARGS-DPO                     | KL       | 0.40  | 1.15  | 2.20  | 3.80  | 5.75  | 7.05  | 8.20  | 9.15  |
| ARGS-DPO                     | Win-Rate | 50.50 | 57.80 | 61.75 | 65.90 | 67.10 | 67.70 | 68.20 | 68.15 |
| $\text{CD}^{--}$             | KL       | 0.50  | 1.25  | 2.35  | 4.20  | 6.50  | 8.75  | 9.35  | 10.75 |
| $\text{CD}^{--}$             | Win-Rate | 50.75 | 62.85 | 68.90 | 72.70 | 75.40 | 76.00 | 76.15 | 76.30 |
| $\texttt{TQ}^{\star}$ (Ours) | KL       | 0.42  | 1.20  | 2.18  | 3.85  | 5.95  | 7.90  | 8.85  | 10.40 |
| $\texttt{TQ}^{\star}$ (Ours) | Win-Rate | **54.30** | **70.90** | **75.70** |**79.60** | **80.55** | **81.95** | **82.95** | **83.25** |





- ***For Evaluation 4 Setup*** (detailed in Paper Table 1): This table shows the value of KL and the corresponding win rate and shows that our proposed method outperforms the existing methods. 
 
 | Method                       |          |       |       |       |       |       |       |       |       |
|------------------------------|----------|-------|-------|-------|-------|-------|-------|-------|-------|
| ARGS-DPO                     | KL       | 0.37  | 1.26  | 2.05  | 3.71  | 5.86  | 7.14  | 8.36  | 9.23  |
| ARGS-DPO                     | Win-Rate | 50.10 | 58.32 | 62.10 | 66.13 | 67.32 | 67.89 | 67.41 | 66.02 |
| $\text{CD}^{--}$             | KL       | 0.45  | 1.32  | 2.39  | 4.36  | 6.58  | 8.85  | 9.50  | 10.89 |
| $\text{CD}^{--}$             | Win-Rate | **51.05** | 63.12 | 69.44 | 73.16 | 75.80 | 76.25 | 77.00 | 77.17 |
| $\texttt{TQ}^{\star}$ (Ours) | KL       | 0.38  | 1.27  | 2.11  | 3.95  | 6.05  | 8.03  | 8.98  | 10.58 |
| $\texttt{TQ}^{\star}$ (Ours) | Win-Rate | 50.86 | **69.45**| **73.28** | **76.19** | **79.20** | **80.19** | **81.00** | **82.16**|


We have thoroughly addressed all concerns and are more than happy to engage in further discussions if any additional issues remain. Thank you so much again for your consideration.
Thank you for your time and efforts in reviewing our paper and rebuttal discussions. 

To address your comment regarding the Pareto front plot in the experiments, we remark that we added the Pareto front plot on Evaluation 1 in the rebuttal pdf (Figure 3) [[link to pdf in openreview](https://openreview.net/attachment?id=6Il3qOI0FO&name=pdf)]. 

***Additional Experimental Results:*** To further strengthen our empirical evaluations, we ran experiments and obtained Pareto front results for two more evaluation setups: Evaluation 2 and 3 (details in the paper Table 1). We present the results in the form of tables here. We are committed to adding them for all the evaluations in the final version of our paper. 

-  ***For Evaluation 2 Setup*** (detailed in Paper Table 1): This table shows the value of KL and the corresponding win rate and shows that our proposed method outperforms the existing methods. 

| Method                       |          |       |       |       |       |       |       |       |       |
|------------------------------|----------|-------|-------|-------|-------|-------|-------|-------|-------|
| ARGS-DPO                     | KL       | 0.40  | 1.15  | 2.20  | 3.80  | 5.75  | 7.05  | 8.20  | 9.15  |
| ARGS-DPO                     | Win-Rate | 50.50 | 57.80 | 61.75 | 65.90 | 67.10 | 67.70 | 68.20 | 68.15 |
| $\text{CD}^{--}$             | KL       | 0.50  | 1.25  | 2.35  | 4.20  | 6.50  | 8.75  | 9.35  | 10.75 |
| $\text{CD}^{--}$             | Win-Rate | 50.75 | 62.85 | 68.90 | 72.70 | 75.40 | 76.00 | 76.15 | 76.30 |
| $\texttt{TQ}^{\star}$ (Ours) | KL       | 0.42  | 1.20  | 2.18  | 3.85  | 5.95  | 7.90  | 8.85  | 10.40 |
| $\texttt{TQ}^{\star}$ (Ours) | Win-Rate | **54.30** | **70.90** | **75.70** |**79.60** | **80.55** | **81.95** | **82.95** | **83.25** |





- ***For Evaluation 4 Setup*** (detailed in Paper Table 1): This table shows the value of KL and the corresponding win rate and shows that our proposed method outperforms the existing methods. 
 
 | Method                       |          |       |       |       |       |       |       |       |       |
|------------------------------|----------|-------|-------|-------|-------|-------|-------|-------|-------|
| ARGS-DPO                     | KL       | 0.37  | 1.26  | 2.05  | 3.71  | 5.86  | 7.14  | 8.36  | 9.23  |
| ARGS-DPO                     | Win-Rate | 50.10 | 58.32 | 62.10 | 66.13 | 67.32 | 67.89 | 67.41 | 66.02 |
| $\text{CD}^{--}$             | KL       | 0.45  | 1.32  | 2.39  | 4.36  | 6.58  | 8.85  | 9.50  | 10.89 |
| $\text{CD}^{--}$             | Win-Rate | **51.05** | 63.12 | 69.44 | 73.16 | 75.80 | 76.25 | 77.00 | 77.17 |
| $\texttt{TQ}^{\star}$ (Ours) | KL       | 0.38  | 1.27  | 2.11  | 3.95  | 6.05  | 8.03  | 8.98  | 10.58 |
| $\texttt{TQ}^{\star}$ (Ours) | Win-Rate | 50.86 | **69.45**| **73.28** | **76.19** | **79.20** | **80.19** | **81.00** | **82.16**|


We believe we have thoroughly addressed all concerns and are more than happy to engage in further discussions if any additional issues remain. Thank you so much again for your consideration.
Dear Reviewer, 

Thank you for your response. We are glad that our rebuttal responses were able to address your concerns. 

Regards,

Authors
Thank you for addressing my concerns with additional experiments and baselines for larger models. I appreciate your commitment to adding comprehensive human evaluation. Based on these improvements, I am satisfied with maintaining my original score for your paper.
> **Comment 1.1:** The author still lacks a theoretical foundation. Whether it's a Q-function or a Value function, they end up having the same representation in this algorithm due to the (both tractable or intractable) normalizer. 

 **Response:** Thank you for your comment. We apologize if we missed anything but this comment is not very clear for us to respond to. We have followed the standard definitions from the reinforcement learning literature [A]. We start by explicity defining the token level MDP in Section 2.1. We emphasize that the Value function is defined for each state (defined in line 210 in our paper), and the Q function is for each state and action (defined in Equation (2) in our paper). 
 
***Request to Reviewer*:** We kindly request the reviewer to please expand on why value and Q function will be the same and what is the meaning of the term "normalizer" in this context. We want to clearly understand the concern before responding. Thank you for your feedback and engagement.  
 
***Our Focus and Contributions:*** In our work, we want to emphasize that our focus is on the estimation of optimal Q* and show that it is better as compared to all existing decoding methods such as CD, ARGS, etc. We show this in theory as well as experiments. 

[A] Sutton, Richard S., and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
 
 > **Comment 1.2:** Furthermore, although a theoretical guarantee has been presented regarding the excessively diverging KL-div, sufficient experiments and defense have not been conducted. Therefore, I maintain my current assessment.

 **Response:** Thank you for your comment. We want to highlight that our work contributes on both theory and experiments side. 
 
***On the theory side,*** our work is the ***first to derive such a theoretical upper bound*** for both (1) suboptimality and (2) KL divergence for a decoding algorithm. There are no theoretical results in any of the existing works (such as CD, ARGS etc), which constitutes a unique and novel contribution of our work on its own. 
 
***On the experimental side,*** we have tested our proposed approach on six evaluations (in the main submission) and added two more large-scale evaluations in the rebuttal pdf [[link to pdf in openreview](https://openreview.net/attachment?id=6Il3qOI0FO&name=pdf)] . 

- For the KL divergence plot, we have Figure 2(b) in the main body and as the reviewer suggested, we added a Pareto front plot for evaluation 1 (***Figure 3 in the rebuttal pdf***) as well in the rebuttal pdf [[link to pdf in openreview](https://openreview.net/attachment?id=6Il3qOI0FO&name=pdf)] which clearly shows the superior performance of our method. 
- Additionally, our current comparison includes win rate, coherence, reward, and diversity, which are designed to approximate human preferences.

We are running more experiments and committed to adding them in the final version of our work. We believe we have addressed all the concerns, and are happy to engage in further discussions if any remain. 

Thank you once again for your time and consideration. Looking forward to your feedback.
The author still lacks a theoretical foundation. Whether it's a Q-function or a Value function, they end up having the same representation in this algorithm due to the (both tractable or intractable) normalizer. Furthermore, although a theoretical guarantee has been presented regarding the excessively diverging KL-div, sufficient experiments and defense have not been conducted. Therefore, I maintain my current assessment.