Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning
Authors: Joey Hong
Anca Dragan
Sergey Levine
Venue: ICLR 2025 Poster (2025)
Keywords: offline reinforcement learning
language models
dialogue
robotics
Primary area: reinforcement learning
TL;DR: We present a new offline RL algorithm specifically to fine-tune pretrained LLMs and VLMs better.

Abstract: Value-based reinforcement learning (RL) can in principle learn effective policies for a wide range of multi-turn problems, from games to dialogue to robotic control, including via offline RL from static previously collected datasets. However, despite the widespread use of policy gradient methods to train large language models for single turn tasks (e.g., question answering), value-based methods for multi-turn RL in an off-policy or offline setting have proven particularly challenging to scale to the setting of large language models. This setting requires effectively leveraging pretraining, scaling to large architectures with billions of parameters, and training on large datasets, all of which represent major challenges for current value-based RL methods. In this work, we propose a novel offline RL algorithm that addresses these drawbacks, casting Q-learning as a modified supervised fine-tuning (SFT) problem where the probabilities of tokens directly translate to Q-values. In this way we obtain an algorithm that smoothly transitions from maximizing the likelihood of the data during pretraining to learning a near-optimal Q-function during finetuning. Our algorithm has strong theoretical foundations, enjoying performance bounds similar to state-of-the-art Q-learning methods, while in practice utilizing an objective that closely resembles SFT. Because of this, our approach can enjoy the full benefits of the pretraining of language models, without the need to reinitialize any weights before RL finetuning, and without the need to initialize new heads for predicting values or advantages. Empirically, we evaluate our method on both pretrained LLMs and VLMs, on a variety of tasks including both natural language dialogue and robotic manipulation and navigation from images.

Published as a conference paper at ICLR 2025
Q-SFT: Q-LEARNING FOR LANGUAGE MODELS VIA SUPERVISED
Joey Hong
Anca Dragan
Sergey Levine
University of California, Berkeley (ixi hong, anca, svlevi @be @warkeeey..eeooo
ABSTRACT
Value-based reinforcement learning (RL) can in principle learn effective policies for wide range of multi-turn problems, from games to dialogue to robotic control, including via offline RL from static previously collected datasets. However, despite the widespread use of policy gradient methods to train large language models for single turn tasks (e.g.. question answering), value- based methods for multi-turn RL in an off-policy or offline setting have proven particularly challenging to scale to the setting of large language models. This setting requires effectively leveraging pretraining, scaling to large architectures with billions of parameters, and training large datasets, all of which represent major challenges for current value-based RL methods. In this work, we propose a novel offline RL algorithm that addresses these drawbacks, casting Q-learning
modified supervised fine-tuning (SFT) problem where the probabilities of tokens directly translate to Q-values. In this way we obtain an algorithm that smoothly transitions from maximizing the likelihood of the data during pretraining to learning a near- meaa-optimal Q-function during finetuning, Our algorithm has strong theoretical foundations, enjoying performance bounds similar to state-of-the-ar Q-learning methods, while in practice utilizing an objective that closely resembles SFT. Because of this, our approach can enjoy the full benefits of the pretraining of language models, without the need to reinitialize any weights before RL finetuning, and without the need to initialize new heads for predicting values advantages. Empirically, we evaluate LLMs and VLMs, on a variety of tasks including both natural language dialogue and robotic manipulation and navigation from images.
method on both pretrained
INTRODUCTION
Recently, some of the most impressive feats in Al have been performed through language models, which are pretrained on large-scale data and adapted to a wide range of downstream tasks (Bommasani et al. 1021 Many of these tasks, such as natural language dialogue robotic control, require
complex sequential decision-making powerful paradigm for solving such tasks (Mnih et al. 2013) Silver al. 2017] AlphaStar 2019).
Reinforcement learning (RL) Sutton & Barto (2018) is a
Furthermore, offline RL [evine et al. (2020) has been shown to do so from only static datasets, such as suboptimal demonstrations from any unknown behavior policy, without the need for any additional interaction. Though offline RL has been used to fine-tune large language models (LLMs) or vision language models (VLMs) (Ouyang al. 2022; Bai al. 20226), its usefulness has been limited to generating better single responses rather than multi-turn, sequential scenarios where RL should theoretically shine. For example, across various dialogue tasks. offline RL fine-tuning of LLMs does not reliably outperform supervised fine-tuning (SFT) (Sodhi et al. [023 Abdulhai et al. 2023). Furthermore, in the realm of navigation and control, popular VL.Ms are still fine-tuned for multi-task
control using SFT (Brohan et al., 2023b a: Collaboration al. 2024).
Single-turn problems, such as answering questions, can be tackled with policy gradient methods (Ouyang al. 2022, Rafailov et al. 2023), but sequential multi-turn problems, such as dialogue or robotic control, require sample-efficies methods that can utilize data to reason about the dynamics of the problem, which typically requires training value functions (Abdulhai et al.; 2023) Hong et al.) 2023). This is in multi-turn problems, the agent must plan their actions to optimize some long-temt objective. Although there many effective value-based RL methods that could be applied to

---

Published as a conference paper at ICLR 2025
Q volues are upčaned through regression staiinatyy values
Our objective is weighted enempy loss similar SFT
logits
Quealax
logits
logits
B-5
DVB
4:09
0.4
Q-leaming head
bor
direcily on head from pretmined model
X|
22
Fyr
from pretrained model adds new value head
4094
0.3
6/5
PM
Transformer
Thank
Our proposed approach allows us to directly leverage the logits from a pretrained model to train value functions. Prior approaches require separately initializing value head.
Figure I:
LLMs and VLMs, in practice such methods have been difficult to adapt to these models with the same effectiveness as policy gradients. We posit that this is due in part to a mismatch between the pretraining objective that these models use, i.e. maximum likelihood estimation, and the fine-tuning objective necessary to train value functions. This discrepancy means that fine-tuning using multi-turn RL may require discarding some of knowledge gained by maximum likelihood pretraining of LLMs and VLMs, including broad understanding of language, vision, and even sequential reasoning.
Specifically. we hypothesize two reasons for why fine-tuning foundation models using offline RL is unsuitable in practice. First, typical offline RL methods require regressing value functions that estimate how appropriate actions, such as an utterance in dialogue, are. Such algorithms, known as Q-learning, have achieved impressive results when applied on small networks (AlphaStar 2019 Mnih et al., 2013), but surprisingly attain disappointing performance when scaled larger ones (Sodhi et al. (023). Recent work has attributed this lack of scaling to instability in the value- learning objective, namely in regression towards non-stationary values (Farebrother et al. 2024). More importantly. major advantage of SFT is the potential to leverage existing capabilities of large pretrained models to drastically improve the efficiency when learning a new downstream task. However, language models are trained to predict likelihoods, but Q-learning instead aims predict action values; therefore, when fine-tuning, Q-learning algorithms discard the learned likelihoods in favor of only utilizing the underlying representations, which eliminates some of the useful prior knowledge within the pretrained models. We illustrate this in Figure] where value functions are trained must be trained via new head with reset weights.
In this work, we propose a new algorithm that remedies both drawbacks. Our key insight is simple: by adding weights to the traditional supervised fine-tuning objective. we can learn probabilities that conservatively estimate the value function instead of the behavior policy. In practice, our approach implemented by adding weights to the maximum likelihood objective, yielding weighted cross entropy loss where weights are target action values computed from the Bellman recurrence relations. By using this objective, we are able to avoid the unstable regression objective commonly used in value learning, as well directly leverage the initial likelihoods resulting from large-scale pretraining. Theoretically, we can show that such objective results in learned likelihoods that are a product the data distribution and Qavalues and that our approach is principled and results in performance bounds competitive with other state-of-the-- art approaches. Empirically, we demonstrate the effectiveness of method variety of tasks involving both LLMs, such language games and dialogue, as well as VLMss such as navigation and robotic manipulation.
2 RELATED WORK
Much of the recent work on reinforcement learning (RL. finetuning of L.LMs and VL.Ms uses policy
gradient methods and reward models learned from human feedback (e.g., RLHF) (Ziegler et al. 2020
Stiennon et al. 2020 Wu et al. 2021 Nakano et al. 2022 Bai et al.) 2022a Christiano et al. 2023
Rafailov et al. 2023), > or from handcrafted Al systems (e.g., RLAIF) (Bai et al. 2022b), to generate better responses to various queries. However, there is large discrepancy in the capabilities required to perform self-contained responses in single-step tasks, such as question-answertin and responses in multi-turn scenarios, such dialogue. Namely, the latter requires planning to optimize long-term
2

---

Published as a conference paper at ICLR 2025
objective, Various prior works provide evidence that existing enable language models with such planning capabilities (Bachmann & Nagarajan 2024).
methods are insufficient to
In principle, value-based RL (Lange et al.) 2012 Levine et al. 2020), specifically Q-learning, can learn effective policies for multi-step tasks that outperform pure imitation via supervised fine- tuning (Kumar al. (022). Many offline RL algorithms exist that reap the benefits of value-based RL using only static datasets, such as those currently used to fine-tune language models. Though offline RL algorithms require handling distribution shift (Kumar et al. 2019), where the learned policy selects out-of-distribution (OOD) actions with unpredictable consequences, many methods exist that effectively tackle this challenge (Kumar et al., 2020; Kostrikov et al. [021] Kidambi et al. 22200] Yu et al., [220]] (02T). Due to the promising benefits of offline RL on learning from demonstrations, algorithms have been proposed for learning L.IM policies to some success in robotic manipulation (Chebotar et al., (023) and language tasks (Snell et al. 2022). However, recent evaluation has shown that, on a variety of natural language tasks, Q-learning approaches are often outperformed by supervised ones (Sodhi et al. 2023 Abdulhai al. 2023). We hypothesize this is due to the mismatch between value-based RL line-tuning and maximum likelihood pretraining, and propose new approach that remedies this core issue.
There also exist a paradigm of supervised approaches called return conditioned supervised learning (RCSL), which learn conditional policies return via supervised learning objective (Brandton brener et al. 2022). The most notable algorithm is Decision Transformer (DT) (Chen et al. (021))
which can train LLM policies that outperform traditional offline RL methods that rely on Q-learning.
Though it performs well in practice, there is theoretical evidence that the ceiling of performance of such algorithms is below that of value-based offline RL. Specifically, Brandfonbrener et al. (2022)
showed that DT and similar approaches can only identify the optimal policy under stronger conditions on the offline data than value-based RL. Our proposed algorithm is similar to RCSL in that we also use maximum likelihood loss, but we learn values and reap the theoretical benefits of other value-based methods.
Recently, prior attempts have also been made to improve value-based RL algorithms for fine-tuning language models. [Chebotar et al.] (2023) propose Q-learning with transformer value functions in manipulation and control tasks by converting actions to sequences of tokens. We adopt their insight when evaluating on robotics tasks, but use a fundamentally different objective to learn values. Most similar to ours, Farebrother et al. (2024) propose replace the regression loss from Q-learning with cross-entropy loss by casting value learning as a classification problem. However, while the proposed method also converts value functions to distributions, these likelihoods are not naturally derived from the logits obtained from large-scale pretraining, and must instead be learned from scratch via separate head with reset weights. Therefore, like traditional Q-learning, they also suffer from being unable to leverage pretraining efficiently, unlike our approach whose likelihoods are directly initialized by the logits of pretrained LLMs VLMs.
3 PRELIMINARIES
Our work proposes a new RL. algorithm for fine-tuning language models, specifically for multi- turn tasks such as dialogue or manipulation and control. Language models operate over discrete vocabulary of tokens V, and are trained to maximize the likelihood the best next-token Im+1 given an input sequence (zo...., xm) of tokens, given by < (m,) In a multi-turn task such a dialogue, the tokens are words that are chained to form utterances, and the best next-token requires complex, sequential reasoning to understand the utterances so far and plan for the next one. Traditionally, this kind of reasoning can be learned via reinforcement learning (RL).
RL fundamentals. RL aims optimize agents that interact with Markov Decision Process (MDP) defined by a tuple (S, A, P. r, #1, (1,7), 7), where represents the set of all possible states, A is the set of possible actions, μ4 is the initial state distribution, and 7y is the discount factor. When action E A is executed at state s E S, the next state is generated according to s' ~ P(-|s, a), and the agent receives stochastic reward with mean r(s, a) [0, 1].
The Q-function Q (s, a) for a policy (-1s)) represents the discounted long-term reward attained by executing a given observation history 8 and then following policy æ thereafter. Q" satisfies the
3

---

Published as a conference paper at ICLR 2025
Bellman recurrence:
(= r(s,a) + +/Bsss (s"((s'aa')
The value function V considers the expectation of the Q-function over the policy V* Bewx(1ls) (Q*(s, a)). Meanwhile, the Q-function of the optimal policy Q" satisfies:
Q'(s, r(),a) PE.sa()) max Q" (s',
and the optimal value function is (s) max, Q* a). Finally, the expected cumulative reward is given by ((r) Exy-a, -m1 [V* The goal of RL is to optimize policy л(. s) that maximizes the cumulative reward J(TT) E,,, [V*(s
In offline RL, we are provided with dataset D {(s,, T,, + of size [D] N. We assume that the dataset D is generated i..dd from an effective behavior policy (a(a))) Many state-of-the-ar offline RL methods build on Q-learning which trains Q-function using parameters on dataset 'D by minimizing temporal difference (TD) error.
Ty (r -
(
CTD (0) = (ss,,,)))
(1)
where Qo(s, a) is the parameterized Q-function, and 0 parameterize a target network and is slow-moving copy of 0.
RL for language generation. Language generation can be viewed as an MDP, where states are sequences of tokens from finite vocabulary V (Ramamurthy agent initially observes are used as our initial state, so = (Zo..... ................................ F. where E V. Ví E [m]. At timestep 4, an action 0, V is some token in the vocabulary. As long as d, not special end-of-sequence <EOS> token, the transition function deterministically appends ar to state to form 81+1. Otherwise, the agent observes (potentially stochastic) responses from the environment, i.c. utterances by conversational partners in the case of multi-turn dialogue, or (Bl0.... Vo),, which also consist tokens in the vocabulary; then, the transition function appends both ae and responses ot state 8t. This continues until the last timestep T where obtain state 8y and the agent receives a deterministic reward r(T)..
al. 2023). All tokens that the
becomes clear that policy π (als) is language model that parses all the language tokens seen so far as the state, and computes a distribution over tokens as the next action to take. Recently, RL has been considered for learning policies that are LLMs VLMMs for difficult tasks such generalist robotic manipulation or dialogue. Because value learning is very different from traditional next-token prediction, preforming such fine-tuning requires reparameterizing the pretrained language model, such by adding value heads with independently initialized weights (Snell et al. 2023).
4 Q-LEARNING VIA SUPERVISED FINE-TUNING
We will now describe our proposed offline RL algorithm, which we dub Q-learning via Supervised Fine-tuning (Q-SFT). Concretely, instead of training value functions by fitting Q-values to their Bellman backup target via regression loss, we instead fine-tune directly on the probabilities learned from aarge-scale pretraining like in SFT- via weighted cross-entropy loss, such that the resulting probabilities also capture the desired Q-values.
4.1 LEARNING VALUES AS PROBABILITIES
Recently, large neural networks such as LLMs and VLMs have been successfully trained and fine- tuned on demonstration data using supervised learning. If we adopt the earlier multi-turn formalism in Section 3 and view these models as agents, such approaches train a policy n (als) with parameters by minimizing cross-entropy loss:
CCe(Ø) = E [log (a s)]
(2)
Because the resulting policy approximates the behavior policy (a]() xe Ft (a]s), this approach has also been dubbed behavioral cloning (BC). While BC scales well to complex tasks and networks, the
4

---

Published as a conference paper at ICLR 2025
resulting policy only be good as the behavior policy, which is insufficient when the dataset is not curated from expert demonstrations.
In contrast, Q-learning enables the learned policy greatly outperform the behavior policy (Kumar ctt al. 2022). by instead having the policy behave according to the estimated Q-values. This can be done via policy extraction, such as (als) 1l (a arg max, Qo(s, d')] or the entropy-regullarized variant π (a]s) x cxp(Qo(e a)). However, as alluded to earlier, the Q-function Qo a) cannot be naturally derived from pretrained language models, which output probabilities, and require modifying their architectures as in Figure
Our goal is provide way to learn Q-values for multi-turn RL problems with language models such that the Q-function can be initialized from a model pretrained via supervised learning (i.c.. maximum likelihood estimation), without the need reinitialize weights or add new heads to represent the Q-values. An autoregressive sequence model (e.g., a transformer) outputs the probability of each token conditioned on the past history. order to avoid adding new heads or reinitializing weights, the Q-values have to also be represented by these same probabilities. Furthermore, to maximize transfer from pretraining, we would like our proposed loss function to also closely resemble the maximum likelihood loss function used for pretraining.
We propose simple modification to the BC objective in Equation 2 Our modification hinges on the following observation. Let pe(a(s) represent the probability of action under state s, and are optimized via the weighted cross entropy loss
LWCE (0) = (e(sa))> [w[s, a) log (a 8) + (1 - w(s a)) log (a {)]
where vsss, a) are weights, and ag is some dummy action. The resulting probabilities that optimize this objective approximate Po (a(ss) se w(s, (a)s) for all a * agg- Our goal is, via proper choice of weights, to learn probabilities that are conservative estimates of the true Q-values Po a) ss Q" (s, a).
In order to do so, we require the following assumption on bounded total rewards:
Assumption 4.1. For any policy π, we have E 2 S 1.
This assumption has been made by multiple prior works without loss of generality (Ren et al. 2021 Kumar al. 2022). rewards can, in theory, be scaled without affecting the optimal policy the MDP. Furthermore, many tasks of interest, such as dialogue, have sparse rewards, where we observe success failure only after the conversation has ended.
Following the above observation, let us define the empirical Bellman probability operator B for transition (s, a, s') as
Po(a' 8¹)
B't (a 8) v y max
T a s')
Note that this is different from the traditional Bellman operator in that we additionally divide by क in the backup. Then, we consider the following weighted cross-entropy loss:
1 -
B' Pa
LwCE(#) E (( i papaa s) log +)+
Σ
log s)
(3)
[A| -1
a' a'sa
Here, we see that our loss an instance of weighted cross entropy loss with weights approximately equal to Bellman target values 8 (alas.). The primary difference is that instead of introducing dummy action, we equally distribute the leftover weight across the remaining actions. As we will show, this acts as a label-smoothing term that ultimately regularizes the probabilities. We will show later that the absence of sampling error, our learned likelihood function Po (a|s) satisfies Q" a) 2 Pa (a|s) 2 ( Q* (s, a). This means that we are able to effectively learn conservative estimation of the Q-function as a likelihood, without the need for optimizing a potentially unstable and poorly- scaling TD objective.
In addition, because probabilities are modeled directly by existing language models, we do not need to modify the parameterization of such models in order to perform such fine-tuning, i.c. by resetting weights or adding a new head. Namely, our likelihood function pa (a) can be directly initialized from the logits of pretrained LLM or VLM.
5

---

Published as a conference paper at ICLR 2025
4.2 THEORETICAL ANALYSIS
In the previous section, we motivated a new objective LWOr given by Equation 3 that learns modified probabilities over actions ps directly from the logits of a base LLM VLM. Here, we will show that such probabilities serve as a conservative approximation of the true Q-values. To simplify exposition, where instead of empirical operator B", we use the we consider a simple modification of Equation 3 true operator:
B' po 8) r(, 7BBy-P(1)a max po(a' s')
TTa(a' ss)
Note that it is simple to adapt our analysis to the empirical operator. Namely, it requires obtaining high-probability bounds of the form v(s,a), pa |s) B po s) S v Vn(s.o)
where C' is constant independent of 7/ This kind of inequality commonly arises in analysis of offline RL algorithms (Kumar et al. 2020 2022).
Our main theoretical result is that our learned probabilities satisfy being conservative estimates of the true value function:
Theorem 4.1. Let po be the likelihood function that arises from optimizing Equation uuing the true Bellman likelihood operator. Then, Pe satisfies
Q' (s, a) () 2 Q' a)Tz(a 8),
for all s E DD and
such that Q" (' a) -1
We defer proof of the theorem to Appendix Note that our probabilities are conservative only over actions that have non-negligible Q-values. In practice, we do not see this as problem as actions with negligible Q-values will not be chosen anyway. Overall, we show that while our objective looks very different from traditional value-based RL, our method still learns conservative value function.
So far, we have shown that theoretically, our algorithm achieves similar theoretical properties as value-based RI methods, even though our objective looks closer to supervised learning. Next, we will compare our approach to other RL algorithms adapted from supervised learning such as filtered conditoond supervised learning, and show why ours is beneficial.
behavior cloning
Filtered behavior cloning. Filtered BC attempts to adapt supervised fine-tuning to non-expert datasets by only training the top p-percent of trajectories by reward for E [0, 1J. While natural, this harms sample efficiency as our method, like value-based RL methods, is also able to extract meaningful knowledge from low-rewart trajectories.
Return- cettunnconddtioned supervised learning. RCSL remedics the issues with filtered BC by condition- ing reward during training extract multiple policies rather than just the expert one. However, argue that RCSL still fails to learn from low-reward trajectories as well as our approach. Specifically, our approach and others that learn value functions can learn from suboptimal trajectories by stitching them into a better policy (Fu et al.) 2020). However, Brandronbrener et al. (2022) showed that RCSL cannot perform stitching in general, which limits its effectiveness learning from suboptimal data.
4.3 PRACTICAL IMPLEMENTATION
Our objective in Equation 3 trains the model such that the predicted token probabilities match their Q-values. Our final step is to choose how to use these probabilities the final learned policy. Prior work performs policy extraction that learns policy regularized to be similar the behavior policy (Peng et al.) 2019 Kostrikov et al. 2021). Namely, is well-known that a policy parameterization 1s solution to the constrained optimization problem (Peng et al.
T(a 8) Tglaas exp( 3 (**((() a)
2019
Brandfonbrener et al. 2021)
arg max Eg-a" aaa (s)
Ex-ars [DKL (- (s) T.( |))] <
where 3 > 0 is hyperparameter derived from the Lagrange multiplier. Recall that we have already approximated Ty by optimizing an (ox()) over parameters Φ. Therefore, without any further training we extract policy whose probabilities of actions follow R(ass) Tz(a ()) exp (3 pe (als)) which can be computed at inference time using our learned probabilities Pa. estimated behavior policy Te. and tunable hyperparameter 0. Note that unlike prior works that explicitly require a policy

---

Published as a conference paper at ICLR 2025
Agent:
animal??
Language Games
WebShop
Observation: No
Observation: would
Agent: an inanimate object
bottle bright citrus deodorant for sensitive akin, and price lower than
Observation: Yes
Agent: is is found outside?
00 dollars
Agent:
bright
deoderant senaltive skin]
M
Observation:
ALFWorld
[8078CWRC1] [J]
Observation:
A
Bright
safe. You are in the middle of room.
Your task
W
O
R
Natural for Sensitive Skin Pregnancy and Breseffeeding, Contains Organic Calendula S-Ounoe $10.99 [B078CTKVXY]
Looking quickly around you, you see drawer 2, shelf 5, drawer 1, shelf 4,
sidetabbe 1, drawer shelf
Ginger Fresh Deodorant by Earth Mema Natural and Safe for Seneilive Skin, Pregnancy and Breaslfeeding, Contains
Robotic Manlpulation
Agent:
go
Organic Calendula 3-Ounce [B08KBVJ4XN]
Observation:
You arrive loc 4 On the shelf you
Agent: click (8078GWAC1.)
Agent: take vase 2 from shelf 6
Figure 2: Overview of all the evaluated tasks, spanning both text and image inputs. Solving all the tasks effectively requires our algorithm to be able be used to fine-tune LLMs, VLMs, and even robotics transformer models.
extraction step with additional training, our policy can be computed at inference-time. Namely, we can express our policy using only the probabilities that had previously learned. An overview of our entire method is provided in Algorithm [ We also provide implementation details such as hyperparameter selection for our experiments in Appendix
Algorithm Q-learning via Supervised Fine-tuning (Q-SFT)
Require: Dataset D {s; as Tay ke(N) hyperparameter / > 0
1: Initialize Φ, 0, from pretrained model.
2: Optimize behavior policy:
3: for each gradient step do
4: Update -
(Ø)
5: end for
6: Optimize likelihood model:
7: for each gradient step do
8: Update 0 + No V CWCCEE
9: Update target parameters: 0
a a8
10: end for
11: At inference time, policy probabilites become: 7(a () aa(a s) exp polaa s))
5 EXPERIMENTS
Our method combines aspects of both SFT and value-based RL training, and therefore compare our method to state-of-the-art methods from both classes, evaluating:
Whether our method improves over SFT methods by taking into account and optimizing over a multi-step task reward.
Whether our method improves on the stability and performance of previously proposed value- based RL methods for training LLMs and VLMs. Whether our method is better able to benefit from the pretraining of large models than previously proposed multi-turn RL methods.
In this section, we perform comprehensive empirical evaluation across a suite of different tasks to find positive answers all the above questions.
5.1 TASK DESCRIPTIONS
Contrary to many existing applications of RL on language models, such as RLHF OOyang et al. 2022) DPO (Rafailov al. 2023), our proposed algorithm is tailored for offline RL on mulfi-step
7

---

Published as a conference paper at ICLR 2025
tasks. Therefore, we consolidate variety of existing benchmarks where language model must make sequential decisions, arriving at the following suite of different tasks.
The first set of tasks include language games from the LMRL benchmark (Abdulhai et al. 2023). which is one of the first benchmarks tailored at evaluating offline RL for language generation.
Chess. This task uses textual representation of the game of chess. The offline dataset consists of trajectories by Stockfish 15.1 simulating various player strengths as the agent, playing against another Stockfish engine with Elo 1200. The reward is 1 for move that results in victory, 0 for a legal move and 11 for an illegal move. Our dataset consists of 625K trajectories of full games, in which the agent achieves an average return of 0.21.
Wordle. In the game of Wordle, the agent is given at most 6 attempts to guess a hidden 5-letter word. After each guess, the agent is told whether each letter in the guessed word is: (1) the hidden word and in the right position (green), (2) in the hidden word but not in the right position (yellow), or (3) not in the hidden word (gray). The agent receives a reward of 11 after each incorrect guess. The dataset consists of 20K trajectories by a suboptimal heuristic policy that achieves an average return of -4,12, originally collected by Snell et al. (2022).
Twenty Questions. The final language task is the dialogue game of twenty questions, where the agent tries to guess what hidden object is by asking a series of yes-or-no questions. The dataset consists of 100K conversations between an agent that is guesser and the oracle that chooses the hidden word. The oracle chooses the hidden work uniformly at random from 158 unique objects. The guesser and the oracle are both simulated using GPT3. 5 (OpenAl 2022). which is prompted to both generate questions and answer them factually. The agent receives a reward of -1 for each question that is not correct guess, up to minimum return of -20. The average return in the dataset is -1..3.
The next evaluation for fine-tuning LLMs as language agents - interactive web-based tasks that require using tools like search.
WebShop. an online shopping website environment where an agent processes unstructured text data (in the form of descriptions crawled from Amazon) purchase a product given some initial user specifications. At the end, the agent receives a reward between 0 and 1 depending on the similarity between the purchased and ground-truth desired item. The benchmark consists of 12k initial user instructions, of which we randomly held out 100 for evaluation. With the remaining instructions, we generate dataset of trajectories where we simulate an suboptimal agent by prompting GPT3. 5 with few-shot examples, following the prompts used by Yao et al. (2022).
Our method can be applied not only language models, but also to multimodal models. In the next experiment, we study the performance of our method on vision-based navigation with VLMs.
ALFWorld. This a popular text-based environment grounded in image observations (Shridhar et al. 2021). In this environment, the agent is tasked with solving one of 6 different task types, ranging from finding, moving, and manipulating different household objects within an embodied environment of 120 rooms. At each timestep, the agent observes textual description of its location and surroundings with an analogous image, and chooses a text action from a set of admissible actions. In this environment, we sample 10k trajectories consisting of random templated task description, and an attempted execution of the task within 30 timesteps by a prompted CPT3. 5 model for data collection. In the dataset, the agent only successfully accomplishes the task 34% of the time aggregated across all task types.
Finally, we also evaluate our method for training policies outside of language generation. Robotics is a popular domain in which offline RL has been proven effective for training per-token Q-values for continuous control (Singh et al.) 2020 Chebotar al., 2023). In these experiments, we do not leverage pretrained language models and simply test the effectiveness of the underlying RL algorithm.
Robotic manipulation. We consider the large-scale robotic manipulation control tasks from Siigh et al.] (222)). In the environment, the agent controls a 7-DoF mobile manipulator in front of a countertop surface to perform two types of tasks: pick up an object and place on the trap in front, or grab an object from the drawer, A sparse reward of is received if the agent accomplishes the task. Following (Singh et al. (020)), we collect 300k trajectories of randomized, scripted policies performing one of the two task types. The scripted policies achieve roughly 40% success rate.
We show illustrations of all the considered tasks in Figure
8

---

Published as a conference paper at ICLR 2025
language games
alfworld
Method
Chess
Wordle
20Q
Pick
Examine
Clean
Heat
Cool
Pick2
ReAct
4.96
- -13.2
45
19
17
7
12
24
SFT
0.11
3.81
17.33
38
0
18
ILQL
0.09
-2.08
-1..2
28
7
0
2
15
Q-SFT (ours)
0.15
-2.1
-13.1
39
19
14
18
21
Table I: Average scores (for language games), and success rates (for ALFWorld tasks) across 100 independent evaluations. Our method performs best or near-best across the table, and competitively with prompting much more complex model.
Method
Score
Method
Pick Object
Place Object Near Target
ReAct
0.60
BC
32
SFT 0.55
CQL
78
57
Offline ArCHer
0.57
QT
92
68
Q-SFT
0.63
Q-SFT
94
64
Table 2:
Average score across 100 held-out instructions in WebShop. Our method performs best, even against prompting much larger model.
Table 3: Success rate for 100 runs across robotic ma-
nipulation tasks. Our general method performs compet- itively with Q-transformer, a value-based RL method specifically designed for continuous control.
5.2 RESULTS
The goal of our empirical results to show positive answers to all the proposed research questions. In order to do so, evaluate multiple state-of-the-art supervised and value-based RL methods.
Evaluation.
We compare our method Q-SFT against three classes of competing algorithms:
Prompting: ReAct (Yao et al. 2022) an extension of chain-of-though prompting (Wei et al. 2023) where the pretrained language model is prompting to think and reason multiple steps in advance. We use GPT3. (OpenAl 2022) the LLM, and GPT4-V (OpenAl 2023) as the VLM.
Supervised learning: Supervised fine-tuning (SFT) on the offline dataset. In the case of using non-pretrained models, such as in the robotics task, we rename the method behavior cloning (BC).
Value-based RI.: Traditional offline RI. algorithms that perform Q-learning to learn value functions. We consider several different algorithms, depending on the task at-hand. In the case of language games and ALFWorld, we evaluate ILQL (Snell et al., 2023). popular approach for language generation. In WebShop, we consider an offline variant of ArCHer (Zhou et al. 2024) that performs best on the task from prior work. Finally, in robotics manipulation, et al., 2020) and Q-transformer (QT) (Chebotar et al. 2023), which are both popular and achieve state-of tatee-of-the-arr performance in continuous control.
consider both CQL (Kumar
Since the state-of-the-un LLMs and VLMs often only expose inference APIs, we instead train our considered methods the GPT2-medi LLM, which consists of 345M parameters (Radford et al. 2019). For ALFWorld, which requires VLMs, we use LLaVA-1. 6 model as the pretrained model (Liu al. 2023). Finally, for robotics, we a randomly initialized Transformer architecture modeled after the popular RT-1 model, which processes images and discretizes the action space into tokens (Brohan al. 2023b). Note that for the Chess and Wordle tasks, because their state and action space are unlike natural language, we replace the pretrained weights with random initialization. Therefore, these tasks, like robotics manipulation, only compares the methods in terms of their effectiveness as RI. algorithms.
Discussion We report results our evaluations in Tables and For fine- seeitunnng LLMs, Table T and 2 show that our Q-SFT method outperforms supervised learning, and different value- based RL methods, sometimes outperforming state-of-ther of-the-art by almost 30%. Similarly. for VLMs, as shown in Table I our approach beats supervised and value-based RL baselines, particularly hard

---

Published as a conference paper at ICLR 2025
0.5
Score (After 10% Training)
-14
Q-SFT
Q-SFT
-1
0.4
Success Rate
QT
ILQL
-66
0.3
-17
0.2
-88
0.1
-9
0.0
-20
20
30
40
50
345
774
1558
Training Steps (x1k)
Number of Parameters (x1M)
Figure 3: Success rate during initial training on the pick object task of the robotic manipula- tion benchmark. Though our method achieves similar final performance as Q-transformer, we perform much better on fewer samples.
Figure 4: Scores after training on 10% of the offline dataset the 20Q task, varying the size of the pretrained model. Our method benefits more from using more sophisticated pretrained models, suggesting our approach scales better.
task types with low success rate in the data. Our approach is also competitive with state-of-the-an prompting of GPT4-V, which contains about 30x more parameters than the base models using during training. Finally, in 3] we see that even without leveraging any pretraining, our approach competitive with state-of-the-art suggesting that our objective is effective for learning values. This can be attributed to the fact that our underlying objective is more stable optimize than traditional Q-learning ones, which require regression to non-stationary Bellman target values. Furthermore, in Figure 3 show the learning curve for our approach compares favorably to QT, showing that our method learns more quickly and achieves better performance in the low-data regime.
Finally, we want answer the last research question. We hypothesize that our approach benefits more from pretraining than existing value-based RL techniques, and verify this with an additional experiment. Specifically, we consider the 20Q task, and train both ILQL and Q-SFT policies against different models of increasing number of parameters, namely the GPT2-laree models, which are 2 and 5x larger than GPT2 -112-mediuu respectively. We also only train 10% of the original dataset, so that retaining prior knowledge from pretraining becomes crucially important. In Figure 4 we show the average return achieved by both methods across the different model sizes. We notice that for larger model sizes, Q-SFT significantly outperforms ILQL, implying positive answer the research question that our method retains knowledge acquired during pretraining better than existing value-bases RL.
and (PT2-xL
DISCUSSION
In this paper, we present Q-learning via Supervised Fine-Tuning (Q-SFT), a new offline RL algorithm where Q-values are learned as probabilities in objective that looks like supervised fine-tuning. Because of this, our objective can be directly optimized over the logits of pretrained LLMs or VLMs To our knowledge, this is the first algorithm that can perform value-based RL fine-tuning without requiring any changes to the architecture, such as adding in new value heads. This has a number of important benefits. First, our objective is instance weighted cross-entropy, which has been shown by prior works to be more stable to train than traditional value-based RL methods that require regression towards non-stationary target values. More importantly, our algorithm fully leverages the advantage of foundation models such as LLMs or VLMs, as our algorithm starts from the pretrained values. Theoretically, we show that our probabilities are conservative estimates of the true value function. Empirically, we compare our approach against strong supervised and value-based RL baselines on a variety of different tasks requiring LLMs, VLMs, and even robotics transformers. As future work, we aim to lision-language-zee aanuuage-actionn (VLA) models, where we expect to see even greater benefit (Kim et al. 2024). Another more interesting direction for futher investigation, is whether method can be adapted to also work online, as many recent works have considered online RI optimization of language agents (hhu al. 2024
probabilities, as opposed to randomly-initializ
approach to also fine-tune
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Marwa Abdulhai, Isadora White, Charlic Snell, Charles Sun, Joey Hong, Yuexiang Zhai, Kelvin Xu, and Sergey Levine. Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models, 2023.
DeepMind AlphaStar. Mastering the real-time strategy game starcraft ii URL: htlps://deepmind. com/blog/alphasttr-maasee reaterin--nnanr-eeee--iiia twwt--tttattg--eeeeee saame-starrrat-ii 2019.
Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield- Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie KerI, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer. Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dode Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022b.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.0723 2021.
David Brandfonbrener, William F Whitney, Rajesh Ranganath, and Joan Bruna. Offline without off-policy evaluation. arXiv preprint arXiv:2106.00 089000, 2021.
David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does return-co conditioned supervised learning work for offline reinforcement learning? In Advances in neural information processing systems, 2022.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang- -Wi Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-2: Vision-aanguage-cttion models transfer web knowledge to robotic control, 2023a.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Decksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Van- houcke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-1: Robotics transformer for real-world control at scale, 2023b.
11

---

Published as a conference paper at ICLR 2025
Yevgen Chebotar, Quan Vuong, Alex Irpan, Karol Hausman, Fei Xia, Yao Lu, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, Keerthana Gopalakrishnan, Julian Ibarz, Ofir Nachum, Sumedh Sontakke, Grecia Salazar, Huong T Tran, Jodilyn Peralta, Clayton Tan, Deeksha Manju- nath, Jaspiar Singht, Brianna Zitkovich, Tomas Jackson, Kanishka Rao, Chelsea Finn, and Sergey Levine. Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions. In 7th Annual Conference on Robot Learning, 2023.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin I Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint wxXi:2216.00344 2021.
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2023.
Embodiment Collaboration, Abby O'Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acomn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew Wang, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang. Chenfeng Xu, Cheng Chi, Chen- guang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng. Giulio Schiavi, Glen Berseth, Gregory Kahn, Guanzhi Wang. Hao Su, Hao-Shu Fang. Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik I Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jaehyung Kim, Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo. Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang. Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang. Jordi Salvador, Joseph J. Lim, Junhyek Han, Kaiyuan Wang. Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen, Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi "Jim" Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip. Mingtong Zhang, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Ning Liu, Norman Di Palo, Nur Muhammad Mahi Shafiulla,, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R Sanketi, Patrick "Tree" Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbecl, Priya Sundaresan, Qiuyu Chen, Quan Vuong, Rafael Rafailov. Ran Tian, Ria Doshi, Roberto Martín- Martín, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Lin, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shuran Song. Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair. Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada. Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davehev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vincent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiaolong Wang, Xinghao Zhu, Xinyang Geng, Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yao Lu, Yecheng Jason Ma, Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yue Cao, Yueh-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang. Yunfan Jiang, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen Zhang, Zipeng Fu, and Zipeng Lin. Open x-embodiment: Robotic learning datasets and rt-x models, 2024.
12

---

Published as a conference paper at ICLR 2025
Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, Aviral Kumar, and Rishabh Agarwal. Stop regressing: Training value functions via classification for scalable deep rl, 2024. URL
bttts:://arxiie wttgabs////ooooooo..ot...o...acs /
/ 22433... .09550
J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven 07219
reinforcement learning. arXiv, 2020. URL
Joey Hong, Sergey Levine, and Anca Dragan. Zero-shot goal-directed dialogue via rl on imagined conversations, 2023.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. arXiv preprint arXiv:2005.0595/, 2020.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishma, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burch- fiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open-source vision aision-language-ate model, 2024. 09246
Ilya Kostrikov, Jonathan Tompson, Rob Fergus, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. arXiv preprint arXiv:2103.08055 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances Neural Information Processing Systems, pp. 11761-11771, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.0477 2020.
Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. When should we prefer offline reinforcement learning over behavioral cloning?, 2022.
Sascha Lange, Thomas Gabel, and Martin A. Riedmiller. Batch reinforcement learning. In Reinforce- ment Learning, volume 12. Springer, 2012.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. arXiv preprint 0xXii:2005.0/643, 2020.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. URL
https: /ttp:v///axvv.../ wwtp//t.///./oooooooo.oo//s..a.a /ta////4/200000 08485
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 112.5513, 2013.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Floundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answering with human feedback, 2022.
OpenAl. Chatgpt, 2022. URL https::////// openai. http://ppaaaaaaaaaaaaaa hat.aaal.ooo//o/o/hhaaaaae chapgpttppcchhtppttt
OpenAl. Gpt-4v(ision) system card. 2023. URL
http://ppp..nnnancccch..oaoeeeee org
CD
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens. Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. 2022.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. srrantage-weightee regression: Simple and scalable off- off-policy reinforcement learning. arXiv preprint arXiv: 1900000///000777... 2019 2019.
13

---

Published as a conference paper at ICLR 2025
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2023.
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In The Eleventh International Conference on Learning Representations, 2023. URL htttps://openrvvii net forum did-mmmddddaaaddau
Tongzheng Ren, Jialian Li, Bo Dai, Simon S Du, and Sujay Sanghavi. Nearly horizon-free offline reinforcement learning. arXiv preprint arXiv:2103. 14077, 2021.
Mohit Shridhar, Xingdi Yuan, Mare-AAexandrr Côté. Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021. URL
https://arxt org/ abs 2010 0. 03768
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, al. Mastering the game of go without human knowledge. nature, 550(7676)3354-333 359, 2017.
Avi Singh, Albert Yu, Jonathan Yang, Jesse Zhang. Aviral Kumar, and Sergey Levine. Cog: Connecting new skills to past experience with offline reinforcement learning. arXiv preprint arXiv:2010. 14500, 2020.
Charlic Snell, Sherry Yang, Justin Fu, Yi Su, and Sergey Levine. Context-aware language modeling for goal-oriented dialogue systems. In Findings of the Association for Computational Linguistics: NAACL 2022, pp.
Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 10.11866//vll http///22..nnnnnnnaaaeee erg/2022. f Findingss ngs naacl... 181
URL htttp::/aclanther
Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language learning. In International Conference on Learning Representations (ICLR), 2023.
Paloma Sodhi, Felix Wu, Ethan R. Elenberg, Kilian Q. Weinberger, and Ryan McDonald. On the effectiveness of offline rl for dialogue response generation. In Proceedings of the 40th International Conference on Machine Learning, 2023.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT Press, second edition, 2018
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023.
Jeff Wu, Long Ouyang. Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2 109. 10862, 2021.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arX1v:2210.03629, 2022.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-basee offline policy optimization. arXiv preprint arXiv:2005.132239 2020.
14

---

Published as a conference paper at ICLR 2025
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. arXiv preprint arXiv:2102.08363, 2021.
Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl, 2024. URL
19446
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020.
15

---

Published as a conference paper at ICLR 2025
A THEORETICAL PROOFS
Here, we provide a proof of Theorem Recall that we are optimizing the following objective:
Lwor(p) = ) p(a 8) logp(a 18)
Σ
1-
B' p(a 5)
log p(a' 8)
(4)
-1
a'sa
Let consider iteration k of training. Setting the derivative of Equation 4 to O, we obtain the following expression of gh+1 in terms of p":
1- 1-B'T0 (a' 8)
Ys EDE,, p 8) = nT(a 8) (a 8)+ +
Σ
no(aa 8)
(5)
[A| -
a'sa
Lower-bound. We will first show the lower-bound part of Theorem 4.1 equation, we see that:
Rearranging the above
pa+ (a =B (a (a)
As(a' ())
1 p² (a' |s)
(6)
TB(a 8)
Tz(a s) [ -1
a'za
Hence, we see that
t B'p (a 8) rs) + +
/ (a b)
max 8taa 84) TT(a' 8r)
TT(a 2
a'
where we substitute the definition of B Finally, taking the fixed point of the above expression yields, P s) (a s) 2 as. = faa s) 2 TT(a (s)),,
as desired.
Upper-bound. Now, show the upper- bouud part of Theorem 4.1
Assume that
B (a))) 2 -1
Then, we can solve for the bound:
1 1 1--BTp*( (a' 1s)
(1 - Ts(a| | 8)) (ap
s) Σ Ts(a' (s)
|A/ -
a'sa
BB 18)
Σ Ta(a' ss
(a 8)
AA
a' za
(11-)
1
1- B (a' (3)
Σ Ta(d (s)
[ [1
2 0.
| -
a'
This means that we have,
1- 1- (a')
ght1(a (a 8) = m s)B'F((a (1))+ + Σ To(a' (8)
AAI -1
a'za
(a' (s)
1 1-BB-p(0' (a'
= =B'ntta 8) - -1-11 111-mph as(a 8)) B"pt(a (a +)+ + Σ
Tolaa (s)
IA|-1 -1
a'zka
< <B |s)
Hence, we see that
(a B (a
((s. a) + P((o,)) max 7t 88) ( ( 8²)
T3(a 8)
n. 8)
Ta 8)
Finally, taking the fixed point of the above expression yields the desired p(a 8) S Q- (s, a). This completes the proof.
16

---

Published as a conference paper at ICLR 2025
B IMPLEMENTATION DETAILS
We use the hyperparameters reported in Table 4 All algorithms were trained on a single TPUv3 on Google Cloud until convergence.
Hyperparameter
Chess
Wordle
20Q
WebShop
ALFWorld
8.0
4.0
1.0
1.0
1.0
Y discount factor
0.99
0.99
0.95
0.9
0.95
Batch size
128
128
128
128
128
Target network update
0.005
0.005
0.005
0.01
0.01
Number of updates per iteration
60
60
60
50
50
Number iterations
100
100
200
200
Ap learning rate
le-4
le-4
le-4
2e-4
3e-4
As learning rate
le44
le-4
1e-4
2e-4
le-4
Table Hyperparameters used during training Q-SFT our experiments.
As shown in Table most hyperparameters were held the same except for 3, where larger 3 results in more deterministic policy. In practice, only had increase 3 for tasks with restricted action spaces (such as games). Hence, we can conclude that in most practical tasks, our method does does require much hyperparamter tuning to perform well.
17


Reviews:
[{"review_id": "lNsnNCpPq1", "reviewer": "Reviewer_oykW", "summary": "This paper proposed Q-SFT, a method that integrates Q-learning with SFT for LLMs in the multi-turn RL setting. To exploit both the representation and the logits in LLMs learned in the pretraining stage and bypass the poor scaling effect of TD-style objectives, the authors proposed a novel weighted cross-entropy loss that embeds the learning of the Q-value into the weights. In this way, the logits prior in the pretrained LLMs can be leveraged, and no new head is needed to learn the Q-values. Theoretical analysis has shown the guarantee of the new learning objective leading to a conservative estimation of the value function. Experiments on several scenarios have demonstrated the effectiveness and efficiency of the proposed method.", "questions": "- While the theoretical analysis shown in Section 4.3 demonstrates that the Q value learned with Equation (3) has the bounds independent from the hyperparameter \\gamma, I wonder what the ablating effect \\gamma brings in practice: Will certain configuration of \\gamma contribute the learning of a better Q-value estimation? This question has another motivation: In Equation (3), some of the probability mass is allocated to the dummy actions a', and the weight ratio between the actions in the demonstration trajectory and the dummy ones has one degree of freedom determined by \\gamma. It would therefore be great to sweep \\gamma to study the ablation effect of the weight ratio.\n   - And by the way, how is the \\gamma set in the demonstrated experiments?\n\n- According to Equation (3), learning the Q-value function requires a learned behavior policy \\pi_\\beta in prior. How will the quality of the behavior policy impact the learning of the Q-values, and the performance of the ultimate policy? While I appreciate the experiments in Figure 4 where only 10% of the training data are used to get the \\pi_\\beta, I wonder how the performance grows with the quality of the initial behavior policy. For example, if the initial behavior policy is near perfect, will it be that the ultimate policy, although weighted with the learned Q-values, performs similarly with the initial one?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The work targets at the multi-turn RL scenario for LLMs, which is vital for the development of complex reasoning and agentic use in LLMs.\n- The proposed algorithm is backuped with a solid theoretical motivation.\n- The experimenting scenarios have a broad coverage, and the performance is well-demonstrated.", "weaknesses": "Please see the Questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "frsoDCjnzD", "reviewer": "Reviewer_WNH5", "summary": "This paper proposes a method that leverages the LLM logits learned via supervised fine-tuning to approximate the Q-learning objective. Authors argue that traditional Q-learning setup suffers from discarding the logit head, and learning a new head for Q values, so the proposed method directly translates the learned logits to Q values, with theoretical analysis on its bound and approximation. Experiments include text-based games, image-based navigation and robotic manipulation, and show some benefits of the proposed method over the prior value-based RL method.", "questions": "Please see the previous section.\n\nResponse to the rebuttal: (posted here since the rebuttal ended)\n\nThanks for all the clarification and discussion -- I raised my scores accordingly. It would be very valuable to revise the paper to address the points we discussed, especially the clear definition on step vs turn, the potential to scale better than existing RL approaches, the setup and comparable work of SFT baselines, motivation to alleviate the downsides of using RL, etc.. The rebuttal was helpful to better understand this work, and it would definitely be better if readers only need to read the paper itself to sufficiently appreciate this work.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "- Using probability prediction in the Q-learning objective is cool, and it sounds particularly beneficial for large vocabulary/action space. \n- The implementation of the method is straightforward: learning two models using different objectives, and use both at inference time", "weaknesses": "- I found the pitch very confusing: it emphasizes multi-turn problems like dialogue, but the actual math formulation treats each turn as a single token in an utterance (i.e., the action space is the vocabulary space). The generation process is to generate an utterance token by token, which is conventional, and it’s confusing where the multi-turn setup plays a role other than using this generative model repeatedly to generate different utterances at different parts of the dialogue. In other words, the value function here is not the reward per utterance as stated in the intro (line 074), and question answering is also not a single-step problem as the answer is also generated token by token.\n- Experimental setup seems simplified in terms of 1) for language-based tasks, the game nature makes the effective vocabulary very small and unambiguous, 2) the well-defined reward function setup in text games do not translate into practical problems, 3) why no supervised learning results from prior work as a direct comparison?; 4) most of the datasets already have RL method outperform supervised learning which is contradictory to the intro “offline RL fine-tuning of LLMs does not reliably outperform supervised fine-tuning”. Therefore, it’s unclear whether the improvement brought by the proposed method should be contrasted to what exactly.\n- The writing can be further improved in terms of clarity, accuracy, and fixing typos. A few examples below: \n   - Table 1: number from the proposed method is bold in the Wordle column, but it’s not the best performance method\n   - Define h used in line 162\n   - Prime should be on a, instead of argmax in line 218\n   - Consistently use “RL” instead of having “RI” occurred \n   - Citation should be within a pair of parentheses: line 037, 039", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MXqgxw2Jj9", "reviewer": "Reviewer_ChGQ", "summary": "This paper proposes a new RL training method for LLMs that take advantage of the pretraining. It is based on Q-learning, but instead of attaching a new value head, the method uses the existing token probabilities as value function. There is some theoretical guarantees and diverse experimental results involving multi-steps interactions.", "questions": "- Some typos: L39 citation, L84 sentence, L194 “performing”, Fig2 top left, \n- Eq2 should have a negative sign? Also tab1 -2.08 should be bold, not -2.11. \n- The figure 1 was bit too small and hard to read.\n- L153 defines RL as MDP, but what about POMDP?\n- About the theoretical guarantee, how good is the lower bound? A probability from a policy can be quite low for large vocabulary, in which case then the bound is not that tight?\n- How many models needed to be loaded during training? There is the main model, also a target model and the original model. So is that 3? How does that affect memory usage?\n- Given that some baselines use different base models, it would be nice to put them in the table. This will help if they perform better because their base model is stronger or not.\n- In fig3, the error bars correspond to different seeds?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Novelty: the method seems to be quite novel. The way it works is quite different from a typical Q-learning where L2 loss is used. Instead it relies on a cross-entropy loss that is modified. Given the importance of RLHF in LLMs and the need of multi-step interaction, a study like this can be impactful and important.\n- The experiments are quite diverse, ranging from LLM-based text tasks to VLM-based robotic manipulation. It is also compared to a diverse set of baselines, which helps to solidify the claims.\n- There is a proof about a theoretical guarantee, which is always nice to have. The method also leads to an improvement in performance in practice too, where the gap increasing as the model scales. Given the lack of value-based approaches in LLMs, it would be interesting to see if this method will adopted widely.\n- Paper is well written and easy to follow.", "weaknesses": "- I felt like the writing of the experimental results were a bit too short. It would have been good to have more ablation studies to understand why the method works, and in which situations it is more optimal. For example, since the main motivation is that the value head is not initialized from scratch, what will happen if we do that but keep the loss the same. Also there is not much on the training details, such as the number of training steps etc. I think the other parts can be condensed to make space. \n- Another important part that was too short was the method itself. The method is introduced between L230-261, which is about only half page. It doesn’t give enough explanation about why this method should work, so it was quite hard to understand it. For example, can you give more insight into why the proposed method should be more stable? \n- Reuse of the pretrained weights is emphasized as the main motivation, but some of the tasks actually train the model from scratch and others are not in natural language, which is a bit conflicting. There is still an improvement when trained from scratch, which is good, but there is a lack of explanation why it works better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qtx0VngKOm", "reviewer": "Reviewer_zr9o", "summary": "This paper proposes Q-SFT, a new approach that enables fine-tuning of LLMs and VLMs within an offline RL framework but using a supervised learning-style loss function. The method matches or outperforms previous approaches across a range of LM fine-tuning benchmarks.", "questions": ".", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation is clear: the instability of the Q-learning objective is a well-known challenge in the RL literature.\n- The proposed method is straightforward and demonstrates strong empirical performance, with improvements in sample efficiency and scalability over previous methods.\n- The evaluation covers a wide range of benchmarks.", "weaknesses": "It seems there are some limitations to the proposed approach that are not addressed in the paper.\n- During inference, the approach requires double the number of forward passes—one for the original pretrained LM and one for the fine-tuned LM. This can be challenging in practical scenarios where real-time decision-making is essential, such as robotic manipulation.\n- In Theorem 4.1., the lower bound for the learned p_theta is Q* multiplied by pi_beta. If pi_beta is small, the resulting pi_theta can be overly conservative compared to the true Q*.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)