Offline RL for Natural Language Generation with Implicit Language Q Learning
Authors: Charlie Victor Snell
Ilya Kostrikov
Yi Su
Sherry Yang
Sergey Levine
Venue: ICLR 2023 poster (2023)
Keywords: offline reinforcement learning
natural language processing
dialogue
controlled generation

Abstract: Large language models distill broad knowledge from text corpora. However, they can be inconsistent when it comes to completing user specified tasks. This issue can be addressed by finetuning such models via supervised learning on curated datasets, or via reinforcement learning. In this work, we propose a novel offline RL method, implicit language Q-learning (ILQL), designed for use on language models, that combines both the flexible utility maximization framework of RL algorithms with the ability of supervised learning to leverage previously collected data, as well as its simplicity and stability. Our method employs a combination of value conservatism alongside an implicit dataset support constraint in learning value functions, which are then used to guide language model generations towards maximizing user-specified utility functions. In addition to empirically validating ILQL, we present a detailed empirical analysis of situations where offline RL can be useful in natural language generation settings, demonstrating how it can be a more effective utility optimizer than prior approaches for end-to-end dialogue, and how it can effectively optimize high variance reward functions based on subjective judgement, such as whether to label a comment as toxic or not.

Published as a conference paper at ICLR 2023
OFFLINE RL FOR NATURAL LANGUAGE GENERATION WITH IMPLICIT LANGUAGE Q LEARNING
Charlie Snell, Πya Kostrikov, Yi Su, Mengjiao Yang, Sergey Levine UC Berkeley; (csne1122, kostrikov, suyi, sherryy, svlevine htt.y//oe.ebrrrko......oooo
ABSTRACT
Large language models distill broad knowledge from text corpora. However, they can be inconsistent when comes to completing user specified tasks. This issue can be addressed by finetuning such models via supervised learning on curated datasets, or via reinforcement learning. In this work, we propose a novel offline RL method, implicit language Q-learning (ILQL), designed for use language models, that combines both the flexible utility maximization framework of RL algorithms with the ability of supervised learning to leverage previously collected data, as well as its simplicity and stability. Our method employs a combination of value conservatism alongside implicit dataset support constraint in learning value functions, which are then used to guide language model generations towards maximizing user-specified utility functions. In addition to empirically validating ILQL, we present a detailed empirical analysis of situations where offline RL can be useful in natural language generation settings, demonstrating how it can be a more effective utility optimizer than prior approaches for end-to-ent dialogue, and how it can effectively optimize high variance reward functions based on subjective judgement, such as whether to label comment as toxic or not
INTRODUCTION
Large language models can acquire remark- able amount of knowledge from large text cor- pora, and can be applied to a wide range of language- based tasks. However, such models are not designed to optimize any user-specified utility, instead requiring considerable trial and error to design prompts that coerce the mod- els into producing desirable outputs (Liu al. 2021:Brrow et al. 2020; Min et al.) 2021). In essence, standard unsupervised language model training only solves part of the problem, being effective at distilling down knowledge in large corpora, but relatively clumsy when applying this knowledge to solve user-specified tasks.
supervised finetuning
offline (this paper)
Task Speallo
lave aici day
Manal ride ciy
Finetuning
Offline RL
L. (4)
Valion Model
Reinforcement learning (RL) in principle provide an effective framework for steering lan- guage models toward user specified tasks long
Figure I: Offline RL differs from supervised learning in that it learns to maximize user-specified rewards from suboptimal data with reward labels.
as the task can be represented by some utility function (i.e., a reward function); however, outlined in gegure@contepporaaa methods suffer from high systems complexity and can require expensive human interaction. We need several conditions make RL practical: (1) Easy use: the underlying learning algorithm and workflow should be simple, stable, and scalable; (2) Able to optimize user specified rewards: the algorithm should be able to steer language model toward maximizing any user-defined reward signal, from high-level task goals (e.g., book flight) to low-level linguistic subtleties (e.g., avoiding rude or toxic speech); (3) Practical for interactive applications: the system should be able to handle variety of tasks,
Cooee httphhtt/////.. snell .github. 10/ 11@1_site/

---

Published as a conference paper at ICLR 2023
from generating text with desired properties sequential turn-taking in settings such as dialogue tasks; (4) Able to leverage existing data: such a system should be able to directly utilize the large quantities of existing data, avoiding expensive and tmm-consuming online human interactions; (5) Temporally compositional (Emmons et al. 2021 Rafols et al. y the method should be able to attain significant improvement over the average behavior in the data not merely copying the best behaviors in the dataset, but actually distilling out underlying patterns in the relationship between rewards, task dynamics, and language to produce near optimal generations, even when the dataset demonstrates only mediocre task performance.
Offline RL provides learning paradigm (Fig- ure I that combines both supervised learning's ability to leverage existing data (criteria 4) with RL's ability to optimize arbitrary rewards and leverage temporal compositionality (criteria 2,
Levtage Inkuatire
Concvvations
V
V
x
x
X
3, 5) (Levine et al. 2020, Kostrikov al., 2021
Kumar et al. 2020; Janner et al.] 2021 Chen
et al.] 2021; Yu et al., 22020; Kidambi et al.
Figure 2: ILQL meets cach of the five criteria for prac-
2020). However, prior offline RL approaches for
tical NLP RL methods we outline in Section [
language tasks are either based on dynamic programming, which enjoy the temporal compositionality
(see Appendix A.2) but suffer from high systems complexity, hyper-paramen
instability, and slow
training times Veerm et al.) 2022 Jaques et al.) 22020) 2017) (meets criteria 5, fails 1), or methods based on conditional imitation or dataset value learning that are simple and stable to train, but do not
provide the temporal compositionality enjoyed by "full" RL methods (meets criteria 1, fails 5) (Chen
2021 Snell et al. 2022 Holtzman et al. 2018: Yang & Klein, 2021 Li et al. 2017: Krause et al.
et al. 2021p. ). Motivated by all these criteria, we design a novel offline RL method based on dynamic
programming with an implicit dataset support constraint (Kostrikov et al. (02T) that enjoys greater stability, fewer training-timi dependencies (such as relying on approximate likelihoods from an external language model during training). and more flexible decoding process than prior approaches (see Sections 4 and(6.))) Our method, ILQL, fine-tunes a transformer language model to predict the state-action Q function and the state value function V each token. During training we perform iterative policy improvement by fitting a value function to an upper-expectile of the Q function, enabling us to learn policies that leverage temporal compositionality, significantly outperforming the data, while avoiding the need to execute expensive training- -ime procedures, such as sampling counterfactual utterances from the language model (Verma et al. 2022) (see Sections 5 and (). Then at inference time we can simply steer a standard language model towards utility maximizing behavior, by perturbing the predicted likelihoods with our learned values functions (see Figure 3)
Our main contribution is twofold: (1) novel offline RL algorithm, ILQL, for language models, that employs a stable optimization process that can flexibly learn nigh-performing policies from sub-optimal data in arbitrary sequential decision making settings, thus meeting each of the conditions laid out above; and (2) a detailed empirical analysis, not only demonstrating ILQL's ability to more consistently and stably adapt to many different utility functions than prior approaches, but also ILQL's unique ability to optimize stochastic or subjective reward functions, and its ability to discover optimal behaviors in the face of sub-optimal generation setting of generating non-toxic text, we demonstrate that ILQL, trained on both toxic and non-toxic comments, learns to produce fewer toxic outputs than the more standard approach of performing standard supervised fine-tuning on only non-toxic comments.
unusual data distributions. In particular, in the controlled
RELATED WORK
number of prior works have explored combining online RL methods with language models for natural language tasks such as machine translation or summarization (Ranzato et al. 2015 Wu et al., 2016, Paulus et al.) 2017 Wu Hu. 2018). These works have demonstrated that RL. can be an effective tool for steering language models towards satisfying utility functions. However, when it comes to settings that require multiple steps of human interaction, e.g., dialogue, these methods can
quickly become impractical (Verma al. 2022 Ghasemipour al. 2020))
Offline RL addresses this shortcoming by removing all need for environment interaction or user
simulators, instead operating purely on static datasets of prior human interaction. Several prior
works have applied offline RL to NLP and more broadly sequence generation problems (Jaques
et al. 2020 Verma et al. 2022
Jaques et al. 2017 Snell et al. 2022 Janner et al. 2021 Chen
2

---

Published as a conference paper at ICLR 2023
al. 2021). The most closely related to work are those methods based approximate dynamic
programming (Jques et al. [020] 201 Verma et al. [002 Jang et al. (022). While all these works present promising offline RL methods for NLP tasks, none of them provide a method that achieves the simplicity, stability, and ease-of-use use aspect at the level of supervised learning. For example, Verma et al. and Jang al. (Verma et al.) 2022 Jang al. 2022) define their action space at the "per-utterance"
level /Verma et al. (022)) resulting in expensive decoding processes during training (Bender et al., 202T; and while Jaques et al. (Jaques al. 2020; 2017) remove this issue by defining actions the "per-token" level, the offline RL algorithm proposed requires querying likelihoods from a language model at RL training time, which adds an additional compounding source of approximation error and increases systems complexity at training time. Our proposed method instead operates both at the "per-token" level and trains a fully self-contained way, without the need to simulate generation at training time query likelihoods from a separate language model. This is achieved by combining an implicit dataset support constraint (Kostrikov ct al. 202I) with a novel policy extraction method that takes advantage of the discrete "per-token" action space. The result of these design choices is a simple, stable, and effective method that is easy for NLP practitioners to pick up and apply to a variety of language- based tasks. In Section [6.4 we demonstrate our method's effectiveness in meeting these criteria through a series of ablations and comparisons.
Much prior work on steering language models towards desired behavior has done so without an explicit utility function, instead focusing on curating finetuning datasets (Zhang et al. 20188] Zellers) al.
2019. Rajpurkar al., 2018). A more closcly related line of work uses classifiers to guide
LMs towards generating desired textual attributes (Yang & Klein, (2021; Ghazvininejad et al. 2017] 2018 Li et al. 2017). These methods are closely related to the prior work on offline RL. In RL parlance, such methods could be considered "policy extraction" methods with Monte Carlo value estimates. This can be interpreted as taking a single step of policy improvement which, though often effective (Brandfonbrener et al. (021), is known to be suboptimal as compared to full dynamic programming methods (i.e., full Q-learning or actor-critic) (Kostrikov et al.) (00T)) We will demonstrate empirically in Section that our offline RL method can lead to significant improvements in final performance as compared to such "single step" approaches, particularly when the training data is highly suboptimal for the desired task.
Holtzman et al.
3 PRELIMINARIES: LANGUAGE GENERATION AS A REINFORCEMENT LEARNING TASK
Token-level POMDP. In this work, we formalize language generation tasks as a partially observable Markov decision process (POMDP). We define the POMDP M at the token level with M (S, O,T, 2, Ho: R, 7). We define the agent's observation h E O as a history of tokens with h4 (to, 12, t3. ..tt-1::: ; the action space a t, E A is the set of possible next-tokens in vocabulary which includes the special end-of-turn token dend (see Figure 3}
Value-based offline RL. In offline RL, the goal is to learn the optimal policy that achieves highest discounted cumulative reward from a static dataset D that was produced by some potentially subopti- mal behavior policy na. In this work, build on the implicit Q-learning (IQL) algorithm (Kostrikov et al. 202T), which approximates the Bellman optimality equation constrained to in-dataset actions
Q' R(s, +7
max a' d'.tt. Ia (=')>0
Q* (s',aa).
Instead of directly implementing the support constraint, IQL approximates the maximization on the right-hand side of the constrained Bellman operator with expectile regression:
Ly (xb) = -E(..)) ~n [1 (Q a) - V=((s))
(1)
where Lz( = |7 1uu < 1) Increasing the hyperparameter т, more closely approximates the maximum. Then this approximation can be used to estimate TD-targets for the Q-networks:
(0) = E(s.,'') (Dlas)">_[[(A (R(s, + +V
- Qo(s, {])
(2)
IQL was designed for fully observable MDPs However, in Section 4.1 we discuss how we adapt this formulation to the POMDP setting described above using sequence models.
3

---

Published as a conference paper at ICLR 2023
Tinget VATM
(3 Vafkel;
Tranaformer Varue Function
Target Net
f. Traneformar
Trastorne Value Functies
HEE Part
9. Tranefumar
Deployment
Figure 3: A diagram of our Implicit Language Q Learning algorithm. Left: ILQL training involves three transformers, each which finetuned from standard pretrained model: (1) A Fe model, finetuned with standard supervised learning. (2) A value function model, with Q and V on two separate heads; the value functions are trained with Bellman backups using combination of conservatism and and an implicit dataset support constraint. (3) A target value network, which is a Polyak moving average of (2). Right: At inference time, we our learned value functions perturb the log probabilities of my towards utility maximizing behavior.
Supervised learning baselines. To align with NLP terminology, we refer to %BC, or supervised learning on curated or filtered data, as "filtered fine-tuning,, and we refer to BC, or finetuning on unfiltered data as "fine-tuning". See Appendix A.4 for filtering details.
4 IMPLICIT LANGUAGE Q-LEARNING
Our main technical contribution implicit language Q-learning (ILQL), an offline RL algorithm for NLP tasks. ILQL is specifically designed to enable simple and efficient training of language models with user-specified reward functions, with a workflow that is similar standard supervised learning.
ILQL builds on the IQL algorithm, extending it to the token-level POMDP that defines NLP tasks via the following modifications: (i) it integrates with sequence models to handle partially observable language generation tasks (Section[4.2 ; (ii) it utilizes a novel policy extraction method that directly perturbs the behavior policy n with our learned value functions, rather than training separate actor æ, significantly improving performance and stability on NLP tasks (Section 4.1) and (iii) it adds conservatism loss term (Kumar et al. 2020) to the Q-function, fixing a calibration issue in the policy extraction step. Figure 3 provides an overview of our method.
4.1 ADAPTING IMPLICIT Q-LEARNING LANGUAGE MODELS
Implicit value function learning. Like IQL, our method learns both a value function and a Q- function, which bootstrap off each other through Bellman backups with implicit maximization through an expectile loss. This recursive process of fitting Q and V corresponds to iterative policy improvement subject to implicit dataset support constraint, specified by the expectile used to fit V. Due to parameter sharing, we combine Eqn. [ and 2 into a single loss function:
(Q) = EEWD
(R(h, a.) + (ho4tz)) - Qo(h,, 0:)) 1=0
L5(0, - vo(h:)) -Vo(h))))))))h)))))
In contrast to IQL, we sample sequences of tokens instead of individual transitions to handle partial observability, such that for each time step, the values are predicted based on full history.
Policy extraction. IQL (Kostrikov al. 2021 uses AWR policy extraction (Peng al. 2019).
which distills the Q-function into a policy with (QQ-V) weighted log-likelihood loss. However, as we discuss in Section we found this somewhat unstable to train on language models, likely due to the high-variance gradients induced by the advantage weights. Fortunately, the value learning procedure in IQL is independent of policy extraction, so instead of attempting to train a model to represent the optimal policy, we use the learned Q and values to directly perturb samples from model finetuned via supervised learning to model Ty (see Figure 3). To this end, we compute modified likelihood for each token by adding its advantage Q(,, a) V(h) to its logits under the Ra model, with multiplier B. We can then renormalize these pseudo-logit and sample them, resulting in
4

---

Published as a conference paper at ICLR 2023
the implicit policy T(alh) mo(aal))e"(((hho)vvii = (x)() -V()))
This does not require training a separate actor, only a behavioral model TT. which can be trained with the standard and stable supervised finetuning objective.
However, naively performing policy extraction in this way can perform poorly due to weer-soothhdd probabilities in Th that may be nonzero for extremely unlikely tokens. In this case, samples from Ty may be out of distribution for Q and V, and might have erroneous values. To fix this calibration issue, we can either further decrease the probability of low probability actions in Ty by performing top-p filtering or tuning a temperature parameter, or we can explicitly push down OOD Q-values during training. We implement the latter by adding a small amount of NLL loss to the values, which corresponds to the additional loss terms introduced by CQL (Kumar et al.) 2020 with a uniform KI. regularizer. Since IIQ actions are discrete tokens, as opposed to the original CQL method (Kostrikov et al. (202T), which operates on continuous action spaces, this CQL. loss term is no more expensive than, and in fact equívalent to, standard cross-entropy loss at the token level. We find that both of these approaches often work in practice, but prefer the latter, finding that it requires less tuning for policy extraction at inference time, Our full loss function is therefore:
(al(sras)
Fq LQ.v (0) - aEE-D log EEvese (lo(ssaa))
In early experiments, we found that decoding using the CQL regularized value functions alone, without ng. required careful tuning of the CQL weight a. When the CQL regularized value function combined with Ty for policy extraction, mitigates this issue with hyper parameter sensitivity, and simply setting the CQL weight a to an arbitrary small value less than I typically works well.
4.2 ARCHITECTURES FOR IMPLICIT LANGUAGE Q-LEARNING
We use GPT-22 small as the base model for all transformers in our experiments. Our value function transformer has three MI.P heads: two independently initialized and trained Q heads and one V head. Each head has two layers, with hidden dimension twice that of the embedding dimension. Our target Q value is parameterized as the minimum prediction of both Polyak averaged target Q heads: Q min Q1, Q2) (Fujimoto et al. 2018). As in standard language modeling, the transformer's causal masking enables us to perform Bellman updates over entire sequences parallel.
5
PROOF OF CONCEPT: MULTI-STEP OFFLINE RL ON WORDLE
The lack of configurable task settings and reliable evaluations has arguably slowed down progress in applying sophisticated RL algorithms to language and dialogue tasks (Tiang al. |021 Deriu et al. 2021 Curry et al.] 2017). To address this, we present the Wordle game (I (Lokshtanov & "Subercaseaux (022)) as an easy-to-use but challenging benchmark task to test the capabilities of
offline RL algorithms. In this section, we use this task to construct situations where we would expect offline RL to lead to significant improvement over simpler methods based on supervised learning or single-step improvement (e.g., single-step RL-style filtered supervised learning methods).
Multi-step RL: a motivating example, General value- RL methods based on solving the Bellman equation described in Section 3 can be viewed as iteratively improving the policy: each update sets the current value Qhhy, ax) to be the reward plus the maximum possible next time step value according to the current value function. This is in contrast to "single-step" update methods, which do not recursively update the value function, instead only learning to estimate the value of the dataset and then greedily selecting the maximal-value action during inference. Classic examples of such methods use Monte Carlo regression or single-step RL (Sutton et al. 1998) to train the value function, and then greedily choose actions at test-time, though a number of different methods of this sort have been proposed for guided language generation in the literature (Yang & Klein T Ghazvininejad et al. 2017 Holtzman et al. 2018 Li et al. (017). Such methods have been referred to in the NLP literature as "reward models" (Young et al., 2017 Gu et al. 2016 [Su et al. (0166) and in the offline RL literature "one step RL" or SARSA (Brandfonbrener et al. (021). Some works also proposed behavioral cloning methods that filter the training data or use conditioning to clone high-reward trajectories (Chen et al. 2021) - though these methods are based on different principles, they also employ Monte Carlo estimates of the cumulative reward in place of value functions learned with dynamic programming. While in principle such methods should not lead to optimal policies, in practice they often constitute an appealing approximation due to their ease of use.
5

---

Published as a conference paper at ICLR 2023
Abstract MOP Diagram
Datasel Composition
Goal: produce culputs that maximally activats a sentiment classifler
sourt man Dlouing creal mapiing
R
10% of coe
s-godd Srme.
reweed: +0.5
masdersiio, onch Bilrrd
The mon wouth good Fisirad
movie
SARSA
45% of EXT
cedilee eatry minite
reward: -1.0
R=.1
R=1
Lava
40% 1.0
The good
regerteno!
reward:
+1.0
The mosil NO Erets sahrr
Figure 4: Left: abstract depiction of MDP where single-step RL fails to discover the optimal policy. Right: notional illustrative example where we might expect full "multi-step" RL methods (such as IIQQI) to perform significantly better than "single-step" methods. In this example, good utterances lend to start with "The movie was...", while bad utterances start with "The movie wasn't... However, the very best examples also start with "The movie wasn't ma.."t...", requiring multi-step planning or multiple steps of policy improvement to derive effective strategies. Methods that implement just single step of policy improvement will fail produce maximally positive sentiment outputs. While this example may appear somewhat contrived, we see in our experiments that multi-stey step RL methods do lead to improvements in a number of more real settings.
Since ILQL performs multiple steps of policy improvement, it can significantly improve over Monte Carlo estimators or single-step RL when the underlying data is sub-optimal. One example corresponds to the notional task in Figure 4 in which the optimal sequence of actions requires traversing a state that's also frequented by sub-optimal examples. In this case, single-step RL will learn to take actions that appear safer according to the dataset- such as the transition "The movie" + "was" in Figure 4 -- whereas full ("multi-step") RI. methods would recover the optimal policy. We demonstrate this empirically the Wordle game below.
Wordle dataset. Our Wordle task is designed allow us use data from real humans in sequential decision-makinn setting while still enabling objective simulated evaluation and the flexibility to compose datasets with different properties, thus providing an effective benchmark for validating a variety of approaches. Wordle is a word guessing game in which the agent gets 6 attempts to guess a certain word, and at each turn receives feedback from the environment about which letters from the guessed word are and are not in the true word (see Appendix A.. for more details). While Wordle may appear distinct from many natural language tasks, it shares a number of high-level properties, such as non-eeterministic dynamics and sequential turn-based structure, with more complex language domains like dialogue, making it well suited for a first evaluation of NLP-focused RL methods.
Synthetic Wordle task. We constructed a synthetic Wordle dataset to serve as a benchmark that is specifically intended to evaluate how well a particular method can perform multiple steps of policy improvement. This task intended to specifically bring out the failure mode of single step methods in a setting suitable for testing offline RL methods with sequence models. The dataset consists of data sampled from three behavior policies, each corresponding to one of the branches in Figure 4 (1) Tupper bound, a high- perh-rerfor policy, corresponding to the path So - Goal. (2) Tadvensarill, which behaves the same as Tupper bound for the first two actions (So- + S1) and then behaves suboptimally (S. - Lava). (3) Tsshoptimal, a policy of moderate performance, corresponding to So Sy. Measuring the predicted Q values from our models trained on this distribution, in Figure 6 (right) we observe that IIQQI assigns higher values to actions corresponding to the paths towards "misleading states" (i.e. S2) than those to the "goal states" (i.c. S1), whereas single-step RL shows the exact opposite preference, confirming both our hypothesis that this type of MDP would be amenable to multiple steps of policy improvement, and that ILQL as an algorithm is able perform such policy improvement. Of course, the structure of real-world NLP tasks might not necessarily reflect this setting as we show in the next section, ILQL still often attains improvement over one-step and BC-based methods, though it is more difficult to discern the particular structure that makes this possible in more realistic tasks.
Validating on natural Wordle data. While the synthetic setting explored above was specifically designed to demonstrate a dramatic difference between ILQL and single-step RL, the findings still transfer to more realistic settings. In Table 6 we demonstrate ILQL outperforming single-step RL on a natural dataset of Wordle games scraped from Twitter (see Appendix A.5 for details).
NATURAL LANGUAGE EXPERIMENTS
Next, we evaluate ILQL on two realistic language tasks. We first identify scenarios in which one might expect offline RL to be particularly beneficial: (1) tasks that demand repeated interactions,

---

Published as a conference paper at ICLR 2023
method
ILQL
-3.21
standard
-5.57
Wn
-6.57 0.118
cons. y/n
Inuinleval standard y/n
standard
y/a
cons. y/a
1-step
0.13
-5.91 + 0.14
-7.63 0.20
-5.21
0013
-11112
-14.97 = 0.36
Filtered FT
.5.07 1: 0.13
-7.48 1: 0.21
-9.13 1: 0.22
-3.41
0.12
-5.57 4 0.13
-8.24 0.22
-5.25 2254:0.111
-0085
0.27
.15.16 0.35
y/n
5.29 1 0.13
-5.42 0.13
-6.57 0.18
Table 1: Left: comparing ILQL to baselines on our Visual Dialogue rewards. "Cons. yn" refers to the "Conservative yl" reward. ILQL successfully optimizes for many different rewards, even those for which the data is sub-optimal (e.g. BC performance). Right: Evaluating each II.QI. agent on other rewards, Agents generally perform worse on rewards for which they were not trained.
like dialogue; (2) data that is highly sub-optimal under its utility function; (3) settings with highly stochastic rewards based on subjective human judgement (e.g., avoiding toxic language). Taking into account these three scenarios, we evaluate ILQL on (1) a goal-directed question asking task based on Visual Dialogue (Das al. 2016), where achieving high rewards on a diverse set of metries during repeated interactions is desirable, and (2) Reddit comments generation task with highly subjective and noisy reward functions (toxicity ratings or upvotes). For general experiment details see Appendix A.4
6.1 EVALUATING DIVERSE REWARDS ON VISUAL DIALOGUE
Visual Dialogue dataset. We use the Visual Dialogue dataset (Das et al. 2016) to evaluate our algorithm's ss ability to optimize many different reward functions in complex dialogue settings. The task involves both a question asking and question answering agent, the latter of which is presented with an image and tasked with answering the former's questions about the image. In- stead of using this task as a question answering task, we follow Das al. (Das et al. 2017) and train our agents to ask questions, with rewards based on how well the ground-truth image can be predicted from the resulting dialogue. For evaluation, we use the model from Das et al. (Das et al., 2017) as our environment simulator. agents to operate entirely in the space
pocts 56% win su phore for whili udinn Net ant Zi sut dessed on place
Quantiennn
Queelueen Te plossee?
Krewaner 1X
Krzwsie 196
YauMo Paraly:
Image Caption: skucc Kyit cand
Qusstionnr wittl calr didae?
Arswener
Quentoone what persor Weawe??
carlt
To allow of natural language, we treat the image embedding as part of the reward function, using the supervised model proposed by Das et al. (Das et al.) 2017) to predict the image embedding from the dialogue. See Figure 5 for example dialogues in this domain. We chose this envi- ronment specifically because (1) has been previously studied in the context of RL (Das et al. 2017); (2) as a dialogue game, automated evaluation more reliable than other tasks; and (3) the Q&A structure enables some temporal compositionality (i.e., the answer to one question may prompt new more specific questions).
Quanttoner Is persoe Fvingal bynng in
Ancwener dank
Srewster
Paraity:
Caption: Quentorre for saeil Arswere::2
wallin
Questioen whit and bose?
Quentoren for Arswere:2
Questinnn gente persler
Queetinen what on wearis?
Arswere: warri ng shonh ond other workng shork
Figure 5: Dialogues from different agents on the Visual Dialogue task. We observe qualita- tive differences depending on the agent's 's reward. The "standard" agent asks many yes/no question, whereas adding a string-match penalty for yes/no questions prevents many such questions from being asked, and adding a more conservative yes/no penalty prevents all of such questions.
Visual Dialogue task. The agent receives reward of -1 for each turn in which the ground truth image can't tt be predicted accurately from the dialogue, otherwise the agent receives a reward of O and the environment ends interaction. For details on the task setup, see Appendix A. Since the Visual-Dialogue dataset was largely designed for supervised learning agents, the data is already near optimal for the original task. However, if we shift the reward function such that the data is no longer optimal, we can observe large gains from offline RL. We therefore use this domain to demonstrate offline RL's flexibility to adapt to different reward functions. We consider three rewards: "standard", "y/n", and "conservative y/n". "Standard" is simply the reward described above and detailed in Appendix A.6 "y/n" adds to the "standard" reward penalty for asking questions that produce yes or answers, by assigning reward of 22 each time the other speaker says "yes" or "no". This is challenging, because while the data contains many yes/no questions, the goal is not for the agent itself to avoid those words, but rather
7

---

Published as a conference paper at ICLR 2023
method
ILQL
0.0=0.00
loacitty
upvotes real
9,33+0.04
upvotes model
10.0==0.0
Iraineeval
locicity
upvokes model
single-stop RL
Fillered FT
0..74:1.007
0.0=0.00
7.0610.14
6.23+0.15
7.86:10.133
10.0==0.0
upvotes gold
toxicity
-5.00::00.00
0.0:00.0
9.07=0.09
9.83-+0004
upvotes model
-5.00=000000
10.0=:0..0
FT
-3.51+00.13
4.87
4.87:00116
Table 2: Left: A comparison of ILQ against baselines on our Reddit comment rewards. ILQ never generales undesirable comments on out of 3 rewards, whereas fine-tuning on filtered data occasionally does. Right: Evaluating each Reddit ILQL agent on other rewards. Agents trained on one reward are less optimal on others. avoid utterances that cause the other speaker to use them. Not all simple questions produce literal "yes/no" answers, so our third reward further penalizes brief responses, such as can't tell", "no isn't", "yes it is", "I don't it know". This reward function assigns a -2 reward to a set of low-information responses using a handful of conservative string matching heuristics, detailed in Appendix [A.6]
Results on optimizing diverse rewards We demonstrate that ILQL learns policy distinct from the dataset behavior policy and can optimize many different rewards in [ where we can see that ILQL is able outperform baselines on most of our Visual Dialogue reward functions. Agents optimized for each reward learn different behaviors as well: in Table right, we see that offiine RL demonstrates this qualitatively: without the "yes/no" penalty, our policies ask many "yes/no" questions, but under its presence policies tend to ask other questions instead. Even when the underlying data is highly sub-optimal for given reward function, ILQ is able to determine the desired behavior. In addition to our reward-based evaluations, we also provide language quality evaluations for all baselines on each reward function in Section AJ]I
agents trained on one reward function are generally suboptimal on others. Figure 5
6.2 SUBJECTIVE REWARDS ON REDDIT COMMENTS
Reddit comments dataset. To evaluate our agents on minimally curated and maximally diverse open-domain text with highly stochastic reward functions based on subjective human judgement, we train ILQ on large dataset of 4 million Reddit comments from P our agents are given parent comment or post as context and then trained to produce replies that satisfy one of two rewards: "toxicity" and "upvotes real". Given that this internet text, the data contains toxic language. so for our "toxicity" reward, we train our agents to satisfy toxicity filter, which gives rewards -10, -5, and 0 for toxic comments, moderately toxic, and non-toxic comments respectively. Our second reward function incentivizes comments that would receive positive number of upvotes, rewarding +10 for positive upvotes and negative. We automatically evaluate our upvote agents with a finetuned RoBERTa- base model, that predicts whether comment will receive positive upvotes. We train this model on a held-out split of the data (see Appendix (A.7 for more details). We train agents on both the ground truth upvotes (denoted "upvotes real") and on this upvote model's predicted reward (denoted "upvotes model").
Results on optimizing noisy rewards. In natural language tasks, we may need to optimize stochas- tic, high- variance reward functions based subjective judgement, such whether a Reddit comment should be flagged as toxic. Such stochastic settings should be expected when multiple users with differing opinions provide reward labels. Offline RI., by design, is robust to environment stochasticity, and therefore should be able to optimize such noisy environments. We use the Reddit toxicity and upvote tasks study how well ILQL can handle such settings. As we can see in Table top. ILQ is surprisingly able to get a perfect or near-perfect score on these more subjective settings, whereas more standard approaches, such as filtered finetuning on only non-toxic or only positive upvote comments, perform significantly worse (i.e. generates more comments flagged as toxic or predicted to have negative upvotes). Additionally, we see in T bottom that agents trained on one reward function are generally less optimal for others, confirming that IIQI is effectively specializing its behavior for each utility function. We have additional complementary experiments studying this effect in Appendix (A.8] In addition to our reward-basse evaluations, we also provide language quality evaluations of our agents in Section A. and a preliminary user study in Appendix A.12
6.3 CHOICE OF OFFLINE RL ALGORITIMM
We compare IIQ to four other RL methods: a per-token version of CQL, an adaptation of the
vi-learning as proposed by Jaques et al. (Jaques et al. 2020 (017)) decision transformer (DT) (Chen) ct al. 2021), ), and single-step RL (Yang & Klein 202 Ghazvininejad et al., 2017 Holtzman et al. 2018, Li et al. 2017). In Table we see that ILQL significantly outperforms baselines, and also has the second lowest hyper-paranee parameter variance, just behind single-step RL, confirming our hypothesis
that IL ILQL can provide both high relative performance and training stability.
8

---

Published as a conference paper at ICLR 2023
6.4 ILQL ABLATIONS
To understand which components of ILQL enable both good results and greater easc-of-use than prior offline RL approaches for language tasks, we abalate ILQL's main design decisions: the choice of a per-token action space, our value learning method, and our policy extraction strategy.
We evaluate these comparisons on the Visual Dialogue "yes/no"" reward because (1) it is im- portant to compare offline RL methods on a chal- lenging and realistic sequential decision prob- lem like dialogue, and (2) since the Visual Dia- logue data is already "near-expert" for the "stan- dard" reward, the "yes/no" reward is better able to differentiate between methods.
method IQQ
max score
w.l hparams
5.57.20.33
0.46
CQL *
.7.32.10.17
-10.05-:0.18
0.60
single-step RL
-5.91=0.144
0.35
DT
-6,70+-0.17
1.15
Filtered Fine -uning
.7.48.10.21
0.72
Fine-tuning
-10.85.10.27
Table 3: Comparison of ILQL and other offline RL
methods on the visual dialogue "y'n" task. ILQL per- forms better with lower hyperparameter sensitivity. Re- sults show hest hyperparameter settings for each method.
vs. per-utterance ac- tions. We hypothesize that, since a per-token Q function enables an efficient search through the utterance action space, performing offline RL the token level, rather than the utterance level, can yield cheaper inference and better performance.
Ablations on per-token
We compare ILQL to: (1) ILQL (utterance): per-utterance adaptation of ILQL that removes the conservatism loss term and performs Bellman backups at the utterance level instead of the token level: (2) single-step RI (utterance): per-utterance version of single-step RI.; and (3) CHAI (Verma et al. 2022): an adaptation of CQL for use on language models at the per-utterance action level. For policy extraction, each baseline uses EMAQ (Ghasemipour al. 2020). where we sample N utterances from a learned behavior policy and then re-rank with the Q function.
We see in Table that ILQL outperforms per-utterance ILQL and single-step RL, while also running inference ~4x faster on single T4 GPU. When tuned well, we see that CHAI perform similarly to ILQL. However, CHAI is less stable with respect to hyperparameters and is >2x slower inference time.
Ablations on policy extraction strategies. In adapt- ing IQL (Kostrikov et al. (202]) to sequence models, we design a novel policy extraction strategy, as described in
method
max score
w.r.l Epurams
ILQ
-5.57=0.113
0.46
ILOL (AWR)
-5.96:20.13
2,82
(GOLD)
-7.58.10.21
0.61
Section 4 We compare our novel extraction procedure
Table 4: Policy extraction techniques for ILQL.
to two baselines: (1) the standard AWR-based (Peng
et al. 2019) policy extraction method used in IQL (Kostrikov et al. 2021) and number of other
offline RL algorithms (Nair et al. 2020 Wang et al. (2200), and (2) GOLD (Pang & He 2020) an off off-policy gradient based method for natural language generation. We expect that our approach should generally yield better performance, while also being easier to tune than these baselines. We apply AWR and GOLD extraction to our best performing ILQL value function, denoting this as "ILQQ.. (AWR)" and "IIQQ. (GOLD)" Table [ demonstratss that IIQQI is both more stable and better at extracting good performance from given value function than the well established AWR-style extraction and GOLD. Additionally, the AWR and GOLD extraction methods require tuning addi- tional parametrrssssrrs at training time rather than just at inference time, decreasing flexibility and increasing the time and effort spent tuning parameters and re-training
CONCLUSION
We proposed ILQL, an offline RL method for steering language generation to fulfill a variety of desirable conversational behaviors. Through experiments ranging from word games and goal-directed question asking to optimizing upvotes and minimizing toxic language, ILQL shows that offline RL can serve as a strong alternative to the method landscape of language generation dominated by language model finetuning on manually filtered datasets and classifier guidance. We hope the positive results from ILQL will inspire more work on offline RL for dialogue, and lead to more controllable language models that directly optimize user-specified utility functions for a wide range of tasks in text generation. Lastly, we acknowledge that our method is generally more computationally expensive than more standard supervised learning approaches to language generation 2xx more training time in our experiments), since it requires 3 separate transformer networks during training and 2 during inference. Future work should study alleviating this cost.

---

Published as a conference paper at ICLR 2023
8 ETHICS STATEMENT
We acknowledge that any utility optimization method be used to aid harm, hope these future works consider ethical uses of offline RL. Additionally, our method also has its limitations: for example, ILQL may not be effective when datasets are highly suboptimal. Offline RL would also not be ideal in settings which require distributional constraints, such as fairness. We hope that practitioners will take these limitations into account when applying our method.
REPRODUCIBILITY STATEMENT
To promote reproducibility, we present extensive results of all hyper-parameter settings we tried for all baselines in the appendix. We also describe all experimental details. Lastly, have also attached in the supplemental source code for reproducing all the experiments in this submission.
REFERENCES
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell, On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference Fairness, Accountability, and Transparency, FAccT 21, pp. 610-623, New York, NY, USA, 2021, Association for Computing Machinery. ISBN 978 9781450383097, doi: (11453344188.3445529 /Rt//ttps////o..o///110 11111111111450 1141
444484 S1 3445922
David Brandfonbrennr, William F. Whimey, Rajesh Ranganath, and Joan Bruna. Offline without pollcy evaluation, 2021. URL
08909
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https:
7/arxiv. / abs 100000111166
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, 2021. URL htctps://axivv.ort wwtprrg/aab/////aa 01345
Amanda Cercas Curry, Helen Hastie, and Verena Rieser. A review evaluation techniques for social dialogue systems. In Proceedings of the 1st ACM SIGCHI International Workshop Investigating Social Interactions with Artificial Agents, pp. 25-26, 2017.
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M. F. Moura, Devi Parikh, and Dhruv Batra. Visual dialog, 2016. URL httpt:://////...t. https::/arxiv..commmm http//aivvvc.//// org. abs hatgg//bs/1611 08669
Abhishek Das, Satwik Kottur, José M. F. Moura, Stefan Lee, and Dhruv Batra. Learning cooperative visual dialog agents with deep reinforcement learning, 2017. URL /tttps://arxllv.ogg/
abs/1703.C 06585
Jan Deriu, Alvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, and Mark Cieliebak. Survey on evaluation methods for dialogue systems. Artificial Intelligence Review, 54(1):755-810, 2021.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2018. URL httpps://arx.v..rg//
abs abs/1810. 04805
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What essential for offline rl via supervised learning?, 2021. URL
10751
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020. /tl.httpp:://rrvv..orr////
10

---

Published as a conference paper at ICLR 2023
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods, 2018. URL /ttp:://axvvvvv org/ http://bs//882// 09477
Felix A. Gers, Jürgen Schmidhuber, and Fred Cummins. Learning to Forget: Continual Prediction with LSTM. Neural Computation, 12(10)2451--477) 10 2000. ISSN 0899. doi: 10.1162/
089976600300050115
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected- max q-learning operator for simple yet effective offline and online rl, 2020. URL https
Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight. Hafez: an interactive po- etry generation system. In Proceedings of ACL 2017, System Demonstrations, pp. 43 48. Vancouver, Canada, July 2017. Association for Computational Linguistics. URL https
//aclanthologe wog..oog..........
Jiatao Gu, Graham Neubig, Kyunghyun Cho, and Victor O. K. Li. Learning to translate in real-time wwtp:///ggg...ooo///////////aeea
with neural machine translation, 2016. URL hRtLhtts:://// http:////aaa.a..oo///////.......
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 2018. URL https
//arxtv.org/aas/180..
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. Learning to write with cooperative discriminators, 2018. URL
06087
Youngsoo Jang, Jongmin Lee, and Kee-Eung Kim. GPT-critie: Offline reinforcement learning for end- to-end task-oriented dialogue systems. In International Conference Learning Representations, 2022. URL
n wttp://wwwwe/foouutmteeee forummiicciooooo (ore:e/ qaxhBGlUUaS
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems, 2021.
N. Jaques, S. Gu, D. Bahdanau, J. Hernandez-Lobato, E. Turner, and Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. International Conference on Machine Learning (ICMI.), 2017.
Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Shane Gu, and Rosalind Picard. Human-centric dialog training via offline reinforcement learning, 2020. /ttpp:::/ap.....///////////o////
05848
Haoming Jiang, Bo Dai, Mengjiao Yang. Tuo Zhao, and Wei Wei. Towards automatic evaluation of dialog systems: A model-free off-policy evaluation approach. arXiv preprint arXiv:2102.10022. 2021.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims, Morel: Model- based offline reinforcement learning, 2020. URL
2005 05951
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning, 2021. URL
06169
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. GeDi: Generative discriminator guided sequence generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 4929-4952, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10, 1/2021 aoddings-emnlp.22.. URL https: http::////.caaatoooooo.o.o////// Findings-emmipp 424
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Proceedings the 34th International Conference Neural Information Processing Systems, NIPS 20, Red Hook, NY, USA, 2020. Curran Associates Inc
ISBN 97817138295466
11

---

Published as a conference paper at ICLR 2023
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement leaming Tutorial, review, and perspectives on open problems, 2020. URL https://axxiv..rppt abs 2005 01643
Jiwei Li, Will Monroe, and Dan Jurafsky. Learning decode for future success, 2017. URL httpa/awwwww....///////////////.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematie survey of prompting methods in natural language processing, 2021. URL
wttp:/iww..rrg/b///.1.000/ 13586
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zetrlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019. URL pnttts://arriis wwtpaaawwwwvoora//////aa/
11 692
Daniel Lokshtanov and Bernardo Subercaseaux. Wordle is np-hard, 2022. URL https:://aaas org/aba/ 2201bb/113311113
Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Noisy channel language model prompting for few-shot text classification, 2021. URL httpp://arxix. org/ abs/
2108. 04106
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets, 2020. URL htlthtsss//axiiii www.:.rgaaaaa....
09359
Richard Yuanzhe Pang and He He. Text generation by learning from demonstrations, 2020. URL
Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization, 2017.
https///axxvvworgaaaaa. /100bs//10.... 04304
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. wevintage-weighttd regression: Simple and scalable off-policy reinforcement learning, 2019. URL /tR/atttp/////x/..org////
01001990000077
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenA/ blog, 1(8):9, 2019.
Eddie Rafols, Anna Koop, and Richard S Sutton. Temporal abstraction in temporal-differee networks. In Y. Weiss, B. Schölkopf, and J. Platt (eds.), Advaaces in Neural Information Process- ing Systems, volume 18. MIT Press, 2005. URL URL/httpss:/pprrceeeee nggs neur Seuuripe pss cc/
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad, 2018. URL 20tp://0001111233110002227733333
Zaremba. Sequence level training Mare' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Zarebch with recurrent neural networks, 2015.
06732
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL httt://e/aartt....r///////....00
Charlie Snell, Sherry Yang, Justin Fu, Yi Su, and Sergey Levine. Context-aware language modeling for goal-oriented dialogue systems, 2022. URL http::://aaxvv..g////////// 220p. 101 98
Pei-Hao Su, Milica Gasic, Nikola Mrksic, Lina Rojas- BarahoBa Stefan Uless David Vandyke, Tsung-Hsien Wen, and Steve Young. On-line active reward learning for policy optimisation in spoken dialogue systems, 2016. URL httpt::://// htttps://arxiv.cot
abs/ 07669
Richard S Sutton, Andrew G Barto, et al. Introduction reinforcement learning. 1998.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017. URL https: :ttpppps://riiii/// abs/ 1706. 03762
12

---

Published as a conference paper at ICLR 2023
Siddharth Verma, Justin Fu, Mengjiao Yang, and Sergey Levine. Chai: A chatbot ai for oriented dialogue with offline reinforcement learning, 2022. URL http::t///s.....ortttt /ta//////////aaaaaa/////////aaa/ 2204 08426
Ziyu Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas. Critic regularized regression, 2020. URL
15134
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc Le, Mohammad Norouzi. Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's 's neural machine translation system: Bridging the gap between human and machine translation, 2016. URL ht https: org/ abs//ab///// 11t./1609.00000
Yuxiang Wu and Baotian Hu. Learning to extract coherent summary via deep reinforcement learning, 2018. URL httppp://aarxv...crr/a/////11..0
Kevin Yang and Dan Klein. FUDGE: Controlled text generation with future discriminators. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Compu- tational Linguisties: Human Language Technologies. Association for Computational Linguisties. 2021. doi: 10. 10.1833/1112021 maaecl-mai.276. 2F2021 nwa.laaaaalaaaaaaaa www.aaaaaaclaamammmmmmm
/tt:::/dooiioo URL /
Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria. Recent trends in deep learning based natural language processing, 2017. URL https: http::://arxxv...o. abs /1708 02709
Tianhe Yu. Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization, 2020. URL https://arxiv org /tg//bs/// 2005. 13239
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?, 2019. URL http:::://ax....orrg////////////
07330
Saizheng Zhang, Emily Dinan, Jack Urbanck, Arthur Szlam, Douwe Kiela, and Jason Weston. Personalizing dialogue agents: have a dog, do you have pets too?, 2018. URL https://
07243
A APPENDIX
A.I JUSTIFICATIONS FOR OFFLINE RL DIALOGUE
Dialogue tasks of the most rich and interactive settings in NLP, and will argue, these properties make it an ideal target for applying offline RL. RL in general presents an elegant and highly desirable utility optimization framework for sequential decision making settings, such as dialogue. However, solving realistic interactive tasks with online RL requires either repeated real- world interaction or building realistic simulator of the environment. In the case of dialogue, such online interaction means communicating with real humans, which may be impractically expensive and time-consuming with contemporary sample-inefficinn 2017, Haarnoja et al. 2018), and building realistic simulator of human responses may be largely intractable in sufficiently rich or complex dialogue settings. Offline RL, on the other hand, avoids both of these heavy requirements, by. just as many recent breakthroughs in the field of NLP (Brown) et al. 2020 Radford et al. 2019 Devlin et al. 2018). operating purely on previously collected data, which is wildly available on the internet in general. Offfine-RI RL therefore presents ideal approach for flexibly steering language models towards the successful completion of dialogue tasks in a way that effectively leverages existing data, just as supervised learning does.
online RI. methods (Schulman al.)
13

---

Published as a conference paper at ICLR 2023
A.2 TEMPORAL COMPOSITIONALITY
has been well studied in the RL literature that fellee-funtion based RL methods are capabale of form of temporal compositionality (Emmons al., 2021; Rafols et al., 2005) or "stitching" in their utility optimization. Specifically this refers to an RL algorithm' 's ability to stitch together the locally optimal parts of suboptimal trajectories to find globally optimal behavior. The result of this stiching is a learning algorithm which can distill the optimal behavior out of a dataset that contains only suboptimal demonstrations.
A.3 FULL TOKEN-LEVEL POMDP FORMULATION
We expand on the POMDP definition presented in Section In order to apply RL to interactive language settings, we need to formalize dialogue generation and other NLP tasks partially observable Markov decision processes (POMDP). We define the POMDP A at the token level with . (S, A. O, T, Z, po, 7) We define the agent's 's observation h; E O to be history of tokens with ft = (oo f1, t2. tg. 11.44-1 the action space a = t E A is defined to be the set of possible next-tokens in our vocabulary, which includes the special end-of-turn token Gend- The agent's s policy then corresponds to a mapping и O - P(A). Many tasks such as dialogue have an underlying state 8, that goes beyond just the sequence history, which can encompass things like the speaker's mental state. The environment transitions T-|ss,, at) are defined as a function of this 8t. In particular, in domains, such as dialogue, the dynamics are trivial within the agent's 's utterance (the selected token is deterministically appended to the history), but when the policy outputs a special "end of turn" token, the other speaker gets turn, which is subsequently appended the history. In other tasks, where the goal is to generate a single utterance, such as generating summary or a single Reddit comment, the episode ends when the policy produces the end token. The agent receives a reward, defined by R(s,, ax) + R. after each action taken. However, in all the settings consider, the agent receives non-zero reward 7r only after producing an "end of turn" token, rather than densely at every token in the agent's utterances.
While some prior works (Verma et al. 2022 Jang et al. 2022) have considered actions at the utterance level, defining decision processes at the token level can yield a more effective search over the exponentially large utterance action space, simply by selecting tokens with high estimated values. Typically, searching over a per-utterance action space requires Monte Carlo process of sampling multiple full utterances and then re-ranking with estimated values, which can generally bring additional computational complexity both training and inference time. In Section [cc4i we demonstrated the effectiveness of learning at the token level through an ablation study that compares with learning at the utterance level.
A.4 GENERAL EXPERIMENT DETAILS
Here we outline architecture and hyper-parameter details of all our models and baselines.
ILQL experiment details. We run all of our experiments on GPT-2 small transformer architectures, with the supervised learning policy on one transformer and Q and V heads separate one. The target Q network is also on a separate transformer. In all our experiments we initialize with GPT-2 pre-trained weights, except in the case of Wordle, where we initialize randomly. Additionally, Wordle uses a different token set: the set of 26 characters, plus an additional token for each "color". We train all RL baselines with double-Q learning, using two separate heads on the same transformer model as the two Q- O-functions. Our target Q networks are the transformer and the Q function head. We use y 0.99 for all offline-RL RL experiments. All value function heads are two layer MLPs with hidden dimension twice that of the transformer's s embedding dimension. Our MLPs used ReLU non-linearities and no dropout. We used the Adam W optimizer for all experiments, with learning rate of 1c-4 on the Reddit and Visual Dialogue tasks and le-5 on the Wordle task. We used no weight decay in the training any of our models, and we used a dropout rate of 0.1 inside the transformer. We trained all Wordle models with batch size of 1024, all Visual Dialogue models with a batch size of 64, and all Reddit models with a batch size of 32. We always truncate token sequences length 1024, except on Reddit tasks, in which truncate length 512.
with decay factor 0.005 for both
Except on the Reddit comment task, we train ILQL on each of T = {0.7, 0.8, 0.9). and we also evaluate each on 3 (4,8, 8, 16, o.). Where = oo refers to acting greedily according only
14

---

Published as a conference paper at ICLR 2023
the Q function. On the Reddit comment tasks, only train with 0.6 and evaluate on B {1,2, 4, 8, 16, 32}. On all tasks, we report the setting with the greatest performance.
For the NLL (CQL) loss term applied to ILQL, we used weight of 1.0 on all VisualDialogue experiments, 0.25 on all Reddit Comment experiments, and 0.0001 on Wordle. These values were tuned by hand. Generally, we find this loss parameter to not be too critical to performance; we tune it a little at first for each task and then don't Ion't worry about it.
All ILQL models and all baselines were trained single GPU until convergence. Training never exceeded three days (or 72 V100 hours).
Evaluation details. During evaluation, we use greedy decoding to generate utterances on all tasks and baselines, except the Reddit Comments tasks, where we sample instead. All experiments are evaluated on 1024 fask-instances from an unseen evaluation set. We use our BC baseline model as Tyy for guiding ILQL's perturbation-based policy extraction.
Fine-tuning (BC) baselines. We train our fine-tuning baselines with the same optimization param- eters (i.e., weight decay, dropout, learning rate, batch size) and initialization ILQLL. We use early stopping: when the validation loss exceeds the training loss, we stop training. Unlike our ILQL value function models, we use linear head on top of the transformer to parameterize our BC policy, as is standard for language model finetuning. The only difference between our fine-tuning and standard language model training is that instead of finetuning the model to predict the whole sequence of states and actions, we only finetune the model to predict the agent's ' own actions or utterances.
Filtered Fine-tuning BC) baselines. For our filtered fine-tuning baselines, we train models on datasets filtered for the top reward trajectories. Specifically filter for the top {10%, 30%, 50%) for Wordle, { {10% 20%, 30%} for Visual Dialogue, and for Reddit, since our rewards are discrete, we define filtered fine-tuning to mean just training on the data-points with the maximum reward label. For each task, we train models on each of these percentages and report the performance of the best setting found. We use the same papperpprramttrss.e our Fine-tuning (BC) baselines for training these models.
Decision transformer baseline. Our decision transformer baseline follows from Chen et al. (Chen et al., 2021), except we initialize with pretrained GPT2 weights. All hyperparameters are iden- tical to those used to train our BC baselines. To evaluate decision transformer on our Vi- sual Dialogue "y/n" reward, we swept over broad range of conditional reward-to-go values: 1-11-10,, -9, -, -7, 6, -5, -4, -3, 22 11 We report the setting with the best perfor- mance.
single-step RL baselines. single-step RL baselines are implemented as ILQL with 7 = 0.5 and all other typer-parametery are identical to those used with ILQL as described above. Except on the Reddit comment tasks, we evaluate all single-step RL models on 3 = (4, 8, 16). and report the setting with the greatest performance. On the Reddit comment tasks, we show the best performance from 3 = 21, 4, 8, 16,32)
Per-utterance ILQL. For "ILQL (utterance)", we train models with r = 0.7,0.8,,..9 We also evaluate each model with number of EMAQ-style (Ghasemipour et al. 2020) samples chosen from N=(4,8, 16). We report the setting with the best task performance. "single-step RL (utterance)" is special case of "ILQL (utterance)" with r 0.5, which we evaluate on each of N=(4, 8, 16) and report the setting with the best performance. The architecture for "ILQL (utterance)" and "single-step RL (utterance))" is largely identical that of per-token ILQL, with the main difference being that Bellman backups are performed at the utterance level instead of the token-level. As result of this difference, Q-heads map to a scalar at the end of an utterance instead of vector at every token with length equal to the size of the vocabulary. We include comparisons to Per-utterance II II.QI. in Table 5
2022) The tasks we CHAI baseline. Our CHAI baseline is adopted from Verma et al. (Verma et al.
consider only require utterance actions; axuiliary actions. like the price proposal action required by that of Verma et al. (Verma et al. 2022)'s 's bargaining task. We therefore only adopt the components from CHAI relevant to utterance level actions. In order to compute CHAI's CQL loss at the utterance
15

---

Published as a conference paper at ICLR 2023
methed
max score
Wint hparams
inference time per-dialogue (sec)
ILQL
-8.57:2:.13
0.46
5.10:20.12
ILQL (utierance)
-3.99-20.144
22.1=:0.47
single step RL (ulterance) CHAT
-7.35 1: 0.17
0.21
20.38:10.44
.3.57440..1
1.11
0.25
Table 5: On the VisualDialogue "ynn" reward, we compare per-token ILQL to per-utterance ILQL, per- per-utterance single-step RL, and CHAI (Verma et al. (022). We observe that per-token ILQL is generally much faster inference time than per-utterance methods, while also outperforning or performing equivalently all per- utterance baselines. All evaluations were performed on a single T4 GPU. All baseline implementations build on the same core code for sampling utterances, with handful of method specific runtime optimizations in cach case.
level, we need to sample counterfactual utterances for each action the training data, which can be highly expensive and can greatly slow training, Following Verma et al. (Verma et al. 2022), we amortize this cost at the risk of inducing some bias by caching 5 counterfactual samples for each action in the training data a preprocessing step. In case, this preprocessing step took over 30 hours to execute on a V-100 GPU for the full the Visual Dialogue training set. As in all our other experiments, we train parameterized as the minimum of both Polyak averaged target Q heads, We train our CHAI models with batch size of 16 and otherwise all other hyperparameters are identical those used with ILQL. We train models with CQL a (0.1, 10.0), and we evaluate each model with the number of EMAQ-style (Ghasemipour et al. 2020) samples chosen from N=(4. 16). We report the setting with the best performance. We include comparisons to CHAI in Table
Q networks (Fujimoto al.) 2018), where target Q value is
Per-token dynamic programming baselines. For our per-token CQL and e-learning baselines in Table 4 tuned the CQL loss weight with {0.1, 1.0, 10.0), and the vi-learning reward scale with {0.1. 1.0, 10.0}. For each baseline agent, we evaluated using ILQL's policy extraction with {4, 8, 16} and also evaluated by greedily selecting tokens with the Q function by itself. We report the setting with the best performance for each baseline.
Our implementation of per-token CQL is identical ILQ with the only exception being that for per-token CQL the loss function is defined as:
T Σ (R(hi,a))) + max Qa(hu+1, 0x41) - LE=
LQv((0) = ETTD
Q0 a;))²
a411EA
Our implementation of gi-learning is adapted from Jaques et (Jaques et al. 2020, 2017) for use on transformer language models (Vaswani et al.) [0117; Radford et al. 20199) instead of RNNs (Gers) ct al. 2000). The architecture is identical to that of ILQL, the main difference in the loss function:
Σ Las R(hi, a.) + (oi,(T a;)) + y log(
IQ.v(0) ETAD
Σ Gc+1EA
Where Ta is our BC baseline model: a transformer language model trained with supervised learning. And Ly defines the Huber loss; we use y 1 in our experiments.
In both baselines, we also fit a value function head to the mean of the Q functions, as in ILQL with - 0.5. Additionally, for both baselines, aside from the parameters mentioned, all other parameters are identical to those used with ILQL, as described above. The only exception being that for d--earning, we used learning rate of le-5 instead of 1e-4 due to training instability with the higher learning rate.
We generally found vi-learning to be highly unstable to train in our experiments, often producing incomprehensible outputs. It is possible that the baseline could work better with even more careful tuning.
AWR extraction abalation details. For "ILQL (AWR)" ablation, we extracted policy with AWR extraction using the best performing ILQL value function out of those trained with {0.7, 0.8, 0.9). We performed AWR extraction from this value function using 3 different settings
16

---

Published as a conference paper at ICLR 2023
synthesize a dataset with trajectories from different Wordle policies.
We see dramatic difference between ILQL and SARSA on this dataset.
i Tupper bound represents GasoGoal average 2.047
T Tadveraaiial represents eoLavv average rewardt
7T suboptiiaa represents 5, 2, overage reward" -40.5% cata
Fraction Valuen Where (5, 1115 15
SARSA
1.00
cara
"upaaa to verte
0.75
0.50
Rices
12115
42165
ridet
0.25
Tea 20451
You mm1 T2a v211
0.00
20000
40000
60000
Figure 6: We empirically validate the setting depicted in Figure 4 on a Wordle task. Left: a visualization of the synthetic dataset distribution we constructed to evaluate the benefits of ILQL's multiple steps of policy improvement over "single step" methods, such as single-step RL. Right: a plot showing that ILQL's Q function learns to more often assign higher Q values to optimal actions than single-step RL.
for beta: 3 = {4,8, 16). Using the same value function, we performed ILQL extraction with T {4, 8, 16). For each, we reported the setting with the best performance.
GOLD baseline. Our GOLD baseline is adopted from Pang et al. PPag & He 2020), with some small modifications. In particular, we use our best performing ILQL Q function as the Q function for GOLD policy learning, and instead of using a manually tuned constant baseline, we use ILQL value-function, V. as the baseline. We updated our target policy every 1.5k steps, as was done on several tasks in the original gold paper. We trained 4 models one with the target-policy weight lower-bound hyperparameter u set to each of (0.00, 0.10, 0.15, 0.203. We reported the setting with the best performance.
A.5 WORDLE TASK DETAILS
Evaluating ILQL on the synthetic Wordle task.
Wordle Background. In the game of Wordle, the agent gets 6 turns to guess a 5 letter word randomly selected from a vocabulary, and the environment responds with one of three "colors" for each letter in the guessed word: "black" meaning the guessed letter is not in the environment's 's word, "yellow" meaning the guessed letter is in the word but not in the right location, and "green" meaning the guessed letter is in the right location. We give a reward of 1 for each incorrect guess and reward of 0 for correct guess, which point environment interaction ends; the agent's goal is therefore to guess the correct word in as few turns as possible, a task for which computing optimal behavior has previously been proven to be an intractable NP-Hard problem (Lokshtanov & Subercaseaux 2022).
Environment details. Our agents observe the game-state history of alternating sequences of 5 letter tokens followed by 5 color tokens. Unlike the actual Wordle game, we do not prevent the agent from generating words that aren't in the vocabulary (i.e., the agent free produce any sequence of 5 letters). For our synthetic experiments, we chose to use the full word list given at https::/giss..githube http://g/gggiiuuubooo///rreeseee wtw////reessmmmmaaa2222oo httpr///aaaaaaaattt////////ttttd /8008//00000007777700aaaaaaa
to its relatively large size and its use in the actual Wordle game. However, the environment can be configured with any provided list of letter words.
Wordle Twitter dataset details. We outline the details of our natural Wordle dataset scraped from Twitter, introduced in Section [S] Due to Wordle's popularity, we have access to a large amount of natural human data for Wordle (specifically 214.930 games), scraped from tweets Existing offline RL benchmarks are composed of purely synthetic data (Fu et al. 2020), and as result, it may be unclear how well offline RI. algorithms work on more natural data distributions. We therefore present this human Wordle dataset as more naturalistic offline RI. task. While the scraped Tweets don't ' display the actual words used by human players, only the sequence of transition colors given by the
we use the wordle tweet dataset provided here: http://www...nggccccmmmeeeeeeeee
Whotebooo
17

---

Published as a conference paper at ICLR 2023
method
Wordle Score
ILQL
-2.13 1 0.03
single-step RL
-2.23 0.03
Filtered Fine-tuning
-2.38 + 0.03
Fine-tuning
-2.61 + 0.03
Tupperr bound
-1.75 1 0.02
Table 6: Comparing ILQL to baselines on Wordle human data. Even on realistic human data, ILQL outperforms "single step" single-step RL.
environment, we can retrofit valid words onto these tweets to produce a dataset of full trajectories. Note that the words we retrofit may not necessarily be the natural words human players would have used, but the dataset still represents the average performance of human players, since the number of turns remains unchanged in this retrofitting process. Additionally, this retrofitting allows to further multiply the size of the dataset, since typically several different sequences of words can be valid for a given tweet. We can also partially control the difficulty level of the task and dataset by specifying the size and composition of the vocabulary used to retrofit words onto Tweets. In our Wordle human experiments in Table 6 we use a random subset of 200 words from the word list given at https://gist..iituut cow
Synthetic Wordle Evaluation. For our synthetic Wordle task, we constructed a training distribution in which demonstrate IIQQI's multiple steps of policy improvement significantly outperforms methods which only perform a single-step of improvement, such as single-step RL. As described in Secrion our training distribution consists of mixture of 3 policies, each of which represents one of the paths through the MDP in Figure 4 (1). Tupper bound: a high- bigh-peeforming policy which myopically selects the word with the highest intormation gain, representing the path from S. - Goal. (2). Tadversariall which behaves the same as Tupper bound for the first two actions (So - Si) and then subsequently words (So + Lava). (3). Tsuhoptimal- which selects random word 50%
repeats these first of the time, and the other 50% randomly selects a word that meets all known letter constraints, representing the path from So + Sj. The relative performance of these policies ordered according to Tupper bound >> sasboptimal > Tadversarial- We construct our dataset with 9% of the data coming from T Tupper bound, 45.5% from suuhoptimal- and 45.5% from aauversarial- In Figure 6 (right), we show that when trained on this distribution, ILQQ assigns higher Q values to actions corresponding to the paths to "misleading states" (i.e. S2) than those to the "goal states" (i.e. S1), whereas single-step RL shows the exact opposite preference, just as our hypothesis predicted.
Human Wordle Evaluation. In Figure we compare ILQL against baselines on our datasct of Wordle games scraped from Twitter. We see that even realistic data, ILQL's multiple steps of policy improvement outperforms methods which employ just a single-step of improvement.
A.6
VISUAL DIALOGUE TASK DETAILS
Here we detail the task setup and reward functions used in our Visual Dialogue experiments in Section 6.1 We use Das et al.'s code P to produce generations and predict image embeddings from the provided supervised learning answer and question bots, respectively. We integrate these components of Das al.'s 's codebase into ours by wrapping the relevant functionality of Das al.'s codebase in a flask webserver interface that is then queried by our system.
As described in Section 6.I our "standard" reward function gives a reward of 1 for each turn in which the true image is sufficiently difficult to predict from the dialogue, otherwise the agent receives a reward of and the environment interaction ends. We firstly formalize this notion of "sufficiently difficult to predict".
The standard reward is based on the relative percentile ranking of the ground truth image's distance from the predicted embedding among a set of images taken from the evaluation set. We give -1 reward to our agent for every turn in which (1 - pt) < (1 po) * 0.5, where pt the ground truth image's 's percentile rank at dialogue turn /. and po is the ground truth image's percentile rank at the beginning of the dialogue, when only the image caption is observed. Otherwise, the agent gets a reward of 0 and the episode ends. This condition effectively rewards the agent once the ground truth image is preferred over 50% of the images that were preferred over it at initialization. The agent should learn to ask many good questions possible to get the episode to successfully end as early
18

---

Published as a conference paper at ICLR 2023
method ILQL single-step RL Filtered Fine-tuning Finc-tuning
toxicity
noised Loxicity
upvotes real
upwoles model
0.0=0.0
0.0.0.0
9.83:10.04
10.0
0.0=0.00
0.010.0
6.2310.15
10,010.0
0.74=10.07
-1.6100.11
7.06:10.14
7.86:10.13
-3.51+0.13
-3.48+00.15
4.87+00.16
4.87+0.16
Table 7: A comparison of ILQL against baselines the various Reddit comments reward functions. ILQL manages never generate undesirable comments on 3 out of reward functions, whereas fine-tuning on filtered data occasionally docs.
as possible. In initial experiments, we found it took a very long time to train good value functions for rewards based on the absolute Euclidean distance alone, as used by Das et al. (Das et al. 2017). so to make it faster to iterate, we used the relative distance formulation described above.
Our "y'n" reward adds, on top of the "standard" reward, a reward of -2 for every question that results in response that exactly matches the strings "yes" or "no""
Our "conservative yn" reward instead aims to provide a more conservative, higher-recall lower- precision penalty to any question which might be yes/no question. This accounts for the fact that people often answer yes/no questions with longer phrases (e.g. "It appears so"). This reward function provides reward of -2 if any of the following words are sub-strings of the response the agent's 's question: "not", "don't i", "can't", "cannot", "fairly", "could", "think so", "okay", "maybe", "yes", "no", "looks", "appears", "tell", "mostly just". All of these were determined by hand to be words/phrases that occur often in answers to questions that are effectively yes/no questions.
A.7 REDDIT REWARD MODEL DETAILS
We outline the details of our reward functions for the Reddit tasks presented in Section 6.2
Toxicity Reward. Our toxicity filter reward uses OpenAl's API which provides a free toxicity filter, meant for developers building applications off the GPT3 API to use to block toxic inputs or generations. We assign reward of -10 for comments labeled as toxic (scored as 2), -5 for comments labeled as moderately toxic (scored as 1), and 0 for comments labeled as non-toxic (scored as 0).
Upvote Model Reward. Our upvote reward function is finetuned from RoBERTa-base (Liu et al. 2019) with learning rate of lc-5 and a batch size of 64. Since our reward functions are binary. we train with binary cross entropy loss. Like our value function heads in ILQL. we predict the reward as sealar from a 2-layer MI. on top the RoBERTa transformer, with hidden dimension twice that of the transformer, ReLU non-linearity, and no dropout. We truncate token sequences to maximum length 256. At inference time, we predict a reward of +10 if the model's reward logit is > 0 and reward of O otherwise. We used binary (positive or negative) rewards for upvotes instead of the more natural cardinal numeric representation, because different sub-reddits can have drastically different upvote counts depending the sub-reddit's population, and our binarization (positive or negative upvotes) is invariant to these differences scale. However, this binarization is not the only normalization that could have used to overcome this issue.
A.8 NOISY REWARDS
we present more detailed visual explanation for why offline RL outperforms filtered finetuning on our Reddit tasks in section 66.2 As our results in Table Lable2show offline RL consistently outperforms finetuning on filtered data on this task. We hypothesize that this is due to offline RL's ability effectively reason about the inherently stochastic and subjective rewards functions present in these tasks. In these high- variance reward settings, simply filtering or curating datasets for exclusively high-reward examples can fail to produce desirable outputs, since such filtered finetuning approaches do not make the model aware of the reward uncertainty. Put another way, training on curated datasets that exclude low-reward examples doesn't t teach the model about what not to generate, whereas models that are aware of the reward, such as Q-learning, directly learn to relate actions to their expected reward values, averaging out uncertainty and stochasticity. This can enable models to avoid
In figure 7
https//openai.com/app//
19

---

Published as a conference paper at ICLR 2023
Stochastic Reward
Key:
expected boundary
policy probsbility density
Filtred Fineluning Palicy
Offiis- Puñcy
Non-stochastic Reward
Key:
boundary
comment
policy probability density
Filtered Finctusing Policy
Offline RL Pelicy
Figure 7: A visual explanation of offline RL's ability optimize high variance reward functions based on subjective judgement, such as whether to label a comment as an example of toxic speech or not. Finetuning on filtered data accidentally generalizes into producing undesirable outputs, whereas offline RL is able to find the "safe" outputs. Top: In the case of stochastic rewards, offline RL learns avoid the highly stochastic regions of the action space, whereas filtered finetuning will explicitly learn to imitate undesirable outputs that were stochastically given positive reward in the training data, thus leading suboptimal behavior. Right: the case of non-stochastic but sharp-boundary reward functions, IIQQI. is still able to integrate into its Q values uncertainty about actions near the sharper parts of the reward function's decision boundary, thus avoiding these regions. Finetuning on filtered data expresses no such preference and thus risks generalizing into occasionally producing undesirable outputs.
ILQL per-token advantages for toxic comments generated by filtered finetuning model
advantage:
0.1
-0.1
-0.5
0.1
0.2
4.99
-1.1
token:
And
they
censor
your
comments
on
this
homrible
site
advantage:
0.1
-1.6
-0.6
-06
0.5
.2.0
-0.6
-.0
-12
-1.0
-10
.21
He
ripped
your
thinking
dead
Figure 8: Two toxic comments incidentally generated by the filtered fine- tieettnnig model. ILQL assigns negative advantages to many tokens, demonstrating how ILQL is more effectively able to avoid such generations.
20

---

Published as a conference paper at ICLR 2023
comment: yes
comment: http:// http://lmgur.oom/ee. gif
comment:
comment: and all you know me...
comment:
comment: Yep. The Oalmeal the right
trying to deny [mns]/ntp:llnggu.coo/jjjee
0.0
comment:
it's not the main menu, it's the
fridge
I did what Idra did, Deep down "I'm cool" and could do was loave.
comment:
0.5
-1.0
comment: All see its Saneeby-lrame summary of its most relevant
toxicity t reward
comments. So heving no thought
-1.5
whatsoever happy.
-2.0
-2.5
-3.0
SARSA
IQLL
Rinettuning Filtered Fine tuning
-3,5-
50
100
150
200
250
300
policy entropy (nats)
Figure 9: By varying 3 at inference time, we can interpolate the trade-off between offline RL "optimization" and output diversity.
outputs that have low but non-trivial probability of undesirable outcomes (e.g., toxicity). Offline RL in this sense is able to find the safest outputs, whereas fine-tuning on filtered data does not explicitly express such preference.
To further test this hypothesis, we artificially add further noise to the toxicity reward function. The standard toxicity reward assigns all comments one of three reward values: 0, indicating the comment non-toxic; -5, indicating the comment is moderately toxic; -10, indicating the comment is highly toxic. We now relabel the reward for all comments originally given -5 reward, randomly to either 0 or 10 with equal probability.
Since some of the moderately toxic comments get relabeled with reward=O, they would be included in the %BC training set, whereas offline RL should learn to represent uncertainty about such comments and thus push away from the stochastic "middle ground" of this reward function. In Table A.8 in the "noised toxicity" column, we see that our offline RL agents learn to never generate toxic outputs despite the additional noise, and Figure g we can see qualitatively that offline RL assigns low advantages to potentially negative or toxic words/phrases that were incidentally generated by the %BC model. All of this goes support our hypothesis that the advantage of offline RL over filtered supervised learning in these settings lies in its improved ability to handle reward uncertainty.
A.9 TRADING OFF OUTPUT DIVERSITY FOR OPTIMIZATION
An advantage of our novel policy extraction mechanism is that we can flexibly tune the parameter | inference time, directly trading off between random generation and optimality. As discussed in Section 9 this parameter controls constraint on our policy's 's deviation from the data distribution. As we increase 8, the resulting policy will be more strongly influenced by the Q-function, and as we decrease 3, it should approach the data distribution. Beyond the risk in diverging too far from the data, another potential downside to increasing 3 is that the resulting policy distribution will become more deterministic. In some settings, such chit-chat dialogue, may desire policies capable of producing diverse and interesting outputs, so such a deterministic, highly-optimized agent would be undesirable.
We demonstrate in Figure using the Reddit Toxicity task, how varying can modulate the diversity of the generations produced by our policy, measured by its entropy. the cost of small decrease in performance. We show that as we increase B, while the policy's 's performance generally increases, the entropy decreases and subsequently so does the interestingness and diversity of the language
21

---

Published as a conference paper at ICLR 2023
model
reward
rougel.
rougel
BERT score
procision
rocal
Fine mine-tuning
standard
27.20.82
26:10.82
0.90-10.0011
0.0910000012
0.90-10.0011
Filtered Fine-tuning
standard
22:30.73
21
0.89-00.0010
0.89-0000010
0.89=
single-stop RL
standard
26=20.82
26:00.80
0.90=+0.0011
0.89=0.0011
0.89:
ILQL
standard
24.70.0.73
23::0.72
0.89:10.00011
0.89 0.0001
0.09:10.0011
Fine tuning
yh
27.10.82
26.10.82
0.90110.0012
000002
0.90
Filtered Fine-tuning
y/n
20--0.83
20:10.69
0.88::0.0010
0.88
single-stop RL
yin 25-0.83
24+00.81
0.89==000012
0.89=000012
0.89=0000011
ILQL
wla
25.20.82
24:00.81
0,88:+0.0012
0.89=0.0012
0.89:0000011
Fine-tuning Filtered Fine-tuning single-step RL
conservative y/n
2720.82
26220.82
0,90=E0.0012
0.89=0.0012
0.90=E0.0011
conservative y/a
22-40.74
21:10.73
0.000
0.88=0000110
0.00
conservative y/n
23-+0.82
23:-0.80
0.88:=0.0012
0.88=0.00122
0.88+0000011
ILQL
conservative y/n 24=0.811
23+00779
0.88=0000011
0.89=0.0012
0.88=0000011
Table 8: Automatic language quality evaluations comparisons between II.QL. and baselines on each Visual Dialogue reward. We see that ILQL still nearly matches standard fine-tuning on these reference- based metrics of language quality and also generally matches or exceeds filtered fine-tuning.
model's outputs. At inference time, we can tune this parameter to trade-off optimization for output diversity as we desire.
A.10 VISUAL. DIALOGUE EXAMPLE DIALOGUES AND HISTOGRAM
In Figures 10 and 12 present set of selected representative example dialogues produced by our best performing IQQ different agents generate yes/no questions as judged by their respective "y'n" or "'ynn conservative" reward functions.
agent on each reward. In Figure [3 we present a histogram showing often
A.11 AUTOMATIC LANGUAGE QUALITY EVALUATIONS
In Tables and we present automatic reference-based language quality scores for our main baselines on each Visual Dialogue and Reddit comment reward.
A.12 REDDIT COMMENTS PRELIMINARY USER STUDY
To validate the automatic evaluations from our Reddit comments task, we ran a preliminary human study, where we got 41 people to answer 48 questions each about generations from our Reddit comment agents. We asked subjects two types of questions: 1) to classify the toxicity of comments generated by agents trained on our "toxic" reward function; and 2) to compare the diversity of outputs produced by our ILQL agent with 3 = 4 and / 32, verifying the claims about output diversity in Appendix A.9 In each case, generations were randomly selected and shuffled for each user from a bank of 1 128 sampled generations for each agent.
We present the results for the toxicity questions in Table 10 We see that the human ratings align very well with those from our automatic evaluations, with ILQL and single-step RL never generating toxic comments, and BC and %BC producing more toxic comments. This further emphasizes the effectiveness of ILQ in optimizing high variance rewards based subjective human judgement.
In the case of our diversity questions, our human raters were given sets of 5 random comments from ILQL with beta=4, and 5 random comments from ILQL with beta=32. They were then asked to select which set of comments was more "diverse". Our subjects were asked to do this task 7 times with the order of the two sets randomly swapped. Raters determined that ILQL with beta=4 generated more diverse comments in 282 out of 289 such cases. This confirms our claim in Appendix (A.9 that by changing beta, the inference time hyperparameter, we can effectively control the tradeoff between output diversity and objective optimization.
In initial investigations, we found that human raters could not provide reliable evaluations for the upvotes reward (regardless of which method produced the comments). This is likely because judging whether a comment will receive an upvote without access to either the comment's conversation context or its broader context in the specific Reddit community is very difficult for human raters (e.g., many of the comments are short phrases like "Yes" or "Thank you"), and the human raters do not necessarily have the right mental model of typical reddit upvote dynamics to make accurate predictions. We therefore consider our automated evaluations to be the closest thing we have to the "ground truth reward" in this case.
22

---

Published as a conference paper at ICLR 2023
model
reward
rougeL
rougel
BERT score
precision
recall
F1
Fine-tuning
upvotes real
74:0.29
6220266
0.000000029
0.82:00.0025
0824000022
Filtered Fine-tuning
upvotes real
9=0.332
7+0.29
0.83+0.00028
0.82++0.002
0.83+0
single-step RL 9
upvotes real
5:20.29
5200277
0.5::0000026
0.81+0.0025
0.0000
single-seep RI. 9
upvotes real
1.10.15
11.0.14
0.84.10.0025
0.79 0.0025
0.81 0.0014
single-step RI. 8
upvotes real
0.76.10.11
0.76.10.11
0.83::000025
0.79::0.0024
0.801:0.0024
single-step RL 3
upvotes real
0.75::0.11
0.75+0.11
0.83+:0.0025
0.79=
0.0024
sinple-step
upvotes real
0.75::0.11
0.754.0.11
0.83:10000025
0.79.10.0002
0.80140.0004
single-step RL / 32
upvotes real
0.75=00.11
0.75+0.11
0.83=:0.0025
0.79=
0.0024
ILQL
upvotes real
5=20.28
5+0.26
0.85+00.0026
0.82++0.0022
0.83+
ILQL
upvoles real
3::00,66
31:0,25
0.87:10.0078
0.811:0.0025
0.0425.0025
IIQQ1,
upvotes real
2.10.22
2.10.22
0.87 0.87.10.0022
0.8:10.0025
0.83.1
ILQL
upvotes real
14:0.12
120.12
0.85::0.0027
0.79=0.00225
0.82+:0.0025
ILQL 9 = 16
upvotes real
0.34::0.08
0,34:20.88
0.84::0.0025
0.79:5
0.0024
I1.Q1, ,
upvotes real
0.15--0.05
0.15+:0.5
0.84::0.0025
0.79:10.0024
0.0044000044
Fine luning Filtered Fine-huning
upsales model
7.10.29
0.26
0.80100.0099
0.0210.00255
0.82.
upvoles model
00:20.34
8:10.31
0.84::0.0027
0.035
0.025
single-step RL
upvotes model
24:024
2.20.33
0.88::0.0029
0.0025
0.84+000002
single-step RL 3
upvotes model
0.33:::.14
0.33
0.84
0.79:
0.8024
single-step RL.
upvotes model
0.31
0.83:10.0025
0.78
single-step Rt. 3
upvotes model
0.31
0.31
0.831:0.0022
0.7824..0024
1.00244
single-slep RL. 8
upvotes model
0.31::0.14
0.31
0.03+
0.78:10.0022
0.81
single stepl RI. f
upvales model
0.31110.14
0.31
0.831.0.0025
0.78.10.002
0,81 0.0024
ILQQ. 3
upvotes model
3::0.24
2.20.2
0.88::0000277
0.8 0-00022
0.85
ILQL
upvotes model
0.42+00.15
0.42::0.15
0.87
0.8:20.0022
0.83
ILQUL 3
upvotes model
0.31:1:0114
0,31
0.83:1:000025
0.78::0.00024
0.80140.00024
ILQL.
upvotes model
0.1310.14
0.83110.0025
0.8.10.0024
11QQ1. 8
16
upvotes model
0.31
0.31
0.821-0.0024
0.78:40.0024
0.8:0.00024
I1QQ1 8
upvotes model
0.31110.14
0.3110.14
0.8210
0.78. 0.70.4..0024
0.8:10.0024
Fine-tuning Filtered Fine-tuning
loxicity
740.29
0.0029
0.82:00.0025
0.02
luxicity
810.32
7::0.29
0.82:10.0026
0.0255
0.82
single-step Rt.
tuxicity
0.811.0.003
0.0255
0.81
single-step RI. 9
toxicity
8=0.31
7+0.28
0.82=#0.003
0.82
0.02260000026
single-step RL
toxicity
74:0299
6+:0266
0.82:40.0031
0.82:+0.0022
0.82+:0.0026
single-step Rl.
toxicity
3::024
32:0223
0.841000051
0.79:
0.8041000088
single-step RL / 16
toxicity
0.02::0.01
0.02++0.01
0.14::0.0099
0.13:t000092
0, 0.1495000095
single-step /
Ixxciity
0.0:1:0.0
0.0::0.0
0.0:1:00
0.0=0.00
0.0:1:000
ILQL. 8 =
toxicity
7.100.11
61.0228
0.81100.000
0.82
0.81
I1QQI.F
toxicity
74.0.32
64:0.29
0.00110.0031
0.82--0.0025
0.8111000026
ILQL 8
toxicity
8=0.3
7=0.27
0.81:00.0031
0.82=:0.0025
0.802700002
ILQL
toxicity
74:0.11
64:0.29
0.83=20.003
0.82=+0.0025
0.82-
ILQL 8
toxicity
0.81+00118
0.8::0.18
0.85::0.0032
0.79=0000228
0.82::0.0029
ILQL 8 32
toxicity
0.35:t0.14
0.35::0.14
0,79::0.0022
0.74
0.77=00006
Table 9: Automatic language quality evaluations comparisons hetween ILQ and baselines the Reddit comment tasks. As we increase our policy diverges further from dataset distribution, and correspondingly the reference-based scores decreases. In general, however, we see that ILQL maintains greater language quality than single-step RL for similar values of B.
method
# toxicity ratings / # ratings
ILQL 5 32 ILQL 3 = BC
0/ 164
4/ 164
43 / 159
%BC
12 / 158
single-step RL.
0 159
Table 10: Human ratings of output toxicity on our Reddit comments task. We see that ILQL and single-step RL never generate toxic comments, whereas BC and &BC occasionally do. These results align very closely with our automatic evaluations, further emphasizing the effectiveness of ILQL for optimizing these kinds of high variance reward functions based on subjective human judgement.
23

---

Published as a conference paper at ICLR 2023
A.13 COMPREHENSIVE RESULT TABLES
In Tables 11
12
13
and 14
present comprehensive results for all hyper-parameter settings for all
baselines on all tasks.
model
standard
Vin
conservative yin
LOL 0.T. -3.23=0133
6.6320.188
-7.6420.21
ILQL =.T. 8
-5.22=0.13
15924...15
-7,05:0..19
ILQL r 0.7. 16
0.18
5.69
6.77:10.188
I1QI. 0.7.8
0.11
1..771..13
6.64.10.18
ILQL 0.8,0
-5.30=0.12
-688
-7.97+0.211
ILQ. 0.8.
I1QL r 0.8, 16
-..38=0.13
5.99
6.665
ILQL r 0.8. = 30
-3.41=0.13
-5.64
-6.71
ILQL 0.9.
-5.35=0.33
17.01
07
ILQL r 0.9,
5.40
6.35110.16
6.72.10.18
-5.45=0.13
66.1740.15
1.
ILQL F 0.9, 00
-5.41
+.89
-6.09
single-sicp RL
-3.20=0.13
6687
9.1220.24
single-siep RL
-5.14-0.113
6.39110.16
8.09:10,21
single.step R1. 8
-5.18-0.13
6.19...155
7..77..0.20
single-step RI
-5.30=0.13
-7.63-0..00
10% Filtered Fine tuning
5.24-0.12
17.8
9.67.100.26
20% Filtered Fine-tuning
-5.07=0.13
-8.9140244
-9,33+0,222
30% Filtered Fine tuming
5.16.0.12
9.1011222
10,52
Fine-tuning
-5.25=0.13
-10.55:00.27
-15.16=00.35
Table 11: All hyper-parameter settings evaluated across all Visual Dialogue tasks for main baselines. The best performing setting for each baseline of abalation bolded. / = oc refers to greedily selecting actions with just the Q function. ILQL is generally better performing and more stable than all baseline approaches.
24

---

Published as a conference paper at ICLR 2023
model
V/a
CQL 0.1,
00.08
CQL 0.1,
0.297
CQLa 0.1. 16
12.9220.233
COL.a 0.1.
10.74
CQL = 1.0, =
-7.84.0.200
1.0,
CQL 1.0, 16
-7.45=0.188
COL 1.0, 06
-7.32=0.17
COLa 9
-11.76=0.29
COL 10.0. 8 =
11.81
CQL. 8
0..
CQL 10.0, 8
-11.81
yc=01.3 =1
-11.99=0.18
$c=01.9
-11.26:1..17
=c= 0.1.6 16
11.42.10.17
#c=
-1.5110.116
6c 1..11
10,33=
10.05500.18
$ 1..1
#c 1.0, 11
10.05
#c= 1.0,
8
10.05500.18
10,0 8
11.33510.18
# 0001 8 8
10.99-10.17
16
10.
V 10.0. 3
-0..78=0.11
DTR 11
DTR
10.58 0.24
DTR = 9
-10.33 0.24
DIR = -7
10.40 = 0.23
DT R -
-10.30 0.23
=
-10.30
DIR -3
9983 = 0.22
DTR -2
-9.54 + 0.21
DI = 0
6.70 0.17
ILQ (AWR) 7 0.7.
1.96=0.13
ILQL (AWR) 0.7.
-1.755:0.23
ILQQ (AWR) 0.7.
12.11110.22
(utterance)
0.7.N
ILQL (utterance) 0.7.N
6,04
ILQL (utterance)
0.7.N
16
ILQL (ullerance)
0.8. N
7.11210.17
11.01. (utterance) T 0.8.
6.15
ILQL (utterance)
14.1
11.Q1. (utterance) 0.9, N
10.1
I1Q (utterance) T 0.9.
16.12.0.15
ILQL (utterance) 0.9.N
-3.93=0.14
single-step RI. (utterance)
-78110.18
single-stop
-7.3920.17
single-step RL (utterance) N 16
-7.35
CHAI a 0.1,N
0..18
CHAI O.I.N
0.1810.15
CHAI a = O.1.N
-562 13
CHAI
-7.03=0.17
CHAI a 1.0,N =
-.17
CHAT 10.0, N
-.43
C 10.0. =8
8.24
CHAT 10.0.N
16
8.28-10.15
GOLD u 0.00
8.33.1021
GOLD 0.15
-8.97=0.255
GOLD 0.20
-9.12=0.25
Table 12: All hyper-parameter settings evaluated across all abalations, and baselines on the Visual- Dialogue "y'n" reward. The best performing setting for each baseline of abalation bolded. & = co refers to greedily selecting actions with just the Q function. ILQL is generally better performing and more stable than all baseline approaches.
25

---

Published as a conference paper at ICLR 2023
example standard reward Visual Dialogues
Image Caption: A man, with other people the background, sitting pienic fable with coll phone front of him while he uses fork and knife dessert plate.
Questioner: is man young?
Answerer: yes
Questioner: is he wearing glasses?
Answerer: no
Questioner: is there food on table?
Answerer:
Questioner: there food table?
yes
Image Caption:
A man with wide eyed look with an apron on in
kitchen.
Questioner: is man young?
Answerer: i would say he
Questioner: what color is man's hair?
Answerer: brown
Questioner: wearing glasses?
Answerer: no
Questioner: is he wearing apron?
Answerer: can't tell
Questioner: is he wearing apron?
Answerer: i can't tell
Questioner: there anything his head?
Answerer: yes, it's UNK and UNK
Questioner: he looking something?
Answerer: he looking at camera
Questioner: is there anything else on counter?
Answerer: yes, there are some papers and papers
Questioner: is there anything on counter?
Answerer: there are some papers and papers
Image Caption: Young edults with surfboards standing on pethway near beach.
Questioner: how many people are there?
Answerer:
Questioner:
male female?
Answerer: male and female
Questioner: they wearing wetsuits?
Answerer: yes
Questioner: what color are they?
Answerer: black and other is white
Questioner: are they wearing swimsuits?
Answerer: yes they are
Questioner: are they wearing wetsults?
Answerer: yes they
Questioner: they wearing sunglasses?
Answerer: no they are not
Questioner: are there any other people around?
Answerer: no there are not
Questioner: are there any animals?
Answerer: no animals
Questioner: is sunny?
Answerer: yes is
Image Caption: Colorful vehicles in roadway near large building in city. Questioner: how many cars are there?
Answerer: i see 3
Questioner: are there any people?
Answerer: yes, there are people
Questioner: are there any parked?
Answerer: yes, there
Figure 10: Example dialogues produced by the best performing ILQL agent on the Visual Dialogue "standard" reward.
26

---

Published as a conference paper at ICLR 2023
example y/n reward Visual Dialogues
Image Caption: Yellow and blue paseenger train rounding curve.
Questioner: how many cars on train?
Answerer: can only see
day or night?
Answerer: day
Questioner: sunny cloudy?
Answerer: sunny
Questioner: there anything else tracks?
birds
Questioner: how birds are there?
Answerer: 2
Questioner: what color are birds?
Answerer: black
what
sky?
Answerer: blue
Questioner: what time of day is it?
Answerer: it is daytime
Questioner: what time of year I!?
Answerer:
Questioner: what
you see?
Answerer: trees, sky, sky,
Image Caption: person flying kite sand dume.
Questioner: what color kite?
Answerer: it's black and white
Questioner: person male or female?
what person wearing?
Answerer: can't
Questioner: what color sand?
Answerer: dark brown
Questioner: person flying flying yes
Questioner: what color kite?
Answerer: black
Image Caption: Little girl reaching through fencee to pet giralfe.
Questioner: how is girl?
Answerer: maybe
what
Answerer: dark brown
Questioner: what she wearing?
Answerer: can only see her from waist up
Questioner: what color her
Questioner: what her eye color?
Answerer: can't toll, it's covered with white and black and red
Questioner: what fence made of?
Answerer: wood
Questioner:
Answerer: metal fence with black metal fence
Questioner: giraffe full grown?
Answerer: yes
Image Caption: number of people in fleid with kite the Questioner: how many people are there?
Answerer: see
Questioner:
Answerer: hard tell, but looks black
Questioner: they male femalle?
Answerer: both are male
Questioner: what are they wearing?
Answerer: they are wearing shorts
Questioner: what
shorts?
Answerer: black
Figure 11: Example dialogues produced by the best performing ILQL agent on the Visual Dialogue "y/n" reward.
27

---

Published as a conference paper at ICLR 2023
example conservative y/n reward Visual Dialogues
Image Caption: Tour buses are lined up an the street wailing for people.
Questioner: many
Answerer:
Questioner: what color are buses?
Answerer: while and red
Questioner: how many people?
Answerer:
Questioner: what gender people?
Answerer: male
female
Questioner: what are they wearing?
Answerer: wearing shorts and other wearing shorts and shirt
Questioner: what
hair?
dark
Questioner: what color their shirt?
Answerer: black
Questioner: what color is their pants?
Answerer: black
Questioner:
Answerer: black
Questioner: what color is their pents?
Answerer: black
Image Caption: living room with table couch guiter and some pictures
Questioner: what color is couch?
Answerer: brown
Questioner: what
pictures are on couch?
UNK other
Questioner: what color guitar?
Answerer: black
Questioner: what color is table?
Answerer: brown
table?
Answerer: brown
Questioner: what color is guitar?
Answerer: black
Questioner: what color are walls?
Image Caption: Little gir reaching through fance to pet giraffe. Questioner: what is little girl wearing?
Answerer: she is wearing long sleeved shirt and jeans
Questioner:
Answerer: it's metal fence with black metal fence
Questioner: what is giraffe doing?
Answerer: is just slanding there
Image Caption: Pictures of living room, dining room, kitchen and bedroom. Questioner: how meny people are there?
Answerer:
Questioner: what are they doing?
top of is side fridge
Questioner: what color fridge?
Answerer: silver
Figure 12: Example dialogues produced by the best performing ILQL agent the Visual Dialogue "conservvtive y'n" reward.
28

---

Published as a conference paper at ICLR 2023
"y/n" reward agents
numer dialogues
1000
750
250
0.0
0.2
0.3
0.5
0.6
0.8
0.9
1.0
fraction of yes/no questions dialogue according to the "yln" reward
Fine-tuning
Filtered Fine-tuning
ILQL
SARSA
"conservative y/n" reward agents
logues
500
250
0.1
0.3
0.5
0.7
fraction of yesmo questions dialogue according to the "conservative reward
Figure 13: Top: Histogram of the fraction of yes/no questions asked per dialogue by Visual Dialogue agents trained on the "y/n" reward. Here yes/no questions are determined by the same exact match heuristic used by the "y/n" reward. ILQL agents ask fewer questions triggered being yes/no than baselines. The best performing agent is used for all methods. Bottom: Histogram of the fraction of yes/no questions asked per dialogue by Visual Dialogue agents trained on "conservative y'n" reward. Here yes/no questions "conservative y'n" " reward. ILQL agents ask fewer questions triggered as being yes/no than basclines. The best performing agent is used for all methods.
determined by the same exact match heuristic used by the
29

---

Published as a conference paper at ICLR 2023
method
toxicity
noised loxicity
ILQL 0.6,
2.15500.007 10.997:10.22
189:0.12/200.442000
ILQL 0.6, 8
-1.16:0.08 282.
0.99:0.09/222.0912223
ILQL 0.6. 4
151700005/151446111
-0.53 156.055771188887.000
II.QI. 8 8
-0.12:::.022799 77.20-5527
-0.23+4-0055 77.112-4.88
ILQI 0.6, =
-0.01 / 1.76
-0.03++0.02 22.05--1111
ILQQ r = 0.6, 3 32
0.00--0.00/2.00-00.22
0.00++0.00/ 9.24
single-step RI,
-.60+00.007240221 + 10.10
-..244.12/233.17-9000
single-step RI. 2
-0.96-+0007 199.0449.24
-1.06++010// 187.3==8822
single-step RL 4
-0.34+0.04/ 111.0004/1111131...
-0.41+00.06 108.76-6.57
single-step RL
1.0710.019/37.5900
-0.13.10.03 41.6
single-step RL 3 16
0.00 4.38-
0.00 1.09
single-step RL 3 32
0.00 0.03
0.00
0,000000000011..00 0,00
Filtered l'ine-tunigg
0.74 80.841319
1.6.L0.11790.5533222
Fine-tuning
0.51 137.705.888
1.48:0115/2665445..
method
upvotes real
upvotes model
ILQ r = 0.6,
1.9
8.38+0.12/25.1552
ILQL 0.6, B
7:0110.1421147:11..
9.53110.07 7.82.0.88
ILQL 0.6.
7.47:0.44799110.63
9.9910.01 1.09
ILQL r = 0.6. 8=
9.05 1.555.00
10.0010.0000.78.0000
ILQL r = 0.6, 3= 16
9.73=0.05 0.55
10.00 0.54
ILQL r = 0.6, = 32
9.83=0.04/0. 10.
10.00=00.0 0.202
single-step RL 3=
/23....15/33.75.1
1. 28.80.L 1.87
single-step RL
4.66.10.16/ 1..801....
8. 9.344
single-step RL
0.81 2.
9.93+0.03 / 0.44
single-step RL $ 8
0.09=0.03 0.0000..00
10.00=0.00 #0.01
single-step RL 8 16
0.09=0.03 0.00=
10.00=10.00 / 0.00
single-step RL 8 32
0.09:00.3//0000-H00
10.00-+0.00/0.00-O0000
Filtered ttnnig
1166:014/100.7744..1
7.86:..3/97.5113.999
Fine-tuning
4.77-0.6//277,5555..2
Table 13: All hyper-parameee settings evaluated across all Reddit Comment tasks, abalations, and baselines. The best performing setting for each baseline of abalation is bolded. The left result in each cell the the agent's 's average reward and the right result is the estimated entropy of the agent's policy, measured in nats. As turned up. performance increases, but the entropy (or diversity) of the policy's 's outputs decreases. ILQL generally learns better performing and more diverse policies than baselines.
30

---

Published as a conference paper at ICLR 2023
method
ILQ 0.7. 8- 4
2.23+0.03
ILQL 7 = 0.7. 8
2.18
ILQQI. 0.7,8 16
2.18+0..3
ILQQ 0.7,8
-6.00+0.00
ILQQ F = 0.8, 8 =
-..3+0.002
ILQL F = 0.8, 3 = 8
2..3
ILQL 0.8. 8 16
20.03
ILQL 0.8.
-6.0010.00
ILQL 0.9, 8
-2.00
ILQL T = 0.9, 8 = 8
-.26-00.03
ILQL = 0.9, $ = 16
-.366
ILQL 0.9, 3
-..00
single-step 8
-.24400.03
single-step RL 8
-2.66+0.03
single-step RL 3 = 16
-.23
single-step RL 3
-6.00+0.00
10%
Filtered Fine-tuning
-2.9310.06
30%
Filtered Fine-tuning -3.01
50%
Filtered Fine-tuning
-2.08
Fine-tuning
-2.61-0.03
Table 14: All hyper-parameter settings and baselines evaluated the human Wordle dataset scraped from Tweets (see Section A.5). The best performing setting for each baseline of abalation bolded.
refers to greedily selecting actions with just the Q function. ILQL is generally better performing than baselines.
31


Reviews:
[{"review_id": "MuzTsxsyml", "reviewer": "Reviewer_jx4F", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an offline RL method for natural language generation, ILQL, which extends a recently proposed offline RL method, Implicit Q Learning. The base method, Implicit Q Learning, uses expectile regression to learn a q function by limiting itself to only observed dataset actions. This paper translates the increased stability from this algorithm  to the NLP setting, while also presenting a few tricks required to making training work for NLG (e.g. pushing down weight of unseen tokens via a regularization term).\n\nAuthors first show a toy wordle example where the multi-step updates enabled by ILQL greatly improve performance in a contrived setting. They then evaluate on more realistic, open-ended visual dialog and reddit comments tasks, showing significant gains over standard finetuning, naive SARSA, and some other offline RL baselines. One of the particularly interesting experiments is that ILQL is robust to changes in the reward function (e.g. avoiding yes/no questions in visual dialog).", "summary_of_the_review": "This paper clearly presents a novel offline RL method, ILQL, which shows strong results across a variety of diverse NLg tasks, especially in more adversarial settings (e.g. reward shift). Although comparison to offline RL baselines are not as thorough as the rest of the paper, I'm inclined to accept.", "strength_and_weaknesses": "# Strengths\n\n- Novel and sensible adaptation of a recent offline RL method, implicit Q learning, to the natural langauge generation setting, as well as various tricks that are needed to make learning work in this setting\n- Thorough experimentation including toy examples as well as more realistic settings in visual dialog and reddit comment generation, with comparison to naive SARSA updates as well as full finetuning/finetuning only on high-reward trajectories.\n- Thorough comparison to other baselines (when such baselines are included)---I appreciate authors' effort to sweep over hyperparameters and explain baselines in detail in Appendix---I think this is much more thorough than your typical ML paper.\n- I like that the method clearly states limitations (e.g. longer training times compared to standard supervised RL, limitations to suboptimal data) in conclusion.\n\n# Weaknesses\n\n- I'm surprised that comparison to other offline RL baselines is deemed an \"ablation\" and only depicted in section 6.3/Table 3, and not throughout. In the main VisDial (table 1) and reddit results (table 2), ILQL is only compared to naive SARSA as well as FT/Filtered FT. Comparison to other offline RL methods used in the literature doesn't feel like an ablation to me - it feels like a fundamental comparison: is the added complexity required by ILQL worth the effort over other (perhaps conceptually simpler) offline RL algorithms? Moreover, how do the other offline RL algorithms handle reward shifts?\n\n## Minor\n\n- I find \"Easy to Use\" to be a fairly subjective/meaningless criterion in Figure 2.\n- Page 6: Appendix ??\n- It'd be great to explicitly point to Appendix A.4 in the paragraph \"Ablations on choice of Offline RL algorithm\".", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "G32yBajxmlp", "reviewer": "Reviewer_LVpS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes Implicit Language Q-Learning (ILQL), a RL method, adapted from Implicit Q-learning, which can learn controlled language generation from a reward signal and a large-scale dataset of mixed quality in an offline fashion. The authors empirically validate their method on a dialogue task and reddit comment generation.", "summary_of_the_review": "This paper introduces and validate a useful and effective new method for using reinforcement-learning to fine-tune large language models in a scalable offline fashion.", "strength_and_weaknesses": "Strengths:\n* The paper is well written and easy to follow.\n* As reinforcement learning gains in popularity for NLP applications, this paper provides an important contribution by introducing scalable offline RL for language.\n* The authors demonstrate that the method is easy to use and effective, and can handle existing data. In particular, it can learn to combine aspects of behaviors to achieve a higher reward than obtainable from the average of the data.\n* The authors validate their method on a goal-directed question asking task (Visual Dialogue) and a Reddit comment generation task.\n\nWeaknesses:\n* ILQL requires a larger computational budget than supervised learning due to the additional cost of the behavioral model.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FMqxzlSSDr", "reviewer": "Reviewer_8Evy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates IQL, an offline RL training algorithm, applied to training a language model to generate a sequence that will maximize a utility function (such as a user reward model). ", "summary_of_the_review": "The paper is generally interesting, and IQIL is sufficiently different from other Offline RL + LM fine-tuning/training papers. However, I don't think the main claim (IQIL/Q-learning > SARSA) in an offline RL setting makes sense. Though this paper has so many interesting ideas, great experimental results, and on a diverse collection of datasets, I cannot recommend an accept at this point.", "strength_and_weaknesses": "Strength:\n- The paper proposes a novel learning paradigm ILQL (Figure 3). The idea of directly \"adding\" advantage Q-V to p(vocab) is very appealing. \n- The paper also re-configured the CQL pessimism penalty and re-adapted it to work with discrete action space, which is novel.\n- The experimental result seems strong to convince me this algorithm works on a wide range of task domains.\n\nWeakness:\n- Section 5 is wrong, which affected all the follow-up experimental comparisons between ILQL and \"SARSA\". There are a few reasons why this is wrong (or \"trivial\"): \n   1. I don't think the author understands the concept of SARSA vs. Q-learning (ILQL). SARSA, first of all, is an on-policy learning algorithm, while Q-learning (ILQL) is an off-policy learning algorithm. In the offline RL setting, an on-policy learning algorithm would never make any sense because on-policy exploration is forbidden. \n   2. SARSA provides a convergence guarantee under two specific conditions: 1). Infinite visits to every state-action pair; 2). The learning policy becomes greedy in the limit. Offline RL setting (and Figure 4 worked example) satisfies NONE of these conditions. If we read the text in the paragraph, the learning policy $\\pi_{\\text{upper bound}}$ was never updated (it stays fixed). The suboptimal result is trivial and obvious. I encourage the author to read [1]. Comparing Q-learning with SARSA in an offline RL setting makes no sense. \n   3. I think this paper tries to follow the original IQL too closely and re-iterate the point made in IQL but without enough finesse and care. The original IQL paper never suggested that SARSA would ever work in an offline RL setting. Also, IQL > SARSA is already shown in the original IQL paper; what is the point of showing this again?\n   4. Overall, I think Section 5 is largely misleading to people who are not familiar with RL and particularly Offline RL.\n\nSuggestion:\n- The equation on page 5, $L^c_{Q, V}(\\theta)$, the author should be able to offer a short proof (of a few lines) that show it's equivalent to CQL pessimism penalty. This would make the statement in the paragraph more concrete: \"this CQL loss term is no more expensive than, and in fact equivalent to, a standard cross-entropy loss at the token level\".\n- I think the authors should consider re-orient this paper to be more NLP-focused and drop the comparison to SARSA in this offline setting. The results aren't surprising to any RL researchers and only serve to confuse them. The proposed algorithm ILQL is novel enough! It doesn't need the motivation to do better than SARSA.\n- I think the authors should rethink the pitch and conduct experiments that make sense to the pitch.\n\nReferences:\n\n[1] Convergence Results for Single-Step On-Policy Reinforcement-Learning Algorithms (https://link.springer.com/content/pdf/10.1023/A:1007678930559.pdf) ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster