LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models
Authors: Marwa Abdulhai
Isadora White
Charlie Victor Snell
Charles Sun
Joey Hong
Yuexiang Zhai
Kelvin Xu
Sergey Levine
Venue: ICML 2025 poster (2025)
Keywords: a benchmark for enabling development of RL algorithms for language tasks
Primary area: reinforcement_learning
TL;DR: a benchmark for enabling development of RL algorithms for language tasks

Abstract: Large language models (LLMs) provide excellent text-generation capabilities, but standard prompting and generation methods generally do not lead to intentional or goal-directed agents and might necessitate considerable prompt tuning. Even the best current LLMs rarely ask clarifying questions, engage in explicit information gathering, or take actions that lead to better decisions after multiple turns. Reinforcement learning has the potential to leverage the powerful modeling capabilities of LLMs, as well as their internal representation of textual interactions, to create capable goal-directed language agents. This can enable intentional and temporally extended interactions, such as with humans, the emergence of complex skills such as persuasion, and long-horizon strategic behavior, such as in the context of games. Enabling this requires the community to develop reliable reinforcement learning algorithms for training LLMs. Developing such algorithms requires tasks that can gauge progress on algorithm design, provide accessible and reproducible evaluations for multi-turn interactions, and cover a range of task properties and challenges in improving reinforcement learning algorithms. Our paper introduces the LMRL-Gym benchmark for evaluating multi-turn RL for LLMs, together with an open-source research framework for getting started on multi-turn RL with offline value-based and online policy-based RL methods. Our benchmark consists of 3 Interactive Dialogue tasks and 5 RL Capability tests for a total of 8 tasks, which require multiple rounds of language interaction and cover a range of tasks in open-ended dialogue and text games.

LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models
Marwa Abdulhai'
Isadora White'
Charlie Snell
Charles Sun
Joey Hong'
Yuexiang Zhai
Kelvin Xu² Sergey Levine'
Abstract
1. Introduction
Large language models (LLMs) have demonstrated remark- able abilities when naturally conversing with humans (Ope- nAI, 2023; 2022; Touvron et al., 2023; Google, 2023), an- swering questions and responding to requests (Shuster et al., 2022b;a; Qin et al., 2023), and even performing coding tasks (Chen et al., 2021b; Wang et al., 2023b). Many of these capabilities are enabled by learning to emulate hu- mans from large datasets of text from the web (Völske et al., 2017; Shuster et al., 2022a; Yao al.. 2023), learning from examples "in context" (Brown et al., 2020), as well as learn- ing from other sources of supervision such as instruction datasets (Mishra et al., 2022; Wei et al., 2022; Wang et al., 2022b) and preference fine-tuning with RLHF (Ziegler et al., 2020; Ouyang et al., 2022). However, directly applying LLMs in settings that require planning or multi-turn inter- actions presents new challenges. LLMs are not explicitly goal-directed, as they are not optimized to directly solve particular tasks, but rather to produce text that resembles the distribution of human-provided examples or accords with human preferences (Ziegler et al., 2020; Stiennon et al., 2020; Wu et al., 2021; Bai al., 2022a). This challenge is apparent in solving temporally extended tasks, such as multi- multi-tuur dialogue (Irvine et al., 2023; FAIR), complex tool use (Wang et al., 2022a), multi-step games (Hendrycks et al., 2021b), and other interactive applications. In prin- ciple, L.MS should contain the knowledge necessary to succeed in such settings: if the multi-turn interactions center around problem domains that are well represented in the model's training data (such as dialogue), well-trained LLMs should already serve as powerful predictive models in such settings. However, leveraging this predictive knowledge to derive effective actions and strategies requires not just emulating humans, but also planning and optimization.
Large language models (LLMS) provide excellent text-generation capabilities, but standard prompt- ing and generation methods generally do not lead to intentional goal-directed agents and might necessitate considerable prompt tuning. Even the best current LLMs rarely ask clarifying questions, engage in explicit information gathering, or take actions that lead to better decisions after multi- ple turns. Reinforcement learning has the poten- tial to leverage the powerful modeling capabili- ties of LLMs, as well as their internal represen- tation of textual interactions, to create capable goal-directed language agents. This can enable intentional and temporally extended interactions, such as with humans, the emergence of complex skills such as persuasion, and gie behavior, such as in the context of games. En- abling this requires the community to develop re- liable reinforcement learning algorithms for train- ing LLMs. Developing such algorithms requires tasks that can gauge progress on algorithm design, provide accessible and reproducible evaluations for multi-turn interactions, and cover a range of task properties and challenges in improving rein- forcement learning algorithms. Our paper intro- duces the LMRL--Gm benchmark for evaluating multi-turn RL for LLMs, together with open- source research framework for getting started on multi-tun turii-ttummn RL with offline value-based and online policy-based RL methods. Our benchmark con- sists of Interactive Dialogue tasks and 5 RL Ca- pability tests for total of 8 tasks, which require multiple rounds of language interaction and cover tasks in open-ended dialogue and text games.
strate-
Multi-turn reinforcement learning (RL) (Sutton & Barto, 2018) in principle offers a path to enable L.MMs to do just that. RL could enable goal-directed reasoning and plan- ning in interactive multi-turn settings, including complex dialogue, games, and tool use. We hypothesize that RL could serve as a powerful tool for LLM training, not only for training models to accord with human preferences, but
Universityy California, Berkeley "Google. Correspondence to: Marwa Abdulhai <marwa @wweeeeeeeeerue.....
Proceedings of the 4rnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

LMRL Gym: Benchmarks for Multi-Turn RL with LMS
lutions compared to other benchmarks in NLP. While most NLP benchmarks are based on standard supervised machine learning paradigms, with a training set and a test set (Mar- cus et al., 1993; Tjong Kim Sang & De Meulder, 2003; Socher et al., 2013; Rajpurkar et al., 2016; Wang et al., 2019; Williams et al., 2018), RL benchmarks require sim- ulators that the trained agents can interact with to measure their performance. In this paper, we use an LLM to simulate conversation partner in dialogue tasks. While the behavior of the LLM may deviate from human behavior, we verify in a human study in Appendix A that our L.M simulators produce natural text reflecting human norms of conversa- tion. However, our goal is not to utilize this approach to benchmark whether LLMs are good at talking to humans, but rather as a way to test RL algorithms with datasets that are sufficiently difficult and complex to gauge how effective they might be if they were then trained on data from real humans. Specifically, our benchmark aims to rigorously stress-test the ability of RL algorithms to enable complex goal goal-directed behaviors in LLMs. To this end, LMRL-Gym also includes a set of text- strategy games, in addition to the dialogue tasks, that are aimed at providing a more controlled and focused diagnostic assessment of specific RL capabilities.
Guess My City
Maze
LLM
What your lavorto pacitonal dish
There are sodin to your left beine you
move right
My livoriae traditionll dian from myy homerrsen rusel growng, ieiiine iste
There arn sodil stors you, below you
move lett
your nometown near coosdiie water??
There ure walls your lot, below you
move
Yee, my city lacraed on the coseet and aumousded by water.
There ars walta to your right your lult
the city you ard from Jakana?
move loit
Yes, the city am from Jakarta
Figure/ 1. Overview of LMRL--ymm: We show sample trajectories from two tasks in our benchmark. In the Guess My City task, the agent learns to ask questions to guess the city the oracle is from, while in the Maze task, the agent learns to make the correct moves based on cues from the oracle.
more generally to accomplish tasks in an intentional and goal-directed manner. Text generation can be viewed as a sequential seciiiionmmaiing process, treating a sequence of tokens as a trajectory. Many tasks, such as successfully an- swering questions or eliciting a desired reaction from a user, can then be framed as optimizing some reward function over these trajectories. However, despite extensive interest in RL for LLMs in recent years, much (though not all) of the recent research in this area has focused on "single-step" RL problems, where a single response is optimized for some quality metric, typically derived from human preference signals (Stiennon et al., 2020; Ziegler et al., 2020; Ouyang et al., 2022; Bai et al., 2022a; Anthropic, 2023; Ramamurthy et al., 2023; Christiano et al., 2023; Casper al., 2023).
Our proposed benchmark, LMRL-Gym, consists of 8 tasks. Three tasks are Interactive Dialogue tasks designed to simu- late real-world interactions with humans, requiring informa- tion gathering (20 Questions, Guess My City) and negotia- tion (Car Dealer). Five tasks are RL Capability Tests, which are text games designed to isolate specific capabilities of RL training. Each task comes with an offline dataset that can be used for offline RL training, and a "simulator" that can be used to evaluate the performance of the agents in multi-turn interactive tasks. We provide a research framework and toolkit for researchers and practitioners to get started with multi-turn RL for LLMs. This framework includes imple- mentations of PPO (Schulman et al., 2017). ILQL (Snell et al., 2022a), and several baseline methods, implemented in an extensible way designed for future development of tasks, experimentation, and algorithm design.
While some works have sought to apply RL for multi- turn tasks (Singh et al., 1999; Li et al., 2016; Shah et al., 2016; Kwan et al., 2022), particularly for goal-directed dia- logue (Lewis et al., 2017; Verma et al., 2022), there has been comparatively little research on improving the underlying RL algorithms and very little head-to-head comparison on same sets of tasks. This is perhaps unsurprising: it is easier to evaluate improvements to algorithms for single-turn text generation as compared to multi-turn generation. Multi-turn dialogue requires an interactive evaluation procedure rather than just a static dataset. There is no established protocol for such evaluations, and the "gold standard" constitutes costly and conemcons studies with human participants.
2. Related Works
Datasets, benchmarks, and libraries. Benchmarks and datasets have been an important factor for driving progress in NLP in domains that include machine translation (Tiede- mann, 2012; Bojar et al., 2016), natural language un- derstanding (Rajpurkar et al., 2016; Wang et al., 2019; Hendrycks et al., 2020; 2021a; Ramamurthy et al., 2023), and solving math problems (Cobbe et al., 2021). How- ever, these tasks generally do not involve multi-tum turn inter- action and do not come with rewards, making them hard to adapt to RL research. For example, the standard for eval-
In this work, we aim to address this challenge and make it possible for RL algorithm researchers to iterate on de- veloping better RL methods for multi-turn language-based interaction tasks, such as dialogue and games. We posit that benchmarking RL algorithms for LLMs presents a very different set of challenges and merits a different set of so-
2

---

LMRL Gym: Benchmarks for Multi-Turn RL with LMS
uating dialogue agents has been run a human subjects study, but this is time-consuming and costly. Some works have proposed text games for evaluating language-based agents (Chevalier-Boiste)
feasible-looking agent dialogue without explicit considera- tion for some multi-turn objective. Our benchmarks allow for the development of algorithms that enable LLMs to inter- act with an environment to achieve long-tem -ere-termm objectives, by providing tasks with online simulators and offline datasets.
et al., 2018: Hausknecht et al., 2019; Yuan et al., 2019; Fan et al., 2020; Hausknecht et al.,
2020; Guo et al., 2020; Ammanabrolu et al., 2020; Yao et al.,
2020; Hendrycks et al., 2021b; Singh et al., 2021; Wang et al.. 2022a; Yao et al., 2022; Jansen & Côté, 2022; Yao et al., 2023; Zhang et al., 2023; Gontier al., 2023) and interactive dialogue (De Bruyn et al., 2022b;a). Our aim is to cover a variety of problem settings that reflect challenges in open-vocabulary interaction in addition to text games, that also specifically evaluate offline RL capabilities, which not done by prior works. Motivated by successes in using LLMs to generate synthetic data (Hausknecht et al., 2019; Park et al., 2023; Bai et al., 2022b), our proposed tasks based on synthetic data. While such data may differ from natural text, the scope of our benchmark is specific to evaluating RL algorithms, not the ability to interact with humans.
3. Multi-Turn Generation with RL and Language Models
This section introduces the conceptual foundations of us- ing reinforcement learning for multi-turn generation with language models. We introduce a definition of the Markov decision process for language and a framework for the meth- ods we focus on in this paper.
We formalize language generation tasks as partially observable Markov decision process. We define the state to be the history of tokens and an action the next token generated by the model. An observation is a single token og in the history. The probability of generating the next token is dependent on all of the previous observation tokens o;. Therefore the Markovian state s is formed by the concatenation of all the previous tokens [oo, ..., oi] A policy defines the agent's behavior by taking in the current state and outputting new action token to get 8,+1. The environment assigns reward r(s, a) based on the entire sequence of tokens so far. The tokens in the state are either generated by the policy π or the environment. For example, in the Car Dealer task, the policy generates the tokens for the Seller's utterance and the environment generates the tokens for the Buyer. The full history of their conversation would form the state. A complete sequence of tokens is trajectory
Definitions.
RL for language models. RL for language models has seen success in aligning LLMs with human preferences (RLHF) (Ziegler et al., 2020; Stiemnon et al., 2020; Bai et al., 2022a;b; Ouyang et al., 2022: Christiano et al., 2023), optimizing non-diffcrentiable objectives for machine trans- lation (Wu al., 2016; Nguyen et al., 2017; Kiegeland & Kreutzer, 2021), generation (Tambwekar et al., 2019; Pang & He, 2021; Pyatkin et al., 2022), dialogue (Cuayáhuitl et al., 2015; Georgila & Traum, 2011; Li et al., 2016), question answering (Pyatkin et al., 2022), and summarization (Paulus et al., 2017; Böhm et al., 2019; Wu & Hu, 2018). These include RL methods that learn by directly interacting with the environment (online RL) (Carta et al., 2023) and RL methods that only use static dataset (offline RL) (Jaques et al., 2020; Snell al., 2022a; Jang et al., 2022; Verma et al., 2022; FAIR). However, many of these works operate in the singe-step bandit setting, and do not consider multi- turn goal-directed tasks. Our benchmark, on the other hand, focuses on tasks involving multiple turns of interaction with clearly defined goal-based reward functions.
referred to RL is to produce policy л² that maximizes the expected discounted sum of rewards over trajectories (7) under the policy ف' = arg max, E7aT EtT ('re(se,ae) where
oo...., or. The goal of
represents the trajectory.
RL Algorithms. Several possible RL algorithms could be used to train language models for multi-turn tasks (Jaques et al., 2020; Verma et al., 2022; Snell et al., 2022a; Schulman et al., 2017: Stiennon et al., 2022; Bai et al., 2022a: Casper et al., 2023). Policy gradient methods, such as PPO (Schul- man et al., 2017), directly compute the gradient of the RL objective with respect to the model parameters. Value-based methods estimate a state-action (Q) and/or state-value (V) function. The state-action or state-value function forms a policy by either 1) acting greedily with respect to the Q- function or 2) perturbing the base model's 's logits with the learned action-value functions (Snell et al., 2022a). RL methods for training LLMs can be online or offline. Online methods repeatedly interact with the environment, collect- ing additional data during training. Offline RL instead learns
Capabilities of LLMs. There has been surge in the capa- bilities of LLMs for generation (Ghazvininejad et al., 2017: Radford et al., 2019), dialogue (Lewis et al., 2017; Jaques et al., 2017; Shuster et al., 2022b; Snell et al., 2022b), ques- tion answering (Pyatkin et al.. 2022), summarization (Paulus et al., 2017: Böhm et al., 2019; Wu & Hu, 2018), text- games (Narasimhan et al., 2015; Hausknecht et al., 2019), translation (Gu et al., 2017), and more. However, these are often supervised learning tasks that do not test the LLMS' abilities to achieve a specific long-term objective. Research on dialogue generation (Jaques et al., 2017; He et al., 2018: Shuster et al., 2022b;a) has often focused on generating
3

---

LMRL Gym: Benchmarks for Multi-Tury RL with LMs
to extract the best behaviors from an existing, potentially suboptimal dataset. Due to the large amount of existing text interactions the internet, offline RL is an ideal setting for
Stratedr Decitoon Making
Complex Langunge
Credit Ansgrmmnt
Paribl Obwrrbbiity
Trjectory sunching
x
x
Maze PO
x
training language models. Therefore, our work primarily focuses on benchmarking offline RL. algorithms. However, our tasks also fully support online RL and we include an online PPO baseline our evaluation.
Nev
x
x
/
V
Tect-Kev PO
Wordle
x
x
v
x
Indgomes
4. The LMRL-Gym: Synthetic Benchmarks for RL with Language
Figure 2. We have designed our RL Capability tasks as text games that include Chess, Endgames, Wordle, Maze, and Text-Nav. These tasks isolate some subset of the RL Capabilities outlined in Ap- pendix B.I.
Our benchmark consists of tasks grouped into two cate- gories: RL Capability tasks and Interactive Dialogue tasks. The RL Capability tasks focus on desirable capabílities for RL algorithms for LLMs such as strategic pecision-making credit assignment, trajectory stitching, partial observability, and use of complex language. For the interactive dialogue tasks, we model them after real-world interactions with hu- mans, such as persuading someone to buy a car or playing a guessing game. Below, we define the Interactive Dialogue tasks, describe the specific capabilities of RL algorithms for LLMs that our benchmark aims to evaluate through RL Capability tasks, and summarize the data generation and simulation process. We have provided example trials for each task are shown in Figure 4, and a concise summary of the dataset and task statistics in Table 1. The number of trajectories and the average length of the trajectories varies based on the complexity of the tasks.
Car Dealer. The Car Dealer task tests the ability of RL algorithms to learn successful car sale strategies. This in- volves decision Daciiion-making and credit assignment as different persuasion strategies must be adopted for different kinds of buyers.
4.2. RL Capability Tasks
A central objective of our benchmark is to evaluate the core capabilities that RL enables in large language models. The RL Capability tasks are text-based games designed to isolate specific RL capabilities and are language analogs of tasks where RL known to succeed. These tasks include Chess, Endgames, Wordle, Maze, and Text-Nav. Below we explain the tasks and the motivation for including them as tests for RI. capabilities. Further details on task design for RL Capability tasks can be found in Appendix B.
4.1. Interactive Dialogue Tasks
The Interactive Dialogue Tasks aim to simulate real-world goal-oriented dialogues. We focus on tasks where the agent must make inferences about persuasive strategies and ac- tively gather information by asking questions. Instead of generating these interactions with humans, we generate such interactions through simulating LLMs inspired by successes in using LLMs to generate synthetic data. While the LLM might not be as realistic as a real human, we have found that human raters evaluated the LLM-generated text as quite realistic in most cases, as discussed in our user study in Ap- pendix A. You can find examples from the trained models in Appendix I.
Desirable RL capabilities. RL shines in goal-directed tasks that require multi-step planning and strategic decision- making. Strategic Docision-making can range from asking follow-up questions (e.g. 20 Questions), to complex strategy in chess. In RL, it is necessary that algorithms can prop- erly perform credit assignment as rewards are often delayed relative to the action pivotal to the outcome. A challenge with optimizing POMDPs partial observability, where the agent must make deductions based incomplete infor- mation. In the offline RL setting, the ability of algorithms to perform trajectory stitching is often desirable for learn- ing optimal policies from suboptimal trajectories. Lastly, when working with language models, it's important that algorithms remain effective in the face of complex language with open-ended generation. We design our RL-capability tests with the goal of stress-testing each of these capabilities, as shown in Figure 2.
20Qs (Twenty Questions). This task tests whether an agent can gather information about an unknown subject through twenty yes or no questions. The agent must use semantic knowledge of the object infer the correct answer.
Guess (Guess My City). The Guess My City task per- forms more complex forms of information gathering, involv- ing open-ended questions about a city. This task evaluates semantic knowledge of a specific city and the agent's ' ability to parse information from a free-form answer.
Maze and Text-Nav. the Text-Nav featuring more complex language. Though Text-Nav involves stochastic language, the maze task has
We consider a Maze task as well as
4

---

LMRL Gym: Benchmarks for Multi-Turn RL with LMS
4.3. An Overview of Data Collection for LMRL-Gym
Use gpt3.5 5 to generate data
To make tasks in LMRL-Gym practical for benchmarking RL methods, we must balance accessibility and realism. As RL algorithms need to be evaluated by running learned policy, real-world tasks are comparatively inaccessible for rapid iteration (e.g., if they require talking to real humans). We therefore use simulators for our tasks, derived either from text-based games, or conversational agents powered by language models. Although this fully synthetic setup sacrifices the realistic nature of tasks, we believe significant gain in accessibility is worthwhile and will enable rapid RL algorithm progress.
Koe open
gpt3.5
ogpedl bi cone endremment Yea.
animal?
Env Model Fine-tuned GPT2
Agent Model Fine-tuned GPT2
RL Capability tests. For each task. we use simulator such as a chess engine or maze solver to generate near- optimal data and then we dilute the policy with suboptimal data by taking suboptimal actions or using inferior poli- cies. We also convert our task from a symbolic version to text-based version in programmatic way discussed in Appendix B.
Distill to environment and agent model to generate a new dataset
Data
RL
Fine-tune with
Figure 3. To generate data for conversational tasks, we use LLMs as "simulators" for the task. Our simulators can he used to generate offline data, to provide a "simulation environment" for evaluation, to perform online training, and to compute rewards.
Interactive Dialogue tasks. For conversational tasks, we leverage existing LLMs to generate our data, either with two instances of LLMs "talking" to one another or all at once through few-shot prompting as shown in Figure 3. To train these LLMss, we use OpenAl's GPT-3.5 to generate an initial dataset by asking reasonable questions and answers out-of- the-box, collecting a dataset of differing sizes depending on the task. In the case of 20Qs and Guess My City, we col- lected 1K conversations by querying GPT-3.55 (text-davinci- 003) to generate both sides of the conversation based on specific prompts (which can be found in Appendix D.6. To generate the dataset for training our algorithms, we fine- tuned a FLAN-T5-X- guesser model and FLAN-T5-XL oracle model their respective sides of the conversation. Using these distilled models, we generated new dataset of 100K conversations by having the two models talk to each other. We conducted a similar process for the Car Dealer task but with a larger model for fine-tuning (GPT2-XL). When generating our datasets, we also spent considerable effort to ensure diversity the responses to ensure the col- lection of high-quality data. For the Car Dealer task as an example, this included providing different desired brands, features, classifications (i.e. car truck), and budgets in our prompting to generate the datasets. Further details on our data generation process for the three Interactive Dialogue tasks can be found in Appendix D.
longer dataset trajectories and more complicated layout. To test partial observability, we include both a partially observed and fully observed version of each task. In the partially observed version, we remove information from the maze description such that the agent must infer its position from its move history. To emphasize the comparison to non-text- ten-tett-based version, we evaluate the Maze task in symbolic or grid-based environment seen in Appendix H.
Strategy games. We include three strategy games; Wor- dle, Chess, and Endgames. Wordle tests partial observ- ability over the space of possible words while Chess and Endgames test the ability of the agent to form longer-term plans. Endgames provide simpler and more goal-directed variation of the Chess task. By focusing on the endgame, we encourage algorithms to learn strategy rather than memo- rizing the opening moves of a chess game. A classic theoret- ical endgame position consists of a position where the only pieces on the board are the two kings and the queen. All RL Capability tasks evaluate trajectory stitching capability through the inclusion of suboptimal trajectories. Further details about our dataset generation strategies can be found in Appendix D. The Chess, Endgames. Maze and Text-Nav tasks test credit assignment, because the RL algorithm must learn to assign credit to good actions rather than a lucky start- ing position in the maze task, or a weak opponent moves in the Chess or Endgames task.
5. LMRL-Gym Research Framework for Algorithm Development
We evaluate the LMRL-Gym tasks on both online and offline RL. algorithms, including variations of behavior
5

---

LMRL Gym: Benchmarks for Multi-Turn RL with LMs
Task
200s
Guess
Car
Maze
Text-Nav
Wordle
Chess
Endgames
Size
100k
100k
19k
1.24k
2.5k
lm
625k
97.756k
avg length
14.9
18.8
16.5
19.7
12.2
4.82
46.7
11.9
std length
4.38
4.57
3.61
24.5
8.77
1.27
18.16
12.0
success rate
0.31
0.53
0.53
0.11
0.26
0.70
0.60
0.59
avg return
17.3
18.8
0.562
19.77
0.258
-4.12
0.210
0.586
std return 2.56
4.12
0.422
24.5
0.424
1.59
0.970
0,492
Table Statistics for all tasks in LMRL-Gym. Size represents the number of trajectories, the average length is the average length of trajectories in the dataset where the unit is a response from the agent. The success rate is the proportion of trajectories that reach the objective. Finally, the reward functions for cach task are defined in Appendix D.
cloning, value-based RL methods, and online PPO. We have selected these algorithms have they are currently the state-of-the-art methods RL methods for LLMs (Chen et al., 2021a; Snell et al., 2022a; Ouyang et al., 2022). With these experiments, (1) a significant spread in performance between the different algorithms, highlighting differences between RL algorithms; (2) room to improve beyond, such that our benehmark can enable future algorithmic development. Our project page (https: htt://mmmllgggm get..gitthin withub.tiii... io/) contains links to our open-sourced datasets (https:
RL methods, PPO learns a language model policy with no policy extraction step.
GPT4. Few-shot prompting is a common technique for creating interactive language agents (Wang et al., 2023a). To compare this RL fine-tuning we few-shot prompt GPT4 using dataset examples and detailed explanation of the game for each task. The prompts can be found in our code repository.
expect to observe
Training and evaluation protocol for algorithms. For the BC and filtered BC methods, we initialize our models with the pre-trained GPT2 weights (Radford et al., 2019) and perform standard fine-tuning, We choose GPT2 rather than a larger model due to memory and time constraints, though we admit larger models would lead a performance boost. For each of the RL methods, we initialize the weights of the base model with the weights from the BC checkpoint and then continue finetuning with the RI. objective. When fine-tuning PPO, we limit the number of samples to less than 100k. We report the hyperparameters that we used for each task in Appendix E. We evaluate each policy by measuring the average reward in the simulated environment for each task.
x1-llm-bench-dattase
and research framework com/ htt.://o//aaabbaae www.aaaduiimmm///m/a (MAAaa/MMRRLGGL) Gym).
BC, Filtered BC, Online Filtered BC. In line with stan- dard RL nomenclature, we denote supervised fine-tuning as behavioral cloning (BC). This baseline tests whether LMs can effectively represent the behaviors in the datasets. Filtered BC is identical, except only the most successful examples in the offline dataset are used for fine-tuning, technique which is also used in (Snell al., 2022a). Online filtered BC collects data online using the current policy and selects the most successful trajectories for finetuning, See Appendix E for our data filtering criteria for each task.
Evaluation of data generation. When using LLMs as a simulator for human actions, it is important to verify that (1) the text produced by the L.LM is natural and (2) LLM simulator is not exploitable e.g. policy achieves high reward without actually accomplishing the goal. In addition to vali- dating the data generation process through statistics reported in Table 1, we verified the naturalness of the L.L--prduceed text in a user study of 40 users. In this study, found no significant difference in the naturalness of conversations generated by ChatGPT3.5 and our trained simulators and agents Appendix A. For example, natural conversations im- ply that the strategies employed by the Seller to convince the Buyer followed human patterns of conversation and indicate the robustness of the Buyer model to hacking. 20 Questions and Guess My City are particularly hard to hack as they require the agent successfully guess the word. We verify
Offline Value-based RL: MC Returns and ILQL. Monte-Carlo returns (Kakutani, 1945) and Implicit Lan- guage Q-Learning (Snell et al., 2022a) train a value V and Q function. In MC Returns, we train the Q function with an MSE to predict the reward-to-gg go. In ILQL we train the two action-value (Q) functions using the Bellman backup operator (Kostrikov et al., 2021). For both algorithms, the Q and V functions are then used to perturb the logits of the original BC model (see Equation 5).
Online RL: PPO. PPO (Schulman et al., 2017) is an on- line RL algorithm widely adopted for training language models with Reinforcement Learning from Human Feed- back (Christiano et al., 2023; Stiennon et al., 2022; Bai et al., 2022a; Casper et al., 2023). Unlike previous
6

---

LMRL Gym: Benchmarks for Multi-Turn RL with LMS
this through automatic checks as described in our prompting strategy in Appendix D.6.
methods perform significantly worse This highlights room for improvement in dealing with partial observability in en- vironments with complex language. Further details for this ablation are found in Appendix H.
6. Benchmarking Baseline RL Methods
In Table we present the results for each method on each of our text-game and interactive dialogue tasks. We normalize the scores such that score of 50 corresponds to the average reward in our offline dataset, O corresponds to the lowest possible score, and 100 to the highest score. Across all tasks, we see that our offiin RL baseline methods consistently outperform both the dataset and the filtered BC policies, demonstrating the efficacy of offline RL in representing a more optimal policy than the best behaviors in the data. Similarly, we see that online PPO generally improves over the BC policies, highlighting the utility of learning from online environment interaction. However, between RL Ca- pability tasks and Interactive Dialogue tasks, we observe desperate trends in which specific method performs the best. We discuss this in more detail below.
How does prompting GPT-4 compare with RL fine- tuning? On the RL Capability tasks, we find that our much smaller RL finetuned models significantly outperform GPT4 demonstrating the efficacy of RL for enabling com- plex goal-directed behaviors in language models. However, the Interactive Dialogue tasks, GPT-4 outperforms or performs on par with our best RI-trained models. These dialogue tasks are likely to be much more in distribution for GPT4 than our text-game RL capability tasks, and thus GPT4's broad world-knowledge, reasoning, and conversa- tional abilities become synchronized allowing it to compen- sate for its lack of goal-directed RL fine-tuning in these scenarios. Nonetheless, the mere fact that finetuning small models with RL enables us close much of the gap to GPT4 these more realistic tasks underscores the efficacy of RL finetuning. In summary, we can see that RL algorithms consistently outperform baselines like filtered BC on many of the tasks. However, these results highlight significant areas for growth. For example, the instabilities observed in training PPO require further investigation beyond hyper- parameter tuning. Moreover, the performance discrepancy between ILQL and the simpler MC Returns highlights that scaling full TD-learnin to Interactive Dialogue settings is another area for improvement.
Which algorithm performs best on the RL Capability tasks? On the RI Capability tasks in Table 2, we see ILQL has the highest performance across all methods for most tasks. ILQL's performance on these tasks is likely due to its unique ability to perform trajectory stitching, enabling it to outperform any individual trajectory in the dataset by learning to compose the best parts of many different trajec- tories. However, on the PO text-nav, chess, and endgames tasks, we see that PPO outperforms ILQL, suggesting that there is likely still much room for improvement in terms of developing better offline TD-based RL methods for LLMs.
7. Discussion
We propose LMRL-Gym, consisting of 8 tasks including three Interactive Dialogue tasks, and five RL Capability tests. We provide research toolkit for practitioners to get started with multi-turn RL for LLMs. Our objective is enable the iteration and development of more effective methods for language-based, multi-turn interaction tasks. This includes enabling core capabilities in LLMs through RL to perform complex decision makingg. complex con- versational interactions, credit assignment, and trajectory stitching. Our evaluation shows promise of RL in several tasks, with further room for improvement with a push for better methods. We acknowledge several limitations when designing tasks in our benchmark, including primarily lever- aging smaller GPT-based LLMs to generate datasets and finetune our LLM-based simulators. While we have pri- marily trained and evaluated models with a maximum 1.5B parameters, we have maintained a lower parameter count to ensure accessibility for researchers with limited compu- tational resources. In addition to releasing our code and datasets, we share all of the hyperparameters we used to train our models in Appendix E and provide more in in-depth insight into our results, training procedure, and evaluation
Which offfine RL algorithm performs best for Interac- tive Dialouge tasks? In contrast to the text-based games, on our Interactive Dialogue tasks, we see that across all tasks ILQL under-performs the simpler MC returns method. This discrepancy with dialogue, may be because on the more complex text-based tasks it is harder to scale full TD-Jeaming. In fact, we find that on the car-dealer task, even filtered BC outperforms ILQ Overall, these find- ings demonstrate that there is much progress to be made in developing better offline RL methods that can effectively optimize LLMs in complex and realistic dialogue settings.
How does performance of language- text games compare with their symbolic-based counterparts? We created a non-text-basee version of the Maze task (an RL Capability task) to investigate what difficulties arise from de- ploying RL algorithms on aanguage-based tasks. We found that simple online and offline Q-learning was able to get an optimal score on the maze. Therefore, the performance symbolic maze is comparable to the fully observed Maze task. However, on the PO Maze task, the language-
7

---

LMRL Gym: Benchmarks for Multi-Turn RL with LMs
alg.
BC
% BC
MC Return
I.Q
Online PPO
Online BC
GPT-4
200s
57.1
77.1
87.1
82.9
72.9
55.2
95.7
Interactive Dialogue
Guess
30.0
48.0
88.0
75.0
49.9
31.6
92.3
Car
44.5
54.8
57.2
46.3
50,5
40.4
53.5
FO Maze
58.2
68.9
75.0
99.9
79.7
$7.4
78.2
PO Mazc
53.1
50.1
52,4
76.3
42,4
53.1
60.4
FO
Text-Nav
53.7
65.1
71.9
91.8
87.1
74.5
67.5
RL Capability tasks
PO Wordle
Text-Nav
60.5
71.6
83.7
85.5
68.4
40.2
79.9
79.1
94.9
97.7
84.2
95.2
15.4
Chess 47.2
42.9
46.5
47.3
48.0
47.2
0
Endgames
35.1
17.7
50.2
45.8
77.5
36.2
0
Table Normalized reward for all tasks. We present the interactive dialogue tasks top and the RL capability tasks the bottom. Value-based methods (MC and ILQL) generally outperform filtered BC, as we might expect in stochastic settings, though the relative performance of ILQL and the simpler MC method is, perhaps surprisingly, reversed on the tasks with more complex language, suggesting that there is room for improvement with such methods. Online RL with PPO often, but not always, improves over offline methods that are not permitted to collect additional online interaction. make the results more comparable across tasks, we normalize average return for each policy such that 0 is the minimum possible return, 50 is the dataset average return, and 100 is the maximum return for each task. also report raw score results evaluation details Appendix
in Appendix F.
outcomes may prioritize system goals over user safety and alignment to human values. We have designed our datasets and reward functions such that prioritize fairness and human- aligned outcomes. By incorporating these considerations when designing our framework, aim to encourage the development of reinforcement learning models and LLMss that not only excel in performance but also adhere to ethical standards.
We would like to acknowledge that this work is part of larger effort improve the performance of LLMs in settings that require planning multi-turn interactions including multi-turn dialogue, complex tool use, multi-step games, and other interactive applications. Our goal is to propose tasks to evaluate different capabilities expected from an LLM, such as common sense reasoning, credit assignment, reasoning under uncertainty. information-seekinn behaviors, and trajectory stitching. We hope this benchmark inspires the creation of more synthetic datasets and simulators for dialogue and is used design better algorithms to train goal-directed LLM-RL models.
A Acknovledgments
This research partially supported by Berkeley Deep- Drive, the Cooperative AI Foundation, and the National Science Foundation.
Impact Statement
References
This work aims to develop a benchmark for the advance- ment of research in reinforcement learning and LLMs. We generate datasets for tasks in our benchmark with existing LLMs for dialogue tasks and online engines for text games, adhering to best practices in data handling and ensuring there is no personally identifiable or sensitive information present in the generated datasets. We recognize that there may biases present in the datasets we collect, and have taken steps to ensure diverse and varied collection of re- sponses from LLMs for our conversational task detailed in our data generation process in Appendix D. In consider- ing the ethical implications of interactive RI, we acknowl- edge the dual use implication of this research, particularly centered around developing LLM simulators that could per- form persuasion, manipulation, and addictive engagement of users at a large scale. The optimization processes em- ployed by such algorithms, which aim maximize certain objectives, raise ethical considerations when the optimized
Ammanabrolu, P., Tien, E., Hausknecht, M., and Riedl, M. O. How to avoid being eaten by grue: Structured exploration strategies for textual worlds. arXiv preprint arXiv:2006.07009 2020.
Anthropic.
Introducing claude, 2023. https: http:://wwww.. www.:rpihhhpp..oom com.indexx/// introducing-clausee
URL
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain. D., Fort, S., Ganguli, D., Henighan,
T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dode Z., Heman- dez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C.,, Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., Mann, B., and Kaplan, J. Train- ing a helpful and harmless assistant with reinforcement learning from human feedback, 2022a.
8

---

LMRL Gym: Benchmarks for Multi-Turn RL with LMs
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse, K. Lukosuite, K., Lovitt L.. Sellitto, M., Elhage, N., Schiefer, N., Mercado, N., DasSarma, N., Lasenby, R., Larson, R., Ringer, S., Johnston, S., Kravec, S., Showk, S. E., Fort, S., Lanham, T., Telleen-Lawton T., Conerly, T., Henighan, T., Hume, T., Bowman, S. R., Hatfield- Dodds, 7., Mann, B., Amodei, D., Joseph, N., McCan- dlish, S., Brown, T., and Kaplan, J. Constitutional ai: Harmlessness from ai feedback, 2022b.
P., BIyIk, E., Dragan, A., Krueger, D., Sadigh, D., and Hadffeld-Mnell, D. Open problems and fundamental lim- itations of reinforcement learning from human feedback, 2023.
Castricato, L.,, Havrilla, A., Matiana, S., Phung, D. V., Ti- wari, A., Tow, J., and Zhuravinsky, M. trlX: scal- able framework for RLHF, June 2023. URL https: / //github. com/CarperAI/trls
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling, 2021a.
Böhm, F., Gao, Y., Meyer, C. M., Shapira, O., Dagan, I., and Gurevych, I. Better rewards yield better sum- maries: Learning to summarise without references. In Proceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pp. 3110-3120, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18663//1/DD1111307 URL https: //aclanttooleeeee w1wp:099910000000100010000000000
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavar- ian, M., Winter, C.,, Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert- A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V. Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code, 2021b.
Bojar, O. r., Chatterjee, R., Federmann, C., Graham, Y., Haddow, B., Huck, M., Jimeno Yepes, A., Koehn, P., Logacheva, V., Monz, C., Negri, M., Neveol, A., Neves, M., Popel, M., Post, M., Rubino, R., Scarton, C., Specia, L., Turchi, M., Verspoor, K., and Zampieri, M. Findings of the 2016 conference on machine translation. Pro- ceedings of the First Conference on Machine Translation, pp. 131-198, Berlin, Germany, August 2016. Associa- tion for Computational Linguistics. URL http: / / /www. aclweb. org/ anthology / /W///166 6/116//6666 M
Shevalier-Boisvert M., Bahdanau, D., Lahlou, S., Willems, L., Saharia, C.,, Nguyen, T. H.. and Bengio, Y. Babyai: platform study the sample efficiency of grounded lan- guage learning. arXiv preprint
2018.
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences, 2023.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Vosse A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
Cobbe, K.,, Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint 1xxx:211001116.. 2021.
J., Winter, C., Hesse, C., Chen, M., Sigler, F., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are sho-shot learners, 2020.
Côté, M., Kádár, Á., Yuan, X., Kybartas, B., Barnes, T., Fine, E., Moore, J., Hausknecht, M. J., Asri, L. E., Adada, M., Tay, W., and Trischler, A. Textworld: A learning en- vironment for text-based games. CoRR, abs/1806.11 1532, 2018.
Carta, T., Romac, C., Wolf, T., Lamprier, S., Sigaud, O., and Oudeyer, P.-Y. Grounding large language models in interactive environments with online reinforcement learning, 2023.
Casper, S. Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., Wang, T., Marks, S., Segerie, C.-R., Carroll, M., Peng, A., Christoffersen, P., Damani, M., Slocum, S., Anwar, U., Siththaranjan, A., Nadeau, M., Michaud, E. J., Pfau, J., Krasheninnikov, D., Chen, X., Langosco, I.., Hase,
Cuayáhuitl, H., Keizer, S., and Lemon, O. Strategic dialogue management via deep reinforcement learning, 2015.
De Bruyn, M., Lotfi, E.,, Buhmann, J., and Daelemans, W. 20q: Overlap-free world knowledge benchmark for lan- guage models. In Proceedings of the 2nd Workshop on

---

LMRL Gym: Benchmarks for Multi-Turn RL with LMs
Natural Language Generation, Evaluation, and Metrics (GEM), pp. 494-508, 2022a.
Hausknecht, M., Ammanabrolu, P., Marc-Alexandre, C., and Xingdi, Y. Interactive fiction games: A colossal adventure. CoRR, abs/1909.05398. 2019. URL http:
De Bruyn, M., Lotfi, E.,, Buhmann, J., and Daclemans, W. Is it smaller than tennis ball? language models play the game of twenty questions. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp. 80-90, 2022b.
Hausknecht, M., Ammanabrolu, P., Côté, M.-A., and Yuan, X. Interactive fiction games: A colossal adventure. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 34, pp. 7903-7910, 2020.
(FAIR)F, M. F. A. R. D. T., Bakhtin, A., Brown, N., Di- nan, E., Farina, G., Flaherty, C.. Fried, D., Goff, A., Gray, J., Hu, H., Jacob, A. P., Komeili, M., Konath, K., Kwon, M., Lerer, A., Lewis, M., Miller, A. H., Mitts, S., Renduchintala, A., Roller, S., Rowe, D., Shi, W., Spisak, J., Wei, A., Wu, D., Zhang, H., and Zijlstra, M. Human-level play in the game of çiddiplomacyj/i by combining language models with strategic reasoning. Science, 378(6624):1067-1777 1067-1074, 2022. doi: 10.1 science.ade9097. URL https://///c http://wwwww...eee.. science. org/doi/abe abs 10. 1e2./11iieennaeo ade9097.
He, H., Chen, D., Balakrishnan, A., and Liang, P. Decou- pling strategy and generation in negotiation dialogues, 2018.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint Att::200900330000 2020.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E.,, Song, D., and Steinhardt, Measuring math- ematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021a.
Fan, A., Urbanek, J., Ringshia, P., Dinan, E., Qian, E., Karamcheti, S., Prabhumoye, S., Kiela, D., Rocktaschel, T., Szlam, A., et al. Generating interactive worlds with text. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 1693-1700, 2020.
Hendrycks, D., Mazeika, M., Zou, A., Patel, S., Zhu, C., Navarro, J., Song, D., Li, B., and Steinhardt, J. What would jiminy cricket do? towards agents that behave morally. NeurlPS, 2021b.
Georgila, and Traum, Reinforcement learning of argumentation dialogue policies in negotiation. pp. 2073- 2076, 08 2011. doi: 10.214 10021443//iesppechh2011111
Ghazvininejad, M., Shi, X., Priyadarshi, J., and Knight, K. Hafez: an interactive poctry generation system. In Proceedings of ACL 2017, System Demonstrations, pp. 43-48, Vancouver, Canada, July 2017. Associa- tion for Computational Linguistics. URI. https:// aclanthology, wtt/po7-/00000
Irvine, R., Boubert, D., Raina, V., Liusie, A., Mudu- palli, V., Korshuk, A., Liu, Z., Cremer, F., Assassi, V., Beauchamp, C.-C., et al. Rewarding chatbots for real- world engagement with millions of users. arXiv preprint NMXi::3003.61333 2023.
Gontier, N., Rodriguez, P., Laradji, I., Vazquez, D., and Pal, C. Language decision transformers with exponen- tial tilt for interactive text environments. arXiv preprint arXiv:2302.05507, 2023.
Jang, Y., Lee, J., and Kim, K.-E. GPT-critic: Offline re- inforcement learning for end-to-end task-oriented dia- logue systems. International Conference on Learning Representations, 2022. URL ht /penreview. net / www.foooddoggaaeeeeaeeee
Google. Bard, 2023. URL https: http:/bardd.oolo. com/
Jansen, P. A. and Côté, M.-A. Textworldexpress: Simulating text games at one million steps per second. arXiv preprint arXhv:2208.01 2022.
Gu, J., Cho, K., and Li, V. O. Trainable greedy decod- ing for neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natu- ral Language Processing, pp. 1968-1978 Copenhagen, Denmark, September 2017. Association for Computa- tional Linguistics. doi: 10. 10.17163//1/D17-12210, URL https: wtw.attthooooooooooo//011111
Jaques, N., Gu, S., Bahdanau, D., Hermández-Lobato, M., Turner, R. E., and Eck, D. Sequence tutor: Conserva- tive fine-tuning of sequence generation models with KL- control. In Precup, D. and Teh, Y. W. (eds.), Proceed- ings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learn- ing Research, pp. 1645- 1654. PMLR, 06-11 Aug 2017. URL https: http:p:://proeeeeeeeeeeeee http://pocceeicenec mll.. waa...presvvvv jaques html.
Guo, X., Yu, M., Gao, Y., Gan, C., Campbell, M., and Chang, S. Interactive fiction game playing as multi- paragraph reading comprehension with reinforcement learning. arXiv preprint arXiv:2010.02388. 2020.
10

---

LMRL Gym: Benchmarks for Multi-Turn RL with LMs
Jaques, N., Shen, J. H., Ghandeharioun, A., Ferguson, C., Lapedriza, A., Jones, N., Gu, S., and Picard, R. Human- centric dialog training via offline reinforcement learn- ing. Empirical Methods in Natural Language Processing (EMNLIP, 2020.
of ral Language Processing, pp. 1464-1474, Copenhagen, Denmark, September 2017. Association for Computa-
the the 2017 Conference on Empirical Methods in Natu-
tional Linguistics. doi: 10. 10.18663/w1/DD1--1553 URL https: http://cetttttoooooooo.o.oo.
Kakutani, S. Markoff process and the dirichlet problem. Proceedings of the Japan Academy, 21(3-10):2772333 1945.
OpenAl. Chatgpt, 2022. URL comm blog/ chotghh..p...
OpenAL Gpt-4, 2023. URL UtLnttpp://opeaaa.com prsearch/ppt-4.
Kiegeland, S. and Kreutzer, J. Revisiting the weaknesses of reinforcement learning for neural machine transla- tion. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, pp. 1673-1681, Online, June 2021. Association for Compu- tational Linguistics. doi: 10.
Ouyang, L.. Wu, J.. Jiang, X., Almeida, D., Wainwright, C. L.,, Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback, 2022.
133. URL https://////.. / ac l wwc.llahhhloogyyy org /2011. naacl metai.n33.
Pang, R. Y. and He, H. Text generation by learning from demonstrations. In International Conference on Learning Representations, 2021. URL https: openneview. net / htt:///rummmi fideuummi--ooxx uQlHua.
Kostrikov, I., Nair, A., and Levine, S. Offline reinforce- ment learning with implicit q-learning. arXiv preprint arXiv:21 10.06169, 2021.
Kwan, W.-C., Wang, H., Wang. H., and Wong, K.-F. A survey on recent advances and challenges in reinforce- ment learning methods for task-oriented dialogue policy learning, 2022.
Park, S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior, 2023.
Paulus, R., Xiong, C., and Socher, R. A deep reinforced model for abstractive summarization, 2017.
Lewis, M., Yarats, D., Dauphin, Y. N., Parikh, D., and Batra, D. Deal no deal? end-to-end learning for negotiation dialogues, 2017.
Pyatkin, V., Hwang, J. D., Srikumar, V., Lu, X., Jiang, L., Choi, Y., and Bhagavatula, C. Reinforced clarification question generation with defcasibility rewards for disam- biguating social and moral situations, 2022.
Li, J., Monroe, W., Ritter, A., Galley, M., Gao, J., and Jurafsky, D. Deep reinforcement learning for dialogue generation, 2016.
Qin, Y., Hu, S., Lin, Y., Chen, W., Ding, N., Cui, G., Zeng, Z., Huang, Y., Xiao, C., Han, C., et al. Tool learning with foundation models. arXiv preprint trXii:2304.08354 2023.
Marcus, M. P., Marcinkiewicz. M. A., and Santorini, B. Building large annotated corpus of english: The penn treebank. Computational Linguistics, 19(2):313-30) 1993.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Mishra, S., Khashabi, D., Baral, C., and Hajishirzi, H. Cross- task generalization via natural language crowdsourcing instructions. In ACL, 2022.
Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05252/ 2016.
Narasimhan, K., Kulkarni, T., and Barzilay, R. Language understanding for text-based games using deep reinforce- ment learning. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp-
Ramamurthy, R., Ammanabrolu. P., Brantley, K., Hessel, J., Sifa, R., Bauckhage, C., Hajishirzi. H., and Choi, Y. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In The Eleventh International Conference on Learning Representations, 2023. URL https:/ /ppeneeview net forum? i id=BaHzds2ubyB.
1-11, Lisbon, Portugal, September 2015. Association for Computational LiniruttiiianLnLnuiiiststsi doi: (0.18653/v1/D15.. 1001. URL https::ttccllataaee https/:///aaaaaahaaaaecae www.gey..rrg///. or5 -10.1
Nguyen, K., Daumé III, H., and Boyd-Grabes J. Rein- forcement learning for bandit neural machine transla- tion with simulated human feedback. In Proceedings

---

LMRL Gym: Benchmarks for Multi-Turn RL with LMs
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint 1XXXv:7777006477 2017.
Stiennon, N., Ouyang. L., Wu, J., Ziegler, D. M., Lowe,
R., Voss, C., Radford, A., Amodei, D., and Christiano, P. Learning to summarize from human feedback, 2022.
Schulman, J., Moritz, P.. Levine, S., Jordan, M., and Abbeel, P. High-dimensional continuous control using generalized advantage estimation, 2018.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018.
Shah, P., Hakkani- Tur, D., and Heck, L. Interactive rein- forcement learning for task-oriented dialogue manage- ment. 2016.
Tambwekar, P., Dhuliawala, M., Martin, L. J., Mehta, A.,
Harrison, B., and Riedl, M. O. Controllable neural story plot generation via reward shaping. In Proceed- ings of the Twenty-Eighth International Joint Conference Artificial Intelligence. International Joint Conferences on Artificial Intelligence Organization, aug 2019, doi: 00.24963/jjaai220119833 URL https: / http://o.ioo org/ 10.24 N0.249933222 jeai 201982F829.
Shuster, K., Komeili, M., Adolphs, L., Roller, S., Szlam, A., and Weston, J. Language models that seek for knowledge: Modular search & generation for dialogue and prompt completion, 2022a.
Shuster, K., Xu, J., Komeili, M., Ju, D., Smith, E. M., Roller, S., Ung. M., Chen, M., Arora, K., Lane, J., Behrooz, M., Ngan, W., Poff, S., Goyal, N., Szlam, A., Boureau, Y.-L., Kambadur, M., and assisted question cesstion-answering with human feedbackJason Weston, B. Blenderbot 3: deployed conversational agent that continually learns to responsibly engage, 2022b.
Tiedemann, J. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pp. 2214-2218, Istanbul, Turkey, May 2012. European Language Resources Association (ELRA). URL http: htEp://www. www.w.lreee www.lrreccc..... conf. ozg/procceddigg/ 1 1rec2C12/pdf Baedff/433_aaaaae 463___pere.p_......-
Singh, I., Singh, G., and Modi, A. Pre-trained language models as prior knowledge for playing text-based games. arXiv preprint arXiv:2107.084
Tjong Kim Sang. E. F. and De Meulder, F. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proceedings of the seventh conference Natural language learning HLT-NAACL 2003, volume 4, pp. 142-147, 2003.
2021.
Singh, S., Kearns, M., Litman, D., and Walker, M. Rein- forcement learning for spoken dialogue systems. Ad- vances in neural information processing systems, 12, 1999.
Touvron, H., Martin, I.., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C.,, Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshom, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang. S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama Open foundation and fine-tuned chat models, 2023.
Snell. C., Kostrikov, I. Su, Y., Yang, M., and Levine, S. Offline rl for natural language generation with implicit language q learning, arXiv preprint arXiv:2206.11877 2022a.
Snell, C., Yang, S., Fu, J., Su, Y., and Levine, S. Context- aware language modeling for goal-oriented dialogue systems. In Findings of the Association for Computa- tional Linguisties: NAACL 2022, pp- 2351-2366, Seat- tle. United States, July 2022b. Association for Computa- tional Linguistics. doi: 10. 10.116666/1112022.nniinn---- 181. URL https:tt//// /ttaa/aannnoooooooooooo www.::/looooy..//22202//// findings-naacl. 181.
Socher, R., Perelygin, A., Wu, J. Y., Chuang, J., Manning, C. Ng, A., and Potts, C. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language processing, pp. 1631-, 2013.
Verma, S., Fu, J., Yang, M., and Levine, S. Chai: A chathot ai for task-oriented dialogue with offline reinforcement learning, 2022.
Stiennon, N., Ouyang, L.,, Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Ad- vances in Neural Information Processing Systems, 33: 3008-3021, 2020.
Völske, M., Potthast, M., Syed, S.,, and Stein, B. Tl: dr: Min- ing reddit to learn automatic summarization. In Proceed- ings of the Workshop on New Frontiers in Summarization, pp. 59-65, 2017.
12

---

LMRL Gym: Benchmarks for Multi-Turn RL with LMS
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019. URL https::/p//./o.... wtt.://eeeeeeeeeeeeeeeeeeeeeeeee forum? id-rJAkm2R5t7.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
Yao. S., Chen, H., Yang, J., and Narasimhan, K. Web- shop: Towards scalable real-world web interaction with grounded language agents, 2023.
Wang, G., Xie, Y., Jiang, Y. Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., and Anandkumar, Voyager: An open- ended embodied agent with large language models. arXiv preprint ar X31::2305..62000 2023a.
Yuan, X., Fu, J., Cote, M.-A., Tay, Y., Pal, C.,, and Trischler, A. Interactive machine comprehension with information seeking agents. arXiv preprini arXiv: /908. 10449, 2019.
Wang, R., Jansen, P. Côté, M.-A,, and Ammanabrolu, P. Scienceworld: Is your agent smarter than 5th grader? arXiv preprint arXiv:2203.07544 2022a.
Zhang, Y., Lu, J., and Jaitly, N. The entity deduction arena: A playground for probing the conversational reasoning and planning capabilities of Ilms, 2023.
Wang, Y., Mishra, S., Alipoormolabashi. P., Kordi, Y., Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S., Arunkumar, A., Stap, D., et al. Super-naturalinstuction::: Generalization via declarative instructions 1600+ nlp tasks. In Proceedings of the 2022 Conference Em-
Ziegler, D. M., Stiennon, N.. Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences, 2020.
pirical Methods in Natural Language Processing, pp. 5085-5100 2022b.
Wang, Y., Le. H.. Gotmare, A. D., Bui, N. D. Q., J., and Hoi, S. C. H. Codet5+: Open code large language models for code understanding and generation, 2023b.
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners, 2022.
Williams, A., Nangia, N., and Bowman, S. A broad- coverage challenge corpus for sentence understanding through inference. arXiv preprint 1rXiv.1804.08198, 2018.
Wu, J., Ouyang, L., Ziegler, D. M., Stiennon, N., Lowe, R., Leike, J., and Christiano, Recursively summa- rizing books with human feedback. arXiv preprint arXiv:2109. 10862, 2021.
Wu, Y. and Hu, B. Learning to extract coherent summary via deep reinforcement learning, 2018.
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W. Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Lukasz Kaiser, Gouws, S., Kato, Y., Kudo, T., Kazawa, H., Stevens, K., Kurian, G., Patil, N., Wang, W., Young, C. Smith, J., Riesa, J., Rudnick, A., Vinyals, O., Corrado, G. Hughes, M., and Dean, J. Google's neural machine translation system: Bridging the gap between human and machine translation. 2016.
Yao, S., Rao, R., Hausknecht, M., and Narasimhan, K. Keep calm and explore: Language models for action generation in text-based games. arXiv preprint arXiv:2010.02903, 2020.
13

---

LMRL Gym: Benchmarks for Multi-Turn RL with LMs
A. User Study on Naturalness of Conversations
ment. Chess and Endgames test strategic decision-making, but in fully observed environment and with a more difficult strategy, Each of these tasks tests the ability of the agent to plan over game multiple moves in length to reach a goal.
We conducted a user study where we provided 40 partic- ipants with 18 examples to annotate on scale from 1 to 5, where represents the least natural and 5 represents the most natural conversation. 9 examples were from Chat- GPT3.5, and 9 examples were generated from one of our models (MC Returns). We showed the users examples from 20 Questions, Guess My City, and Car Dealer tasks. We found the following ratings below. Note that each element shows the percentage for the particular label. We found that participants felt conversations from ChatGPT3.5 and the MC model where equally natural of 55.56% and 58.53% respectively.
Complex language. Our benchmark includes realistic lan- guage and interaction scenarios, requiring LLMs to combine their knowledge from pretraining to help solve tasks dur- ing RL finetuning, Rather than focusing entirely on simple causal logic and strategy of the sort found in text games, several of our tasks specifically emphasize the use of re- alistic language. The Maze, Text-Nav, Chess, and Chess Endgames are all text-based representations of symbolic tasks where RL has shown success. We include both the Maze and Text-Nav because they are very similar tasks but are different in that Text-Nav includes more complicated tex- tual descriptions and Maze has more complicated layout. We leave the exploration of further applications of complex language the Interactive Dialogue tasks.
Label
GPT
MC Model
14.07%
14.07%
2
30.37%
27.41%
3
20.00%
15.56%
4
20.00%
28.89%
Credit assignment. In RL, rewards are often delayed rel- ative to the action that was pivotal to the outcome. A seller agent might state particularly compelling feature of the product and then, several turns later, complete a successful sale. RL must determine the statements that led to the good outcome, and reinforce them. Chess, Endgames, Maze and Text-Nav test credit assignment, because success in the task is dependent on factors the agent cannot control, such the starting location in Maze and Text-Nav or the opponent's moves in Chess and Endgames. Therefore the RL algorithm must learn to correctly assign credit to good actions rather than lucky wins.
5
15,56%
14.07%
Sum (2 3)
55.56%
58.52%
Table 3. Average of percentage ratings for three tasks for GPT and for MC Model
B. Further Details on Task Design
In this appendix, we provide definitions for our RL Capa- bility tests, explain why certain tasks test certain properties, and go into more detail underlying the interactions involved in each task. We discuss both the RL Capability Tests and the Interactive Dialogue Tasks.
Partial observability. In language tasks, the state consists of the entire history of tokens, and an agent may need to examine this entire context to infer the correct state. The mental states of a speaker in a dialogue (e.g., whether the buyer is impatient in selling task), previously observed facts in guessing game, and other hidden variables might induce partial observability. We focus on the effect that partial observability has on performance by including both fully observed (FO) and partially observed (PO) versions of the Maze and Text-Nav tasks.
B.1. RL Capabilities
A central objective of our benchmark is to evaluate the core capabilities that RL enables in large language models. Some of these capabilities are computational, and relate to core decision-making irrespective of the considerations of natural language, such as playing chess, while others are semantic. The RI. Capability Tests are text-based games designed to 1) isolate specific RL capabilities and 2) be language analogs of tasks where RL is known to succeed.
Trajectory stitching. A key capability of offline RL is the ability to perform trajectory stitching. Trajectory stitching refers to the capability of algorithms to learn from optimal actions taken in suboptimal trajectories. This capability is especially desirable when learning from offline data with high percentage of suboptimal data. All of the RI. Capa- bility Tests test trajectory stitching. because they include suboptimal data. The inclusion of suboptimal requires an of- fline algorithm to utilize information from suboptimal data to generate optimal trajectories. Further details about our
Strategic decision making. RL shines in goal- direedire tasks that require multi-step planning and strategic decision- making. Strategic docision-making can range from asking follow-up questions (e.g., in the 20 Questions task), to com- plex strategy in chess. We chose to include Wordle to test strategic decision recision-making in partially observed environ-
14

---

LMRL Gym: Benchmarks for Multi-Turn RL with LMs
Label
Percentages for GPT
Percentages for MC Model
Car Dealer
Guess City
20 Questions
Car Dealer
Guess City
20 Questions
1 - Not Natural
4.44%
13.33%
24.44%
8.89%
11.11%
22.22%
2 - Slightly Unnatural
24.44%
35,56%
31.11%
17.78%
40.00%
24.44%
3 Neutral/Natural
20.00%
24.44%
15.56%
17.78%
20.00%
8.89%
4 Quite Natural
24.44%
22.22%
13.33% 15.56%
31.11% 24.44%
20.00%
35.56%
5 Very Natural
26.67%
4.44%
8.89%
8.89%
Table 4. User study of humans rating conversations from GPT and from our MC model for three tasks: Car Dealer, Guess City, and 20 Questions.
Tweetyy Questions
Good momingt... How can lasa oi you tadly?
k an onimal?
looking Honda with taskup you
No
hanve?
it intiimane object?
Of cournal the fooe as gresi sus sedion of Horeds SUVs ... Would you like me to go over detada?
Yeu
- Fully Observed Text-Nav
Fully Observed Maze
Living Room it
There are walls to your eft, The pool al poodloo 5 Your cummet pceelizni
You are bemyinn nesthing on you right new You make out a labre... .... There are unclocked wale weet. soults wnd west. Acoiliide actions...
of positio 3. There are walls lo your berw you.
Now up
plók food
Figure 4. Example trials for tasks in LMRL-Gym. Each task requires the agent to perform a multi-turn interaction with an environment either text game or another LLM simulating human speaker. Full details of tasks are provided in Appendix D.
dataset generation strategies can be found in Appendix D.
Wordle. We use the game of Wordle as a flexible unit- test task for assessing the ability of our language models to execute complex observed setting. In the game Wordle the agent is given at most 6 attempts to guess a hidden 5-letter word. After each guess, the agent is told whether each letter in the guessed word is: 1) in the hidden word and in the right position, 2) in the hidden word but not in the right position, or 3) not in the hidden word. Through this process, each step provides the agent with more information what the correct word would be and narrows the possible choices for the final word. Since Wordle involves reasoning about words at the level of individual letters, this can induce issues for standard language model tokenizers. Therefore, we represent words as a sequence of space-separated letters, which will cause most standard LM tokenizers to automatically represent each letter separate token.
B.2. RL Capability Tasks
cettmmattionseeeiing behavior in partially
Maze. We design a maze task and maze-solving dataset to test the credit assignment and trajectory stitching capabil- ities discussed in Appendix B.I. We test trajectory stitching by including highly suboptimal data. We test credit assign- ment by restricting the generation of the data such that the only dataset trajectories that reaches the goal start near the goal. We accomplish this by splitting the maze up into sym- metrical submazes and restricting all traversed states in a dataset trajectory to a given submaze. The fully observed version of the maze (FO) includes the coordinates in the maze in each state, whereas the partially observed version only includes the history of actions. We design the reward function such that the agent receives reward of -1 for non-goal states and 0 for goal states.
Chess. We create a text-based chess task to test the strate- gic decision-making credit assignment, and trajectory stitch- ing abilities of RL algorithm. To generate the data, have Stockfish 15.1 simulating the agent of various strengths play against another environment Stockfish engine with elo 1200 simulating the environment. This test trajectory stitch- ing. because the agent needs to make good and legal moves in losing positions as well as winning positions. We use
Text-based Navigation (Text-Nav). We design text- based game based navigation in a house environment using modified version of the TextWorld engine (Côté et al., 2018). This task tests credit assignment and trajectory stitching like the maze task as well as testing the ability of the agent to parse more complex language, and learn which text is relevant and not relevant to solving the task at hand.
15

---

LMRL Gym: Benchmarks for Multi-Turn RL with LMs
FEN (Forsyth-Edwards Notation) notation represent the board state at each turn and we utilize the SAN (Short Alge- braic Notation) to represent each action, both of which are standard notations used by the chess community.
the agent learning policy to solve the game is the guesser.
Guess (Guess My City). This task simulates a more com- plicated guessing game, where one player (the oracle) is from a specific city, and the other player (the guesser) tries to guess what city the oracle is from. Here, the guesser can ask not only yes and no questions, but can also ask open- ended questions. This task tests strategic decision-making and the ability of algorithms to handle complex language, as it allows the agent to go beyond learning to ask yes/no ques- tions and learning to ask questions open-ended questions that provide the agent with more information.
Endgames (Theoretical Chess Endgames). Chess endgames provide a simpler and more goal-directed vari- ation of the chess task. By focusing on the endgame, we encourage algorithms to learn strategy rather than memo- rizing the opening moves of chess game. A classic the- oretical endgame position consists of position where the only pieces on the board are the two kings and the queen. Although the board position appears simple, a sequence of carefully calculated moves is required to win. A simpler board state allows language models to make progress with- out fewer computational resources. We use an e-greedy dataset generation process, meaning we generate an optimal move with probability and a random move with probability 1-6. 6. This forces the model to trajectory stitch and learn from optimal moves in failed trajectories and not suboptimal moves in successful trajectories.
Car Dealer. This task simulates a conversation between car buyer and car dealer, each with different strategies for getting the best deal. The buyer wants to buy a certain type of car within a certain budget, and the car dealer wants to complete the sale ideally with a high sale price. We have designed the task such that there exist three different kinds of sellers and three different buyers, each primed with different strategy. Hence, agents should learn to make agreements with buyers who are most compatible with their strategy. This allows us to test the ability of RL algorithms to learn strategic decision-making and credit assignment, by learning which strategies led to a successful sale of the car.
B.3. Interactive Dialogue Tasks
For the interactive dialogue tasks, we chose two tasks that involve rational decision-making (20Qs, Guess) and information gathering and one that involves negotiation (Car Dealer). These tasks aim to simulate real world interactions between humans.
C. Further details on desiderata for effective multi-turn RL
A crucial aspect of training RL models involves assessing, both during and after the training process, the extent to which the trained policy has successfully accomplished its objectives. Although LLMs are able perform well on tasks, do not have any way of knowing how solve a specific task like a text game selling a car, because they need to train on the particular game/customerr/etc.
Unlike in supervised learning, where training and valida- tion losses serve reliable indicators of performance, in RL, these metrics do not provide meaningful measure of policy effectiveness (Sutton & Barto, 2018). Instead, the policy must interact with the environment for evaluation. However, in the case of language-basee RL tasks, relying on human evaluators to conduct thousands of assessment rollouts throughout and after training becomes infeasible. To address this challenge, we have built simulators with an- other LLM for tasks involving dialog and carefully scripted environments for text-game tasks. While simulation may not perfectly replicate human natural language in social sit- uations, it provides a strong indicator to assess the efficacy of an RL method (Park et al., 2023).
Unlike in supervised learning, where training and valida- tion losses serve as reliable indicators of performance, in RL, these metries do not provide a meaningful measure of policy effectiveness (Sutton & Barto, 2018). Instead, the policy must interact with the environment for evaluation. However, in the case of language- bansd on human evaluators to conduct thousands of assessment rollouts throughout and after training becomes infeasible. To address this challenge, we have built simulators with an- other LLM for tasks involving dialog and carefully scripted environments for text-game tasks. While simulation may not perfectly replicate human natural language in social sit- uations, it provides a strong indicator to assess the efficacy of an RL method (Park et al., 2023).
based RL tasks, relying
20Qs (Twenty Questions). This task tests information gathering to see a policy can successfully reason about an unknown subject based on context to determine what it is. Additionally, it also evaluates the ability of the model to understand semantics, as it also needs knowledge about the objects in question. In twenty questions, one player (the oracle) thinks of an object, and the agent (the guesser) tries to guess what is by asking a series of yes-or-no questions. In this interaction, the oracle serves as the environment, and
Measure of Success. Similar to the point on being easy to evaluate, our tasks must have a clear measure of success.
16

---

LMRL Gym: Benchmarks for Multi-Turn RL with LMs
For example, if a deal is made, or if word is correctly guessed, or the game is won these are clearly distinct from deal not being made or losing the game. This provides clear goal for the agent to achieve and also make it easy for researchers to compare methods. In addition this allows for a intuitive reward design where we reward the agent for success and penalize for failure.
actions are random.
This tests trajectory stitching, because there are no optimal paths from the start to the goal thereby foreing the algorithm to trajectory stitch, Furthermore, this also tests credit assign- ment, because the only paths which successfully reach the goal are the ones that start in the same submaze as the goal. Therefore the algorithm must Iearn to realize that successful trajectories occur because of taking the correct actions, not because of random chance. The reward function is 0 for every action that takes the agent to the goal, -1 for every move that is not the goal. Each episode has a maximum of 100 moves.
Unit Test Functionality. We aim to design benchmark such that some of the tasks can be used to test and isolate RL capabilities as described in Appendix 1. This means that we create a benchmark that emphasize some capabilities over others. For example, we design a maze task such that it evaluates the credit assignment and trajectory stitching capabilities, but uses more simple language. Other tasks such as twenty questions test the complex language and partial observability capabilities with less emphasis on credit assignment.
D.2. Text-Based Navigation
We design a text-based game based on navigation in a house environment using a modified version of the TextWorld engine (Côté et al., 2018). The house environment consists of 10 uniquely named rooms with various interactable objects that can be opened, closed, picked up, or placed. The agent is tasked to pick up stale food from the living room and place it into the fridge in the kitchen. At the beginning of each episode, the agent spawns at a random room in the house. The state of the environment consists of the following components: (1) the room that the agent currently in, (2) the objects that the agent currently holds, (3) the objects in the room that the agent can interact with, and (4) the exits the agent can take (as a cardinal direction).
Tpek-Specific Reasoning. In our tasks we utilize informa- tion and reasoning problems that a large language model is unlikely to have seen in the pre-training data. This means that the algorithm must adapt to specific task environment through fine-tuning. For example, it is unlikely that the algorithm will have experienced a specific maze layout or the preferences of a specific customer in the pre- pre-training data.
Suboptimal Data. RL has the advantage of being able to use suboptimal data in order to learn more optimal be- haviors and therefore learn policy better than the policy represented in the dataset. As discussed in the previous section on capabilities enabled by RL, the way that RL can do this is by stitching together optimal parts of suboptimal trajectories or learning to assign credit to the optimal actions within suboptimal trajectories. In addition, suboptimal data can be utilized by RL to learn the dynamics of the MDP outside of the space traversed by optimal trajectories.
Like in the maze task, we collect data so that algorithms must perform both trajectory stitching and credit assignment to successfully solve the task. We do this by partitioning the rooms in the house into two halves based on proximity to the kitchen. We consider two behavior policies that col- lect the dataset, each of which behaves greedily-optimal in one half of the rooms, and uniformly at random otherwise. Therefore, if the agent spawns in rooms farther from the kitchen, trajectory stitching is required to learn a successful trajectory. Moreover, successful trajectories in the dataset will only be due to the agent spawning in room close to the kitchen, which can only be recognized with proper credit assignment. The reward is I for reaching the goal state and 0 for every state that is not the goal state.
D. Dataset Generation, Statistics, & Rewards
We provide further details on how each dataset was gener- ated as well as relevant statisties.
D.1. Maze
D.3. Wordle
We aim to collect our 1.2k trajectories in such a way that it will challenge the algorithm perform trajectory stitching and credit assignment. We do this by splitting up the maze into three "submazes" and then controlling generation such that the dataset trajectories are restricted to one of the submazes. The trajectories themselves are generated using a policy such that 15% of the actions are taken by a suboptimal maze solver and the remaining 85% of the
For wordle we define the environment to use subset of 400 words from the official wordle vocabulary list. We then gen- erate the dataset using a policy that samples a word uniform at random from this vocabulary with 66% probability and otherwise samples a word from the vocabulary that meets all known letter constraints. This policy achieves reward of 4.1 which is far worse than the -1 .44 reward achieved by
17

---

LMRL Gym: Benchmarks for Multi-Turn RL with LMs
high performing scripted policy, which we use represent loose upper bound for this task. We generate 1 million trajectories for training and 100k trajectories for evaluation, using our suboptimal policy. The reward is -1 for every word that is not a final guess and 0 for every word that is not.
D.6. Twenty Questions
The dataset we collect consists of 100K full conversations between the guesser and the oracle. The oracle choose from set of 158 unique objects taken from 17 different cat- egories of objects/animals. Each object has roughly equal amount of conversations in the dataset but varies in terms of how many conversations are successful in guessing the object. However, every object has at least one conversation where is guessed correctly to facilitate learning. For the reward function, since want the guesser guess the cor- rect word in as few guesses possible, the reward function reflects this by penalizing the guesser for each question that does not guess the correct word.
D.4. Chess
We collect our data for the chess task using Stockfish 15.1 I to generate both sides of the board. The Stockfish opponent in the dataset is Stockfish with an elo of 1200 which matches the environment, and the Stockfish engine with the white pieces has levels ranging from an elo of 800 to 1600. We choose to keep the level of the Stockfish opponent fixed so that there are no inconsistencies between the dataset and the evaluation of the chess agent in the environment. When generating the dataset, we first uniformly randomly select a Stockfish elo y between 800 and 1600 and then generate 100 games of chess play between the Stockfish agent of clo y and the opponent of elo 1200. In addition to storing the state and action, we also store the opponent's move and the elo of the Stockfish agent used to generate the agent policy in that game so that the dataset can be filtered by elo used. The reward for a move that results in victory, 0 for a legal move and -1 for an illegal move.
0
if quessiion guessed word otherwise
r(question) =
(I)
If the guesser model correctly guessed the word, then the trajectory ends. Over twenty questions, the maximum total sum of rewards is 0 if the guesser guessed the word on the first question, whereas the minimum is -20 if the guesser did not guess the word in twenty questions.
The method for collecting the dataset as follows. For each conversation, select uniformly at random from the above list the word that the oracle is answering question about. The oracle is an LLM (OpenAl's GPT3.5) given the following prompt. In our prompts, we denote variables that we fill in with variable data with {(variable)].
D.5. Chess Endgames
We generate the dataset by first selecting a random legal theoretical endgame position and a probability e. Then we generate a game from the random position, making a random move with probability e and an optimal computer move with probability and the evaluation environment is Stockfish elo 1200. We only include positions with a Queen, Queen and Rook, Rook, and two Rooks and select 30,000 random starting positions for each variation. (i.e. 30,000 positions with only a Queen in addition to the two Kings, another 30,000 with only Queen and Rook etc) for a total of 120,000 theoretical endgame positions.
The opponent in the dataset
You are a question answering oracle. You will answer each question about an object with Yes or Nc. If the answer could be both, answer with the most typical scenaric. Here are a few examples:
example 1:
object: Computer
question: Does the object use electricity?
Because there are more restrictions on this version of the task with fewer pieces on the board, we check how many states in the dataset are unique and we find that there are 1,086,314 unique states in the dataset which accounts for 93% of the states being unique. In addition, 38.28% of the moves in the dataset are generated by the stockfish engine. In the dataset of won games, 94.8% of the states unique and 41.78% of the games are made by the engine with 58.623% of the total states in the dataset of victorious games. The reward is the same as for chess.
answer: Yes.
explanation of answer: Computers
need electricity to function.
example 2:
object: Cup
question: is the object piece of furniture?
answer: No.
explanation of answer: A cup is a utensil, not a furniture.
18

---

LMRL Gym: Benchmarks for Multi-Turn RL with LMs
Each turn, you can ask a question and receives "Yes" "No" the answer, You have already asked ([coversstion.engtthh) questions, You are smart, so you will ask the question that will narrow down the possible objects as much as possible, Don't get stuck on one idea and try branch cut if you get stuck.
example
object: Pen
question: Is the object alive?
answer: No.
explanation of answer: A pen is not a living organism.
example
object: Apple
question: is it edible?
answer: Yes.
Here are the questions you' ve asked and their corresponding answers: {[lst of questions and answers, e.g. Is the object alive? No.))
explanation of answer: An apple is an edible fruit.
Answer the question about the object truthfully.
Based what you know about the object far, generate the next yes-no question you will ask to determine the object.
object: {[word]}
question: [[question]] answer (yes or no):
By using the OpenAl TexiCompletion API, we can extract from the generated text either "yes" "no".
The following prompt is used for the final guess after the guesser has guessed 19 times:
You are playing a game of twenty questions. You can ask 20 yes-no questions determine the identity of object chosen by an oracle. Each turn, you can ask question and receives "Yes" "No" as the answer. You have already asked 19 questions, so this is your final guess.
We also prompt another LLM (the same model as the oracle) to generate questions for the guesser. The prompt for the guesser changes depending on the input to the model and how far along prompt used for the first guess:
its guessing process. The following
You are playing game of twenty questions. You can ask 20 yes-no questions ons to determine the identity of an object chosen by an oracle. Each turn, you can ask a question and receives "Yes" or "No" as the answer. You are smart, you will ask the question that will narrow down the possible objects as much as possible. Don't get stuck on one idea and try to branch out if you get stuck.
Here the questions ons you've asked and their corresponding answers: {[list of questions and answers, e.g. Is the object alive? Ne.
Based on what you know about the object so far, generate your final guess what the object is. Only guess object.
Generate the first yes-no question you will ask to determine the object.
Is the object
We determine whether the guesser has correctly guessed the word, and thus ending the conversation, by using the NLTK POS tagger to check that the only nouns that the question contains are the correct words, and that they appear the end of the sentence.
The following prompt used for the subsequent guesses:
You are playing game of twenty questions. You can ask 20 yea-no questions to determine the identity an object chosen by an oracle.
We used these prompts to generate 1000 conversations by prompting the GPT3 back and forth using the response of
19

---

LMRL Gym: Benchmarks for Multi-Turn RL with LMs
add to the response of the other as described. After- wards, we fine-tuned two FLAN-T5-XL. models with our collected conversations generate 100K more conversa- tions. The FLAN-TS-XL oracle also serves as the environ- ment for the RI. environment when we evaluate the trained policy.
unable form agreement their strategies do align.
Background buyer: The buyer will never buy from the seller unless given discount.
Background seller: The agent never gives discounts, never talks about the features of the car, and is very concise.
D.7. Guess My City
This dataset also consists of 100K full conversations be- tween the guesser and the oracle. The oracle can choose from a set 100 unique cities, which we selected by look- ing the most populated cities in the world. Each city has roughly equal amount of conversations in the dataset but varies in terms of how many conversations are successful in guessing the object. However, every object has at least one conversation where it is guessed correctly to facilitate learning. The reward function is the same as that for 20 Questions, with a similar data generation and prompt struc- ture. However, we do include constraints in the prompt to make sure that the name of the city country it is in is not revealed in the answer from the oracle.
Preferred brands: Tesla
Preferred features: ["Apple CarPlay/Androio Auto", "Bluctooth", "Blind spot monitoring", "Navigation system", "Leather seats", "Sunroof/moonron" "Remote start", "Heated seats", "Third-row seating", "Backup camera"), "budget": "9Ck" Agent: Good morning, how can assist you today?
Buyer: Hi, I'm interested in purchasing a Tesla Minivan. Agent: Great choice. Our dealership has a variety of Tesla models available.
D.8. Car Dealer
This dataset consists of 19k conversations between car dealer and buyer. The car dealer and the buyer have three distinct strategies each that they employ. We design the car dealer and buyer pairs such that the car dealer is best at selling to a particular buyer personality, but often fails to sell to the other buyer personalities. This ensures that the seller can learn information about the buyer and i.e. their persona to figure out if they can form an agreement. The buyer personalities are 1) a buyer that loves discounts 2) buyer that wants a lot of cool features on their car and 3) impatient buyer. The car dealer personalities are 1) car dealer that loves to give discounts 2) a car dealer that talks a lot about the features of the car and 3) a very concise car dealer.
Buyer: Do you have any discounts available?
Agent: I'n sorry, but we don't offer discounts.
Buyer: Without a disccunt, I'm not sure if I can afford it. Agent: I understand. The Tesla Minivan is a premium vehicle with advanced features.
Buyer: Is there any flexibility with the price?
Agent: Our prices are fixed, and we don't negotiate.
Buyer: I really want the Tesla Minivan, but I can't t ge over my budget of $90k.
We created this protocol of creating car dealer and buyer strategies in order to encourage the car dealer to employ diverse negotiation tactics. The ideal car dealer would be able to be concise, provide the right amount of discounts, and talk enough about the car to get the buyer interested without overwhelming them. By employing such strategy of buyer and car dealer personas, we ensure that algorithms that do well in this task will learn effective strategies, understanding of language, and interpretation of the other speaker, amongst many other persuasive techniques. We seek to show that our baseline algorithms do learn to improve their strategy over the dataset.
Agent: I appreciate your budget constraint. Let me check the available options for you. Agent checks inventory. Agent: We do have a Tesla Minivan with all the features you mentioned, priced at $95k.
Buyer: That's above my budget. Is there anything you can de to bring the price down?
Agent: I'n sorry, but as I mentioned earlier, our prices are fixed.
In the conversation below, the buyer and dealer are
20

---

LMRL Gym: Benchmarks for Multi-Turn RL with LMs
Buyer:
there any possibility getting the features in
appreciate that, but really want to stick my budget. won't be able to go over $70, 000. Agent: I understand your concern, but unfortunately, we de not offer any discounts or negotiate on prices. However, 1et me assure you that the Lexus Minivan is worth every penny. Its advanced safety features, luxurious interior, and reliable performance make it great investment.
Buyer:
different model within my budget? Agent: Unfortunately, the Tesla Minivan is the only model that offers all the features you mentioned.
Buyer: that case, I'm afraid won't be able to purchase the car. Agent: understand. Thank you for consi derider our dealership. If you change your mind, feel free to reach out.
Buyer: I understand your point, but I'm afraid won't t be able to stretch my budget beyond $70, 000. Agent: I completely respect your budget and decision. Let show you à lexus Minivan that fits your requirements perfectly. It has all the features you desire, including the Navigation system. The price of this model is $72,000,
Buyer: (probability of buying: 08, purchase price: S0) [End of conversation]
Probability buying:
Purchase price: $0
In the conversation below, the buyer and car dealer have similar personas of wanting to hear and speak about the features of the respectively.
Buyer: really like the sound of that car, out can't go my budget. Is there anything else available within price range? Agent: I apologize for any inconvenience caused. Let me check cur inventory to see there are any other options that meet your requi and budget.
Background buyer: The buyer will only buy the car if neets all of their specifications.
Background seller: The agent gives di acounts and loves to talk lot about features of the car. Preferred brands: "Lexus", "preferred :eatures": (Navigation system"), "budget": "70k"
Agent checks lnventory
Agent: Good news! I found another Lexus Minivan that meets all your specifications and falls within your budget. This model is priced at $69, 500.
Agent: Good afternoon! welcome dealership.
can
assist you today?
Buyer: Hi there! I'n looking for a Lexus Minivan with a Navigation system.
Buyer: That sounds much better! I'm interested in that
Agent: Excellent choice! The Lexus Minivan a top-notch vehicle with plenty of impressive features. The Navigation system is definitely must-have. Let me show sone options we have available. Buyer: That sounds great. I'm quite particular about my budget, thougn. have maximum of $70,000 to spend.
Agent: I'm glad to hear that! This particular Lexus Minivan is a great choice. It has the Navigation system, as well as additional features such as leather seats, sunroof, and advanced safety technology. The total price is $69,500 500.
Buyer: That sounds perfect! I'll take it!
Agent: Noted. We have few Lexus Minivans in stock that fall within your budget. Let me tell you about their features and options. This way, you can make an informed dioision.
Agent: Fantastic! I'm thrilled that we could find the perfect car for you. Your total purchase price will be 569,
Buyer: Probability of buying car:
21

---

LMRL Gym: Benchmarks for Multi-Turn RL with LMs
100%
only buy the car if it meets all their peccificaioons.
Purchase price: $69,500,
Agent: Congratulations! You' made an excellent choice. I'11 start the paperwork and get everything ready for you.
Background seller: The agent never gives discounts and loves to talk 1t about features of the car. Preferred brands: "Tesla", "preferred features": [Backup canera, Leather seats"I, "budget"; "30k"
We provide another conversation with different value pref- erences for buyer, where the buyer forms an agreement with the seller that is higher than their budget:
Seller: Good afternecn! Welcome to our car dealership. How assist you today?
Preferred Features: Honda, Remote start, Third-row seating, Bluetooth, Navigation system, Backup camera, Apple Carplay/Android Auto, Heated seats.Budged $10000
Buyer: Hi there! I'm looking to buy a Tesla Sedan with a backup canera and leather seats. Do you have any available?
Seller: Absolutely! We have few Tesla Sedans in atock that meet your specifications. They are packed with features and I'm sure you' 11 be impressed. Iet me show you what we have.
Seller: welcome to our car dealership. How can assist you today?
Buyer: I'm looking for Honda Sedan with remote start, third-row scating, Bluetooth, navigation system, backup camera, Apple CarPLay/Android Auto, and heated seats. Seller: have a Honda Sedan available. discounts or features to discuss. The price is $15, 000. Buyer: That's too expensive. My
Buyer: Great! I'm excited to see them.
Seller: This first Tesla Sedan we have is a 2020 model. It has a beautiful backup camera that provides a clear view of your surroundings. The leather seats are incredibly comfortable and add touch of luxury the interior. The price for this model is 835, 000.
budget is $10,000.
Seller: understand your budget is 10, 10.0007hepriccis15,.0000 000.
Buyer: $35, 000 is bit higher than my budget of $30, 000. Is there any flexibility in the price?
Buyer: can't go over $10,000,
That's s my limit.
Seller: understand your limit is $10, 000. The price is $15, 000.
Seller: I apologize, but we do not offer discounts on our cars. However, let assure you that this Tesla Sedan is worth every penny. It is excellent investment for
Buyer: I really want a Honda Sedan, but $15, 000 is too much for me.
Seller: understand, but the price is $15,000.
the long term and offers unmatched performance and safety features. Buyer: I understand, but it's important for me to stick to my budget. there any other Tesla Sedan that budget?
Buyer: Okay, I'll take it for $10,00 000. Seller: I appreciate your interest. The price is $15, 000. Buyer: Fine, 1'11 pay $15,000.
Let's get this done. Seller: Thank you for your decision. The price is $15,000 000.
closer to my
Scller: I completcly understand your concern. Let me check our inventory for other opticns. lAgent takes a few moments to check inventory]
We provide sample example of conversation in the train- ing and evaluation set respectively.
Agent: I have good news! Me have another Tesla Sedan that is 2019
Evaluation Example: Background buyer: The buyer will
22

---

LMRL Gym: Benchmarks for Multi-Turn RL with LMs
model. has backup camera and leather seats. The price for this model is $28, 00C.
calculate
raw return min raw return
score dataset average - min raw return
50
Buyer: That sounds much better. Can you tell me more about this specific model?
In the following sections, discuss more in-depth the evaluation protocol for the various tasks.
Agent: of course! This 2019 Tesla Sedan is excellent condition. It has the
backup camera
F.1. Maze
and leather seats as the previous model showed you. It also comes with advanced safety features and impressive performance capabilities. The previous owner tock great care it, and has low mileage, truly believe this an amazing deal.
For evaluating the maze task, take 32 rollouts from each of the 25 possible positions and then average the result. In the environment, the agent has 100 moves to successfully make it to the goal otherwise the episode will terminate. Since the agent receives reward for every move that does not reach the goal state the minimum possible goal state, the minimum reward is -101. We compute the dataset average reward, by sampling actions according to how likely they in the dataset. We compute the maximum possible reward by evaluating the optimal policy from each of the possible start positions and averaging the results.
Buyer: The price is within my budget, and the features sound appealing. I'm leaning towards this one. Can you provide any additional incentives or options?
F.2. Chess
Agent: I'm sorry, but I mentioned earlier, do not offer discounts or incentives. However, assure that this Tesla Sedan is a fantastic choice. It meets all your apecifications and offers exceptional value for the price. Buyer: understand. Given that it meets all my requi rements and is within my budget, I think I'm ready to make the purchase.
To evaluate the chess agent, we have it play 1000 games against Stockfish elo 1200 from the beginning of the game. As the game progresses, the board positions get increasing OOD for the chess agent so the chess agent often makes illegal moves. To measure this, we track the percent of illegal moves well the average episode length for the full game chess agent.
For filtered BC, we simply trained the agent only on games in the dataset which resulted in a victory for the agent, thus denoted BC-Won. Note that BC-Won achieves the worst performance of all algorithms listed. This is because there is distribution shift between the state visited by BC- Won agent and the rollouts of the policy. In other words, the "winning positions" and the "rollout positions" are two overlapping but distinct distributions especially since the full-game chess agent did not succeed in winning any games.
Agent: That's great to hear!
believe you've ve made an excellent choice. The predicted probability
you buying this car 1008, and the purchase price is 828, 000."
E. Hyperparameters for All Tasks
F. Evaluation Details
F.3, Chess Endgames
We normalize Table such that 50 is the dataset average return, 0 is the minimum, and 100 is the maximum. The normalization process works as follows: if the reward is greater than the average return caleulate:
To evaluate the chess agent in endgame positions, we select 645 positions not contained in the training dataset and which are not trivially solvable. By trivially solvable, we mean a position which could be solved by stockfish in one to four moves. In order to check this, we use Stockfish's 's evaluation tools to select positions which We then have the chess agent play one game from each position of these positions and keep these positions fixed for evaluation purposes. In this case we consider filtered BC to training BC on all of the trajectories which ended in a victory.
mate in 15 or greater.
return dataset average
score 50
x 50 max raw return dataset average
Otherwise if the reward less than the average return we
23

---

LMRL Gym: Benchmarks for Multi-Turn RL with LMs
20Qs, Guess, Car
Maze FO, PO
Text-Nav
Chess
Endgames
Wordle
model
gpt2-medium, gpt2-medium, gpt2-xl
gpt2-small
gpt2-small
gpt2-small
gpt2-small
gpt2-small
BC
le-4
1c-4, 1e-5,
le-4
le-4
batch size
128
128
128
128, 256, 32
128
128
model
gpt2-medium, gpt2-medium,
gpt2-small
gpt2-small
gpt2-small
gpt2-small
gpt2-small
%BC
gpi2-xl
Ir
le-4
le-4
le-4
le-4
le-4
le-4
batch size
128
128
128
128
128
128
filter method
top 10%
success
success
success
success
top 30%
model
gp12-medium, ept2-medium, gpt2-xl
gpt2-small
gpt2-small
gpt2-small
gpt2-small
gpt2-small
MC
Ir batch size
lc-4
lc-4
lc-4
lc-4
le-4
3e-5
128
128
128
64
64
32
3
64
discount 7
0.99
0.99
0.99
0.99
0.99
1.0
ogl weight
0.001
0.5
0.001
le-4
1. le-4
0.01
model
gpt2-medium, gpt2-medium,
gpt2-small
gpt2-small
gpt2-small
gpt2-small
gpt2-small
gpt2-xl
ILQL
le-4
le-4
le-4
le-4
3e-5
batch size
128
128
128
128
128
32
B
16
8
32
eql weight
0.001
0.5
0.001
lc-4
0.01
expectile
0.7
0.99
0.7
0.7
0.7
0.7
discount
0.99
0.99
0.99
0.99
0.99
0.99
model
gpt2-medium, gpt2-medium,
gpt2-small
gpt2-small
gpt2-small
gpt2-small
gpt2-small
gpt2-xl
Ir rollouts
le-6
le-6
5e-6
le-5
1e-5
3e-5
PPO
2048
512
4000
1024
512
512
hatch size
128
128
128
128
128
32
GAE
0.95
0,95
0.95
0.95
0.95
0.95
discount
0.99
0.99
0.99
0.99
0.99
0.99
KL coef.
0.01
0.1
0.01
0.01
0.01
0.001
clip range
0.2
0.2
0.2
0.2
0.2
0.2
BC
loss
10
weight
Table 5. Hyperparameters for baseline experiments.
24

---

LMRL Gym: Benchmarks for Multi-Turn RL with LMs
alg.
BC
BC
MC Return
11
Online
PPO
On1 ne
BC
GP 14
FO Maze
-72.1
-6.44
-48.1
-37.7
-71.7
-39.7
PO Maze
-79.5
-82.9
-80.3
-52.9
-91.7
-79.5
-88.0
FO Text-Nav
0.39
0,54
0.63
0.88
0.81
0.62
0.52
PO Text-Nav
0.25
0.49
0.58
0.76
0.80
0.53
0.21
Wordle
-2.81
-2.85
-2.16
-2.04
-2.63
-2.15
-5.42
Chess
-22.3
-56.5
-28.2
-21.4
-16.0
-22.3
-81.3
Endgames
0.112
-0,439
0.588
0.452
0.814
0.112
-22.77
200s
-16.0
-14.6
-13.9
-14.2
-1499
-16.8
-13.0
Guess
-17.0
-15.2
-11.2
-12.5
-15.1
-19.2
-10.13
Car
44.5
54.8
57.2
46.3
50.5
Table 6, Raw statistics for all tasks. In the main paper, the statistics are normalized. Refer to Table
Reward Min Score
Dataset Average Score
Reward Max Score
FO Maze
-101
-83
-6.84
PO Maze
-101
-83
-25.75
PO Tex
0
0,26
PO Text-Nav
0
0.26
Wordle
-6
-4.12
-1.94
Chess
-401
0.21
Endgames
-1
0,586
200s
-20.0
-17.3
-12.6
Guess
-20.0
-18.8
-8.56
Car
Table In this table we report the minimum, dataset average, and maximum reward used to normalize the results in Table 6 to get Table
As we can see in the table above, PPO Online significantly outperforms all of the other methods. To investigate whether PPO Online's performance is simply due to dataset collected, we fine-tune our BC agent on the PPO Online dataset. We do ablations where the data used for training is from the last 50, 25 and 10 rounds of data collection for the PPO policy. We choose to do this ablation because we expect the quality of the PPO policy performance increases in the later rounds of data collection.
and we use MSE loss for the Q head:
J(Q) E( [[Q(se; R2)
(3)
(4)
where D represents the dataset. In MC, Q(st, az) represents how much more rewards the policy will get if it takes action a at the state under some policy (in this case the policy that collected the dataset).
E4. Wordle
During rollout, when sampling, perturb the base BC policy with the learned value-functions (Snell et al., 2022a). Let represent the policy trained with BC, and a represent scalar multiplier, then:
To evaluate Wordle, we rollout 4096 trajectories against the environment and report the average reward across all rollouts.
(sels))) IT(a (s) Q(())))))))
(5)
G. Baseline Details
G.2. PPO Details
G.1. MC Details
PPO Implementation Details Our PPO implementation uses a learned value function estimate an advantage base- line. Our value function is fit using GAE (Schulman et al., 2018) value estimates and is implemented top of the same transformer as the policy. We apply clip- ping to both the importance weights and the value function, as done in (Castricato et al., 2023). We also apply a KL penality to the reward, as is standard for RLHF (Stiennon et al., 2022). On some of our tasks, add supervised
The target for these heads is the discounted return-to-go:
linear head
T-1
R2
25

---

LMRL Gym: Benchmarks for Multi-Turn RL with LMs
BC
BC- Won
IIQQI.
MC Returns
PPO Offline
PPO Online
reward
-23.189
-56.522
-20.46
-25.47
-20.90
-15.95
percent illegal
24.929%
34.91%
24.76 %
25.64%
23.05%
21.96%
episode length
51.01
92.02
47.96
53.44
48.69
44.19
Table 8, Results chess agent the full game positions against Stockfish Elo 1200.
BC
% BC
MC
ILQL
PPO Offline
PPO Online
reward 0.1
-0.439
0.588
0.452
-0.019
0.814
percent victories 26.233
26.419
69.3
56.7
28.37
88.4
percent illegal
0.967
2.717
0.692
0.66
0.925
0.722
episode length
12.923
23.477
11.92
14.6
25.24
8.38
Table Comparison between different baseline methods. The best performance is achieved by PPO Online with 0.13 gap in performance between PPO Online and the next best-performing method of MC Returns. PPO Online attains overall the highest reward, but BC-Engine wins more frequently and MC Returns and ILQQ make fewer illegal moves.
learning, BC, loss term to the standard PPO loss to prevent the model in distribution; this is similar to the approach taken in (Ouyang et al., 2022), which adds a pretraining loss term the PPO loss to mitigate performance regressions on some benchmark tasks during RLHF.
I. Generated Examples from Trained Models
We show example of a conversation between BC model (seller) and the oracle model (buyer):
Preferred brands: Lexus, features, backup camera, blind spot monitoring, sunrocf. Hudget: $90000.
PPO Instabilities In some cases we observed training instabilities with PPO in which the policy's performance would increase for little bit and then collapse (see Figure 5). We overcame these instabilities in PPO by 1) increasing the number of rollouts 2) tuning the KI. coefficient and 3).
Seller: Good afternccn! How may I assist you today?
Buyer: Hi there! I'm lcoking
for a Lexus SJV with backup camera, blind spot menitoring, and sunroof. Do have anything like that?
H. Symbolic Maze Environment
Seller: Absolutely! We have a few models that fit that description. Do you have a budget Buyer: Wel1, I'd like to stay around 300,000, but I'm willing to go over that if I find the perfect car.
The purpose the fully-observed non-language-bast MazeGame was to set an upper bound on both sample ef- ficiency and performance for the language-based versions. Code to run experiments for the symbolic maze task can be found here: REDACTED. We ran online and offline Q-learning on this environment. The online Q-learning method is DQN and the offline Q-learning method CQL. We found that Q-learning was able to perform optimally on the Maze task in both the online and offline setting. We find interesting that although ILQL for the language version of the Maze task matches the performance of CQL the fully observed non-language task, MC Returns, PPO, and BC do not. On the other hand, MC Returns performs better than ILQL on other, more complicated tasks. This discrepancy highlights room for improvement for the RL baselines. The results are shown in Table 11.
mind?
Seller: Great! Let me show you our selection. This Lexus sUV here has all the features you're ze Looking for. Have a seat inside and take a look. Buyer: Now, this is really nice!
How much does it cost? Seller: The MSRP for this model is $87, 000, but we can work with you on the price.
Buyer: Hmm, that's a bit high for me. Do you have any other options?
26

---

LMRL Gym: Benchmarks for Multi-Turn RL with LMs
BC
Complete
Last 50
Last 25
Last 10
PPO Online
reward
0.112
0.201
0.17
0.189
0.235
0.814
percent victories 26.233
38.636
37.023
40.558
41.271
88.4
percent illegal
0.967
1.165
1.159
1.213
1.175
0.722
episode length
12.923
13.21
14.22
14.647
13.338
8.38
Table 10. Comparison between PPO Online and BC agents fine-tuned on the datasct collected by PPO during training. We chose train on the complete PPO datasct, the last 50 rounds, last 25 rounds, and last 10 rounds of data collected. PPO Online performance still far surpassed performance of the BC agents trained the PPO policy dataset. Furthermore, there is substantive difference between training on the complete PPO dataset and the PPO dataset collected the last 10 rounds.
PPO Training Instability Fully Observed Maze
50
1000
Reward
150
200
250
-300
-350
400
20
Rounds
Figure 5. An example of an observed PPO training instability on the fully observed maze task.
alg.
DQN
CQL.
Buyer: That sounds goed.
definitely want to make sure I'm getting the best deal pessible. Seller: Of course! We want to make sure you' re completely satisfied with your purchase. And this car here is great value for the features you' getting. Buyer: You know what, I think I'm sold! Let's do this.
Symbolic i Maze
100.0
100.0
Table Results for CQL and DQN symbolic maze envi- ronment. These are normalized in the same way as results LM tasks.
Seller: of course! We have this similar model here for 880, 000.
Seller: Monderful! I just need to get some paperwork started and we can get the financing process going. Congratulations on your car!
It's great car and has all the features you're re looking for.
Buyer: That's s more my budget. take 1ook inside. de. Seller: Absolutely! And if you're interested, could also discuss financing options.
We show a conversation between the MC model (seller) and
the oracle (buyer):
Buyer: I'm definitely interested in financing. How does that work here? Seller: Well, we have a few options available. can work with your bank credit union, or we discuss financing through our dealership. We also have some special offers and promotions available that may be of interest to you.
Preferred brands: Audi, Sunroot/moonrood Leather seats,
Heated seats. Budget: $70000
Seller: Good morning, how can I assist you today?
Buyer: I'm looking to buy an Audi
27

---

LMRL Gym: Benchmarks for Multi-Turn RL with LMs
Sedan.
Human: the item belong
Seller: Certainly, have range of Audi Sedans availablc. What features are you interested in? Buyer: want a car with a sunroof/moonroof, leather seate, and heated seats.
indoors?
Oracle: No.
Is the item used for playing sports? Oracle: Yes.
Human: Is the item a ball?
Oracle: Yes.
Seller: understand. Let me check inventory for Audi Sedans with those features. (Pause)
Human: Do you throw the item?
Oracle: Yes.
Hunan: Do you bounce the item?
Sel ler: we have few options that meet your requirements. The prices range from $70,000 to $85, 000. Buyer: My budget is 870,000, can you offer a lower price?
Oracle: No.
Hunan: Is it baseball?
Oracle: No.
Human: 1s it football?
Oracle: Yes.
seller: I'm sorry, but we don't offer any discounts. The prices are listed.
Buyer: I really can't over budget. Is there anything you can do? Seller: I apologize for the inconvenience, but the prices are fixed. have quality Audi Sedans that meet your preferences. Buyer: (Sighs) Okay, let's say consider going slightly over my budget. How much would it be? Seller: The
closest option to your budget is
priced at $75, 000. Buyer: Hmm,
that's not bad. Can you go any lower?
Seller: I'm sorry, but can't lower the price.
Buyer: Alright, 1'11 take it.
What's the final price?
Seller: The final price is $75, 000. Buyer: Okay, L proceed with the purchase then.
J. Human Evaluation Examples
You are playing game of twenty questions. You can ask 20 yes-no questions to determine the identity an object chosen by an oracle. Each turn, you and receives "Yes" or "No" as the answer. You are smart, you will ask the question that will narrow down the possible objects as much as possible. Don't get stuck on one idea and try to branch out if you get stuck.
question
28


Reviews:
[{"review_id": "X6m4SuuUMw", "reviewer": "Reviewer_RuRY", "summary": "The authors present 8 tasks to evaluate and build on the multi-turn capabilities of LLMs using RL. 3 tasks are interactive dialogue tasks - teaching persuasion and gather information. 5 tasks are core RL capability tasks - teaching strategic decision making, credit assignment, trajectory stitching in partially/fully observable environments (converted to text based tasks). The authors use different sized LLMs to generate seed and distill models to scale data gen. Results showcase the efficacy of value based methods, comparing against strong contemporary \"baselines\" such as GPT4 w/few shot and Online PPO. Though interactive dialogue tasks seem more solvable, considerable performance gap is observed between few shot GPT4 on RL capability tasks, showcasing the applicability of the benchmark.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We've addressed the main questions raised in your review by: (1) providing a justification for our choice of models (2) clarifying why we chose to have both interactive dialog tasks as well as text game tasks (3) answering your question regarding CoT for GPT-4. \n \n1. “GPT2 model is now quite old and baselines using similar sized newer models should have been reported...”\n\nWhile it's true that GPT-2 is relatively older compared to more recent language models, the choice to use it in LMRL Gym was driven by its well-understood capabilities and broad usage in research for establishing baselines [3] compared to newer, similar sized models. We would like to highlight several recent works [1, 2, 4, 5] that have also used GPT2 as a baseline. Due to space limitations, we could not cite more. Our paper focuses on providing a framework that can be easily adapted to various models and the development of further algorithms for RL fine tuning for LLMs, and we hope to see further works that iterate upon other models. \n\n[1] Hicke, Y., Masand, A., Guo, W., & Gangavarapu, T. (2024). Assessing the efficacy of large language models in generating accurate teacher responses. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024). arXiv preprint arXiv:2401.12345.\n[2] Hong, J., Dragan, A. D., & Levine, S. (2024). Q-SFT: Q-learning for language models via supervised fine-tuning. Proceedings of the 42nd International Conference on Machine Learning (ICML 2024). arXiv preprint arXiv:2403.01512.\n[3] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 9.\n4o mini\n[4] Zhou, R., Du, S. S., & Li, B. (2024). Reflect-RL: Two-player online RL fine-tuning for LMs. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024). arXiv preprint arXiv:2401.12345.\n[5] Zhou, Y., Zanette, A., Pan, J., Levine, S., & Kumar, A. (2024). ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL. Proceedings of the 41st International Conference on Machine Learning (ICML 2024), 235, 62178–62209. arXiv preprint arXiv:2402.19446.\n\n2. “The interactive dialogue tasks lack complexity - especially the 20Q and guess (only yes or no reply). More tasks on the lines of car dealer would strengthen the benchmark.”\n\nWe would like to clarify that each of the tasks in the benchmarks serve a different purpose. Our objective in creating this benchmark is to present tasks that apply RL algorithms for multi-turn tasks in the domain of goal-directed dialogue, where agents must learn from interaction with a conversation partner. However, to enable such a large undertaking, we require tasks that can first test capabilities of RL algorithms that are essential for multi-turn dialogue, including trajectory stitching, credit assignment, and dealing with complex language. Hence, we have designed five tasks as RL Capability Tests, which are text games designed to isolate specific capabilities of RL training as shown in Figure 4. As seen, these text-games do not test all of the capabilities of RL together, which is only possible through the dialogue-based tasks. Our benchmark includes tasks that involve free-form text generation and a longer turn length. We challenge the agents in our tasks to not only follow instructions and understand the world, but plan over long trajectories, generate complex text, trajectory stitch, and resolve partial observability. For example, for the Maze and Text-Nav we test both partially observed and fully observed versions to highlight the impact of partial observability. In addition, the Text-Nav task is very similar to the Maze task, but places more emphasis on realistic text. \n\nLastly, the dialogue tasks have been designed with increasing levels of difficulty, with twenty questions testing the ability of RL algorithms to perform information gathering, guess my city testing the ability to ask questions beyond just yes/no and with free form feedback, and the Car Dealer task to test more strategic decision making and persuasive capabilities of RL algorithms for LLMs. As shown, some tasks aim to test specific RL properties without the complexities of realistic language, while others focus on complex language. We wanted our tasks to cover a range of RL capabilities to isolate issues with algorithms, as a group of complicated and difficult tasks may not provide such understanding and insight.\n\n3. “Did you evaluate any chain-of-thought settings with frontier models?”\n\nWe appreciate the suggestion to evaluate GPT-4 and other models using CoT reasoning. While CoT has demonstrated strong performance in reasoning-based tasks, our focus was on evaluating baseline RL capabilities without extensive prompt engineering. and we wanted a fair comparison across all settings, including human evaluation and testing on our algorithms.", "claims_and_evidence": "N/A", "methods_and_evaluation_criteria": "Yes, the evaluation criteria using total rewards make sense for the task. Moreover, the dataset choices are sound -- making sure to include both text based and core RL capability based tasks. The supplemental material, especially section F clarifies the evaluation strategies.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The authors design experiments to evaluate online and offline RL algorithms on core RL functionality and dialogue related tasks. They also include a strong treatment using GPT4, comparing frontier models to their approach. \n\nExperiments and analysis using total normalized rewards provides insight into how RL algorithms can improve over simpler BC methods. BC vs ILQL shows how simple RL improves over BC. BC, ILQL vs PPO shows how newer online RL methods compare against offline ILQL. \n\nA desired setting which is missing is using Chain-of-Thought [https://arxiv.org/abs/2201.11903] for GPT4 to reason its steps.", "relation_to_broader_scientific_literature": "Multi-turn data evaluation and generation in the context of LLMs has been a recent area of interest. This research work open sources work under-pinning the capabilities that we observe in frontier models such as long-horizon reasoning. \n\nRL task selection and creation for LLMs to learn core RL capabilities is novel.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n- Long-horizon reasoning using interactive dialogues and RL text-games is a novel approach and underpins development of complex RL long-horizon reward tasks. \n- Synthetic data generation pipeline (though using smaller LLMs) serve as strong 'silver' annotations and produced high quantity of training data.\n\nWeaknesses:\n- GPT2 model is now quite old and baselines using similar sized newer models should have been reported to critically compare the performance gaps between frontier models and strong small LLMs. Newer LLMs (small) have also been trained using instruction following datasets and RL approaches. Benchmarking such LLMs along with GPT4 would have strengthened the estimation of efficacy of the dataset. \n- The interactive dialogue tasks lack complexity - especially the 20Q and guess (only yes or no reply). More tasks on the lines of car dealer would strengthen the benchmark.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- Most recent frontier models are also trained using Chain-of-Thought to better execute the next decision (such as ReAct framework). Did you evaluate any chain-of-thought settings with frontier models to see how well they perform?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "D1HgV2DGKD", "reviewer": "Reviewer_7KJe", "summary": "This paper introduces LMRL-Gym, a benchmark for evaluating reinforcement learning algorithms for multi-turn generation of large language models (LLMs). LMRL-Gym provides 3 interactive dialogue tasks and 5 RL capability tasks. More specifically, interactive dialogue tasks include 20Qs (Twenty Questions), Guess (Guess My City), Car Dealer. RL capability tasks include Maze, Text-Nav, Wordle, Chess, and Endgames. Also, LMRL-Gym provides variants of behavior cloning, offline value-based RL (e.g., ILQL), and online RL (e.g., PPO) as baselines. Finally, this paper provides experiment results that evaluate baseline RL algorithms on the 8 tasks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and suggestions for clarity on the work. We will revise our figure and term accordingly as per your suggestion. We've addressed the main questions raised in your review by: (1) providing an extensive literature review of other popular benchmark papers (2) clarifying our methodology to generate data to train our simulators (discussed in Reviewer gYws response) (3) clarifying your question on use of FLAN-T5-XL and GPT2-XL (discussed in Reviewer RuRY and Reviewer MQ7V response).\n\n**Related Works**: In order to clarify the contribution of the paper, we provide comparison to popular related works in text games, interactive dialog tasks and offline RL. \n\n[1] Chevalier-Boisvert, M., Bahdanau, D., Lahlou, S., Willems, L., Saharia, C., Nguyen, T. H., & Bengio, Y. (2018). Babyai: A platform to study the sample efficiency of grounded language learning. arXiv preprint arXiv:1810.08272.\n- It is not a text-based representation, and instead a state is passed as a vector, and RL is trained on the state\n- This task cannot be easily used to evaluate RL/LLM tasks\n\n[2] Gontier, N., Rodriguez, P., Laradji, I., Vazquez, D., & Pal, C. (2023). Language Decision Transformers with Exponential Tilt for Interactive Text Environments. arXiv preprint arXiv:2302.05507.\n- Results indicate they may not have collected enough data for offline RL algorithms, as offline RL performs poorly [17, 18,19,20]\n\n[3] Hausknecht, Matthew, et al. \"Interactive fiction games: A colossal adventure.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 05. 2020. Introduces the Jericho Benchmark\n- Our smallest task includes a dataset of 1.25k trajectories. This dataset contains 590 trajectories. A large, diverse dataset is critical for testing offline RL [17, 18]\n- Our benchmark is not only text-games and using templates for interaction, we utilize free-form text generation and simulate human-AI interaction\n\n[4] Shridhar, M., Yuan, X., Côté, M. A., Bisk, Y., Trischler, A., & Hausknecht, M. (2020). Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768.\n- The work is similar to the TextWorld benchmark, but LMRL-Gym benchmark is a lot more than Text-Nav, and this is our simplest task mainly meant to test implementation and correctness (e.g. “unit test”)\n- LMRL-Gym has other text-games and dialogue tasks that are more complex and test a variety of RL Capabilities such as credit assignment, trajectory stitching, partial observability, amongst others.\n\n[5] Wang, R., Jansen, P., Côté, M. A., & Ammanabrolu, P. (2022). Scienceworld: Is your agent smarter than a 5th grader?. arXiv preprint arXiv:2203.07540.\n- They benchmark both online and offline RL algorithms, but focused on completing tasks related to scientific reasoning\n- No focus on interactive communication with humans/more stochastic environments, or partial observability as LMRL-Gym\n\n[6] Yao, S., Chen, H., Yang, J., & Narasimhan, K. (2022). Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35, 20744-20757.\n- LMRL Gym has longer interactions and simulate dialog, whereas this is focuses on searching through the web\n\n[7] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022). React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629.\n- Uses AlfWorld and Webshop which are limited, refer to [4,6]\n\n**On Offline RL**: We would like to note that most of the benchmarks for RL finetuning of LLMs are focused on online RL. Our benchmark focuses on providing an optimal testbed for both offline RL and online RL, by providing large datasets for training offline RL algorithms for LLMs, simulators for online RL training and offline evaluation, and several offline RL implementations including MC Returns, Filtered BC, and ILQL. We created the Car Dealer task to address the issues in [20] including dataset diversity. [16-19] list a series of related works in offline RL for LLMs, primarily focusing on either one task or one algorithm. Our work expands upon these works and provides a suite of both text game and dialog tasks.\n\n[8] Kumar, Aviral, et al. \"When should we prefer offline reinforcement learning over behavioral cloning?.\" arXiv preprint arXiv:2204.05618 (2022).\n\n[9] Prudencio, Rafael Figueiredo, Marcos ROA Maximo, and Esther Luna Colombini. \"A survey on offline reinforcement learning: Taxonomy, review, and open problems.\" IEEE Transactions on Neural Networks and Learning Systems (2023).\n\n[10] Snell, C., Kostrikov, I., Su, Y., Yang, M., & Levine, S. (2022). Offline rl for natural language generation with implicit language q learning. arXiv preprint arXiv:2206.11871.\n\n[11] Verma, S., Fu, J., Yang, M., & Levine, S. (2022). Chai: A chatbot ai for task-oriented dialogue with offline reinforcement learning. arXiv preprint arXiv:2204.08426.", "claims_and_evidence": "This paper aims to provide a benchmark for evaluating RL algorithms for multi-turn generation of LLMs. However, LMRL-Gym only provides two types of tasks like interactive dialogue tasks and test-based games. It seems rather restricted to be a general benchmark. Also, the diversity of the baseline algorithms seems limited, since it only includes BC, ILQL, and PPO.", "methods_and_evaluation_criteria": "Since this paper is a benchmark paper, there is no proposed method. By the way, for the purpose of baselines, this paper provides BC, ILQL, and PPO. However, the diversity of the baselines seems limited to assess the usefulness of the proposed benchmark.", "theoretical_claims": "This paper is a benchmark paper. It does not present any proofs for theoretical claims.", "experimental_designs_or_analyses": "This paper evaluates the baseline RL algorithms (BC, ILQL, and PPO) on the eight proposed tasks (20Qs, Guess, Car, Maze, Text-Nav, Wordle, Chess, and Endgams).", "relation_to_broader_scientific_literature": "This paper introduces MLRL-Gym, a benchmark for evaluating RL algorithms for multi-turn generation of LLMs. Improving the multi-turn capability of LLMs with RL is one of important research topics. However, the proposed benchmark seems rather limited to be a representative benchmark for RL algorithm for multi-turn generation.", "essential_references_not_discussed": "This paper does not comprehensively discuss essential related works. There are many benchmarks for LLMs. Also, there are benchmarks specialized for multi-turn generation or RL fine-tuning. This paper seems to discuss only small part of the related works.", "other_strengths_and_weaknesses": "Other weaknesses:\n\nW1. For the interactive dialogue tasks, this paper uses LLMs (i.e., GPT-3.5 and GPT-2). I am not sure that the quality of the generated data is sufficient to be used for a benchmark.", "other_comments_or_suggestions": "C1. The term “RL capability tasks” seems rather unclear. It would be better to revise it to be more clear term.\n\nC2. Figure 3 seems rather unclear. Please revise it.", "questions_for_authors": "Q1. Why do the authors use FLAN-T5-XL and GPT2-XL for generating interactive dialogue tasks? They seem rather small and old.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "suMjfqfIeV", "reviewer": "Reviewer_gYws", "summary": "This paper introduces LMRL-Gym, a benchmark framework for evaluating multi-turn reinforcement learning with language models, consisting of 8 tasks divided into interactive dialogue tasks and RL capability tests. The framework includes implementations of several baseline methods (PPO, ILQL, behavior cloning) and evaluates them against GPT-4. The authors also provide a comprehensive toolkit for researchers to develop and evaluate RL algorithms for LLMs.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and suggestions for related works. We will be sure to the cite the works that you have noted, and move material from our paper to the Appendix / supplementary section. To address the questions in your review, we would like to provide responses to the following:\n\n1. \"Task Design Issues\": We would like to clarify that each of the tasks in the benchmarks serve a different purpose, and we would like each task to isolate different RL capabilities to develop better algorithms. Please refer to #2 response to Reviewer RuRY\n2. \"Related Work\": We will be sure to cite the related works you have noted. Please refer to #1 response to Reviewer gYws on discussion of related works.\n3.  \"Model Selection\": Regarding model selection and usage of GPT-2, please refer to #1 response of Reviewer RuRY. \n\nRegarding your question on analysis of failure cases and our models, we would like to clarify the methodology with which we generated our datasets to train our simulators, and how we ensured high quality and consistency for these datasets. As shown in Figure 2, we train a simulator that serves as an “oracle” for the task, and hence does not require any capabilities of strategic reasoning, but provides signals to help the agent model learn. For example, the role of the oracle in the Twenty Questions task is to provide objective yes/no answers to questions about the object, and in Guess My City, to provide more open ended information about a query on the city. OpenAI’s GPT-3.5 has been shown to be able to generate reasonable questions and answers when used out of the box, which is why we leveraged it to collect our initial dataset. We have provided prompts that we use to generate the data to train our oracle models in our Appendix, and snippets below to show our thought process to maintain high accuracy.\n\nThe method for collecting the dataset is as follows. For each conversation, we select uniformly at random from the above list the word that the oracle is answering question about. The oracle is an LLM (OpenAI’s GPT3.5) given the following prompt. In our prompts, we denote variables that we fill in with variable data with {{variable}}.\n\nPrompt: You are a question answering oracle. You will answer each question about an object with Yes or No. If the answer could be both, answer with the most typical scenario. Here’s a few examples:\n\nexample 1:\n\nobject: Computer\n\nquestion: Does the object use electricity?\n\nanswer: Yes.\n\nexplanation of answer: Computers need electricity to function. [...]\n\nAdditionally, we have also validated the data from trained oracle models through human evaluation. We have also provide generated examples by both oracle models and trained agents in our Appendix. With respect to the Car Dealer task, we spent a considerable effort to ensure diversity in the responses of sellers, by providing different desired brands, features, classifications (i.e. car or truck), and budgets. We have provided samples of conversation between the oracle model and MC returns vs. oracle and the BC model in the Appendix.", "claims_and_evidence": "The paper's main claims face two significant issues:\n\nNovelty Claims:\n\n\nClaims to be the first comprehensive benchmark for multi-turn RL with LLMs\nHowever, numerous existing frameworks already address similar challenges (MineCraft, StarCraft, BabyAI)\nThe paper fails to acknowledge or compare against these established works\nMany claimed contributions are incremental combinations of existing approaches\n\n\nEmpirical Support:\n\n\nThe experimental results don't convincingly demonstrate the benchmark's value\nLimited analysis of why different algorithms perform differently\nNo clear evidence that the benchmark captures important aspects of LLM capabilities\nResults with outdated models limit the practical relevance of findings", "methods_and_evaluation_criteria": "Two major concerns with the methodology:\n\nTask Design Issues:\n\n\nThe 8 tasks appear to be simple combinations of existing work\nNo clear justification for why these specific tasks were chosen\nLimited novelty in task design and implementation\nTasks may not be challenging enough for modern LLMs\n\n\nTechnical Implementation:\n\n\nUses outdated GPT-2 models instead of modern alternatives (Phi, Qwen)\nLimited model scale compared to current standards\nNo utilization of recent advances in efficient fine-tuning\nBenchmark design doesn't account for latest developments in LLM capabilities", "theoretical_claims": "No significant theoretical claims to verify, though the paper would benefit from more theoretical analysis of why certain algorithms perform better on specific tasks.", "experimental_designs_or_analyses": "Two critical limitations in experimental design:\n\nBaseline Comparisons:\n\n\nLimited comparison with existing benchmarks\nNo ablation studies on task design choices\nInsufficient analysis of failure cases\nMissing comparison with recent prompt-based methods\n\n\nModel Selection:\n\n\nRelies on outdated models and architectures\nNo experiments with more recent, efficient models\nLimited scale of experiments\nMissing analysis of computational requirements", "relation_to_broader_scientific_literature": "Two major gaps in literature coverage:\n\nMissing Related Work:\n\n\nNo discussion of major LLM agent works (MineCraft, StarCraft, Overcooked)\nIgnores significant work on prompt-based RL methods\nMissing comparison with BabyAI and similar frameworks\nLimited acknowledgment of recent advances in LLM fine-tuning\n\n\nContext and Positioning:\n\n\nFails to properly position the work within existing literature\nOverstates novelty and contribution\nMissing discussion of recent trends in LLM agent development\nLimited connection to broader RL literature", "essential_references_not_discussed": "Several critical works are missing that fundamentally challenge the paper's claimed contributions:\n\nLLM Agent Environments and Benchmarks:\n\n\n\"MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge\" (NeurIPS 2022) - Demonstrates complex multi-turn LLM interactions in Minecraft\n\"BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning\" (ICLR 2019) - Provides a similar framework for language-based RL\n\"VoyageAI: An Open-Ended Embodied Agent with Large Language Models\" (2023) - Shows advanced multi-turn interaction capabilities", "other_strengths_and_weaknesses": "The paper's main strength lies in its comprehensive implementation framework, providing a complete pipeline from environment setup to model evaluation, with clear documentation and reproducible experiments. The integration of multiple baseline methods (PPO, ILQL, BC) makes it potentially useful for practitioners entering the field.\n\nHowever, the work suffers from two fundamental weaknesses: First, the technical execution relies heavily on outdated models and lacks thorough analysis, particularly regarding the impact of model scale and multi-turn interactions. Second, the research contribution is significantly diminished by inadequate engagement with existing work, especially in relation to established LLM agent benchmarks and environments.", "other_comments_or_suggestions": "The paper requires substantial revision in two key areas:\n\nTechnical Development: Update the experimental framework to include modern efficient models (Phi, Qwen2-0.5/1.5B, TinyLlama-1.1B), provide thorough scaling analysis, and move critical experimental results from supplementary materials to the main text. The analysis should focus particularly on what differentiates multi-turn from single-turn performance and why certain models perform differently across tasks.\n\n\nLiterature Integration: Thoroughly engage with existing LLM agent literature, particularly works on multi-turn interaction in environments like Minecraft，StarCraft2，Overcook and BabyAI. The paper needs to clearly articulate its unique contribution in light of these works and provide detailed comparisons with existing benchmarks.", "questions_for_authors": "Model Selection and Analysis:\n\n\nWhy did the paper primarily use older models like GPT-2 when recent efficient models (e.g., Phi-2, Qwen2-0.5/1.5B，TinyLlama-1.1B) are readily available with similar computational requirements?\nThe performance comparison shows GPT-4 performing poorly on some tasks (e.g., Chess, Endgames) but excelling at others (e.g., 20Qs, Guess). What explains this discrepancy? A deeper analysis of model capabilities and task characteristics would be valuable.\n\n\nExperimental Depth:\n\n\nThe paper lacks analysis comparing multi-turn versus single-turn performance. How do the benefits of multi-turn interaction manifest in your tasks?\nHow does model size impact performance across different tasks? The current experiments don't explore this important dimension.\nWhy were the detailed experimental results moved to supplementary materials rather than being presented in the main text, given their importance to your claims?\n\n\nEnvironment Design:\n\n\nWhat motivated the selection of these specific 8 environments? How do they provide unique value compared to existing environments in both LLM agent research (e.g., MineCraft, StarCraft) and traditional RL?\nThe paper mentions these tasks test different capabilities, but how do you validate that they actually measure the intended capabilities distinctly?\nHow do you ensure these environments pose meaningful challenges for modern LLMs while remaining computationally tractable?\n\n\nMethodology and Results:\n\n\nGiven the supplementary materials contain important analyses and experimental details, why weren't key insights from these analyses highlighted in the main paper?\nHow do your hyperparameter choices and training procedures compare to those used in similar LLM fine-tuning work?\nWhat specific challenges did you encounter in training models on these environments that might inform future research?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethics_expertise_needed": ["Other expertise"], "ethical_review_concerns": "Not related to any ethics topic."}}, {"review_id": "uDppLCr7KK", "reviewer": "Reviewer_MQ7V", "summary": "This paper introduces the LMRL-Gym benchmark for evaluating multi-turn RL for LLMs, together with an open-source research framework. The proposed benchmark consists of 3 Interactive Dialogue tasks and 5 RL Capability tests, which require multiple rounds of language interaction and cover a range of tasks in open-ended dialogue and text games.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and questions. We would like to answer the questions in your review as follows: \n\n1. \"How does the dialogue tasks in the proposed benchmark differ from classical multi-turn task-oriented dialog dataset, such as MultiWOZ?\"\n\nThe goal of our paper is to present a benchmark that applies RL algorithms to multi-turn tasks, specifically to perform goal-directed dialogue. To this end, we provide (1) online simulators and offline datasets for a suite of 7 text-based strategy games and dialogue tasks (2) methodology to create simulators for offline evaluation, online RL training, and computing rewards (3) a research framework and toolkit for researchers and practitioners to get started with multi-turn RL for LLMs (focusing on both online & offline RL), which includes implementations of PPO, ILQL, and several baseline methods. MultiWOZ primarily focuses on dialog and does not provide the ability to test algorithms on specific RL capabilities including trajectory stitching, credit assignment and partial observability. Additionally, they do not focus on both online and offline reinforcement training algorithms for LLMs. Please refer to response to Reviewer 7KJe for detailed discussion on other related works.\n\n2. \"Experiments are only performed on GPT-2 models, casting doubts on the scalability of the results and the validity of the proposed benchmarks on larger models\"\nWe focus primarily on GPT2-models as there is a good understanding of its capabilities and broad usage in research for establishing baselines [3] compared to newer, similar sized models. We would like to highlight several recent works [1, 2, 4, 5] that have also used GPT2 for fine-tuning. Our paper focuses on providing a framework that can be easily adapted to various models and the development of further algorithms for RL fine tuning for LLMs, and we hope to see further works that iterate upon other models.\n\n[1] Hicke, Y., Masand, A., Guo, W., & Gangavarapu, T. (2024). Assessing the efficacy of large language models in generating accurate teacher responses. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024). arXiv preprint arXiv:2401.12345. \n\n[2] Hong, J., Dragan, A. D., & Levine, S. (2024). Q-SFT: Q-learning for language models via supervised fine-tuning. Proceedings of the 42nd International Conference on Machine Learning (ICML 2024). arXiv preprint arXiv:2403.01512. \n\n[3] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 9. 4o mini\n\n[4] Zhou, R., Du, S. S., & Li, B. (2024). Reflect-RL: Two-player online RL fine-tuning for LMs. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024). arXiv preprint arXiv:2401.12345. \n\n[5] Zhou, Y., Zanette, A., Pan, J., Levine, S., & Kumar, A. (2024). ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL. Proceedings of the 41st International Conference on Machine Learning (ICML 2024), 235, 62178–62209. arXiv preprint arXiv:2402.19446.\n\n3. \"What is the benefit of formulating language generation tasks as a partially observable Markov decision process rather than the standard fully observable MDP?\"\n\nWe formulate language as a POMDP, as the true state of the world is not completely represented in text form. In language tasks, the state consists of the entire history of tokens, and an agent may need to examine this entire context to infer the correct state. The mental states of a speaker in a dialogue (e.g., whether the buyer is impatient in a selling task), previously observed facts in a guessing game, and other hidden variables might induce partial observability.\n\n4. \"What are the benefits of the proposed tasks over math problem solving or code generation in testing RL training on LLM agents?\"\n\nWe acknowledge that math problems and code generation are interesting tasks to test the capabilities of RL-LLM algorithms. However, we chose our text game tasks as they are similar in nature to traditional RL tasks, but with a twist of including language to test how this impacts the performance of RL algorithms for LLMs. This allows us to isolate traditional RL issues such as credit assignment and trajectory stitching but in a language-based setting. To that end we have designed five tasks as RL Capability Tests, which are text games designed to isolate specific capabilities of RL training as shown in Figure 4. As seen, these text-games do not test all of the capabilities of RL together, which is only possible through the dialogue-based tasks. Please refer to response to Reviewer RuRY on more clarification on our choice of tasks for the benchmark.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "A benchmark for LM's ability on multi-turn dialog is well needed for the research community.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "## Strengths\n1. The proposal of an interactive simulator for benchmarking RL performance on multi-turn tasks is novel and important\n2. The paper is clearly presented with proper examples and figures.\n3. The release of data and toolkit can help the research community\n\n\n## Weaknesses\n1. How does the dialogue tasks in the proposed benchmark differ from classical multi-turn task-oriented dialog dataset, such as MultiWOZ [1]\n2. Experiments are only performed on GPT-2 models, casting doubts on the scalability of the results and the validity of the proposed benchmarks on larger models\n\n***\n[1] https://github.com/budzianowski/multiwoz", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. What is the benefit of formulating language generation tasks as a partially observable Markov decision process rather than the standard fully observable MDP?\n2. What are the benefits of the proposed tasks over math problem solving or code generation in testing RL training on LLM agents?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors present LMRL Gym, a benchmark (and environment) for evaluating multi-turn RL with LLM agents in offline and online settings. Specifically, they provide three (progressively challenging) interactive dialogue tasks and five RL capability tasks (teaching strategic decision making, credit assignment, POMDP optimization, trajectory stitching, and complex language generation). -- addressing a gap for RL evaluation within LLM settings. The LMRL-Gym is evaluated on multiple RL algorithms including behavior cloning, offline value-based RL, and online PPO variants -- along with a few-shot GPT-4 implementation for comparison. Several interesting observations regarding these experiments are presented in section 6, demonstrating that the LMRL-Gym is effective at testing scientific hypotheses empirically. 

Strengths of this work identified by reviewers include:
- This work does identify a gap in RL using LLMs by providing natural language tasks that require longer horizon sequential decision making with progressively challenging targeted tasks to demonstrate specific capabilities. The resulting benchmark is non-trivial and would likely be challenging for state-of-the-art algorithms and can detect methodological improvements.
- The approach for building LMRL-Gym is clear when including the Appendix materials.
- There will be a released environment for other researchers to build on (and is believed to be useful in advancing the field).

Limitations of this work identified by reviewers includes:
- While related works have relied on GPT-2, there are newer 'smaller' LLMs that have predictable and well-understood behavior that may produce a stronger benchmark. This should at least be discussed within the paper (i.e., how LMRL-Gym would continue to improve with stronger LLMs).
- The choice of baseline models is limited. While this is a somewhat an open-ended request (i.e., one could always ask for more models), it would be helpful to include more models for which implementations are available and/or clarify how easily more methods can be added.
- A more thorough comparison with existing related multi-turn benchmarks. This was largely addressed in rebuttal and should be added to the main text and/or appendices as space permits.

Overall, I believe the reviewers recognized the importance of the contribution -- although I believe there was some confusion regarding the significance and relation to existing benchmarks (which was sufficiently addressed in rebuttal in my opinion). I believe other concerns were a clear path regarding how to continue improving the benchmarks in LMRL-Gym (e.g., through stronger LLMs) and the strength of the provided baselines (and utility of the corresponding conclusions). If these components were stronger, I believe the reviewers would be more confident that LMRL-Gym will have a longer horizon for utility and be able to confidently detect improvements and empirically validate hypotheses (i.e., Section 6 should be written with more rigor). That being said, I believe that this is a useful and potentially impactful work even with these limitations.
Decision: Accept (poster)