The Synergy of LLMs & RL Unlocks Offline Learning of Generalizable Language-Conditioned Policies with Low-fidelity Data
Authors: Thomas Pouplin
Kasia Kobalczyk
Hao Sun
Mihaela van der Schaar
Venue: ICML 2025 spotlightposter (2025)
Keywords: Large Language Models
Goal-conditioned policy
Offline policy learning
Decision Making Agent
Goals generalization
Domain generalization
Primary area: deep_learning->large_language_models
TL;DR: We train LLM agents as Language-conditioned policies without requiring expensive labeled data or online experimentation. The framework leverages LLMs to enable the use of unlabeled datasets and improve generalization to unseen goals and states.

Abstract: Developing autonomous agents capable of performing complex, multi-step decision-making tasks specified in natural language remains a significant challenge, particularly in realistic settings where labeled data is scarce and real-time experimentation is impractical. Existing reinforcement learning (RL) approaches often struggle to generalize to unseen goals and states, limiting their applicability. In this paper, we introduce $\textit{TEDUO}$, a novel training pipeline for offline language-conditioned policy learning in symbolic environments. Unlike conventional methods, $\textit{TEDUO}$ operates on readily available, unlabeled datasets and addresses the challenge of generalization to previously unseen goals and states. Our approach harnesses large language models (LLMs) in a dual capacity: first, as automatization tools augmenting offline datasets with richer annotations, and second, as generalizable instruction-following agents. Empirical results demonstrate that $\textit{TEDUO}$ achieves data-efficient learning of robust language-conditioned policies, accomplishing tasks beyond the reach of conventional RL frameworks or out-of-the-box LLMs alone.

The Synergy of LLMs & RL Unlocks Offline Learning of Generalizable Language-Conditioned Policies with Low-fidelity Data
Thomas Pouplin"
Katarzyna Kobalczyk
Hao Sun'
Mihaela van der Schaar'
Abstract
tion data is infeasible. Moreover, current RL-based agents often struggle to generalize to new goals and environments beyond their training data (Yang et al., 2023). These chal- lenges underline the need for a change of focus in the area of instruction-follovii agents: from reliance on online interactions in environments with rich forms of feedback to the learning of generalizable policies from unlabeled obser- vational data. We illustrate the problem setting considered with the following hypothetical scenario:
Developing autonomous agents capable of per- forming complex, multi-step meaision-making tasks specified in natural language remains a sig- nificant challenge, particularly in realistic settings where labeled data is scarce and real-time ex- perimentation is impractical. Existing reinforce- ment learning (RL) approaches often struggle to generalize to unseen goals and states, limit- ing their applicability. In this paper, we intro- duce TEDUO, a novel training pipeline for of- fline language-conditioned policy learning in sym- bolic environments. Unlike conventional methods, TEDUO operates on readily available, unlabeled datasets and addresses the challenge of generaliza- tion to previously unseen goals and states. Our ap- proach harnesses large language models (LLMs) in a dual capacity: first, as automatization tools augmenting offline datasets with richer annota- tions, and second, as generalizable instruction- following agents. Empirical results demonstrate that TEDUO achieves data-efficient learning of robust Ianguae--conitionn policies, accomplish- ing tasks beyond the reach of conventional RL frameworks or out-of-the-box LLMs alone.
Imagine recording a human using their smartphone in daily tasks, such text messaging or calendar manage- ment. How can we train an an Al assistant following natural language instructions, e.g. "Book restaurant for 7 PM" or "Send an email to John.", by just record- ing the actions taken by the human and not having to manually label each state or action recorded? How can make the learned policies generalize commands that have not been explicitly observed in the data?
Problem setting. We formalize our setup by framing it as an offline language-conditiones We assume access a pre-collected dataset of state-action transitions, D consisting of triplets (z, a, 2'). where z and x' belong to an observable state space X', and represents actions within an action space A. To enable the grounding of the environment dynamics with the space of natural lan- guage we additionally require: 1) that the individual states and actions are representable in textual format; 2) access to an unordered list of goals, G expressed in natural language that are plausibly achievable within the environment of inter- est. We posit that both D and G are often easy to obtain by simply recording agents interact with the environment and curating a list of natural language commands corresponding to the tasks typically performed within that environment. Without constraining assumptions regarding the optimality of the data collection policy with respect the goals G. nor access the ground-truth state-transition dynamics or rewards, our aim is to learn a language connuage-onndtiooned policy, "." that can determine optimal actions with respect to new states x & DD and new goals g" & grr, where G C G is the subset of goals used for training (see section for details).
policy learning problem.
1. Introduction
Motivation. Enabling Al agents to follow human-provided natural language instructions is a critical step toward build- ing intelligent systems that can perform complex, multi-step tasks in dynamic environments. However, existing methods often require vast quantities of annotated data or rely on costly online interactions with the environment to train ef- fective policies. Such requirements limit their scalability to real-world scenarios where collecting annotated or interac-
'Equal contribution 'Department of Applied Mathematics and Theoretical Physics, University of Cambridge. United King- dom. Correspondence to: Thomas Pouplin <tp531 @cam.ac.uk> Katarzyna Kobalczyk <knk25 @ca..acccm.......
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Challenges. The task of learning n² from D and gtr alone

---

The Synergy of LLMs& & RL Unlocks Offline Learning of Generalizable Language-Condttioned Policies with Low-fidelity Data
might seem impossible without resorting human supervi- sion. We can immediately identify the following challenges: CI) Unlabeled data. The dataset D lacks explicit labels linking states I E X' to the goals E G. Nor does it include any rewards indicating the optimality of actions in relation to these goals. C2) Limited exploration. We are in an offline setup with our knowledge of the environment dynam- ies being constrained to the state-action transitions observed in D. C3) Unknown data collection. We make assump- tions regarding the optimality of the data collection policy with respect to the training or testing goals. The actions in DD could be entirely random or generated by policies aimed at solving goals with an unknown relationship to those in g. C4) Generalization. Beyond solving goals from G and taking optimal actions in previously observed states x E D, we want our agent to generalize to new states and language commands corresponding to genuinely novel goal states.
Significance for the RL Community: Offline Learning from Low-fidelity Data: We introduce the first method to enable learning generalizable Inguage-ccoditiione policies using only an unlabeled dataset of state-action transitions and an unpaired set of language commands. This addresses a critical bottleneck of conventional RL by eliminating the need for expensive labeled datasets or real-time experimen- tation. > Enhanced Generalization and Data Efficiency: TEDUO significantly improves both the generalization ca- pacity and data efficiency of offline training, outperform- ing conventional RI. approaches. We provide empirical evidence that our method scales effectively and facilitates robust policy learning in symbolic environments.
2. Problem Formalism
We are given a dataset D of past interactions of an agent acting according to data collection policy n². This dataset is represented as a collection of trajectories:
Solution: LLMs& & RL synergy. Recent advances in LLMs offer a promising solution to these challenges. LLMs, pre- trained on vast amounts of Internet data, possess the requi- site prior knowledge to understand natural language and fol- low simple instructions. However, while LLMs excel at gen- eral language comprehension, their ungrounded knowledge is insufficient for executing complex, multi-step decision- making tasks in dynamic environments (Finn, 2024; Szot et al., 2024). In this paper, we propose a novel, sequen- tial training pipeline for offline language-conditiong pol- icy leaming-TEDUO: Teaching the Environment Dynam- ics from Unlabeled Observations. As displayed in Fig- ure 1, TEDUO distills knowledge of the environment dy- namics into a pre-trained LLM through supervised fine- tuning (SFT). This knowledge is obtained by learning opti- mal policies with traditional RL. based on the offline dataset augmented with LLM-generated labels and optional state abstractions. Within TEDUO, LLMS fulfill the dual role of cheap data enhancers and flexible generalizers, elevating conventional RL to address challenges C1-C4.
D = [m)EEI, T; = (111, at, Je+1 )
30 ~ P, T++1 ~ P(- at), at T (H(ze)
where is the state transition function determining the next state given an action at A and state de X and p represents the distribution of initial states. Alongside D, we are given an unpaired set of goals G split into training G and testing grest subsets, describing in natural language tasks an agent may attempt to solve within the environment.
Denoting by P(X) the powerset of X, we assume there exists a ground-truth mapping Φ G + P(X) associat- ing each goal g with subset of the state space, d(g) Xg C. X. We say that g is achieved at time step L, if zi lies in Xg.'. Then, the cumulative discounted reward: Ee 7" Ra( at, (2+1gg) with Rø at, ;t+ I [ E d(g)} measures the optimality of actions taken by an agent with respect to achieving the goal g, where 7 E [0, 1) is the discount factor penalizing long sequences of actions. We make no assumptions regarding the optimal- ity of the data collection policy T² with respect to G and thus, in what follows, we will view our pre-collected data as an un-ordered collection of state-action-state transitions, in short denoted as D = {(z, a, 2')}. We also do not assume access to either of the ground-truth state-transition dynam- ies P or the goal-to- statte mapping o and consequently the reward R.- We only require that G contains goals corre- sponding to states that have been visited in D'.
Contributions. We categorize our contributions as having significant impact both for the fields of L.MMs and RI.
Significance for the LLM Community:: > Grounding LLMs for Multi-Step Decision Making: In line with previous research, we demonstrate that standalone L.LMs surprisingly fail even in simple multi-step decision- makinn tasks. We identify this limitation stemming from the lack of ground- ing in environment dynamics. Crucially, we show that such grounding can be successfully achieved through SFT, en- abling LLMs to integrate their background knowledge with actionable policies. > Core Skill Acquisition: Our analysis reveals that fine-tuned LLMSs acquire core Decision-making skills rather than merely memorizing optimal actions. This highlights the potential of LLMs to generalize effectively to previously unsolved tasks.
The goal. Given D and GTT, our objective is to learn a
'In this paper, we focus simple goals representable as subset of the state space. This can be extended to more complex goals using temporal logic which we leave for future work.
IIn practice, G can consist of much larger set of training goals. This set will be effectively reduced to the set of visited goals after the first step of our training pipeline.
2

---

The Synergy of LLMs & RL Unlocks Offline Learning of Generalizable Language-Cnddttooee Policies with Low-fidelity Data
Evaluption novel goals
as Enhancers
as Generalizers
New goal g $ g
Unseen state 2: ş D
((x, a, a,,z)) Unlabeled dataset of atate-action transitions
LLM hi desight tabel ling
(+ optionnll state ols absstrcctionn
Labeled dotaset abstract
Supervise finee-innniee cee-unnnnnng of a base LLM
state transitions
Policy leaming with offine RL
Learned policies
Fine-tuned LLM as general languege conditioned policy
LLM
n [9:}
TO gren
Fg1
set training gools in natural language
Optimal
Minimal requiiementts
Figure 1. Overview of TEDUO, ① The unlabeled dataset of state-action transitions is pre- pre-proeesed with LLM-automated hindsight labeling and state abstraction. @ The resulting labeled dataset of abstract transitions is used the input offline RL algorithm to learn the optimal goal-conditioned policies for the finite set of training goals. ( Knowledge of the optimal actions for each observed training goal is distilled into base LLM via SFT. The fine-tuned LLM acts as language conditioned policy generalizing previously unsoon states and language commands.
Lunguage-conditionee policy x" maximizing the expectation of the cumulative discounted rewards averaged across all g E G. Crucially, n/ should generalize to novel goals g & 9 and previously unseen states z & DD We require that π not only generalizes to synonymous language commands, but also previously unvisited goal-states.
and thus, enabling generalization to new, previously unseen states and goals, addressing C2 and C4.
3.1. Step 1. Construction of solvable MDPs
Given that in many symbolic environments the original state representations X' is high-dimensional, our pipeline contains an optional state abstraction step that maps raw observations x E X' goal-dependent abstract states s SS For each g G we construct an MDP {M² : g E G where .M (S", A,, P9, R2, p. with pa standing for the induced transition operator and Rt9 the reward function with respect to the goal g approximated with scalable LLM- based proxies. The state abstraction stage is entirely optional and if not employed, we simply take S = X' and P9 = P.
3. The Method: TEDUO
To address the problem of learning language-conddtioned conditioned policies solely based on the inputs DD and G'r, we must overcome the challenges C1-C4 outlined in the introduction. While conventional RL methods are successful learning optimal policies within well-explored environments, they typically require additional data labeling and are limited in generalization to new, previously unseen language com- mands and states. In contrast, although LLMs can under- stand the meaning of sentences in natural language describ- ing each goal, their skills lack grounding in relation to the environment's is dynamics. Our pipeline, TEDUO, employs LLMs to enhance conventional RL, effectively addressing challenges C1-C4. TEDUO consists of three main steps:
1.1.1. STATE ABSTRACTION (OPTIONAL)
The goal of state abstraction to reduce the size of the state space by grouping together similar states in way that reduces the complexity of the MDP (Li et al., 2006). With well-designed abstraction, conventional RL algorithms can learn more efficiently, requiring fewer samples of data, which is particularly relevant in the offline setup. Let F X x g Sa be an abstraction operator that given a goal 9 maps single observation x X to an abstract state 8 F(x; g). This map should be such that |So| << |2'].
① Construction of solvable MDPs. For each goal, E g", we construct an MDP by employing LLM- automated hindsight labeling and optional state abstrac- tion, addressing C1 and C2.
In the context of text-based environments, we can leverage natural language to guide state abstraction, so that the result- ing abstract states contain only the goal-relevant information. Given an environment that can be essentially represented in d-dimensional feature space, X' = Xx' x x² x ... x X we assume that only a relatively small subset of these vari- ables is relevant for solving a specific goal g and an smaller subset is required to identify the goal states Φ(g). For details on the implementation of the LLM state abstrac-
Offline Policy Learning. After obtaining labeled dataset for each goal in g E Gtr, we solve the set of abstract MDPs using out-of-the-box offline RL algo- rithm. As a result, we obtain set of learned policies (Ty 9 E G"r.. The learned policies improve on naive imitation learning, addressing C3.
3 LLM supervised fine-tuning.
We distill the knowledge of the environment dynamics into a pre-trained LLM via SFT, grounding the prior knowledge of the base L.LM
3

---

The Synergy of LLMss & RL Unlocks Offline Learning of Generalizable Language-Conditione Policies with Low-fidelity Data
Goal g: Put the purple ball next the purple chest
LLM (x)
Abstract state 8
Abstracted features relevant for identifying finginn the completion of the goal
A goal object 15 on tile: (6,18)
A goal location tile: (28.14)
Inventory:
The agent 1s currently at the tile: (13,10)
Figure An example of state abstraction for a grid world. I.I.-induced abstraction function reduces the complexity the original state by treating irrelevant distractors walls, disregarding the color of opened doors, and identifying the object to be picked up (marked with "O") and its designated location (marked with "L").
tion operator refer to Appendix C.4. Figure shows an example effect of state abstraction on grid-world from the BabyAl environment.
step, we do not require at this stage that the learned policies generalize to unseen states. Given these considerations, we primarily employ tabular Q-learning (Watkins & Dayan, 1992) solve the set of abstract MDPs. To demonstrate the flexibility of our approach, also conduct experiments with Deep Q-L.earning (Appendix B. B.4) and filtered Behav- ioral Cloning (Appendix B. At the end of this stage, we obtain a set of learned policies { E g"r). These policies are limited to the set training goals and the set of states observed DD
3.1.2. GOAL-CONDITIONED HINDSIGHT LABELING.
Following recent works (Kwon al., 2023), hypoth- esize that the existing abilities of LLMs are sufficient to perform the simple task of identifying whether a particu- lar state belongs to the set of goal states ((g) associated with the goal description g. In order to perform hindsight labeling of our dataset we wish to approximate é, and thus the reward Ra(::)), with prompted language model LLM" [g achieved in ,²), where we assign the rewards in the abstracted spaces 8 instead of X. Given the large number of goals and states, reduce the number of LLM calls needed, instead of using language models directly, we train proxy reward models- lightteitttt neural networks Ro( :9) S + {0, - predict the labels generated by the prompted language model, LLM" Details can found in the Appendix C.5. The proxy rewards functions provide much more cost-effective way to perform hind- sight labeling compared to labeling all states from DD for all goals from G directly by LLM prompting let alone by human annotators. Appendix B.5 shows that for the BabyAl Al environment proxy rewards reach near 100% accuracy in comparison to the ground truth rewards of the environment.
3.3. Step 3. Training the LLM as generalizable policy
To enable generalization to previously unseen states, and more importantly, generalization to novel goals, the final step of our method distills the knowledge of the optimal ac- tions per each abstract state and goal into pre-trained LLM. We build supervised dataset DSFT consisting of goal com- mands, initial abstract states and the sequence of optimal actions with respect the learned policies. Concretely,
+. D SPT : ((g si (d
(1 1) i gtT, si DD, arg max 7 (a s?), aeA
a,
5.9
8 = arg max P0 0, ))
1+1
SES#
Ny s.t. Rg +1, +1g) 1),
3.2. Step 2. Offline Policy Learning.
where P9 is the empirical state transition function based on the abstract datasets D9, obtained in step 2. We then fine- tune pre-trained large language model on DSFT using the standard next-word prediction objective. We integrate description of the goal and the state si into a prompt and
After the first step, for each goal g gtD we obtain of- fline dataset D : {sS a, 88, r.)). Given these data, we can apply any offline reinforcement learning method to learn optimal policies T², for each goal E Gtr practice, how- ever, to learn the goal-conditioned policies, the chosen RL method should be scalable, as we must solve multiple MDPs, one for each goal in Grr. Therefore, in our instantiation, we discard computationally intensive methods. Furthermore, as the generalization to new states is tackled by the next
....,, set the sequence [a *.9 "o..,,,,,.. a;;9] as the expected completion. We expect that the fine-tuned language model combined with the state abstraction function LLMahstrat can effec- tively act proxy for the general, goal-conditioned policy π", generalizing any new goal g $ G and previously unobserved low-level state x E X.

---

The Synergy of LLMs& & RL Unlocks Offline Learning of Generalizable Language-Condttioned Policies with Low-fidelity Data
4. Related Work
pler prompting-based methods and alternative approaches to inguage-conditionee RL? (Q3) As a result of SFT, does the language model memorize the optimal actions or does it learn generalizable and compositional skills? (Q4) How does our method scale with computer power and what is the effect of language abstractions on data efficiency?
LLMs for decision making. There is growing interest in using general-purpose LLMs directly as socision-making agents (Yao et al., 2023b). Various prompting techniques, such chain-of-though (CoT) (Wei et al., 2023) and self- reflection (Ji et al., 2023), have been developed to enhance LLMs' abilities in long-term planning tasks. Yet, prompting alone is insufficient for solving complex tasks in dynamic environments (Szot et al., 2024; Finn, 2024). To effectively utilize the prior knowledge of L.LMs, they must be grounded in the environment dynamies. This can be achieved either through in-context learning (Wang et al., 2023; Wu et al., 2023) fine-tuning (Carta et al., 2023; Tan et al., 2024; Brohan et al., 2023a). A key limitation of in-context learning is its restricted window size. In this work, we focus on fine-tuning; however, unlike prior studies, we significantly reduce the requirements on the input data.
Experimental Setup. We require a controlled with environ- ment where a wide variety of distinct goals can be specified, whose state-spaces can be represented in natural language. We choose the BabyAl Al (Chevalier-Boisver et al., 2018) environment as our main benchmark and the Webshop en- vironment (Yao et al., 2023a) as an additional environment with an increased complexity of goals and actions, demon- strating the adaptability of TEDUO beyond grid-world en- vironments. A detailed discussion on these choices can be found in Appendix A. BabyAl is a grid world platform for instruction following where an agent receives natural lan- guage goal instructions such as: "Go to the tile (3,2)" or "Look behind the green locked door". The grids can con- sist of multiple rooms connected by open or locked doors and different distractor objects that the agent can interact with (boxes, keys, balls, etc.). The action space A consists of several navigation primitives (forwa rd, pickup, etc). Webshop is simulated e-commerce environment where an agent given product requirements must locate correspond- ing items by navigating a website. This environment is particularly relevant due to its dynamically evolving action space. The available actions vary by state due to changes in the website's UI elements. Furthermore, the agent must generate linguistic inputs (e-g., search queries) to the search bar. This highlights the necessity for the fine-tuned LLM to avoid catastrophic forgetting (Luo et al., 2025) and produce coherent keywords for narrowing searches. We work with the simplified HTML representation provided in the environment as our abstracted state space. Due to space constraints, results for the Webshop environment are presented in the Appendix B.I.
LLMs as data enhancers. In conventional goal- conditioned RL, datasets of state-action transitions require augmentation with goal-dependent rewards, often through human annotation or learning from demonstrations (Ziebart et al., 2008; Fu et al., 2018 8; Bahdanau et al., 2018) 8). Recent studies show pre-trained LLMs can generate task-specific rewards (Yu et al., 2023b; Ma et al., 2023; Xie et al., 2023), though most rely on costly iterative prompting. We reduce these costs by approximating I.LM-induced reward func- tions with proxy neural networks and assigning rewards in abstracted state spaces, lowering labeling needs. Similar to Peng et al. (2023), our approach uses natural language to guide the elimination of irrelevant state features.
conduigge RL. Prior work on language- conditioned policies often assumes access ground-truth rewards (Jiang et al., 2019; Co-Reyes et al., 2018), real-time experimentation (Fu et al., 2018; Bahdanau et al., 201 8; Mir- chandani et al., 2021), or expert demonstrations paired with language annotations (Stepputtis et al., 2020; Xiao et al., 2023; Brohan et al., 2023b;a). Our approach learns from offline, unlabeled datasets that are potentially suboptimal and lack reward signals. While most language-conditionel RL studies evaluate on synonymous commands seen during training (Lynch& & Sermanet, 2021; Nair et al., 2022), we focus on novel instructions corresponding to previously un- solved goals which is a significantly more challenging setup that only a few related works attempt to address (Xiao et al., 2023; Brohan et al., 2023a; Jang et al., 2022).
Metrics. We rely on the following metrics to evaluate our learned policies: success rate: proportion of attempts in which the agent achieves the goal within the time limit (500 steps); episode length: the average number of steps taken to reach the goal or the time limit; invalid actions: ratio of invalid actions (c.g.. moving into a wall) to total actions.
5.1. Q1: Online Evaluation: Generalization Benchmark
See Appendix A for an extended discussion of related works.
Setup, We choose the collection of Synth environments- the most complex environments not requiring state memory from AlbyAl-as the main test bed for TEDUO. All envi- ronments are constructed as 22x22 grid and containing 9 rooms. They differ in the type, position, and color of the distractors. The tasks include goals such as "go to the (color) {object)", "pick up the (color) (object)", or "put
5. Experiments
Questions. In our experiments we aim to answer the follow- ing questions: (Q1) Does the use of a pre-trained language model enable generalization to new language commands and new states? (Q2) How does our method compare to sim-
5

---

The Synergy of LLMs& & RL Unlocks Offline Learning of Generalizable Language-Conditioned Policies with Low-fidelity Data
Table 1. Online evaluation of generalization performance. Results averaged over 400 (g, sg) pairs
Method
Environ-
Goals
Success Rate
Episode Length
Invalid Actions
ment
[%]
[%]
Llama-3-8B (vanilla)
train/test
train/test
17 (+0.9)
444 (+3.2)
42 (=0.1)
Llama-3-70B (vanilla)
train/test
train/test
14 (+0.7)
452 (+3.0)
55 (=0.2)
Llama-3-8B (
train/test
train/test
16 (40.7)
443 (43.3)
42 (20.1)
Llama-3-70B (in-context+CoT)
train/test
train/test
21 (40.9)
432 (+3.8)
47 (+0.3)
DeepSeek-RI (distilled to Llama-8B)*
train/test
train/test
32 (43.6)
379 (+14.4)
40 (20.5)
train
train
69 (+1.2)
248 (+4.9)
17 (+0.6)
test
train
45 (+1.2)
344 (+4.)
19 (=0.6)
TEDUO: steps 1 & 2 Baby-lllLL
train
test
15 (40.8)
453 (+2.9)
44 (+0.7)
test
test
16 (10.8)
447 (43.1)
36 (20.6)
train
train
65 (+1.4)
203 (+6.7)
21 (+0.7)
TEDUOO-Lama---8B
test
train
53 (+1.1)
257 (+5.4)
27 (+0.7)
train
test
55 (a1.6)
241 (47.5)
22 (a1.1)
test
test
45 (+1.3)
286 (46.1)
31 (a1.2)
DeepSeek-RI generates a reasoning trace per action, resulting in 100x longer responses by token count and inference time. Consequently, results are averaged over 50 (g, +g) pairs.
the {color) {object} next the {color) (object)". A list of 500 goals is used as Gtr The set of testing goals contains 100 goals that are semantically distinct from those in G The set of testing goals is augmented by asking GPT-4 to paraphrase the original commands provided by BabyAI. We train a Llama-3-8B model with TEDUO based on a dataset DD containing 800k non-unique state-action-state triplets generated according to policy that is a random mixture of default policies from BabyAl (see Appendix C.3 for details).
and high invalid action ratios indicate the inability of LLMs to understand the dynamics of the environment. Common failures include only using the action "move forward" with- out considering the agent's direction or attempting final actions (e.g. door opening) without first navigating to the correct location. This underscores the need for develop- ing data-efficient methods for distilling knowledge of the tatroomenn-dynaatien into LLMs. Our fine-tunning strat- egy brings the success rate from 17% to 65% in its training setting and 45% for testing on novel goals. Unlocking generalization. We evaluate the generalization abilities of the fine-tuned TEDUO-LLama-3-ID model to new environments and goals. When tested on new envi- ronments unseen during training, a performance drop of 12% is observed, significantly smaller than Baby-I-IL- with drop of 24%. This can be explained by the over- fitting of the baseline due to the limited offline training data. L'EDUO-Llama-3-4 benefits from the zero-shot ca- pabilities of the pre-trained LLM. This effect is even more pronounced with new 8% decrease in success rate, compared to the 40% drop for the Baby-AI-IL-bot. Overall, TEDUO achieves nearly three times better performance than the RL baseline when generalizing to both new natural language commands and environments. Appendix B.6 shows success rates by goal type. Additional experiments demonstrating TEDUO's gen- eralization to larger, unseen Baby grids after training on smaller grids are presented in Appendix B.7. Performance remains robust for grids three times larger than those seen during training, confirming TEDUO's generalization ability.
Baselines. We compare our fine-tuned Llama-3-8B agent with non-fine-tuned LLMs: DeepSeek-R1 (DeepSeek- AI, 2025), Llama-3-8B and Llama-3-70B (Llama Team, 2024) using a) vanilla and b) CoT prompting (Wei et al., 2023) with additional demonstrations provided in-context (in-context+CoT)). The latter integrates expert demonstra- tions generated during step 2 of TEDUO to test the in- context learning ability of the LLM. Following recent works (Mezghani et al., 2023; Li et al., 2022; Cao et al., 2023), we also compare against BabyAA-IL-Ba by the authors of BabyAl Al Sndevalier-Boisver
experiences only an
the baseline proposed al., 2018), which is the combination of GRU encode the instruc- tion, CNN+FILI layers to encode the grid and an LSTM memory. We train this method via imitation learning on the policy generated by TEDUO, steps 1&2. Implementation details of can be found in Appendix C.
Results. Based on the results presented in Table we make the following observations:
Prior knowledge of LLMs is insufficient. We find that non-fine-tur tuned LLMs, irrespective of their parameter count or prompting method struggle in solving the seemingly sim- ple tasks from the Baby Al environments. Low success rate
5.2. Q2: Online Evaluation: Ablation Study
Setup. With the same experimental setup, we compare our full fine-tuning pipeline with its ablations. After obtaining the abstract datasets D with the first step of TEDUO, we
Cooebase to reproduce the main results of this paper be found at this link.
6

---

The Synergy of LLMss & RL Unlocks Offline Learning of Generalizable Language-Conditione Policies with Low-fidelity Data
Table Ablation study. Results averaged over 400
(g.0%) pairs.
Table 3. Performance on type C test tasks. TEDUO and TEDUO
Method
Success
Episode Length
Invalid
B have been trained in only one environment whereas TEDUO
Rate [%]
Actions [%]
A&B has been trained both.
Method
Success Rate [%]
Episode Length
Invalid
Step + GCBC
7 (#0.6)
474 (+2.3)
11 (+0.1)
Actions [%]
Steps 1+2 (GCRL)
16 (=0.8)
430 (#3.9)
10 (10.1)
LLM (vanilla)
0 (#0.0)
20 (40.0)
75 (#0.5)
All steps
65 (=1)4)
203 (#6.7)
21 (+0.7)
TEDUO
0 (40.0)
(40.0)
(#0.7)
Llama-3-8B
TEDUO
0 (20.0)
20 (20.0)
28 (20.4)
TEDUO A&B
60 (=2.1)
16 (20.2)
37 (#1.0)
generate goal-conditioned policies with naive behavioral cloning (step 1 + GCBC). We also compare our fine-tuned Llama-3-8B against the performance of the GCRL policies obtained with offline Q-learning in step Note, neither of GCBC nor GCRL generalize new, previously unseen language commands. Therefore, in this study, we are only looking at performance on goals from G Ablation of the abstraction function is delayed to the next section.
training and type C environments for testing. We note that tasks from type C environments require the internalization of three core skills: moving to a given location, opening door, picking up a box. The skill of moving to a location can be obtained from both environments A and B, but the skill of picking up a box or opening the door can only be obtained from one of the environments, A or B, respectively. This setup allows us to investigate the transferability of learned skills across environments and their compositionality.
Results. The results of GCBC and GCRL can be seen as ablations of our pipeline. We first note that the success rate of naive behavioral cloning is low, indicating low fidelity of the data collection policy and highlighting the need for in- corporating offline policy-learning methods. Moreover, the significantly improved performance of the Q-learning poli- cies (GCRL) validates the effectiveness of the first two steps within our pipeline. The synthetically constructed MDPs are meaningful offline constructs that yield policies effective during online testing. Finally, the improved performance of our fine-tuned Llama-3-8B over the Q-learning/GCRI policies training goals and environments confirms the importance of the third step of our method and suggests that the ungrounded, prior knowledge of large language models improves generalization to new previously unseen states.
Results. Table 3 shows that agents trained on one type of en- vironments only fail to generalize. TEDUO A reaches a 99% success rate in tasks without closed doors (type A grids), but consistently fails when the goal is behind a door. TEDUO achieves an 81% success rate in new grids from Type but cannot generalize to Type C. The fact that TEDUO A&B can generalize to the environment C that requires combina- tion of both skills independently seen during training with a high success rate of 60% indicates that the fine-tuned LLM does not merely memorize optimal trajectories for individ- ual tasks. Instead, learns core, generalizable abilities that can be combined to solve novel tasks. This result empha- sizes the significance of multi-skill learning for successful generalization that TEDUO enables.
5.3. Q3: Learning and exploiting core skills
5.3.2. INTERNALIZATION OF CORE SKILLS.
We wish to investigate if by learning the optimal policies for diverse goals and environments, the LLM can integrate core skills required to achieve these goals and whether such skills can be transferred across tasks. We also investigate the aspect of skill compositionality. Does the prior knowledge of the LLM, now grounded in the environment dynamics, suffice to compose together learned skills?
One of the core skills required to successfully solve the tasks is to identify whether the agent at its location is facing an object or a wall, or is free to move forwards. This section provides additional insights into the behavior and internal state representation of the LLM fine-tuned with TEDUO in comparison to base LLM.
Setup. As in the main evaluation benchmark, we operate within the Synth environments and generate a dataset with 10 random goals and 512 states per each goal. We embed each goal-stat pair into the prompt template for eliciting actions and LLM fine-tuning and pass them through both the base and fine-tuned Llama-3-8E from experiments 5.1 and 5.2. We record the logprobabilities of the tokens [0, 1, .... 6] as well as the hidden representation of states at each layer. We label our dataset according to whether at the given state the agent is facing a wall or an object. For each layer, we fit two linear probes on top of the hidden representations: one for wall and one for object detection.
(a) Type A Figure 3. Three simple environment types.
(b) Type
(c) Type C
5.3.1. 1. SKILL TRANSFER AND COMPOSITIONALITY
Setup. We are working with three types of simple envi- ronments illustrated in Figure 3. The position and color of the door and box vary across different instantiations of the environments. We use type A and B environments for

---

The Synergy of LLMss & RL Unlocks Offline Learning of Generalizable Longuage-Condttioned Policies with Low-fidelity Data
Vanilla 1ama-3-88
TEDUO 1ama--388
Is facing an object
Is facing a wzll
Mein Actie Delaales
ROC-AUC
0.4
0.2
3
2
4
20
Layer indax
10
Facing
Action index
Object
TEDUO
Vanill
(a) Action probabilitiex. Action codes: O: left, I: right, 2: forward, 3: pick up an object, 4: drop an object, 5: toggle an object, 6: done
(b) ROC-AUC score of the linear probe.
Figure 4. Interpretability results for detection of walls and objects.
First, from Figure 4(a), we observe that a non- fine-tuned Llama-3-8B puts a high probability on the action 'move forwards' irrespective of whether the agent is facing an obstacle not; this results in high ratio of invalid ac- tions, previously observed in the benchmark experiments. After fine-tuning with TEDUO, the probability of moving forwards when facing an obstacle is significantly reduced, putting more weight on the actions of moving left or right to avoid the obstacle. We also observe, that our TEDUO method taught the LLM that objects can be picked-up (ac- tion 3), only when the agent is directly facing it. From the linear probe experiments (Figure 4(b)) we observe that after fine-tunning, the internal representations of states directly encode the information of whether the agent is facing an ob- stacle. At the final layers, the ROC-AUC score of predicting both types of labels is near 100%, in sharp contrast with the score of around 80% for the non- con-finetuunet model. Yet, the score of 80% is still relatively, high, indicating that the orig- sufficient identify whether the agents is facing an obstacle, but, since the non-fine-tuned LLM lacks grounding of this knowledge with respect to the environment dynamics, it struggles to translate it into an op- timal action to be taken. This result underscores the claims of previous works that out-of-the-bbx LLMs struggle to translate their prior knowledge into low-level actions within dynamic environments (Finn, 2024; Szot et al., 2024).
Results.
without the abstraction function during step 1. As antic- ipated, the efficacy of the learned policies improves with increasing size of the dataset D. On average, the LLM-based state abstraction reduces the number of unique states by 10% (see Fig. B.2). The reduction in state space size significantly enhances data efficiency of our training method across all three performance metrics. Furthermore, the size of the state spaces 5% corresponding to the subset of features relevant for identifying the completion of the goal is reduced to just around 20% of the original state space size (Fig. B.2 in the Appendix). This reduces the size of the datasets used for training and the subsequent goal coal-identification 5-fold.
Compute power is the new bottleneck. Given a fixed observational dataset D, we can expand at no extra cost the finettuning dataset DSFT by introducing more training goals in G Yet, larger DSFT necessitates more compute power for training the LLM agent. Table demonstrates the
inal state representations
Table 4. Performance vs. compute power.
TFlops
|g'hl
Success
Episode Invalid Length
Rate [%]
Actions [%]
5.2e7
266
33
342
32
8.6e7
372
36
330
40
1.4e8
534
45
286
scaling of our method with compute power. As expected, training on a wider range of goals results in an improved performance on unseen test goals. We do not observe a plateau in performance metries, suggesting that with addi- tional compute further gains may be possible. Consequently, our approach shifts the bottleneck from the limited avail- ability of real observational data computational power.
5.4. Q4: Data efficiency and Scaling of TEDUO
Success Rate
Episode Length
Invlid Actions
1.00
50
0.75
40
0.10
0.50
30
0.25
20
0.05
0.00
10
1000
2000
3000
1000
3000
1000
2000
6. Discussion
Number of unique state satte-action pairs collected Original
Limitations. Leveraging LLMs' prior knowledge enables efficient policy generation with minimal data. However, some applications may benefit more than others. First, cer- tain scenarios may be out of distribution even for LLMs trained on extensive Internet data. Second, we assume that the environment state can be represented textually, which, although feasible for many applications due to language's ' expressiveness, may not be ideal in all cases (this can be
Abstrocted
Figure 5. Performance vs. offline dataset size. The abstraction function enhances data efficiency.
Impact of state abstraction. Figure 5 shows the perfor- mance of the Q-learning policies (i.e., policies ny obtained at the end of TEDUO step 1+2) against the size of the ob- servational dataset D. This experiment is realized with and
8

---

The Synergy of LLMs& & RL Unlocks Offline Learning of Generalizable Language-Condttione Policies with Low-fidelity Data
owever mitigated by employing VLMs which we leave as future work). Third, due to the discrete nature of L.M tok- enization, using fine-tuned LLMs to directly output actions requires discretization of the action space, which can hin- der performance in continuous control tasks. Lastly, while data requirements are minimal, they still assume some prac- titioner knowledge of the environment and the data D to propose the list of goals G (see Appendix B.2 for details).
house, J., Reyes, D., Sermanet, P., Sievers, N., Tan, C., Toshev, A., Vanhoucke, V., Xia, F,, Xiao, T., Xu, P., Xu, S., Yan, M., and Zeng, A. Do as i can, not as i say: Grounding language in robotic affordances, 2022. URL https: http://axxvvooog..e///
2204. 01 691.
Allen, C., Parikh, N., Gottesman, O., and Konidaris, G. Learning Markov State Abstractions for Deep Reinforce- ment Learning. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural In- formation Processing Systems, 2021. URL https: / //openreview.. net/ werrumidddVVzzglaee
Conclusions. We introduced novel framework for training natural language cestruction-ollowwiin agents capable of generalizing to new states and instructions in a zero-shot setting. This is the first method to learn natural language goal-conditioned policies in an offline setting using obser- vational data that both unlabeled and non-expert. The success of TEDUO relies on the synergy between the mu- tual strengths of conventional RL and LLMs. Given the demonstrated flexibility of this method in generalizing to new goals and environments, future work could explore its potential in learning across multiple environments with distinct action and state spaces.
Andre, D. and Russell, S. J. State abstraction for pro- grammable reinforcement learning agents. In Eighteenth National Conference on Artificial Intelligence, pp- 119- 125, USA, 2002. American Association for Artificial In- telligence. ISBN 0-262-51129.4 event-place: Edmonton, Alberta, Canada.
Bahdanau, D., Hill, F,, Leike, J., Hughes, E., Hosseini, A., Kohli, P., and Grefenstette, E. Learning to Understand Goal Specifications by Modelling Reward. September 2018. URL https://openreeeee i id=H1 x JC9Ym.
Impact Statement
net forum?
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., Florence, P., Fu, C., Arenas, M. G., Gopalakrishnan, K., Han, K., Hausman, K., Herzog, A., Hsu, J., Ichter, B., Irpan, A., Joshi, N., Julian, R., Kalashnikov, D., Kuang, Y., Leal, I., Lee, L., Lee, T.-W. E., Levine, S., Lu, Y., Michalewski, H., Mordatch, I., Pertsch, K., Rao, K., Rey- mann, K., Ryoo, M., Salazar, G., Sanketi, P., Sermanet, P., Singh, J., Singh, A., Soricut, R., Tran, H., Vanhoucke, V., Vuong, Q.. Wahid, A., Welker, S., Wohlhart, P., Wu, J., Xia, F. Xiao, Xu, P., Xu, S., Yu, T., and Zitkovich, B. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control, 2023a. URL https: /abps://axxiie http///xxivv..// orw.ivorrrraaoe aarg/abbs2300.. 15818. eprint: 2307.15818.
Acknowledgments
This work was supported by Azure sponsorship cred- its granted by Microsoft's Al for Good Research Lab. Thomas Pouplin's research is supported by funding from AstraZeneca. Katarzyna Kobalczyk's research is supported by funding from Eedi. Hao Sun's 's research is supported by funding from the Office of Naval Research (ONR).
References
Abel, D., Arumugam, D., Lehnert, L., and Littman, M. State Abstractions for Lifelong Reinforcement Learning. In Dy, and Krause, A. (eds.), Proceedings of the 35th Interna- tional Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 10-19. PMLR, July 2018 URL https://///... http///poceeeeeeeee/ mlr. wwwpsp/epepe.pp. / v0/ // abe 1aae..1a..o html.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., Ibarz, J., Ichter, B., Irpan, A., Jackson, T., Jesmonth, S.. Joshi, N. J.. Julian, R., Kalashnikov, D., Kuang, Y., Leal, I., Lee, K.H,,, Levine, S., Lu, Y., Malla, U., Manjunath, D., Mordatch, I., Nachum, O., Parada, C., Peralta, J.. Perez, E.,, Pertsch, K., Quiambao, J., Rao, K., Ryoo, M., Salazar, G., Sanketi. P., Sayed, K., Singh, J.. Sontakke, S., Stone, A., Tan, C., Tran, H., Vanhoucke, V., Vega, S., Vuong, Q. Xia, F., Xiao, T., Xu, P., Xu, S., Yu, T., and Zitkovich, B. RT-1: Robotics Trans- former for Real-World Control at Scale, 2023b. URL https: /al:://axxxe /ta//vivvv...////////a/////avtva wwwpxvv.orgg.o. org/ http/bb//22111 06817. eprint 2212.06817.
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Ho, D., Hsu, J., Ibarz, J., Ichter, B., Irpan, A., Jang, E., Ruano, R. J., Jeffrey, K., Jesmonth, S., Joshi, N. J., Julian, R., Kalashnikov, D., Kuang, Y., Lee, K.HH., Levine, S., Lu, Y., Luu, L., Parada, C., Pastor, P., Quiambao, J., Rao, K., Retting-
9

---

The Synergy of LLMSs & RL Unlocks Offline Learning of Generalizable Language-Conditoone Policies with Low-fidelity Data
Cao, T., Wang, J., Zhang, Y., and Manivasagam, S. Zero- shot compositional policy learning via language ground- ing, 2023. URL htwps: http://awwww....//////////.////. 20044 07200.
Finn, C. "What robots have taught about machine learn-
ing", July 2024.
Fu, J., Korattikara, A., Levine, S., and Guadarrama, From Language to Goals: Inverse Reinforcement Learn- ing for Vision- Instruction Following. September 2018. URL https: http:///ppeervveeeee net forum? i id=r11g1hRep
Carroll, M., Shah, R., Ho, M. K., Griffiths, T. L., Seshia, S. A., Abbeel, P., and Dragan, A. On the utility of learning about humans for human-ai coordination, 2020. URL https: //tp//aavvvv/v////////ee//e/aav. wwwp:vvooor./voooooooooo abs / 1190 05789.
Givan, R., Dean, T. and Greig, M. Equivalence notions and model minimization in Markov de- cision processes. Artificial Intelligence, 147 (1):163-223, 2003. ISSN 0004-3702.
Carta, T., Romac, C., Wolf, T., Lamprier, S., Sigaud, O., and Oudeyer, P.-Y. Grounding large language models in interactive environments with online reinforcement learning, 2023. URL pestps://arxia org/ abs/ 2302. 02662.
doi:
URL ccleene/aattidee//////0000222220
com/
Chebotar, Y., Hausman, K., Lu, Y., Xiao, T., Kalashnikov, D., Varley, J., Irpan, A., Eysenbach, B., Julian, R., Finn, C., and Levine, Actionable models: Unsupervised of- fline reinforcement learning of robotic skills, 2021. URL https: / arxiv. org/ abs 44.04.. 07749.
Gur, I., Nachum, O., Miao, Y., Safdari, M., Huang, A.. Chowdhery, A., Narang, S., Fiedel, N., and Faust, A. Understanding html with large language models, 2023. URL https: arxiv. www.org///aooooo 2aabs/21000 2220..3915.......
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling, 2021. URL https: /estps://arxiee http//axvvv.o/// wwwpxvvooor/aa.oo abs/ 2106. 01345.
Han, B., Zheng, C., Chan, H., Paster, K., Zhang, M. R., and Ba, Learning domain invariant representations in goal-conditioned block mdps, 2021. URL https: / //arxiv.org/ abs 22.20 14248.
Harnad, S. The symbol grounding problem. Phys- ica D: Nonlinear Phenomena, 42(1-33:335-344 June 1990. ISSN 0167-2789, doi: 10.1016/0167-2889990) 90087-6. URL http: //dx: doi wow.ooooo.....11111111111 0167-2789 (90) 90087-6.
Chevalier-Boisvent, M., Bahdanau, D., Lahlou, S., Willems, L., Saharia, C., Nguyen, H., and Bengio, Y. BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning. September 2018. URL https:
//openreview / http:/ooummmmxreeeooo0000
Hong, Z.-W., Yang, G., and Agrawal, P. Bilinear value networks, 2023. URL
Co-Reyes, J. D., Gupta, A., Sanjeev, S., Altieri, N., An- dreas, J., DeNero, J., Abbeel, P., and Levine, S. Guiding Policies with Language via Meta-Learning. September 2018. URL https: http///ppeeeeeeeeeeeeeeeeeeeeeee net/ forum? id HKgHSSSSSA
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models, 2021. https://aaaae org/ abs / 221/2216.... 09685.
Côté, M.-A., Kádár, A., Yuan, X., Kybartas, B., Barnes, T., Fine, E., Moore, J., Tao, R. Y., Hausknecht, M., Asri, L. E., Adada, M., Tay, W., and Trischler, A. Textworld: A learning environment for text-based games. CoRR, abs/1806.1 11532, 2018.
Islam, R., Zang, H., Goyal, A., Lamb, A., Kawaguchi, K., Li, X., Laroche, R., Bengio, Y., and Combes, R. T. D. Discrete factorial representations
abstraction for goal conditioned reinforcement learning, 2022. URL /ttps://arxivvorr//abb///// 22tp://bb/221....0.0 00247.
DeepSeek-AI. Deepseek-rl:
Incentivizing reasoning ca- pability in Ilms reinforcement learning, 2025. URL abs// 12948.
https: / aaaaiiiiv...//../ ott////////////oo
James, S.. Ma. Z., Arrojo, D. R., and Davison, A. J. RI- bench: The robot learning benchmark & learning envi- ronment, 2019. URL https: http:/:///wwwww.///////// 1 12271.
Fan, J. and Li, W. Dribo: Robust deep reinforcement leamn ing via multi-view information bottleneck, 2022. URL https: / araii.ii...///.. org/ hbcg://bb// /2t.2.222... 13268.
Fan, L.., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., Tang, A., Huang, D.-.., Zhu, Y., and Anandkumar, A. Minedojo: Building open-ended embodied agents with internet-scale knowledge, 2022. URL https:// arxiv. wr.oorggaaoo 2206. 2285308833......
Jang, E., Irpan, A., Khansari, M., Kappler, D., Ebert, F., Lynch, C., Levine, S., and Finn, C. BC-Z: Zero- Shot Task Generalization with Robotic Imitation Leam- ing, 2022. URL https: : /aaxiii.. org/ 02005. eprint: 2202.02005.
orggabs/////ooooooooo..../o...rr abs/ 2202.
10

---

The Synergy of LLMss & RL Unlocks Offline Learning of Generalizable Language-Condttions Policies with Low-fidelity Data
Ji, Z., Yu, T. Xu, Y., Lee, N., Ishii, E., and Fung, P. Towards mitigating LLM hallucination via self reflection. In Bouamor, H., Pino, J., and Bali, K. (eds.). Findings of the Association for Computational Linguisties: EMNLP 2023, pp. 1827-1843, Singa-
MDPs. In International Symposium on Artifi- cial Intelligence and Mathematics, Al&Math 2006, Fort Lauderdale, Florida, USA, January 4-6, 2006, 2006. URL hstp://anytime.. cs. umass. pdf.
a wwwchoo/ppooeeednnooooc
pore, December 2023. Association for Computational Linguistics. doi:
Li, S., Puig, X., Paxton, C., Du, Y., Wang, C., Fan, L., Chen, T., Huang, D.-A., Akyürek, E.,, Anandkumar, A., Andreas, J., Mordatch, I., Torralba, A., and Zhu, Y. Pre-trained language models for interactive decision 2022. URL https: / arxiv. wwg/oogg///.... /2022.222222222 01771.
220d2i///.22....dinneeemmllpp 123. URL https::////.cc. ////aaaaanie /aaaaaaanntttooyooeee org/2033. Eindings-emnipp 123.
Jiang, Y., Gu, S., Murphy, K., and Finn, C. Language as Abstraction for Hierarchical Deep Reinforcement Lear- ing, November 2019. URL ht ht:p:/// /twpiv.rivvvv.////////..//aaav org/ abs/ 1 1901.07363.. 1#Xivv1106.073:: 07334 [es, stat].
Llama Team, A. M. The llama 3 herd of models, 2024. URL https:: /tae:////xxiii wttaaaiivvvoo /abs/ 220024000000 217833.
Luo, Y., Yang. Z., Meng, F., Li, Y., Zhou, J., and Zhang, Y. An empirical study of catastrophic forgetting in large language models during continual fine-tuning, 2025. URL https: /ttpss///arivv.oo org/ abs 200 230..08744 08747.
Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, T. Morel Model-based offline reinforcement learn- 2021. URL www.httss//arxiv..rrgge 05951.
Lynch, C. and Sermanet, P. Language Condi- tioned Imitation Learning over Unstructured Data, July 2021. URL http:// /ttp:///aavvvvvv wrgg/bs//2055 2005 07 07648. arXiv:2005.07648 [cs].
Klissarov, M., D'Oro, P., Sodhani, S., Raileanu, R., Bacon, P.-L., Vincent, P., Zhang. A., and Henaff, M. Motif: Intrinsic Motivation from Artificial Intelligence Feedback. October 2023. URL hitps: /atee:///pppeeee opwnreviee..w.....m// / forum?id=tmBKIe
Ma, Y. J., Yan, J., Jayaraman, D., and Bastani, O. How far i'll go: Offline goal condiionnd reinforcement Iearn- ing via f-advantage regression, 2022. URL https: / arxiv. www. abs
Knox, W. B., Allievi, A., Banzhaf, H., Schmitt, F., and Stone, P. Reward (mis)design for autonomous driv- ing, 2022. URL https: 13906.
/ www.aawwww....///.///...
Ma, Y. J., Liang, W., Wang. G., Huang, D.-A., Bastani, O., Jayaraman, D., Zhu, Y., Fan, L., and Anandkumar,
Kostrikov, I,, Nair, A., and Levine, S. Offline reinforcement learning with implicit q-learning, 2021. URL ht //arxiv. org. abs/ 20bs//1100 2210..1100000 06169.
A. Eureka: Human- Level Reward Design via Coding Large Language Models. December 2023. URL https:
/ //openreview net
Mazoure, B., Kostrikov, I., Nachum, O., and Tompson, J. Im- proving zero-shot generalization in offline reinforcement learning using generalized similarity functions, 2021. URL https: //rxxvv org/ aaaabba211111 14629.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conserva- tive learriigg for offline reinforcement learning, 2020.
URL https::///// http//aaxivvv/..///vv/ org. abs//000ab/0/5 04779.
Kwon, M., Xie, S. M., Bullard, K., and Sadigh, D. Reward design with language models, 2023. URL https:// arxiv. org/ abs/ 00001.
Mediratta, I., You, Q., Jiang, M., and Raileanu, R. The generalization gap in offline reinforcement learning, 2024. URL https: / aawiiv.vvvv..///vv///./////avvvv www.aawwwwaaaaaaaeaeeaeeeareeeee wtw//aa/////eaaoooae.ag/oeeaeaee /2t..22122.00 05742.
Küttler, H., Nardelli, N., Miller, A. H., Raileanu, R., Sel- vatici, M., Grefenstette, E., and Rocktäschel, T. The nethack learning environment, 2020. URL https: //arxiv. orroorgg//b.............
Mezghani, I.., Bojanowski, P., Alahari, K., and Sukhbaatar, S. Think before you act: Unified policy for interleaving language reasoning with actions, 2023. URL https: wwwp:/iw.wra/a///////0/0 110663........
abs www.///....... 13760.
Laskin, M., Wang, L., Oh, J., Parisotto, E., Spencer, S., Steigerwald, R., Strouse, D., Hansen, S., Filos, A., Brooks, F.,, Gazeau, M., Sahni, H., Singh, S., and Mnih, V. In-context reinforeement learning with algorithm distilla- tion, 2022. URL httpsttps.. / arxiv. wrtp:/s//21//00 14215.
Mirchandani, S., Karamcheti, S., and Sadigh, D. ELLA: Exploration through Learned Language Abstraction.
November 2021. URL /tttps://openrevit/ net/forum ittoruummmidioooll23232...
Mnih, V., Kavukcuoglu. K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Play- ing atari with deep reinforcement learning, 2013. URL https: /tt///aavvvvooaa//////aa///e/ 5602.
Li, L., Walsh, T. J., and Littman, M. L. To- wards Unified Theory of State Abstraction for

---

The Synergy of LLMs & RL Unlocks Offline Learning of Generalizable Language-Conditione Policies with Low-fidelity Data
Nair, S., Mitchell, E., Chen, K., Ichter, B., Savarese, S., and Finn, C. Learning Language- CondiggeCnn Robot Behavior from Offline Data and Crowd-Soureed Annotation. In Proceedings of the 5th Conference on Robot Learning, pp. 1303-1315. PMLR, January 2022. URL http:: ptapttp:://proceeee /pppoceeeiiggssssee mwr.m.r..... presss/ v1 v164/nair222 html.
Szot, A., Mazoure, B., Agrawal, H., Hjelm, D., Kira, Z., and Toshev, A. Grounding Multimodal Large Language Mod- els in Actions, 2024. URL https: / wawaiiivvoooo org/
abs / 07904. .eprint: 2406.07904.
Tan, W., Zhang, W., Liu, S., Zheng, L., Wang, X., and An, B.
True knowledge comes from practice: Aligning llms with embodied enviromments via reinforcement learning, 2024. URL https: / arxiv. wwwg/orgg........ abs 2401 14151.
Peng, A., Sucholutsky, I., Li, Z.., Sumers, T., Griffiths, T. L., Andreas, J., and Shah, J. Learning with Language- Guided State Abstractions. October 2023. URL https: //openrevies net htt////rrummooeeee/ d= 5xa2c0%g*.
Todorov, E., Erez, T., and Tassa, Y. Mujoco: physies engine for model-based control. In 2012 IEEF/RSJ Inter- national Conference on Intelligent Robots and Systems, pp.
Pouplin, T., Sun, H., Holt, S., and van der Schaar, M. Retrieval augmented thought process for private data handling in healthcare, 2024. URL
IEEE, 2012. doi: (009//RRS..012 6386109.
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., and Anandkumar, A. Voyager: An Open-Ended Embodied Agent with Large Language Mod- els, 2023. URL https: //axxivv...////////////////vaavv org/ abs/2305. 16291. .print: 2305.16291.
org/abs/ 2402.07812 07812.
Puig, X., Ra, K., Boben, M., Li, J., Wang. T., Fidler, S., and Torralba, A. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Con- ference Computer Vision and Pattern Recognition, pp. 8494 8502, 2018.
Watkins, C. J. C. H. and Dayan, P. Q-learning. Machine Leurning, 8(3):279-292, May 1992. ISSN 1573-0565, doi: 10. 10.1007/BF009992698 URLtttps://doi.oo 10. 1001 BBD0//9F0992....
Putterman, A. L., Lu, K., Mordatch, I., and Abbeel, P. Pre- training for Language Conditioned Imitation with Trans- formers, 2022. URL https: /ttp////ppeeeeeeeeeeeeee / forum? d=eCPCn25gat.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F. Chi, E., Le, Q. and Zhou, D. Chain-of- thought prompting elicits reasoning in large language models, 2023. URL https: /ttpaaaivvvorr./..a............a orr//bs/////oooo////////o//////s 2201 11903.
Raparthy, S. C., Hambro, E., Kirk, R., Henaff, M., and Raileanu, R. Generalization to new sequential deci- sion making tasks with in-context learning. 2023. URL https: /ttp//arvvvv///////////////aaav. org/ org/abs/////eeooo..r....////...r 231223312222 03801.
Wu, Y., Prabhumoye, S., Min, S. Y., Bisk, Y., Salakhutdinov, R., Azaria, A., Mitchell, T., and Li, Y. Spring: Study- ing the paper and reasoning to play games, 2023. URL https: /tt////avvvv//.////////ee////.aa www.xi..oog/aaaa abs/
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algo-
rithms, 2017. URL https: //tp://aaavvvv/aa/eaaeeeeeaaaa org/ abs/
1707.06347.
Xiao, T., Chan, H., Sermanet, P., Wahid, A., Brohan, A., Hausman, K., Levine, S., and Tompson, J. Robotic Skill Acquisition via Instruction Augmentation with Vision- Language Models, July 2023. URL http:// //arxvvax.vvvv/////// org/ abs abs//22/20 22111.211111 1 11736. arXiv:2211.11736 [cs].
Shah, D., Equi, M., Osinski, B., Xia, F., Ichter, B., and Levine, S. Navigation with large language models: Se-
mantic guesswork heuristic for planning, 2023. URL https: / /aaxiiiiiiv////i///aaa/a/aivavvv org/ arg/abs//// 2210..0.. 10103.
Shridhar, M., Manuelli, I.., and Fox, D. CLIPort: What and
Xie, T., Zhao, S., Wu, C. H., Liu, Y., Luo, Q. Zhong, V., Yang, Y., and Yu, T. Tex:2Reward: Reward Shaping with Language Models for Reinforcement Learning. October 2023. URL httph:ttp///...... /ttp///pppweeeeee////e/ee htpp::///////eeeeeeelee.. id-t0M39YTRxE.
Where Pathways for Robotic Manipulation, 2021a. URL https:/ /ttss///rxiiie http///wwa..g///aaa...//aaaooott /rgg 2119. 12098. .pprint: 12098.
forum??
Shridhar, M., Yuan, X., Côté, M.-A., Bisk, Y., Trischler, A. and Hausknecht, M. Alfworld: Aligning text and embodied enviromments for interactive learning, 2021b. htRps://///a orgp:b////l//b///oob 03768.
Xie, T., Zhao, S., Wu, C. H., Liu, Y., Luo, Q. Zhong, V., Yang, Y., and Yu, T. Text2reward: Reward shaping with language models for reinforcement learning, 2024. URL https: http:///axivvvv... www.wwwwwww..aaaaa.a.a.aee wwwp/os.///.oo...ooo/...oea.eese hbtp://b//20000 2209 2199.///.....
11489.
Stepputtis, S., Campbell, J., Phielipp, M., Lee, S., Baral, Engguage-Conditionee Imita-
C., and Amor, H. B. tion Learning for Robot Manipulation Tasks, Octo- ber 2020. URL http: www.ii.rg//as///.....o 12083. arXiv:2010. 12083 [cs].
Xu, M., Shen, Y., Zhang, S., Lu, Y., Zhao, D., Tenenbaum, J. B., and Gan, C. Prompting decision transformer for few-shot policy generalization, 2022. URL https:/ a arxiv. www.vv.orgrbb//eee /206 13499.
12

---

The Synergy of LLMSs & RL Unlocks Offline Learning of Generalizable Language-Conditione Policies with Low-fidelity Data
Yang, R., Fang, M., Han, L., Du, Y., Luo, F,, and Li, X.
Mher: Model-based hindsight experience replay, 2021.
URL https: /ttpxxiivvvoo///a org/ abs/ 00306.
Yang, R., Lu, Y., Li, W., Sun, H., Fang. M., Du, Y., Li, X., Han, L., and Zhang, C. Rethinking goal-conditioned supervised learning and its connection offline rl, 2022. URL httph:tt:///emm aaaiiv.iii......ii.a.aaaa/aiiav. org/ abs/ /202. 04478, 78.
Yang, R., Lin, Y., Ma, X., Hu, H., Zhang, C., and Zhang, What is essential for unseen goal generalization of offline goal-conditioned rl?, 2023. URL https: / arxiv. org/abs/ 2ttpa//355.18888
Yao, S., Chen, H., Yang, J., and Narasimhan, Web- shop: Towards scalable real-world web interaction with grounded language agents, 2023a. URL https:// arxiv. org/ abs 22077 01206.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. React: Synergizing reasoning and acting language models. In The Eleventh In- ternational Conference Learning Representations, net/ forum?
2023b.
Ed-RE_vluYUL X.
Yu, T., Thomas, G., Yu, L.. Ermon, S., Zou, J. Y., Levine, S., Finn, C., and Ma, T. Mopo: Model-based offline policy optimization. In Larochelle, H. Ran- zato, M. Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 14129-14142. Curran Associates, Inc., 2020. URL http:::: neurips. /c//apeee_..eeeepppp//////////// 5322852cc0EFs0000077777ceeeoooo3 pdf.
Yu, W., Gileadi, N., Fu, C., Kirmani, S., Lee, K.-H., Are- nas, M. G., Chiang, H.-T. L., Erez, T., Hasenclever, L., Humplik, J., Ichter, B., Xiao, T., Xu, P., Zeng, A., Zhang, T., Heess, N., Sadigh, D., Tan, J., Tassa, Y., and Xia, F. Language to rewards for robotic skill synthesis, 2023a. URL /attps://ariiv. wwwaa.avvvooo..... org/ abs/2306... 08647.
Yu, W., Gileadi, N., Fu, C., Kirmani, S., Lee, K.-H., Are- nas, M. G., Chiang, H.-T. L., Erez, T., Hasenclever, L., Humplik, J., Ichter, B., Xiao, T., Xu, P., Zeng. A., Zhang, T., Heess, N., Sadigh, D., Tan, J., Tassa, Y., and Xia, F. Language to Rewards for Robotic Skill Synthesis, June 2023b. URL /tatt::://xxiiii org/ hbap/abs/23330 2306. 08647. arXiv:2306.086/4 [cs].
Ziebart, B. D., Maas, A. L., Bagnell, J. A., Dey, A. K., et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, IL, USA, 2008,
13

---

The Synergy of LLMs& & RL Unlocks Offline Learning of Generalizable Language-Condttioned Policies with Low-fidelity Data
A. Extended Related Work
A.1. Generalization in offline Reinforcement Learning
Following the work of Mediratta et al. (2024), we separate the generalization abilities of offline reinforcement learning algorithms into two categories: new instruction following and adaptation to new states or environments.
Goal-conditioned RI. Goal-conditioned Reinforcement Learning (GCRL) is subfield of RL dedicated to developing policies capable of achieving multiple goals within the same environment dynamies. These policies are conditioned on an additional input, g. indicating the goal that the next action should aim to achieve. While most recent research has focused on online settings (Islam et al., 2022; Han et al., 2021; Hong et al., 2023; Yang et al.. 2021), only a few methods have addressed the offline GCRL problem (Yang et al., 2022; Ma et al., 2022; Chebotar et al., 2021). (Yang et al., 2023) offers a comparison of these methods and highlights the key challenges in offline GCRL Additionally, these approaches typically restrict goal representations to those expressible as a single state in the state space (Chebotar et al., 2021). scalar parameter (Ma et al., 2022), or a fixed set of known goals (Yang et al., 2022).
Language-conditioner RL. Our work addresses the problem of goal-conditioned RL, where goals are expressed in natural language. While using language to specify goals is natural and broadens the range of possible goals it comes with the challenge of grounding the semantics of language in the environment state space and dynamics. Such language-instructin following agents have been widely studied in both reinforcement learning and imitation learning contexts. However, most existing methods either rely on access to an online environment for interaction (Fu et al., 2018; Bahdanau et al., 2018: 8; Jiang et al., 2019; Mirchandani et al., 2021) or require costly, goal-annotated expert datasets of offline demonstrations (Stepputtis et al., 2020; Brohan et al., 2023b;a). In contrast, our approach does not assume any anvironment-provide reward signal access real-time exploration. Furthermore, in terms of generalization to new natural language instructions, we distinguish between evaluation on instructions that simply paraphrase the training goals (Nair et al., 2022; Lynch & Sermanet, 2021) from those that represent semantically novel goals. Similar to the works of Xiao et al. (2023); Brohan et al. (2023a); Stepputtis et al. (2020); Shridhar et al. (2021a); Jang et al. (2022), our focus is on the latter, more challenging scenario.
Domain Generalization. While the previous section addressed generalization to new goals, this section focuses on generalization to novel state-action transitions. This type of generalization extends beyond goal-conditioned RL, it is essential even for single-goal RL. It has been widely studied and observed that Offline RL methods often overfit to the training distribution of state-action transitions, resulting in poor performance when the test distribution differs. Various approaches have been proposed to address this distribution shift, including regularization techniques (Kostrikov et al., 2021; Kumar et al., 2020), model-based RL. (Yu et al., 2020; Kidambi et al., 2021), and enhanced representation learning (Mazoure et al., 2021; Fan & Li, 2022). In TEDUO, we intentionally avoid domain generalization when solving the abstract MDPs in step 2 to prevent providing incorrect examples to the LLM in step 3. However, our method achieves domain generalization by leveraging the zero-shot capabilities of the fine-tuned L.M. Future work could enhance TEDUO by replacing tabular Q-learning in step 2 with method that generalizes to new state-action transitions.
A.2. Offline policy learning with minimal data requirements.
This paper focuses on realistic requirements regarding the training inputs. We work under offline setting, with limited number of unlabeled environment transitions (i.c., (ze, at Iz++1) triplets) and without any assumptions about the policy that generated the actions. To address this scenario, we employ LLMs to label the data, enabling the use of RL methods, and as an abstraction function to enhance sample efficiency. Below we discuss the related work regarding these two steps.
Hindsight labeling. Labeling data for goal-conditioned RL requires the design of reward functions for each goal. The most common approach for designing the rewards relies on handcrafted methods that are often require multiple refinements through trial and error (Knox al., 2022). With a large number of goals, manual reward design becomes infeasible. Inverse Reinforcement Learning (Ziebart et al., 2008; Fu et al., 2018) attempts to generate reward functions directly from data, but it requires a large amount of expert demonstrations. Recent studies have explored the use of LLMs and VLMs as reward functions. These methods typically involve creating a preference dataset (Klissarov et al., 2023), comparing the cosine similarity between natural language goals and state representations, or leveraging the coding abilities of LLMs (Yu et al., 2023a), especially in an online iterative fashion (Ma et al., 2023; Xie et al., 2024). These approaches are however aimed at densifying the reward signal. In contrast, our method requires generating reward labels for a large number of goals (approx. 100-1000), making the scalability of the process crucial. Therefore, we focus on generating rewards with limited number
14

---

The Synergy of LLMs& & RL Unlocks Offline Learning of Generalizable Language-Condttioned Policies with Low-fidelity Data
of LLM calls. Our approach relies on LLM-based detection of task completion, which has been proven effective by (Kwon al., 2023). We further reduce the number of LLM calls by approximating the I.MM-generatd rewards with lightweight proxy neural networks.
State abstraction. State abstraction aims to reduce the complexity of the state space by eliminating irrelevant features, thereby improving the efficiency of learning algorithms. Harly work in this area focused on state aggregation, where similar states are grouped together to form more compact representations, with state similarity defined through the transition dynamics, value- or Q-functions (Andre Russell, 2002: Li et al., 2006; Givan et al., 2003; Abel et al., 2018). Recent advancements have explored more sophisticated methods, such as deep learning- based state abstractions, employing neural networks to learn abstract representations of states (Allen et al., 2021). In this work, explore the use of LLMs to accomplish the task of state abstraction. Our approach relies on prompting pre-trained LLM to remove the features of a state that are irrelevant in solving the given goal. Such LLM-based state abstraction has been previously shown effective in the context of robotics by Peng et al. (2023) who employ LLMs to translate the language command into a binary mask highlighting the location of the goal-object.
A.3. Large Language Models for Decision Making
Decision Transformers. Pre-trained models based on the Transformer architecture have been widely used to address decision-makine problems. However, this paper does not focus on Decision Transformer (DT) models (Chen et al., 2021), Although DTs have been applied in goal-conditioned RL and IL. (Xu et al., 2022; Raparthy et al., 2023; Putterman et al., 2022), the joint modelling of goal, state, and action representations remains challenging and requires large labeled datasets. Instead of training decision transformer, this papers leverages the prior knowledge accumulated in LLMs trained on Internet data to a) enable effective use of the limited offline, unlabeled data, b) enable generalization to previously unseen goals and states.
General-purpose LLMs for decision making, Utilizing off- shell LLMs has gained significant attention due to its simplicity. In decision-making. LLMS have been used to create assistance functions within training pipelines to enrich data (Klissarov et al., 2023; Yu et al., 2023a; Ma et al., 2023; Xie et al., 2023; Laskin et al., 2022), and as high-level planners during inference to guide traditional RL policies (Shah al., 2023; Ahn al., 2022). Additionally, there has been growing interest in using reneral-purpost LLMs directly decision-making agents (Yao al., 2023b). Improving the reasoning capabilities of L.M agents is now an active research area, focusing on methods that are independent of traditional RL., These include iterative prompting techniques such as self-reflection (Ji et al., 2023), CoT reasoning (Wei et al., 2023), and integration with planning algorithms like Monte Carlo Tree Search (Pouplin et al., 2024). Nevertheless, such methods have been shown inefficient in completing complex, multi-step decision- -aaiing tasks in dynamic environments (Finn, 2024; Szot et al., 2024). To effectively use the knowledge embedded LLMs for solving RL. problems, these models need to be grounded in the dynamics of the environment.
Grounding LLMs with the environment dynamics. An LLM agent grounded in an environment can link the semantics of both observations and possible actions to its internal representation system, enabling appropriate decision-making (Carta et al., 2023; Harnad, 1990). One approach to achieve such grounding is through in-context learning. For instance, Voyager (Wang et al., 2023) pushes the concept of an L.M agent to its limits by developing an automatic curriculum for GPT-4, supported by a library of executable programs, to play Minecraft. Another method involves providing the LLM with a game manual (Wu et al., 2023). However, these approaches either rely on extensive expert knowledge, such as carefully designed prompts, on game manuals, which may not always be available. Additionally, in-context learning has limitations in data-driven scenarios, partly due to the restricted context window size, which is insufficient for incorporating entire datasets. An alternative approach involves fine-tuning LLMs to achieve grounding. Studies such as (Tan et al., 2024) and (Carta et al., 2023) use Proximal Policy Optimization (PPO) (Schulman et al.. 2017) to propose online fine-tuning of LLMs. In the robotics domain, RT2 (Brohan et al., 2023a) demonstrates that co- co-fine-tuning on both web-scale data and expert robot demonstrations improves performance of VLMs for decision making in the context of robotics. Our method differs from previous work by significantly lowering the requirements on input data, as we do not need online interaction or labeled expert demonstrations. Furthermore, while RT2 implements co-fine-tuning, our method utilizes an off-the-shelf pre-trained LLM, which is then fine-tuned.
15

---

The Synergy of LLMss & RL Unlocks Offline Learning of Generalizable Language-Condttionee Policies with Low-fidelity Data
B. Additional Results
B.1. Webshop results
Main Results. The results averaged over 200 instructions are shown in Table B.1. Unlike Minigrid, Webshop does not feature multiple environment types, prohibiting generalization evaluation across environments. The scores represent the true environment reward (scaled to [0, 100]).
Table B.I. Online evaluation of gemeralization performance for the Webshop environment. Results averaged over 200 instructions.
Method
Goals
Score
Episode Length
ReAct-Llamm-3-88
Training/Testing
8.4 (+0.8)
14.1 (+0.1)
RcAct-Llama-3-70B
Training/Testing
13.8 (+0.9)
14.2 (+0.1)
TEDUO-LLama-3-8D
Training
52. (+0.6)
6.6 (+0.1)
TEDUO-LLama-3-8B
Testing
44.4 (+0.6)
6.9 (+0.1) I)
This benchmark demonstrates that TEDUO successfully learns from unlabeled trajectories in the Webshop environment. TEDUO significantly outperforms the ReAct prompting approach, particularly when applied smaller models (compared to GPT-3.5 in the original paper). While ReAct prompting provides some improvement over standard prompting, its performance remains limited, whereas TEDUO achieves substantially better results.
Table B.2. Ablation study for the Webshop environment. Results averaged over 200 instructions.
Method
Goals
Score
Episode Length
Data collection policy (LLM-random)
Training/Testing
5.6 (a0.5)
12.3 (a0.2)
Step 1 (GCRL)
Training
56.2 (+0.6)
6.9 (+0.1)
T'DDUO-Llama-3-8B
Training
52.1 (+0.6)
6.6 (+0.1)
TEDUO-LLaam----8
Testing
44.4 (40.6)
6.9 (+0.1)
Ablation study: Similar to the approach in the paper, table B.2 provides an ablation study below that summarizes the performance improvements achieved after each TEDUO step. The results demonstrate that TEDUO Steps I & 2 effectively produce improved policies over the data collection policy for the training goals. Furthermore, TEDUO Step successfully achieves its generalization objective, extending the learned policy's s performance to new instructions. Compared to the BabyAl environment, the fine-tuned LLM does not outperform the Q-learning policies trained on the training instructions. This limitation can be attributed to the lack of diversity the initial states. In the Baby AI environment, agents can start from any grid position, including configurations unknown to the tabular Q-learnig policies, often resulting in failure trajectories. In contrast, the fine-tuned LLM demonstrates successful generalization in such cases. However, the webshop environment presents a single initial state-thh the search bar webpage. To surpass the Q-learning policies in this setting, the LLM would need to contradict its learned behaviors, making superior performance unattainable.
16

---

The Synergy of LLMss & RL Unlocks Offline Learning of Generalizable Language-Conditionee Policies with Low-fidelity Data
B.2. The impact of the data collection policy
We examine the effect of the data collection policy on our pipeline's performance. Specifically, we demonstrate that our pipeline remains effective irrespective of the optimality of the data collection policy with respect to the set g
Given the observational dataset D collected under a policy x², let GP represent the set of goals corresponding to goal states that have been visited in D. This set is defined as:
g = [g E 9 3(,, s.t. 2,(2, =1.
(I)
We can measure the alignment between the dataset DD and the training goals Ger by the size of G ng j.ee the set of goals from G that have been visited in D. A key point is that, in step 2 of TEDUO, we cannot generate a policy σ for any goal 9 not present in GP_ As discussed in section 5.4, the performance of the fine-tuned LLM depends on the size of the synthetically generated dataset DSFT, making |GP n G an import ant metric for evaluating the fidelity of our training inputs: and G
To empirically analyze this, we consider two randomized policies:
A) Goal-oriented policy: This is the policy used for data collection in the main experimental section. For each trajectory, a random goal from a set of goals Gd is drawn and the agents acts according to the goal-oriented policy provided in the Baby Al environment in order to achieve it. This policy simulates agents attempting to accomplish multiple task within the environment. Examples of real-world unlabeled data that could be generated from such policy include CCTV footage of employees at work, logs of medical procedures performed on patient, or YouTube videos.
B) Random policy: Actions are drawn uniformly at random from the action space. This policy represents agent that explores the environment without a specific goal. Although this scenario is less common in real-world settings-where agents typically pursue objectives-it remains applicable to batch RL, particularly when learning from untrained agents with no prior knowledge.
1.0
Goal
1.0
5
8o hall
LS (min max staled
0.8
False
a key
0.6
go to yellow ball
0.6
pick nod bey
pick purple ball
0.0
056
collected
25%
50%
goal-orismtad policy
75%
100%
# 5x
c 6.
9% of colllcttedd with the goal-criented policy
25
50
(a)
(b)
Figure B. Impact of data collection policy. The x-axis shows the proportion of data D collected with policy A vs. policy B for a fixed size of 'D. a) The y-axis shows (88). i.c. the number of unique initial abstract states o for which g is reachable with the learned policy =", Values are min-max normalized across all 5 mixture policies. b) The y-axis shows the same values as plot a), averaged across 14 goals, bars represent the standard error.
The metric of fidelity. The likelihood of the final fine-tuned LLM in solving given goal g from any starting position is directly linked to the fidelity of the corresponding Q-learning policy, T9. Thus, we chose to measure the fidelity of our starting point inputs by computing the number of initial abstract states sÿ. from which g can solved using T We define a goal g as reachable from oi if by iteratively selecting the actions at arg max 7 9 (-(sg). ) the agent can eventually reach g. We denote this number of states with | The set of candidate initial states to compute |S is the entire abstract state space.
Figure 1.I illustrates that the optimal data collection policy varies by goal. For some goals policy A works better, while for others it is the fully random policy. Importantly, a comparable amount of synthetic action sequences for fine-tunning the LLM can be extracted using either policy A or B. Averaging across all goals, we find that policy A tends to perform better for goals in 9. than those not Gx.
17

---

The Synergy of LLMSs & RL Unlocks Offline Learning of Generalizable Language-Condttione Policies with Low-fidelity Data
Explanation: We note that only subset of initial states o has been used during data collection. Using goal-oriented policy to collect trajectories results in limited exploration of the state space. Consequently, the learned policies x expected to excel at solving when started from state that has been visited in D, but are not guaranteed to succeed when started from si not visited in D. This is why, including more exploration during data collection can help in ensuring that the learned policies can reach their respective goals from anywhere in the state space. However, relying solely on random exploration is only effective for simple goals that are likely to be reached by chance. Instead, in our main experiments, we use a goal-oriented collection policy that uniformly samples across goals of varying difficulty, This results in a more diverse dataset that includes complex behaviors and better reflects realistic settings, where completely random agents don't 't' exist.
Future work could explore optimizing the set of training goals g maximize the alignment of G with a given dataset D. Yet, the necessity of aligning DD and G is moderated by two factors. First, as shown in subsection 5.4, the abstraction function reduces the complexity of the abstract MDPs, requiring fewer data samples. Second, since extending the list of goals in G is computationally inexpensive, we can continually seek better alignment.
18

---

The Synergy of LLMss & RL Unlocks Offline Learning of Generalizable Language-Conditionee Policies with Low-fidelity Data
B.3. Abstraction function
Figure B.2 presents the performance of the abstraction function used in the Baby AI environment.
go to red key
pick blue ball
put purple ball
yel key
go to a ball
pick
key
put yellow key next a ball
grey key next to purple box grey next
yellow
put a yellow key next to yellow box
go
tile (15.15)
go to the tile (17.16)
go to the tile (18.19)
go
(17.
to the tile [19,8)
go the (8.12)
[
0.00
0.50
0.75
1.00
Ratio of original abstract states
Figure B.2, Reduction in count of unique states due to applying the (g) abstraction functions and the relative size of the reduced abstract feature space Sg. containing only features necessary to identify the completion of goal g.
B.4. TEDUO step 2: Deep Q-Learning.
The step 2 of the TEDUO framework is independent of the specific offline reinforcement learning algorithm used to solve the abstract MDPs generated in step 1. Each abstract MDP is represented by a labeled transition dataset D9, consisting of tuples (s², a, 8'9, rr),, making compatible with any standard offline RL algorithm. In the implementation presented in Section 3, we used tabular Q-learning as it was most suitable for the environment under consideration. However, to illustrate the scalability of the approach, we show in table B.3 the performance of Deep Q-learning (DQN) (Mnih et al., 2013), demonstrating that it can effectively replace tabular Q-learning in our framework. The deep learning models for Q-value estimation combine CNN and dense layers, consistent with the lightweight models used for reward shaping (see Appendix Chil. While tabular Q-learming is limited to discrete state and action spaces and becomes impractical as their dimensions grow, DQN handles continuous spaces and scales effectively without such constraints.
Table B.3. Ablation study with DQN. Results averaged over 400 (g. sg) pairs.
Method
Success Rate [%]
Episode Length
Invalid Actions [%]
Step 1 + GCBC
7 (#0.6)
474 (+2.3)
11 (#0.1)
Steps 1 & 2 )
16 (+0.8)
430 (+3.9)
10 (#0.1)
Steps 1 & 2 (GCRL- DQN)
15 (+1.2)
446 (+5.9)
15 (#0.1)
All steps Llama-3-8B
65 (#1.4)
203 (+6.7)
21 (#0.7)
19

---

The Synergy of LLMSs & RL Unlocks Offline Learning of Generalizable Language-Conditionee Policies with Low-fidelity Data
B.5. Reward shaping evaluation
This section evaluates the performance of the reward-shaping step. We utilize pre-trained LLMs identify states where specific goal g is achieved. As discussed in Section 3.1.2, the large number of states (around 200k) for each goal makes direct LLM usage impractical due computational constraints. Therefore, the process is divided into two steps: (a) constructing supervised dataset by labelling subset of states (5k) using an LLM, and (b) training lightweight neural network Ro (o(:0) on this dataset.
Table B.4. Reward Shaping Benchmark, The accuracy, precision, and recall metrics are computed with classification threshold ensuring
least 95% precision.
Goals
ROC-AUG
Accuracy (%)
Precision (%)
Recall (%)
Go to box
0.90
89
96
38
Pick up a ball
0.75
98
95
83
Open door
0.92
85
95
85
Go to red door
0.98
94
100
0.2
Go to the tile (5,6)
1.0
100
100
100
Put a box next to a blue ball
0.64
100
100
25
Table B.4 shows the performance of Ra(::) for various types of goals compared to ground truth rewards. The benchmark
setup is consistent with the main experiments; details are provided in the Appendix C.5. All goals achieve 95% precision, crucial metric since false positives lead to generating incorrect data points for DSFT in TEDUO's step 2. Conversely, false negatives only reduce data points in D which is less critical given our synthetic data abundance (see Section 5.4). Performance varies across goals; for instance, "go to the red door" has low recall (0.2%), likely due to limited positive examples the dataset. Expanding the dataset could improve such outcomes.
Table B.5. I.IM-only Reward Shaping Benchmark. Accuracy, precision, and recall are computed with respect L..M-gencrated labels.
Goals
Accuracy (%)
Precision (%)
Recall (%)
Go to box
100
100
100
Pick up ball
100
100
100
Open door
100
100
100
Go green door
99
88
100
Go to the tile (5,6)
100
100
100
Put a box next to a blue ball
100
100
100
Table B.5 reports the performance of LLM-only reward shaping across a variety of goal types. The model achieves near-perfect accuracy, precision, and recall, indicating strong ability to generate consistent goal labels from environment states. The slight drop in precision for the Go to a green door task highlights occasional LLM hallucinations, though such cases are rare. Consequently, most residual inaccuracies in the overall reward shaping pipeline come from lightweight downstream models used to reduce computational overhead, rather than from the LLM-generated annotations themselves.
20

---

The Synergy of LLMSs & RL Unlocks Offline Learning of Generalizable Language-Conditioned Policies with Low-fidelity Data
B.6. Benchmark results per goal category
Table 6. Online evaluation of generalization performance split per goal category. This the success rate [%] presented in Table with the 400 (g, sū) grouped by goal category.
Method
Environ-
Goals
Pick up a X
Go to the X
Open a
Put an next
ment
to a Y
Alama-3-88 (vanilla)
train/test
train/test
11 (1
35 (#
(2 (.1)
(=00)
Llama-3-70B (vanilla)
train/test
train/test
13 (.77
33 (+
2 (2:05)
(= 0.0)
Llama-3-8B (In-context+CoT)
train/test
train/test
6 (+ 1.3)
(...))
10 1.4)
0 0.0)
Llama-3-70B (in-contex(+CoT)
train/test
train/test
9
45 1.7)
14 1.2)
0 0.0)
train train
46 (+ 2.01
92 (+ 1.2)
100 (=00)
(+ 2.9)
TEDUO: steps & 2 BabyAI-
test train
30 (+ 1.6)
58 (+ 1.7)
100 (=000)
(# 2.1)
IL--bot
train test
(a 1.2)
44 (.5).
(a 0.99
(a 0.0)
test
test
(+ 1.2)
40 (+:2.)
5 (s
(+ 0.0)
train
train
46 (.3)
85
100 (20)
test
train
39 (+4)
65 (+: 2.0)
100 (.0)
(+ 0.0)
TEDUO (Llama-3-8B)
train
test
20 (.8)
87 (+:)22
83 (# 1.8)
(= 0.0)
test
test
26 (=:0.)
70 (+))
61 (.66
(= 0.0)
21

---

The Synergy of LLMss & RL Unlocks Offline Learning of Generalizable Language-Conditionee Policies with Low-fidelity Data
B.7. Generalization to Grids of Varying Sizes
In this experiment, we fine-tune ama 3-8B- II-ttrutt using the TEDUO pipeline, leveraging training data collected from environments with grid sizes 1, 4, 6, and 9. Here, the grid size refers the number of rooms it contains. Figure B.3 illustrates example of 5 5x5 grid (size 25). We evaluate the resulting model on larger environments with grid sizes 16 (4x4), 25 (5, and 49 (7 x 7), reported in Table B.7.
fail to generalize to larger grids due to the fixed size of their convolutional layers, which constrains them to fixed grid size. contrast, the vamilla LLM exhibits consistent performance across grid sizes, albeit a lower level. The model fine-tuned with TEDUO achieves substantially improved success rates on grid sizes close those during training. This demonstrates TEDUO's ability facilitate generalization more complex environments. However, the grid size increases beyond the training distribution, the performance gradually declines, ultimately approaching that of the vanilla LLM on 7 x grids.
Classical CNN-based RL agents such as BabyAI-III-
Table B.7. Online evaluation of generalization larger grid sizes. Results averaged over 200 (g. sy) pairs.
Method
Environ- ment
Goals
Success Rate
Episode Length
Invalid Ac- tions
4x4
train/test
13 (#0.8)
905 (+6.2)
33 (+0.2)
Llama-3-8B (vanilla)
5x5
train/test
16 (+0.8)
901 (+6.3)
35 (+0.2)
7x7
train/test
19 (#0.8)
874 (#6.8)
39 (+0.1)
TEDUO: steps 1 BabyAl-IL-bot
4x4 / 5x5
7x7
train/test
(+0)
500 (+0)
NA
4x4
train
53 (+1.8)
597 (#16.1)
33 (+0.6)
4x4
test
36 (#1.6)
726 (#14.3)
38 (+0.4)
5x5
train
35 (=1.8)
739 (#16.3)
36 (+0.6)
REDUO-Llama-33
5x5
test
31 (=1.6)
757 (#13.3)
36 (+0.4)
7x7
train
19 (+1.3)
867 (=10.2)
31 (#0.4)
7x7
test
25 (+1.3)
852 (#10.1)
35 (#0.4)
Figure B.3. Example of a BubyAl grid of size 25 (5x5).
22

---

The Synergy of LLMs & RL Unlocks Offline Learning of Generalizable Language-Conditionee Policies with Low-fidelity Data
C. Experimental details
C.1. Choice of Environments
BabyAl. This paper primarily uses the Minigrid-Babyaa environment to benchmark its method. This choice was motivated by several factors. Most importantly, we require a sandbox environment in which a wide range of goal reaching tasks can be expressed in natural language. Robotic environments ((Todorov et al., 2012; James et al., 2019)) were excluded due precise control of robotic components being beyond L.M''s prior knowledge and the need to discretize continuous actions match LLM's tokenized output. Additionally, 3D environments ((Fan et al., 2022; Puig et al., 2018)) were not considered due to computational constraints. Text- bext-based games (CCôté et al., 2018; Shridhar et al., 2021b) were also excluded as they involve high-level text interactions, contrary to this paper's focus on low-level control task for language models.
Given the significant computational resources and time required to perform all three steps of our pipeline, in particular fine-tuning an LLM agent, our current scope is necessarily limited BabyAL. Nonetheless, the insights derived from this controlled setting are broadly applicable and provide a foundation for future work in environments with similar tabular structures, such NetHack ((Küttler et al., 2020)) and Overcooked ((Carroll al., 2020)), which differ mainly in thematic focus (video game dungeon crawling and collaborative cooking, respectively).
Webshop. To demonstrate the ability of TEDUO to scale beyond grid-world environments, this section proposes an additional benchmark of TEDUO in the Webshop environment (Yao et al., 2023a). The development of digital agents which are can excel at tasks performed on computers, such as web navigation, coding, and operating office software is one of the most promising applications emerging from the Mei-reeeforrement Learning synergy. Webshop is simulated e-commerce environment where an agent given product requirements must locate corresponding items by navigating a website. This environment is particularly relevant due to its dynamically evolving action space, presenting two key challenges:
Dynamic UI Actions:
The available actions vary by state due to changes in the website's UI elements. We address this challenge by concatenating the available actions with the state description. Linguistic Action Requirements: The agent must generate linguistic inputs (c.g., search queries) to use the search bar. This highlights the necessity for the fine-tuned L.M to avoid catastrophic forgetting (Luo et al., 2025) and produce coherent keywords for narrowing searches.
C.2. Guide for Practitioners
This section provides a high-level summary of the adaptations required to deploy TEDUO in a new environment. The key modifications involve adjusting the stages where domain knowledge incorporated. Specifically, the following prompts must be adapted:
1. The prompt for generating the abstraction function (Appendix C.4).
2. The prompt for reward labeling (Appendix C.5).
3. The prompt used at inference time (Appendix C.7).
At these stages, prior knowledge about the environment's dynamics, the semantics of instructions, or any contextual information that can facilitate conlicondi policy learning in Step 3 can be integrated via natural language. The provided prompt examples illustrate how such adaptations be implemented for two different environments.
Additionally, the offline Reinforcement Learning method used to solve the abstract Markov Decision Processes in Step 2 should be tailored to the characteristics of the environment. This includes considering factors such as modality, the scale of the state and action spaces, and the number of initial states.
C.3. Data Collection
As mentioned in the main body of this paper, our pipeline makes no assumption on the data collection policy used.
C. BABYAI ADAPTATION
In the experiments with the BabyAl enviromment, we rely on the default gral-oriented policies from the BabyAl environment. We denote these policies by 3 (:) Our data collection policy then random mixture of the policies x (::g). Given a
23

---

The Synergy of LLMs& & RL Unlocks Offline Learning of Generalizable Language-Condttione Policies with Low-fidelity Data
randomly sampled initial state 2i E X' and an unknown goal y randomly sampled from the of original BabyAl language commands, we let the agent interact with the environment according to x (55g) until either g is reached the limit of 500 steps is reached. This policy simulates agents attempting to accomplish multiple task within the environment. Examples of real-world unlabeled data that could be generated from such policy include CCTV footage of employees at work, logs of medical procedures performed on a patient, or You'l YouTuue videos.
Refer to Appendix B.2 for an analysis of how the data collection policy affects our pipeline, comparing goal-oriented data collection with fully random data collection.
C.3.2. WEBSHOP ADAPTATION
In the experiment with the Webshop environment, to demonstrate that our pipeline can learn from non-expert demonstrations, we design goal-conditioned data collection policy, (:: (::g), which relies on pre-trained LLM and random actions. When encountering a "search" webpage that allows keyword input for locating the target object, T (::g) prompts a pre-trained LLM with instruction g to generate relevant keywords. In all other states, x (::g) selects a random action from the set of available options provided by the environment. The pre-trained LLM is Llama-3-8B Baama----B-Instruuue with the following parameters: temperature: 1, top k: 40, maximum number of tokens: 200.
Each episode terminates either upon purchasing an object reaching the step limit of 200. Using n (:(g)) collect 5,000 unlabeled trajectories spanning 1,500 unique instructions.
C.4. Step 1: Abstraction Function
The abstraction function utilizes contextual understanding of LLMS to identify goal-relevant features. The state abstraction operator is implemented as collection of Python functions built top of the feature selection made by a prompted language model, gg) LLMabbttot (g)(-). Using LLM powered Python functions instead of directly applying the LLM to create an abstraction of each state reduces the number of LLM calls from [ to |G*r and ensures that the abstraction is consistent across all states. The prompt for generating the code includes contextual information about the environment, a list of features, and a description of the goal, instructing the LLM to create a function that removes features of a state that are irrelevant to achieving the specified goal. Additionally, our state abstraction functions separate the set of relevant features into two subsets: sg and s. that sy U sg. The features in sy are the ones which are necessary for identifying if the underlying low-level state achieves g. i.e. if x E ((g). The remaining features, which are relevant for solving the task specified by g but not strictly necessary for identifying if this goal is achieved are found in s We introduce this separation of abstracted features to even further reduce the dimensionality of the state space for hindsight labeling (see next section).
C.4.1. BABYAI ADAPTATION
To adopt the state abstraction function in for the BabyAl environment, we prompt an LLM with the given goal, a randomly sampled state representation as an example, and two in-context examples of the expected output. Figure C.4 shows the prompt template used. The LLM returns the goal relevant features which are then passed to python function that processes states according to the following rules:
Distractors identified in the selected features are labeled as either "goal object" or "goal location." "
Distractors not included in the selected features are labeled as obstacles.
Doors not referenced in the selected features are assigned uniform colors.
If all relevant objects are within the agent's current room, the environment outside the room is disregarded.
In our experiments, we the Llama-3-70B-Inate Instrucct language model with the following parameters: (temperature: 0, top k: 1, maximum number of tokens: 8000}.
C.4.2. WEBSHOP ADAPTATION
The WebShop environment includes predefined state abstractions that simplify the original HTML representation of the website. In our experiments, we utilize these abstractions, which are shared across all goals rather than being goal-specific,
24

---

The Synergy of LLMSs & RL Unlocks Offline Learning of Generalizable Language-Conditoone Policies with Low-fidelity Data
in BabyAI. Furthermore, given the demonstrated ability of LLMs parse HTML (Gur al., 2023), we confident that TEDUO Step 1 could replicate a similar abstraction. Since WebShop is non-Markovian, ensure Markovian properties by using historical states-concatenattin all states from the initial to the current state. References to "states" in the following discussion refer to these
states.
<lbegia of [sstat header idf>sysiem <|end header iou are helping a Reinforcement learning agent in the minigrid environment. Always answer as helpfully possible, while being truthful. trutbful.
hww.eee-ddlduusee end header Given grid. features and goal, you simplity the features the by detecting all geal and necessary geal location. necessary. make sure flag relevant object and
abjects related not just one.
giving you examples the
Grid : "It is 22 by 22 tiles grid. The features the eaviconment are:
The following tiles are wall: (1,2) (1,14) (2.7) (2,14) (3.7) (3,14) (4.7) (5.7) (5.14) (6.14) (7.3) (7.2) (7.3) (7,4) (7.5)
(7.6) (7
(7.10) (7.11) (7.13) (7,14)
(7.16)
(7.19) (7.20)
(10,7)
(10.14) (11.2) (11.14) (12.7) (13.2) (13,34) (14,1) (14.2) (14.3) (14,4) (14.3) (14.0) (14.7) (14.9) (14.10) (14.11)
(14.12) (14.13) (14.14) (14.16) (14,17) (14,18) (14,19) (14,20) (15.) (15,14) (16.7) (16,14) (17.)) (17,14) (18,7)
(18.141 (19.14) (20.7) (20.14)
open purple
sile
open green
file (5.8)
3. open yellow box is on tile (6.3)
4. blue box on tile (8,13)
open purple
on 7. open red box is on tile (20.19)
open grey
(18.10)
8. clased yellow door is on tile (4.14)
purple
10. lacked grey
lile (7.12)
11. closed red door is on tile (9.7)
12. closed yellow door
tile (12.14)
closed grey
(14.8)
14. closed grey
(14.15)
15. A closed red door is on tile (19.7)
16. bine key is tile (3.5)
grey key
(8.10)
18. blue key is file (11.4)
19. A purple ball is on tile (1.16)
20.
ball is tile (2.20)
(3.19)
22. red
is tile (9.12)
23. A grey ball is file (9,13)
24. yellow ball
tile (13.1)
25. grey
(13.6)
26. yellow ball on lile (17.6)
27. Inventory
The goal is "Pick up key".
Following the indications, the correct output is these simplified features ("goal object" (3,5) (11.4)
Example 2: goal
green
grey
Following the indications, the correct output is these simplified features
("goal object" (18.10),
gooa location" : (9.13) (13,5).)
Now, goal "(aoal)" and "It 22 grid. The features environment
following
(sate)
Let's think step by step- First, tell about knowledge the Minigrid/Babse BabbyAl reinforcement learning environment. Then, provide analysis environment goal. Finally. simplified features the same formal the example <leotiid b=/sstrtt.eaderr.d assisiant < beader-ee
Figure C.4, Prompt template for selecting the relevant features to achieve the goal in the Baby Al environment.
C.5. Step 1: Reward Shaping
As detailed in Section 3.1.2, the reward shaping process involves two stages.
In the first stage, a large language model LLM, here labeled goals, {(50, r9)= : r -LM (s9: g). 8 E S9) for each g E Gtr. We choose Sa as a small (up to 5000 states), diverse subset of the abstract space S" and 1 E 10, The subset $9 is chosen so that for any two abstract states, the
-OB-Inrt is utilized to generate a supervised dataset of
25

---

The Synergy of LLMss & RL Unlocks Offline Learning of Generalizable Language-Conditoone Policies with Low-fidelity Data
following liles are wall: (1.7) (1.14) (..7 (.34) (32) (3.4) (4.14) (3.7) (6.7) (6.14) (7.1) (3.2) (7.3) (7.4) (7-3) (7.6)
(7.7) (7.9) (7, (7.10) (7.11) (7,13) (7.14) (7.15) (7.16) (7.17) (7.18 (7,19) (7.20) (8.7) (8.14) (8.44))199)))) (9.7) (10.7) (10.14) (11,7)
(11,14) (12.7) (12,14) (13.14) (14.0) (14.2) (14.)) (14.4) (14.6) (14.7) (14.8) (14.9) (14,0 (14.11) (14.12) (14,14) (14,15)
(14,16) (14,18) (14.19) (14.20) (15,14) (16.7) (16,14) (1.7)
(30.7) (20.14)
open
en
(1.2)
2. open yellow box is on tile (4.9)
bine box on file (6.8)
open grey
open grey box is on lile (17.1)
open red hos is on tile (20,6)
closed blue door is tile (4.3)
door
closed purple door
(7.8)
10. A closed blue dour is on tile (9,14)
11. closed yollow door
file (13.7)
closed yellow door
13. A clused red door is tile (14.13)
14. A closed red door is on tile (14.17)
15. closed
door
tile (15.7)
grey
17. yellow key on (9.15)
18. A green key is on tile (15.5)
19. yellow key
tile (16.12)
grees
(17.15)
21. red ball is en tile (4.19)
22. A purple ball is on tile (9.5)
23. purple ball
(12.2)
(16.19)
25. Inventory
26. The agent is currently at the following tile: (6.10)
27. The facing
Figure C.5. An example of BabyAl textualized state before state abstraction.
The following tiles are wall: (1.7) (1.14) (2.7) (2 (3.7) (3.14) (4.14) (5
(6.7) (6.14) (7.1 (7 (1.3) (7.4) (7,5) (7.6)
(7.7) (7.9) (7.10) (7.11) (7,11)-(7.11) (7.13) (7,13) 07
(7
16
(7 10) (7.18)
17 (7,20)
(8.7) (8.14) 19. (10.7) (10.14) (11.7)
(11,14)
(12,14) (13,14)
(14.11)
(14,15)
(14,16) (14,18) (14.19) (14,20) (15.14) (16.7) (16.14)
(18
(19
14) (20.7) (20.14).
The following tiles are obstacles (1.2) (4.9) (16.18) (17.0) (20 (5.20)
(9. (15.5) (16.12) (17.15)
The following tiles
closed doors (6.8) (4.7) (5.14)
(6.8)
(4.7) (5.14)
07
(9.14) (13.7) (14.5) (14.13) (14.17) (15.7).
goal abject
(4.19).
A poal abject is
(9.5).
A goal abject is on the rile (12,2).
A goal object
the tile (16.19).
Inventery D..
The agent carrently
(6,10).
The agent is facing up.
Figure C.6. Textualized state from C.5 after applying abstraction for the goal "pick up ball".
26

---

The Synergy of LLMSs & RL Unlocks Offline Learning of Generalizable Language-Conditioned Policies with Low-fidelity Data
set of features relevant for goal-identification is distinct, i.e. Vs4 * $ E S9, o 62.66 This maximizes the chances of including goal-states in S9, mitigating the potential issue of generating highly-imbalances dataset for training our proxy neural networks.
In the second stage, a collection of neural networks Ro(::)): : - {0, 1} indexed by g E G is trained on the constructed supervised datasets.
C.5.1. BABYAL ADAPTATION
In our experiments with the Baby Al environment, the state-labeling LLM is configured with parameters (temperature: 0, top-k: 1, max tokens: 8000), and we use the prompt template shown in Figure C.7.
The state representations are transformed from text to a grid format. The network architecture consists of small convolu- tional neural network with one convolutional layer (output dimension: 32, kernel size: (2,2)), followed by two linear layers (hidden dimension: 32, output dimension: 1). A Sigmoid activation function is applied after the final linear layer, and ReLU used after all other layers. Dropout layers are added before each linear layer. The network is trained with the following hyperparameters: learning rate of le-5, maximum of 3000 epochs, and dropout rate of 0.1. The dataset split into training and validation sets (90% and the model weights with the lowest validation loss retained.
</begia, of heeeestttttteddrdddppp
judge good progress in Minigrid/ BabyAl reinforcement learning environment with respect to a specific COAL, Always answer as helpfully as possible. while heing truchful, simple and coneise. If you don" i know the answer to a question doe'l share false information. seeeereddleseeseee..eee..
helpful
will present you COAL to be achieved and the descriptions of STATE of "go specific location". "patting an object next another
gual are "opening ather" or picking up abject".
envirunment. Examples
First. tell me about your knowledge of the Misigrid /BayAl reinforvement learning environment related to the goal. Then, write analysis deseribing the semanties of the state strietly using information from the description and your /Maigrid/RabyAA.
knowledge
Finally , respond by explicitly declaring if the state indicates that the GOAL has been achieved at any point in the past. writing either (" ("goal achieved"; ; True), ("geal achieved"; False). If you have doubl, you could also say ("gual achieved " NA).
The environment is a 22 by 22 tiles grid. An object that has been picked up is placed in the agent inventory.
The agent object is considered
adjavent tile the object (for example. (4.2) and their Manhattan dixtance [4-5] [2-3] is strietly suporior 1}
object location if
is If the goal explicitly mentions the agent going to an object or putting an object near another object compute the Munhattan distance, show the details of the computation, explicill compare the result to and then verify your rensoning does not have mistakes and hase your decision only the Manhattan distance. Don't say they adjacent I. Don't forget check inventery. the coordinates the destination are mentioned, the agent must go to this exact tile.
(5.3)
adjacent
inventory
their Manhattan distance higher
For other types of goals, do compute them and ignore the previous paragraph.
("STATE": {
("GOAL"; (pell)<ceorsiddpp< start wtw..heeeeer..d.aainnnnnaaaee clend
Figure C.7. Prompt template for labeling states goal states not for the Baby Al environment.
C.5.2. WEBSHOP ADAPTATION
In TEDUO Step I, goal-conditioned reward labeling is performed by prompting an LLM evaluate the alignment between the instruction and the purchased product based on four criteria: category, attributes, options, and price. These criteria directly influence the true reward function defined by the environment. The resulting scores are combined using the formula specified in the WebShop environment to generate the synthetic reward. The prompt includes five example instruction evaluations (see Figure C.8). We use 1.1-a----------m- tokens: 100). Since only terminal states (i.e., post- post-purchase states) require labeling in WebShop, the computational cost of this step is significantly reduced.
with the following parameters: ftemperature: 0, top-k: 1, max
27

---

The Synergy of LLMss & RL Unlocks Offline Learning of Generalizable Language-Conditione Policies with Low-fidelity Data
<|bogia of text|><<sttrrt header header.iddpsyteeeee id]=systm <|end header You are helpful and bonest judge of the fit between instruction purchasing objeet and propesed objeet. Your evaluation based whether abject criteria: category. attributes options. price. Follow steps:
Identify Attribares and Options
List attributes mentioned
instruction.
List the options selected in the instruction.
Evaluste Each Criterios
Category: Assiga a value of the proposed object 's product entegory matches the iastruction; otherwise assign 0.
Attributes: Coust the number of attributes from the instruetion that are carrectly matched by the praposed object.
Options: Count number of options selected by
that the proposed object's 's price is lower than or equal the price specified the instruction:
matched proposed objeat.
Price: Assign value
atherwise,, asaign
,id (Example Instrctionn []]___o__
<lstari_header.ff Dssssssttt <|end, header (Exmmll []]<eett.dd
[xnser be
Instnctioon []]_]eor_dd]>>
<
[iart_header_dd332 aassiitaatt beader-iddl>- suliiddl--aapppppp Inctuctionn []]]
hendd_eadder_dd (Exumpll [Example crnera 2/]<|eot_id]]
<lend wllndddeeeae header
[start_header_id
<lend_header,
(Fxample (r coj
haarthhadderidd [eeuser-eeedddde headeree (Euample ())
<tarr
[saart header id aaassisttant </end header baraaderrddda. iiddddffiampa criteri </] id|>
start_header_it [>-serr</eed wwwdeerddddd..... (Example instuction </jot_iddd]>>
hhstart Iasssistant <lend (e-.dr.rddl///mmlleeeee
[Exmmpl S|
Cttart_hender_il [Suser headeddd ( eat,id|>
<[star_hhadde [passistant hidder-e
Figure C.8. Prompt template for synthetic reward in the Wehshop environment.
C.6. Step 2: Offline Reinforcement Learning
C. BABYAI ADAPTATION
In TEDUO's step 2, the abstract MDPs are solved using tabular Q-learning. For each goal g, a Q-value table Q9 of size | x |A| is constructed. The Q-values are updated iteratively using the Bellman equation:
Phew +- aa)Q%(ss ail (19(s arl y max Q" @71s441,
In our experiments for each environment, the learning rate a is set to 0.1, and the discount factor ry is set to 0.7. Subsequent states 81+1 are restricted to transitions observed in DD Iterations stop when | Q9 10 < where = 1 x 10-6
C.6.2. WEBSHOP ADAPTATION
In the WebShop variant, given the deterministic nature of the environment and its unique starting state, we employ an improved filtered Behavioral Cloning method solve the abstract MDPs. For each training goal g, we identify the terminal state with the highest reward 7 among the collected samples. The corresponding trajectory is then refined by removing any potential loops. Finally, trajectories with synthetic reward below a predefined threshold (0.6) are discarded.
C.7. Step 3: LLM Fine-tuning
TEDUO's step involves fine-tuning large language model using the generated supervised dataset DFF In this paper, the fine-tuned model is (aamaa---B-Insteeee We use Low-Rank Adaptation ((Hu et al., 2021)) to reduce the compute cost. The hyperparameters used for the fine-tuning step are detailed in Table C.8. The model weights with the lowest validation loss are retained. The fine-tunings have been realized on a cluster of 4 A100 (80GB VRAM). The computing power provided in figure is determined by multiplying the number of GPU hours by the peak Tflops (312 for A100 in bf16) and the estimated utilization rate (90%).
28

---

The Synergy of LLMs & RL Unlocks Offline Learning of Generalizable Language-Conditioned Policies with Low-fidelity Data
Table C.8. Fine-tuning hyperparameters
Hyperparameter
Value BabyAI
Value Webshop
Batch size (per device)
10
Learning rate
2e-5
2e-6
Maximum Gradient norm
0.3
0.3
Warmup ratio
0.01
0.01
Maximum number of epochs
3
20
LORA rank
512
512
TORA alpha
512
512
LORA dropout
0.1
0.1
Split train/val ratio
0.1
0.1
Tensor type
bf16
bfl6
</begin texi]a</start header idjasystem clend header Yol are Reinforcement learning agent is the minigrid envirenment You seleet the sequenee of optimal actions to sehieve the COAL. Always answer as helpfully as possible while being leat.id< aactarrtheeeaee he.deee jilessser
hedd_headdr.dddp.. state of the environment is gixen by the STATE. The forward
environment is by tiles grid. possible actions
right 2: direction faced by agent, 3: pick up objeet. drop object, toggle/activate as abject 6: dose completing the task }.
You only output the list of numbers associated with the optimal sequence of action achieve the GOAL.
STATE {sute)
GOAL
[poal)
ij
Figure C.9. This prompt template is employed to generate sequence of optimal actions achieve the given goal while being in the given state for the BabyAl environment.
C.8. Evaluation Setup
C.8.1. BABYAI ADAPTATION
Environments. An environment in this context refers to grid setup, which includes the arrangement of rooms, doors, and objects. The training environments consist of the grid setups included D. This implementation uses 40 distinct environments for training the model. Testing environments entirely grid setups not encountered during training. For this benchmark, we utilize 2 different grid setups for testing.
Goals. Training goals are defined as the goal contained in g",, a subset of natural language instructions provided by BabyAl without any modifications. Testing goals differ both grammatically and semantically from training goals. They are derived from BabyAl's original instructions, distinct from g"r, and reformulated using alternative phrasings and synonyms. Tables C.9, C.10, and C.11 provide the alternative formulations and synonyms for objects and colors used in these reformulations.
Baselines:
LLMs (vanilla). The vanilla Large Language Model baseline utilizes Llama-3 3-8B-Instruct or 1imma-3-70B-Instie prompted with the template shown in Figure C.9. This prompt provides basic information about the environment, current goal, and a textual (non abstracted) representation of the state.
LLMs (in-context + CoT). This baseline extends the vanilla LLM approach by using the prompt in Figure C. which includes detailed environment information, similar game manual, as described in (Wu et al., 2023). It also integrates expert demonstrations using textual grid examples and goals with their optimal action sequences. The CoT prompting technique is employed to guide the LLM through multi-step reasoning and self-reflection.
Ba-y-b---- This bascline employs the official implementation from Sndevalier-Boivved et al., 2018) using Imitation Learning (IL) with the largest default model parameters: memory dimension = 2028, recurrence = 80, batch size = 768, instruction architecture = AttentionGRU, instruction dimension 256, learning rate = 5 x 1 Training is performed on the supervised dataset DSFT from TEDUO step 2 instead of an expert demonstration dataset.
29

---

The Synergy of LLMss & RL Unlocks Offline Learning of Generalizable Language-Conditionee Policies with Low-fidelity Data
The state of the environment is givea the STATE. The environment [eav]0]] (eavil]) tiles grid. The pessible actions left object, toggle/activate right
is the direction faced agent 3:
object, drop been picked up placed
abject. 6; completing agent inventory. The agent or as object is considered at an object location if is en an adjacent tile to the abject For example, (4.2) and (5.3) are not adjacent their Machattos distance [4-5] + [2-3] = 2 strietly superiar to 1) or tile.
t.sk). object
is the inventory. sure you are facing the right direction before esing the actios "2",
the coordinates of the destination
mentioned, the agent go
You only output the list of aumbers associated with the optimal sequence of action achieve the GOAL.
help you achieving GOAL, provide examples optimal sequences actions multiple examples different examples STATE.
sERExomple
COAL {Example godl I}.
[Exmmple })
Sequence of actions (Example action I]
NuRExample
COAL: (Example goal 2}.
[Exumple state 2}.
Sequence of actions (Exumple action 2]
your knowledge reinforcement learning environment Second, explaia how you use proposed actions move around grid. Third, similar the example, output Python list that contains the sequence of action keys (1-6) chusen to achieve the goal.
present you
achieved.
{pral).
STATE
[sure):
Figure C. 10. This prompt template is employed to generate a sequence of optimal actions to achieve the given goal while being in the given uses in-context learning and CoT prompting for the BahyAl environment.
C.8.2. WEBSHOP ADAPTATION
Environment. Since the WebShop environment does not provide multiple configurations, our evaluation focuses on generalization to new goals.
Goals. The training goals gtr are subset of the natural language instructions provided by the WebShop environment, used without modification. Similarly, the testing goals form a distinct subset of instructions generated using different seeds while ensuring no overlap with g Given the grammatical and semantic diversity of the instructions, we do not perform any goal reformulation.
Baselines:
ReAct. This baseline leverages general-purpose LLMs with the ReAct prompting technique (Yao et al., 2023b), which is -fate-of-the-at for low-data settings in this environment. ReAct enhances reasoning in LLMs by integrating reasoning traces with action steps within the same prompt, enabling the model to reason through complex problems while simultaneously retrieving and validating relevant information. This feedback loop improves both accuracy and coherence. Vanilla prompting was excluded, as failed to reach the purchasing step within the step limit (15), preventing it from achieving valid rewards.
Table C.9. Alternative formultiion for the natural language commands.
Original instruction
Alternative formulation
Go to the tile (X,Y)
Move the location at the coordinate (X,Y) / Reach the position at (X.Y) Navigate to the point (X,Y)
Pick up X
Grab X Acquire X collect a X
Go to
Move to X Reach a Naviguate a
Open X
Push X open Swing open X
Put a next
X and Y next each other Position X alongside a Place beside Y
30

---

The Synergy of LLMss & RL Unlocks Offline Learning of Generalizable Language-Conditione Policies with Low-fidelity Data
Table C.10. Synonyms used for the objects.
Original word
Synonyms
Box
Container / Crate / Chest
Key
Passcode / Lock-opener/ / Unlocker
Ball
Sphere / Globe / Orb
Door
Portal Gate / Hatch
Table C.I1. Synonyms used for the colors.
Original Color
Synonyms
Blue
Azure / Cobalt / Navy
Red
Scarlet / Crimson / Ruby
Green
Emerald Jade / Time
Yellow
Golden Amber Canary
Purple
Violet / Lavender / Mauve
Grey
Ash Charcoal / Silver
31


Reviews:
[{"review_id": "kqKr1LzhZc", "reviewer": "Reviewer_azTQ", "summary": "This paper introduces TEDUO, a training framework that aims to enhance language-conditioned policy learning in autonomous agents while reducing the reliance on extensive data. The framework is structured around three key stages, each leveraging the capabilities of large language models (LLMs).\n\nFirst, data enhancement is performed by using LLMs to abstract states and apply hindsight labeling to an unlabeled dataset of state-action transitions, resulting in the creation of labeled datasets. Next, policy learning is conducted by utilizing offline reinforcement learning (RL) algorithms to develop optimal policies tailored to a finite set of training goals. Finally, the framework emphasizes generalization, where a base LLM is fine-tuned to encode knowledge of environment dynamics and optimal actions, enabling it to adapt to unseen states and interpret novel language commands effectively.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and constructive feedback. We appreciate your engagement with our work and have carefully addressed your concerns below:\n\n---\n\n### P1 Task Complexity and Scalability\n\nSee P2 in the answer to reviewer `zuF6`, including **new results on generalization from simple to complex tasks**.\n\n### P2 Scalability of Q-Learning\n\nTabular Q-learning was used in our main experiments as it suits the BabyAI environment. However, TEDUO’s pipeline is agnostic to the choice of the learning algorithm in step 2. For larger state spaces, it can incorporate more scalable offline RL methods like DQNs, which our ablation study (Appendix B.4) shows perform comparably on BabyAI. In WebShop, we instead used filtered behavioral cloning, as Q-learning would be ineffective due to the limited number of trajectories in comparison to the dimensionality of the state space. WebShop was intentionally chosen to demonstrate TEDUO’s flexibility, reinforcing that it is agnostic to the policy-learning algorithm.\n\n### P3 Trajectory Rewriting and Offline RL\n\nWhile trajectory rewriting indeed cannot introduce new exploration, development of flexible offline RL methods is critical for real-world settings where exploration is unsafe or impractical (e.g., robotics, healthcare). TEDUO’s focus on *offline* training aligns with such requirements. That said, we find the extensions of similar hybrid approaches employing LLMs as RL agents in online settings a highly promising area of research, which is beyond the scope of this work.\n\n### P4 Additional RL baselines and comparison to DeepSeek R1\n\nWe agree that comparisons to state-of-the-art baselines are essential. However, as TEDUO focuses on *offline* language-conditioned RL, to the best of our knowledge, there are no directly comparable prior works beyond the ones already included in our benchmarking. We are open to suggestions regarding additional baselines suited for the offline language-conditioned RL setting. Regarding generalization to more complex tasks, we have now extended our evaluation with a new experiment demonstrating our method’s success (see P1). Please note, testing RL agents on more complex tasks than the ones seen at training time is non-conventional and we find the presented result very exciting.\n\n**Comparison to DeepSeek R1.** The reviewer rightly notes the success of recent works employing GRPO for *online* policy improvement in the context of LLM training. However, such methods require access to real-time experimentation to enable policy rollouts. The focus of this paper is on learning from passively collected observational datasets. We find employing ideas similar to the ones observed in Deepseek R1 a very promising direction for follow-up works. \n\n- **New result:** We included a comparison of TEDUO to the baseline of prompting Deepseek R1, see P1 in the answer to reviewer `zuF6`. \n\n### P5 Question: State abstraction\n\nWe agree with the reviewer’s suggestion to clarify the role and implementation of state abstractions. State abstraction serves two purposes: (1) transforming the states from their initial modality into text and (2) filtering all irrelevant information to achieve the goal. For a web agent like Webshop or WebArena, the first purpose is achieved by transforming the HTML code of the webpage into a curated text report, where possible actions (button/search bar, etc.) are identified. The second purpose is achieved by removing irrelevant information such as advertisements or clearly irrelevant buttons etc. In our case for Webshop, the first part is natively supported by the environment, and the second part is not needed due to the noiseless nature of the provided state. \n\n### P6 Question: Direct (online) RL training\n\nTEDUO prioritizes *offline* RL because many real-world applications (e.g., healthcare, education) prohibit online exploration. While online RL excels in simulated settings, our framework addresses practical constraints. The cited work ([1]) focuses on online RL, which is orthogonal to our setting. \n\n[1] SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training\n\n---\n\nThank you for suggesting the areas for improvement, which has helped us refine and better articulate TEDUO’s contributions. Your points inspired us to:\n\n- **Push generalization further** with new results showing the promise in TEDUO’s ability to train LLM agents to generalize to new, more complex tasks.\n- **Clarify scalability** by emphasizing TEDUO’s compatibility with more scalable offline RL algorithms, like DQNs.\n- **Highlight the focus** on real-world data-constrained applications and discuss exciting areas for **future work,** extending LLM+RL hybrids to online settings.\n\nWe hope you’ll find our revisions compelling and reconsider your score. Thank you for helping us in making a meaningful contribution to the field.", "claims_and_evidence": "The claims are generally well-supported.", "methods_and_evaluation_criteria": "1. The pipeline for data enhancement is reliant on the symbolic nature of the tasks. For example, BabyAI is a easy to define symbolic task where changing the shape or color of objects and doors would result in new tasks. Also Webshop tasks are primarily focused on browsing objects, with actions mainly limited to search [A] and buy [A]. However, this would be more difficult to enhance data for more complex environments, e.g. Webarena, Osworld.\n\n2. Also, the method lacks scalability due to reliance on memory-intensive tabular Q-learning. It would suffer if the task is very complicated and requires many steps to finish.\n\n3. Rewriting trajectories as data enhancement is very limited due to the extent of change you could perform when rewriting trajectories. For example, no additional exploration and trial and error are added.", "theoretical_claims": "No theoretical claims are involved.", "experimental_designs_or_analyses": "1. More experiments on complicated tasks (with diverse goals, and larger action space) to prove the applicability of this method is necessary. Please see Methods for details. \n\n2. Comparison to more simplified RL methods. e.g. Deepseek R1 like methods that directly use a task success verifier as reward to train the policy model.", "relation_to_broader_scientific_literature": "The method proposed by the paper is novel, yet it's conclusion is not solid due to lack of comparison on more complicated tasks and up-to-date baseline models. Also the improvement in generalization ability is limited to generalization to similar environments and lacks discussion of generalization to more challenging tasks (e.g. pick up red ball -> pick up green ball, v.s. pick up red ball -> boss level in BabyAI).", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Could you elaborate on how you could create abstract states and templates for a quite complicated task, e.g. webarena? Note that the website for browsing could be written as abstract state, but the next state after clicking on the website could be very different. \n\n2. Direct RL training has been shown more effective in generalization than trajectory SFT in recent work [1]. Why is it necessary to use abstract template generated trajectories for SFT rather than directly perform RL training. \n\n\n[1] SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NTbah2Pspa", "reviewer": "Reviewer_vbjR", "summary": "The paper studies the problem of training generalizable RL policies with the help of language models. RL policies can generally achieve impressive performance given enough exploration/coverage over the (state, action) space, but if the RL policy networks are trained from scratch on a particular environment, they can perform very poorly at new environments. On the other hand, pre-trained large language models show remarkable capabilities for generalization and are becoming more and more popular at decision making tasks. \n\nThis paper provides a recipe for training a language model on decision making tasks. Specifically, the paper uses LLMs to convert classical RL tasks into a text based representation and constructs solvable MDPs, next it uses tabular Q learning to solve these MDPs and learn optimal actions, and finally teaches an LLM these optimal actions for a given set of states and goals. The paper demonstrates that the resulting LLM learns generalizable strategies that can then transfer zero-shot to similar but unseen environments.\n\n# Update after rebuttal\n\nI like the paper's method and results --- and also the authors answered my concerns during the rebuttal process. \n\n**I maintain my score at 4. I think this is a strong paper, and would strongly recommend its acceptance**.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your enthusiastic review and constructive feedback! We’re delighted by your positive assessment and have addressed your questions below to strengthen the manuscript further.\n\n---\n\n### P1 Task Complexity\n\nSee P2 in answer to reviewer `zuF6`, showing **new results** on generalization from simple to more complex tasks.\n\n### P2 Questions: Base vs. instruction-tuned models\n\n1. We used **Llama-3-8B-Instruct** throughout all experiments (Appendix A.1), which we will clarify in the main text to avoid confusion.\n2. Since we already use the instruction-tuned version, this question does not apply.\n3. Similarly, since our experiments already use the instruct model, we think a comparison with the base model is not necessary.\n\n### P3 Invalid action rate\n\nPure RL policies (e.g., GCRL, tabular Q-learning policies) are designed to only take actions that have been seen in the training dataset (unless they are in a previously unseen state where the action is taken at random), minimizing invalidity. In contrast, in TEDUO, the LLM’s prior over the action space (which may contain invalid actions) is merged with the learned policies via SFT. While, on the one hand, this enables the LLM to flexibly generalize to new tasks and environments, it may occasionally lead to proposing invalid actions. As demonstrated in our experiments, the invalid action rate increases as the inference setting diverges from the training distribution; a non-fine-tuned LLM has a very high rate of invalid actions. This reflects a trade-off between generalization and memorization of the observed behaviors. When the agent is in a previously seen state, a potential workaround could integrate action masking by pruning options deemed invalid by the Q-learning policies. **Action:** We will discuss these observations in the limitations section of the revised manuscript, particularly in relation to high-stakes environments where safety is a concern.\n\n### P4 Paper Title\n\nWe appreciate the reviewer’s feedback regarding our title. While it was not LLM-generated (we promise!), we are open to suggestions for improving clarity in the camera-ready version.\n\n### P5 Training directly on collected trajectories vs. TEDUO.\n\nWe interpret the reviewer’s question as suggesting an alternative approach where the LLM is fine-tuned directly on observed trajectories. However, this approach has several limitations:\n\nFirstly, the original data collection policy used to generate the set of trajectories used for subsequent offline RL may be highly suboptimal with respect to the test time goals. In fact, it can even be random (see Appendix B.2. for an experiment with randomly collected observational data). \n\nSecondly, the generated trajectories a priori are not goal-labeled. While we could, in theory, fine fine-tune the LLM on sequences: $s_0, a_0, s_1, a_1, \\ldots$, the LLM could not be made aware of which goal is the given sequence solving. This challenge is addressed by the hindsight labeling step of our pipeline. \n\nFinally, skipping the policy learning stage and fine-tuning the LLM on just the trajectories matched with the goals after step 1 is expected to perform strictly worse than fine-tuning the LLM on the Q-learning policies. This is a consequence of the suboptimal nature of the data collection policies. Importantly, this Q-learning step is also the most computationally inexpensive part of our pipeline, as it does not involve any LLM calls, so including it only marginally increases the overall compute.\n\n## P6 Concurrent work\n\nRegarding the concurrent work cited by the reviewer [1], we find it an interesting idea, but we also highlight some key differences:\n\n- In the first part of this method, the collection policy (an LLM) is assumed to be good enough to generate a large number of trajectories with positive reward. (in our paper the step 2 (offline RL) goal is to obtain more positive trajectories based on the suboptimal offline data).\n- In the second part, they shift to online interaction, generating trajectories from their trained models, whereas TEDUO remains strictly offline throughout.\n- This method does not handle unlabelled observational data, while TEDUO explicitly addresses this by leveraging hindsight labeling to infer goals. As a result, the approach in [1] is not applicable to the data regimes considered in our work.\n\nTo teach the LLM [1] uses DPO (with both positive and negative trajectories in a contrastive way) while we only teach the positive trajectories via SFT. This alternative approach is interesting and could be used in TEDUO step 3. We are happy to cite this work in the related section of the updated version of our manuscript.\n\n---\n\nThank you once again for your feedback. We’re thrilled by your excitement about TEDUO’s potential and hope these revisions solidify its contribution. Thank you for your support!", "claims_and_evidence": "Yes, the paper makes sound claims supported by detailed experiments, at least according to my opinion.", "methods_and_evaluation_criteria": "Yes, the proposed methods and/or evaluation criteria makes sense.", "theoretical_claims": "The paper has no theoretical claim of note to discuss.", "experimental_designs_or_analyses": "I checked the experimental design/analysis part of this paper, it made sense to me.", "relation_to_broader_scientific_literature": "# Language Conditioned RL\n\nEarly works established that language can specify goals for RL, but they often relied on expensive data gathering or manual labeling. For an example work on using language abstraction for decision making tasks (or the general flavor of work in this area), please see [4].\n\n# LLM grounding in real tasks\n\nWith the rise of large language models, researchers began using off-the-shelf LLMs as decision-makers in interactive tasks. For example, Yao et al. (2023) proposed treating a general-purpose LLM as an agent that can choose actions by generating and evaluating plans (the ReAct framework) [2]​. Techniques like chain-of-thought prompting (CoT) and self-reflection have been shown to improve LLMs’ planning and reasoning on complex, multi-step tasks​. Despite these improvements, recent analyses (e.g. by Szot et al., 2024 [3]) found that prompting alone is insufficient for long-horizon decision-making in dynamic environments. One can use in-context learning or fine-tuning to achieve grounding in an LLM. Due to certain limitations, the authors of this paper move away from this direction, and propose the grounding via fine-tuning direction.", "essential_references_not_discussed": "Nothing that I can think of.", "other_strengths_and_weaknesses": "# Strengths\n\nI think the most important strength of this paper is that they show generalization across new environments --- a key feature that traditional RL policies seem to lack. Pretrained LLMs have world knowledge and can learn strategies from a few environment but then are able to dynamically adapt to similar but new environments. These remarkable generalization potential can lead to general purpose decision making agents. Though the paper demonstrates this in a very toy setup, I think it is a powerful result and should be studied more by subsequent work.\n\n# Weaknesess\n\n1. Experiments are in very toy setup. It would be interesting to see if these results also hold in a more versatile set of environments.\n\n2. (**Minor**) The paper title is very hard to understand and honestly, it seems to be LLM generated. Could the authors come up with a more suitable title?", "other_comments_or_suggestions": "None that I think of, please see my questions below and I would appreciate if the authors can address them!", "questions_for_authors": "## Questions about base/instruction tuned models\n\n1. Is the paper using the pretrained models (Llama-3-8B) or the instruction tuned versions (Llama-3-8B-Instruct)?\n\n2. If the paper is using pretrained models, any particular reason why instruction tuned versions were not used? Could the authors give a comparison with the instruction tuned model as well?\n\n3. It is possible that the base pretrained models are just bad at following instructions and hence we see a large performance improvement in Table 1. If one fine-tuned Llama-3-8B-Instruct instead using this paper’s method, how much improvement would one observe?\n\n## Questions about Table 2\n\nDo the authors have any insights on why the invalid actions for Llama-3-8B are double that of invalid actions? Can it be problematic in certain situations/can it be improved in some ways.\n\n\n## Question about training on these tasks directly\n\nI understand that for certain tasks, interacting with the environment or making a simulator is very difficult. But for the tasks that this paper experiments with, I imagine one could directly generate trajectories from the task and train the model on them. How would it perform compared to this paper’s complicated procedure/what benefit does this paper’s method have over that? For example, [1] (**concurrent work, so the authors need not cite it**) has similar ideas of using LLMs for general sequential decision making agents, but their pipeline seems significantly simpler compared to this paper’s method. Could the authors discuss this issue?\n\n**Overall I am excited about this paper, and happy to recommend its acceptance pending my questions above are answered satisfactorily.**\n\n# References\n\n[1] Training a Generally Curious Agent, https://arxiv.org/abs/2502.17543\n\n[2] ReAct: Synergizing Reasoning and Acting in Language Models, https://arxiv.org/abs/2210.03629\n\n[3] Grounding Multimodal Large Language Models in Actions, https://arxiv.org/abs/2406.07904\n\n[4] ELLA: Exploration through Learned Language Abstraction, https://arxiv.org/abs/2103.05825", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "788rP4F7Y5", "reviewer": "Reviewer_GwQb", "summary": "This paper introduces TEDUO, a method for fine-tuning instruction-following LLM agents using an unlabeled dataset of interactions (i.e. environment transitions without instructions or rewards).\n\nTEDUO operates in two key stages:\n1. The LLM labels the dataset by determining whether any possible goals are reached in each transition.\n2. Tabular Offline Reinforcement Learning (RL) is then applied to learn one policy per goal. These policies generate a new expert demonstration dataset, which is used to fine-tune the LLM via imitation learning.\n\nEmpirical results on BabyAI with Llama-3-8B demonstrate strong generalization and impressive performance.\n\n## update after rebuttal\nAs mentioned in my rebuttal comment, I deeply appreciate the effort put into this rebuttal, including the additional experiments and several updates to the manuscript. While I believe some empirical results—such as applying Imitation Learning directly to the demonstrations—might still be useful (even though the authors’ explanation for why this would lead to poor results is reasonable), the authors have addressed most of my concerns. I am therefore now recommending acceptance of this paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback and valuable suggestions. Below, we address each of the key concerns and clarify aspects of our work.\n\n---\n\n### P1 Hindsight Labelling\nWe acknowledge that LLM-based hindsight labeling itself is not novel, and we did cite prior work (Appendix A.2). The reviewer’s suggested references (MineDojo, LMA3) will also be added. However, these operate online, while TEDUO is fully offline. Our contribution also lies in scaling LLM-based reward functions by approximating them with NNs to reduce LLM calls. **Action:** We will refine our contribution statement to avoid overclaiming and emphasize these specific contributions.\n\n### P2 LLM as a Goal Labeler\nWe clarify that the results presented in B.5 report the accuracy of the rewards predicted by the NN reward approximators **with respect to the environment ground-truth rewards.** **Action:** We have additionally included the accuracy, precision, and recall metrics comparing LLM reward labeling to ground truth rewards. See this [link](https://imgur.com/a/Q2fixHr) for results.\n\n### P3 Appendix B.2\nWe acknowledge the reviewer's difficulty in interpreting Appdx. B.2, and we agree that the definition of $|\\mathcal{S}_0^g|$ was unclear. **Action:** We have provided a more detailed definition and the motivation behind this metric. The revisions can be found under this link: [link](https://imgur.com/a/8pGgMIx). Thank you for drawing our attention to this part!\n\n### P4 Experimental designs\n**Unlabeled Demonstrations** The claim that \"none of the experiments leverage a true dataset of unlabeled demonstrations\" is incorrect. We always act as if the data collection policy were completely unknown. Even if the policy used is a mixture of goal-conditioned policies, we do not assume knowledge of which trajectory has been generated with respect to which goal. \n\n**Semantically Distinct Goals** We define two \"semantically distinct\" instructions as two goals with different goal states, rather than paraphrased versions of the same task. (E.g., ”Pick up a red box” is distinct from “Pick up a yellow key” and is semantically equivalent to “Collect a red container”). While prior works have mostly focused on paraphrased instructions, we specifically test whether TEDUO can generalize to goals that differ in key task attributes. While this may seem incremental, we note that conventional RL approaches have struggled with this form of generalization. Please also consult P2 in the answer to reviewer `zuF6` for **new results** on generalization to more complex tasks.\n\n**Construction of $\\mathcal{D}^{SFT}$.** We confirm that no additional interaction with the environment occurs in the construction of $\\mathcal{D}^{SFT}$; an empirical transition function is used. This is due to our focus on purely offline RL. \n\n### P5 Fine-tuning with Imitation Learning.\nSee P5 in the answer to reviewer `vbJR`.\n\n### P6: Directly applying Offline RL\nWe appreciate the reviewer’s suggestion of applying offline RL directly to the LLM. While promising, this poses challenges due to scale and the natural language goal-conditioned setting.\n\nILQL is an interesting adaptation of Implicit Q-Learning, but it targets standard RL tasks with a single reward function, not GCRL, which requires learning multiple reward functions and generalizing across tasks. Scaling value estimation in GCRL remains an open challenge.\n\nRecent efforts to adapt IQL to GCRL (excluding the natural language aspect) have succeeded only in maze navigation [1]. To our knowledge, ILQL has not been extended to natural language GCRL, likely due to scalability issues. We agree this is a valuable research direction.\n\n### P7 The necessity of Offline RL\nWe would like to highlight that filtered BC significantly improves performance (from an average score of 5 to 50 out of 100), demonstrating its effectiveness. A detailed discussion on the necessity of offline RL can be found in P5 in the answer to reviewer vbJR.\n\n### P8 Performance of offline RL Policies vs. Final Results\nWe would like to clarify that the average performance of the offline RL policies is low due to evaluation on *unseen initial states,* i.e. states that have not been visited in $\\mathcal{D}$.  In such states, the Q-learning policies can only take random actions, leading to poor performance. For fine-tuning the LLM agent, we only use successful transitions to perform imitation learning via SFT. That is, $\\mathcal{D}^{SFT}$ only includes trajectories that lead to goal achievement, which we can generate without access to online interaction with the environment by relying on the empirical state transition function and the learned reward functions. \n\n### P9 State abstraction\nSee P5 in the answer to reviewer `aztQ`.\n\n---\n\nWe thank the reviewer for their detailed and insightful feedback. We are confident that the revisions we plan to incorporate will further clarify and strengthen the paper and we are happy to answer any further questions you may have!", "claims_and_evidence": "One important contribution of the paper is proposing an approach to leverage an unlabelled dataset of demonstrations. This approach leverages an LLM to label data given a known set of goals. First, several prior works proposed similar approaches, such as MineDojo (Fan et al., 2022) or LMA3 (Colas et al., 2023). Then, none of the experiments really leverages a true dataset of unlabelled demonstrations as demonstrations were collected using goal-conditioned policies. In comparison, MineDojo did use unlabelled recordings of humans. Finally, as the true labels are known, several natural questions arise e.g. What is the accuracy of the LLM as goal labeller? Did the goal labeller also found intermediate goals achieved? However, no analysis is provided on this key part of the method. Appendix B.5 indicates the performance of the lightweight neural networks trained to reproduce the LLM's labelling, but the accuracy seems to use the LLM's labels as ground truth. \n\nThe authors also argue that TEDUO is very robust to the quality of demonstrations. Appendix B.2 proposes an analysis of this, and I am really struggling to understand the results. In particular, the authors seem to say that collecting demonstrations with a random policy leads to better results than using an optimal policy for some goals. This appears surprising, and I do not understand how to relate the metric studied (the \"number of unique initial states for which a goal is reachable\") to TEDUO's performance. Additionally, even though these results seem to show that using an optimal goal-conditioned policy to collect demonstrations is not optimal for TEDUO, the authors still used it in the experiments in the main paper.\n\nFinally, the authors argue that they study, in comparison to prior works, generalization on \"semantically distinct\" goals. While they do not define what they mean by \"semantically distinct\", Table B.5 hints that all types of goals were in the training set for their experiments on BabyAI, meaning that what differs between the training and test set is the name of the objects that agent must interact with (which is not different from what prior works did). Moreover, the authors argue that prior work mostly studied generalization on synonymous instructions, but they also introduce such goals by asking GPT-4 to paraphrase the original commands from BabyAI.", "methods_and_evaluation_criteria": "The baselines chosen to compare TEDUO against, as well as the ablations with TEDUO + BabyAI-IL-bot (instead of the LLM) and the ones in Section 5.2, appear well chosen to me and very insightful.\n\nI believe other key ablations would be very important. For instance, what would the performance of the LLM directly finetuned with Imitation Learning be on the demonstrations (which should be of much lower quality than the ones collected with the policies trained with Offline RL)? What would be the performance of directly applying Offline RL to the LLM based on the demonstrations instead of first learning intermediate policies? A method such as ILQL (Snell et al., 2023) could be used here.\n\nLittle is said about how the policies trained with Offline RL are used to produce the dataset for Imitation Learning. Section 3.3 seems to indicate that no interaction with the environment is performed and that an \"empirical transition function\" is used. More explanations on this part would be more than helpful.\n\nI also have concerns regarding the experiments with WebShop. In particular, Appendix C.6.2 indicates that the Offline RL part of TEDUO was replaced by \"filtered Behavioral Cloning.\" This raises a question about whether Offline RL is really necessary for TEDUO and also questions the choice of the WebShop if parts of the method are not suited for it.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "Results from Section 5.2 indicate that the policies learning with Offline RL perform poorly. Yet, using them to collect demonstrations for the LLM to perform Imitation Learning leads to impressive results. This seems quite surprising. One could say that, even though the trajectories generated by the policies are of poor quality, it is sufficient for the LLM to adapt its strategy successfully. But even more surprisingly, using these demonstrations to perform Imitation Learning with BabyAI-IL-bot also leads to good results (on the training environment/goals at least). Could the authors comment on this?", "relation_to_broader_scientific_literature": "As explained before, I believe the contribution of hindsight relabelling is overstated, given how close it is to prior works.\nApart from this, the literature is well-covered, including techniques similar but not applied in this paper, such as Inverse RL.", "essential_references_not_discussed": "I do not see any essential references that were not discussed.", "other_strengths_and_weaknesses": "I appreciated the authors performed multiple ablations and generalization experiments.", "other_comments_or_suggestions": "The state abstraction part is said to be optional, yet it is the first subsection of the method section and abstract states are often referenced. I think this could be much clearer by either not mentioning abstract states too much in the main paper or always using abstract states (and not saying this part is optional) and putting the ablation on their usefulness in the main paper.\n\nTheir are a few minor typos I spotted:\n    - l.223: \"We require a controlled with environment\"\n    - l.881: \"an import ant metric\"", "questions_for_authors": "I do not have any further questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethics_expertise_needed": ["Responsible Research Practice (e.g., IRB, documentation, research ethics, participant consent)"]}}, {"review_id": "kZVPctHDpV", "reviewer": "Reviewer_zuF6", "summary": "Existing reinforcement learning (RL) approaches often struggle to generalize to unseen goals and states. To address it, this paper propose TEDUO,  a training pipeline for offline language-conditioned policy learning in symbolic environments.  TEDUO employs large language models (LLMs) as generalizable instruction-following agents. Experimental results demonstrate that TEDUO outperforms baselines.\n\n## update after rebuttal\nI have carefully read all the reviewers' comments as well as the authors' rebuttal. Most of my concerns have been addressed. So I rise the score to 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback on our work. Below, we would like to address your questions and concerns. \n\n---\n\n### P1 Additional baselines.\n\nWe understand that a key concern in your review is the lack of additional baseline comparisons. However, to the best of our knowledge, no existing standard RL methods operate under the exact same setting as TEDUO beyond those already included in our benchmarking. Prior work typically requires either access to an online environment, something we explicitly rule out due to our focus on offline training paradigms or relies on labeled demonstration data, whereas we work with only an unlabeled set of state-action transitions.\n\n- **New result:** As suggested by reviewer `AzTQ`, we include an additional comparison with DeepSeek-R1, not available at the time of our original submission. We have incorporated its performance into our benchmark; results under this [link](https://imgur.com/a/wxLwFgh).\n\nIf the reviewer is aware of any other works that can be applied in our setting, we would be eager to include them in our comparisons to further strengthen our evaluation. We would be grateful for any suggestions! \n\n### P2 The complexity of tasks and environments.\n\nWe appreciate the reviewer’s concern regarding the complexity of our evaluation environments. BabyAI and WebShop were deliberately chosen as two distinct yet illustrative environments: a grid-based world and a text-based web interaction task. This illustrates TEDUO’s adaptability across symbolic environments of different kinds. Additionally, as detailed in Appendix C.1, WebShop presents significant challenges, such as dynamic UI-based actions and linguistic action requirements, making it a non-trivial testbed.\n\nTEDUO's modularity makes it readily generalizable to other symbolic environments, as outlined in Appendix C.2 (*Guide for Practitioners*). While environments such as RLBench habitat and Minecraft are indeed complex, they are also image-based, which diverges from TEDUO’s focus on symbolic, text-based tasks. As acknowledged in our limitations section, continuous control tasks like RLBench may not be best suited for LLM-based agents. However, we recognize the potential for extending TEDUO’s applicability via, e.g., VLMs. We leave such extensions for future work.\n\nWhile scaling to more complex settings is an important goal, it is a common practice in RL research to focus on simple environments for a detailed and interpretable analysis. Notably, many works recently published at top-tier venues still focus on toy environments. For instance, this includes the relevant related work [1] cited by reviewer **azTQ,** which uses a card game as their primary environment for experimentation, and the related works cited by reviewer **vbjR** [1,3,4], which use environments much simpler than Webshop. Our focus on these settings allowed us to better understand TEDUO’s core mechanisms, providing a foundation for future extensions to more complex settings.\n\nFinally, while BabyAI and WebShop may appear toyish, we respectfully argue that the significance of our results goes beyond the complexity of these benchmarks. To the best of our knowledge, TEDUO is the first method capable of learning generalizable, language-conditioned policies from fully unlabeled data, which we find a significant and exciting result. The presented results demonstrate the promise of combining LLMs with RL-based approaches to enable flexible generalization.\n\n- **New Results:** To deepen the argument about flexible generalization, we have performed a new experiment demonstrating TEDUO’s generalization to *larger, unseen BabyAI grids* after training on smaller grids. Performance remains robust for grids three times larger than the grids seen during training. Results are available under this link : [link](https://imgur.com/a/kjgK47J).\n\n### P4 How are the goals in $\\mathcal{G}^{tr}$ obtained?\n\nThe goals are manually designed (see  Appendix C.8.1 and C.8.2, paragraphs “Goals” for details).\n\n### P5 What is the temperature setting for the LLM?\n\nWe used the default temperature of the Llama-8B-Instruct model: 0.7 (both for TEDUO-fine-tuned and baselines).\n\nMotivated by your suggestion, we ran an additional small-scale experiment using online evaluation. Results, available at this [link](https://imgur.com/a/gxPcSbQ), show a slight improvement for temperature=1.2. We interpret this as a higher temperature promoting more diverse actions, helping the agent not to get stuck in suboptimal behaviors.\n\nPlease note, tuning of the temperature is feasible when simulators are available. In offline RL scenarios, model selection is complicated, and hyperparameter tuning may not be possible.\n\n---\n\nWe sincerely appreciate the reviewer’s time and thoughtful feedback. We hope that our clarifications and new results address your concerns and strengthen the contributions of our work. Please let us know if there are any additional points that would benefit from further elaboration!", "claims_and_evidence": "The main claims of this paper are that TEDUO can: 1) Ground LLMs for Multi-Step Decision Making, and 2) Enhance Generalization and Data Efficiency. These claims are supported by the experimental results.", "methods_and_evaluation_criteria": "The proposed method is reasonable and expected to be effective. However, in the experimental design, comparisons are only made with the non-finetuned LLM backbone and certain baselines, lacking sufficient RL-based policies as baselines.", "theoretical_claims": "The theoretical claims explained in Section 2 do not present any obvious issues.", "experimental_designs_or_analyses": "As stated in the Methods and Evaluation Criteria section, the paper lacks sufficient RL-based policies as baselines, especially prior SOTA methods on BabyAI and Webshop.", "relation_to_broader_scientific_literature": "I did not find particularly relevant literature, aside from works on large language models and reinforcement learning policies.", "essential_references_not_discussed": "The paper should introduce more RL-based policies and clearly differentiate them from the RL method used in TEDUO.", "other_strengths_and_weaknesses": "Strength\n\n1. This paper is well-written, with good presentation and clear motivation, making it easy to follow.\n\n2. The paper conducts sufficient ablation studies to support its claims and demonstrate the effectiveness of the proposed method.\n\n3. Using RL-based policies to distill LLMs is an interesting approach and may provide insights for future LLM-based policy research.\n\nWeakness\n\n1. The paper conducts experiments on relatively simple environments such as BabyAI and Webshop, which are insufficient to demonstrate the proposed method's generalization to more challenging online environments like Habitat, RLBench, or Minecraft.", "other_comments_or_suggestions": "In Figure 1, the text describing the three key steps could be arranged horizontally to improve readability and help readers quickly understand the process.", "questions_for_authors": "1. In line 078, how are these goals obtained? Are they manually designed?\n\n2. What is the temperature setting for the LLM? Would a high temperature setting affect the LLM's performance (either improving or degrading it)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces TEDUO, a training framework designed to enhance language-conditioned policy learning in autonomous agents while reducing reliance on large-scale labeled data. The framework proceeds in three stages, each leveraging the capabilities of large language models (LLMs). First, LLMs are used to abstract states and apply hindsight labeling to an unlabeled dataset of state-action transitions, generating labeled data. Second, offline reinforcement learning (RL) algorithms are employed to learn optimal policies for a finite set of training goals. Finally, the framework focuses on generalization by fine-tuning a base LLM to encode environment dynamics and optimal actions, enabling adaptation to novel language instructions and unseen states.

As noted by Reviewer vbjR and others, there are several existing approaches that use LLMs to enhance the generalization of RL policies. Beyond those cited in the paper, related methods from the Decision Transformer family—such as Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformer; Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglement—also leverage pre-trained transformers for in-context learning across tasks. Including a discussion of such work in the related literature would help further contextualize the proposed method.

Despite the breadth of existing literature in this area, the reviewers found the approach in this paper—using LLMs to transform classical RL problems into a text-based representation and construct solvable MDPs—to be novel and refreshing. The paper is clearly written and well-presented, with the main claims supported by thorough ablation studies.

A minor limitation is that the experiments are conducted in relatively simple environments with modest task variation. Nonetheless, the results are solid and convincing, and the method shows potential for broader impact. Overall, I agree with the reviewers that this is a strong submission. I recommend a strong acceptance of this work.
Decision: Accept (spotlight poster)