KALM: Knowledgeable Agents by Offline Reinforcement Learning from Large Language Model Rollouts
Authors: Jing-Cheng Pang
Si-Hang Yang
Kaiyuan Li
Jiaji Zhang
Xiong-Hui Chen
Nan Tang
Yang Yu
Venue: NeurIPS 2024 poster (2024)
Keywords: reinforcement learning
large language models
knowledgeable agents
Primary area: reinforcement_learning
TL;DR: This study investigates developing knowledgeable agents with RL and LLMs, which achieve low-level control and adapt to novel situations.

Abstract: Reinforcement learning (RL) traditionally trains agents using interaction data, which limits their capabilities to the scope of the training data. To create more knowledgeable agents, leveraging knowledge from large language models (LLMs) has shown a promising way. Despite various attempts to combine LLMs with RL, there is commonly a semantic gap between action signals and LLM tokens, which hinders their integration. This paper introduces a novel approach, KALM (Knowledgeable Agents from Language Model Rollouts), to learn knowledgeable agents by bridging this gap. KALM extracts knowledge from LLMs in the form of imaginary rollouts, which agents can learn through offline RL. To overcome the limitation that LLMs are inherently text-based and may be incompatible with numerical environmental data, KALM fine-tunes the LLM to perform bidirectional translation between textual goals and rollouts. This process enables the LLM to understand the environment better, facilitating the generation of meaningful rollouts. Experiments on robotic manipulation tasks demonstrate that KALM allows agents to rephrase complex goals and tackle novel tasks requiring new optimal behaviors. KALM achieves a 46% success rate in completing 1400 various novel goals, significantly outperforming the 26% success rate of baseline methods. Project homepage: https://kalmneurips2024.github.io.

KALM: Knowledgeable Agent by Offline Reinforcement Learning from Large Language Model Rollouts
Jing-Cheng Pang, Si-Hang Yang, Kaiyuan Li, Xiong-Hui Chen, Nan Tang, Yang Yu" National Key Laboratory for Novel Software Technology, Nanjing University, China & School of Artificial Intelligence, Nanjing University, China Polixir Technology
(pangja, yangsh, liky, chenxh, tangn ê ammdaa nju. nju.edu.cee. edu.en cn, joy@nju.edu.com
Abstract
Reinforcement learning (RL) traditionally trains agents using interaction data, which limits their capabilities to the scope of the training data. To create more knowledgeable agents, leveraging knowledge from large language models (LLMs) has shown a promising way. Despite various attempts to combine LLMs with RL, there is commonly semantic gap between action signals and LLM tokens, which hinders their integration. This paper introduces a novel approach, KALM (Knowl- edgeable Agents from Language Model Rollouts), learn knowledgeable agents by bridging this gap. KALM extracts knowledge from LLMs in the form of imaginary rollouts, which agents can learn through offline RL. To overcome the limitation that LLMs are inherently text-based and may be incompatible with numerical environmental data, KALM fine-tunes the LLM to perform bidirectional translation between textual goals and rollouts. This process enables the LLM to understand the environment better, facilitating the generation of meaningful rollouts. Exper- iments on robotic manipulation tasks demonstrate that KALM allows agents to rephrase complex goals and tackle novel tasks requiring new optimal behaviors. KALM achieves 46% success rate in completing 1400 various novel goals, signif- icantly outperforming the 26% success rate of baseline methods. Project homepage: https://kalmmeuris22222..tt.
1
Introduction
Developing knowledgeable agents that complete diverse manipulation tasks is a hallmark of machine intelligence. Reinforcement learning (RI.) has emerged as a powerful mechanism for training intelligent agents to acquire such abilities in interactive environments [ 3. In this approach, agents can learn from the environmental interaction data. Although RI. has shown promise many challenging tasks, it is limited by the scope of the available interaction data: agents often struggle to accomplish tasks not covered by the interaction data and lack the ability to generalize to new or slightly altered tasks. For instance, policy trained to move the block to the left often fails to complete the instruction move the block to the right, despite them being highly similar tasks.
On the other hand, advancements in large language models (LLMs) have opened new opportunities for developing intelligent agents that solve general tasks in text domain 5 g 9. Trained on extensive text corpora, I.LMs embed a wide array of general world knowledge. Developing approaches effectively leveraging such knowledge build knowledgeable agents for interactive and physical tasks recently presents promising research frontier [IO]. Fortunately, existing research evidence has indicated LLMs' utility beyond textual domains. They successfully utilize LLMs
"Corresponding: yuy @nju.edu.cn.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

$
d1
Large Language Model
Large Language Model
Supervised fine-tuning
Generate a rollout for the following goall. Goal: open the door, Rollout:
Explain the following rollout for Rollout:
Generate roilout for the following goal Goal; close the door. Rollout: Sq
[5g.9g:55p (2 Gg.... Explanaation
[Sp. Ca. Sq, ...]
the light...
(I) LLM grounding
(2) Rollout generation
Figure I: Illustration KALM utilizing LLM to generate environmental rollouts the numerical vector level. (I) Grounding phase that fine-tunes LLM with supervised fine-tuning on the environmental data. (2) Generation phase that prompts LLM to generate data for novel skills. KALM modifies the input/output layer of LLM, enabling it process and interpret non-textual data.
to decompose complex tasks and make high-level decisions in interactive, physical environments [2 [1]. However, they are constrained by set of available skills and gap still exists between LLMs and the environments: the LLMs are inherently text-based, while the physical environments often operate with numerical data. As result, previous methods with LLMs tend focus on high-level, text-based decision-making and do not adequately address low-level control, especially when handling numerical vector inputs and outputs.
This study investigates developing knowledgeable agents with RL and LLMs, which achieve low-level control and adapt to novel situations. We introduce KALM (Knowledgeable Agent from Language Model rollouts) method, which employs a pre-trained LLM to create imaginary rollouts for novel skills, which agent easily learn through offline RL techniques. The motivation behind KALM is that LLMs, with their extensive knowledge repository, are ideally suited for generating sequences that simulate the completion of novel goals. However, key challenge is that LLMs inherently designed to process textual data, while the environmental data is often in numerical vectors. To address this, KALM applies techniques ground L.MM and process the environmental data: (1) adapting the LLM's architecture to handle environmental states and actions, and (2) fine-tuning the LLM in a supervised manner, e.g., predict execution rollouts and translate between natural language goals and their corresponding rollouts. Subsequently, KALM uses the LLM to generate imaginary rollouts for various goals, including rephrasings of existing goals and completely novel tasks not present in the interaction data. Fig. [ depicts the process by which KALM utilizes an LLM to generate imaginary rollouts. Finally, KALM integrates offline RL techniques to acquire novel skills. This novel combination of LLM- --Mee--tettedd imaginary rollouts with offline RL potentially yields more versatile and knowledgeable agents.
We introduce an effective way to integrate RI. and LLMs for low-level control, enhancing the RL training with LLM knowledge through imaginary rollouts. Unlike previous works that primarily leverage LLM for text-based high-level control, we demonstrate that LLM knowledge can be utilized in interactive environments that operate on numerical vectors. Additionally, we present a technique for effectively aligning LLMs with environments, enabling LLMs to comprehend environmental data from different modalities. Lastly, we verify the efficacy of KALM on two robotics manipulation environments: CLEVR -oooot [14] and Meta-world [15]. Experiment results show that KALM successfully fine-tunes the LLM to generate meaningful rollouts. In the CLEVR-Robot simulation environment, it achieves 46% success rate on tasks with 1400 various novel goals, significantly outperforming the baseline offline RI. method's 26% success rate.
Our contributions are as follows:
Background and Related Work
2.1 Background
Reinforcement learning. We consider an RL problem where the agent completes goals assigned by natural language. We model the environment goal-augmented Markov Decision Process

---

(MDP) 17 [8], represented by the tuple M AS. In this tuple, S denotes the state space, A the action space. P the transition function of the environment, R the reward function that evaluates the quality of the agent's 's action, 7 the discount factor which balances the immediate and future rewards, and & the set of natural language goals. policy π S x 9 - A(A) defines the agent's strategy, mapping states and goals to a distribution over possible actions. The interaction between the RL agent and the environment proceeds as follows: at each timestep t, the agent observes state 8; and goal G' from the enviromment. It then selects an action or based on the policy T T-|18,, G). Upon executing this action, the agent receives a reward r(,,, ae, G) and the environment transitions to a state $1+1 according to the transition function P(-]s,, az). The objective of RL is to find a policy that maximizes the expected sum of rewards over time. In this study, we call the state and action data of the environment the environmental data.
Large language model. I.L.M refers to an autoregressive text generation model that predicts future tokens in sequence, where each token is predicted as 1++1 (M(Fr(o)). ... (r( )), conditioned on all prior sequence of tokens. Here, Ey is the token embedding layer that converts the token into a D-dimensional embedding, 4 E Σ, and 2 denotes the vocabulary of the LLM. Specifically, the token embedding layer converts tokens into embeddings e = ET((4) ) E IR_ and the output layer of a LLM classifies tokens. In this study, we consider LLM, which operates on textual data, environmental state, and action data. To process state and action data, we modify the LLM's architecture by replacing its original input and output layers with additional multi-layer perceptrons (MLPs), thereby enabling the integration of non- non-textual environmental data.
2.2 Related Work
enables agents to learn from static dataset of pre-collected experiences without real-time environment interaction. The core challenge in offline RL is to derive effective policies from a dataset that may be biased or has limited data. Behavior cloning (22) is a straightforward solution by mimicking the behavior present in the dataset. Besides, prior studies have introduced novel techniques such as importance sampling, conservative policy evaluation, and representation learning to address these challenges [23] 24 25 26). Trajectory
Offline Reinforcement Learning. Offline RL 19 20 21
transformer (TT) [27 treats offline RL as a sequence modelling problem, aiming to produce a sequence of actions that leads to high rewards. Unlike TT, KALM utilizes a pre-trained LLM to generate rollouts instead of as a policy. Despite offline RL's successes, it limited by the diversity of the dataset: if the dataset lacks specific experiences, the agent may fail to perform adequately in those scenarios. Model-based RL (MBRL) methods [28 29 learn a dynamic model from offline data, which can then be used by any policy learning algorithm to recover the policy. MOPO 30 and MOReL 331 use uncertainty quantification to construct lower bound, aiming to avoid issues like model bias and distribution shift. COMBO
32 employs both the offline dataset and model-generated rollouts to train a value function and regularize it on out-of-support state-action tuples. Despite both MBRL and KALM methods utilized generated rollouts for policy training, they are different from motivation. MAPLE (33] and RcDM [34] attempt to overcome this by training multiple diverse environment models to simulate wide range of environmental rollouts, thereby enhancing policy robustness in unfamiliar scenarios. These environment models, however, are typically learned from scratch and prioritize extensive data coverage, which may not align with real-world data distributions. In contrast, KALM leverages a pre-trained LLM to facilitate the generation of imaginary rollouts, taking advantage of the general knowledge from the pre- pre-trained LLMs.
Large Language Models. Large language models (LLMs) exhibit remarkable proficiency in process- ing and comprehending natural language (35] 36 37 38). More exciting, their capabilities extend to tasks beyond basic language understanding problems, including dialogue 4
[] multimodal vision-language tasks (6), logical reasoning 7 8) and mathematical pooleem-solving [40]. Recent advancements, such as GPT-4 [35), have pushed the boundaries further, not only in terms of text processing but also in exploring the capabilities of LLMs in interactive environments 4] 42 43
39
capitalizing on their embedded knowledge of the world. How to effectively utilize this knowledge for decision-make making has emerged a promising research area 44 42]. In this work, we propose a novel idea of leveraging the knowledge in LLMS as imaginary rollouts to develop knowledgeable agents.
LLMs for RL. A promising area of study is how to effectively leverage LLMs to enhance RI in interactive tasks. Research this domain has taken several approaches to leverage L.MMs. One approach is built on hierarchical framework, utilizing LLMs to decompose complex tasks and
3

---

action
(A)
(B)
Po
Real data ((5b (6t. }.,],,
Collect data
LLM grounding
Environment
Large Language Model
Offline RL.
LLM
Agent
t
Goal: pick up the cup
Rollout: [5g. Gax
Rollout generation
Pick the
Imsgginary rollout
Goal description
[C45 .ag 6
Overall framework of KALM method. (A) Overall training procedure, consisting of three key steps: LLM grounding that fine-tunes LLM with environmental data, rollout generation that prompts the LLM generate imaginary rollouts novel tasks, and offline RL training line (- ->) represents an optional online process, allowing for continuous improvement through iterative data collection and training.
Figure 2:
facilitates skill acquisition.
dashed
(B) The adapted network structure of the LLM. KALM adapts the architecture of LLM process both text and environmental data.
Another approach uses LLMs to design the reward function [46 47 48), which streamlines the otherwise laborious process of reward function formulation. Recently, some works have investigated directly employing LLMs as the behaviour policies, training them with RL to facilitate direct interaction with the environments. For example, GALM [4] and TWOSOME (49] ground LLMs in text-based games, and LLaRP 42 trains LLM to output actions for embodied tasks with RL. Different from these studies utilizing LLMSs the text level policies, in this work, propose novel approach to utilizing LLM, which extracts knowledge in the form of imaginary rollouts.
generate high-level plans, which are then executed by low-level controller
12 45
3 Method
This section presents the proposed Knowledgeable Agents from Language Model Rollouts (KALM) method. Fig. shows the framework of KALM. We first give a formal definition of the problem and then elaborate on the three critical steps of KALM: (1) LLM grounding that enables LLM to understand the elements of the environment, (2) rollout generation that generates imaginary rollouts for novel skills, and (3) Skill acquisition that trains the policy with offline RI.
3.1 Problem Formulation
Consider that we have an offline dataset D, which is collected from the environment. This dataset consists of pairs of goals and corresponding interaction rollouts: {G*, (sf og. ...)) In this context, the sequence (st ab. sf. ...). represents rollout detailing the sequence of states and actions (sf,at" required to complete the goal G*. The primary objective here is to obtain policy that achieves high rewards on unseen goal distributions, represented as G', thus ensuring its madiinnn-maingg ability beyond the constraints of the available interaction data. We will define the "unseen goal distributions' in Sec. 3.3
3.2
LLM Grounding by Supervised Fine-tuning
The first step of KALM is to ground LLM in the environment. The purpose of this module is to enable LLM to interpret the meaning of states, actions, dynamics and rollouts of the given environment. To achieve this, we train the LLM using the dataset D to perform three different tasks via supervised fine-tuning (SFT):
Dynamics prediction: The LIM predicts environmental dynamics changes. Given the current state s, and action de the LLM predicts the next state $1+1.

---

The LLM is presented with a rollout sequence So. do, and it is required to describe the rollout with natural language.
Rollout explanation:
Rollout generation: The L.M generates a rollout that aligns with specified goal G.
We can construct the supervised training data based on the data from D. Here, KALM regards problem: LLM demonstrates excellent performance following given natural language instructions to generate the desired answer. This way, we can adjust the instruction prompt input to the LLM to better utilize it and specify its generation objective. The instruction prompts we use for SFT are presented in Appendix A.4
the LLM grounding problem as an instruction-followee
Given that LLMs are initially trained on textual data to process and predict sequences of text tokens, they can not be directly utilized handle numerical vectors. To overcome this, KALM introduces modifications to the LLM's network architecture. As shown in Fig. B).. we use pre-trained I.M as the backbone model and modify with additional layers to handle environmental data. For inputs such as states and actions, we incorporate learnable embedding modules, F.s S - TRD and F - DR,,, which transform these inputs into embeddings of the same dimensionality as the token embeddings. For outputs, employ learnable modules, Og IR - S and OĄ R + A, which map the LLM's output into state space S or action space A. This framework can be easily extended to visual observation tasks by integrating appropriate neural network architectures.
3.3 Rollout Generation with Goal-oriented Prompt
After fine-tuning with environmental data, the LLM acquires the capability to interpret the states, actions, and dynamics within the environment. Given that LLMs possess broad spectrum of world knowledge, they have the potential to generate imaginary rollouts for a diverse range of novel skills. To this end, we employ the fine-tuned LLM to generate imaginary rollouts given the initial state so and the goal: {ag. $1, "....... } + M(GOP, so). Here, GOP stands for goal-oriented prompt: "Generate rollout for the following goal: [GOAL]. Rollout: ". where "[GOAL]" is a placeholder for various goals that reflect different skills. Following prior research that studies policy generalization under natural language goals
we measure the novelty of the goal along two dimensions:
Paraphrastic Robustness: This dimension assesses the agent's consistency in optimal behavior when faced with linguistically diverse goals that share the same underlying intent as previously seen goals. It includes alternative phrasings for identical actions or re-expressing the name the objects entities.
Novel Task Generalization: Here, we investigate the agent's proficiency in performing tasks that require the formulation new optimal behaviors. Such tasks do not exist in the dataset D. For instance, the dataset includes tasks related making robot walk, while novel task enables the robot to run. To generate effective rollouts for these tasks, the LLM is required to understand the entity's 's meaning and the environmental dynamics correctly.
We will elaborate how we construct novel tasks that align with these dimensions in Sec.
3.4 Offline Reinforcement Learning for Skill Acquisition
KALM employs offline RL approach to train a policy π (-|s, G). utilizing both the real and imaginary rollouts generated from LLM, with the same proportion of two sources of rollouts. To build a policy network, BERT [50] serves as the encoder for processing natural language goals due to its proficiency text encoding. The goal encoding integrated with state representations form the input for the policy network. For policy optimization, KALM is compatible with any offline RL algorithm, such as TD3+BC [5T) and CQL (23,, leveraging the combined data from the offline dataset and imaginary rollouts. Furthermore, comprehensively analyse KALM's performance when integrated with different offline RL algorithms Sec. 4.3
Experiment
In this section, we conduct experiments evaluate the efficacy of the KALM method. We aim to address the following important questions: (1) How does KALM perform novel goals compared to
5

---

Door close
Drawer open
Push
(A) CLEVR-Robot
(B) Meta-world
Figure 3: A visualization of the environments in our experiments. (A) In the CLEVR- Robot environment, the agent (silverpoint) manipulates five movable balls reach goal configuration. (B) Meta-world, the agent controls a Sawyer robot to manipulate various objects.
How are the rollouts generated by the fine-tuned LLM (Sec. 4.3)? (3) How does KALM ground LLM in the environment (Sec. 4.3 ? (4) What is the impact of each component in KALM on the overall performance of the algorithm (Sec. [4.4] We first introduce
existing baseline methods (Sec. 4.2 ?
the environment used for experiments and the specific settings employed in the evaluation.
4.1 Experimental Setting
Evaluation environments. We conduct experiments two benchmark robotics manipulation environments built based on MuloCo physics engine [52] Meta-world [15] and CLEVR-Robot 53], as depicted in Fig. 3] In the Meta-world, the agent controls Sawyer robot manipulate various objects, e.g., doors, drawers and windows. The target configurations in the offline dataset involve: reach, push, pick-place, button-press, door-unlock, door-open, window-open, faucet-open, coffee-push and coffee-button-pres arm's 's state and the different objects' location & orientation. The action space IR denoting the gripper movement and open/close.
The state space is defined in IR representing the robotics
In CLEVR- Roboo, the agent (silverpoint) manipulates five movable balls. The target configuration in the offline dataset involves moving a specific ball in target direction (front, behind, left, or right) relative larget ball. The state space defined in R representing the positions of the balls, while the action space is TR 40 where the action is a one-hot vector representing the movement of a specific ball to direction.
Novel task for evaluation. To evaluate the efficacy KALM and the learned policy's 's generalization to novel tasks, we define novel tasks with three levels of complexity as follows:
Rephrasing goal: The agent performs the same manipulation tasks as offline data but receives paraphrased goals which not present in the data. For example, the goal offline data is move the blue ball to the front of the red ball, while the paraphrased goal could be really dislike how the red ball is positioned in fromt of the blue ball. Could you exchange their places?
Unseen (easy): The agent is tasked with different manipulation tasks that do not exist in the dataset, requiring the LLM to understand the environmental data well. To correctly generate
imaginary rollouts, the LLM must understand the meaning of state, action, dynamics and their relation with the goals.
Unseen (hard): The agent faces tasks substantially different from those in the offline dataset, which require complex composition of behaviors, such as "Gather all balls together". and "Move five balls to a straight line" the CLEVR-RE Robot environment, For these tasks, LLM needs to understand the meaning of environmental data and create a novel combination of state and action to generate meaningful rollouts.
Due to the space limitation, present detailed descriptions and examples of novel goals in Tab. Before utilizing LLM to generate rollouts, we prompt ChatGPT [9] to output the natural language goals that describe different novel target configurations (which are not present in the offline dataset). Then, we prompt the fine-tuned LLM to generate imaginary rollouts for various novel goals.
Dataset collection and rollout generation. For Meta-world and LARVR-Roboo the pre-collected offline dataset consists of 100,000 rollout-goal pairs, each corresponding to the state, action, and reward sequences for completing the natural language goal. Here, we employ the environment-buil--nn reward function to obtain and incorporate rewards for offline datasets and imaginary rollouts. The
dataset encompasses 80 unique target configurations. Following prior research [13], we let ChatGPT

---

Meta- Reta-world
FLEVR-Robor 1
Rute (N)
Suncos Buts (N)
Stocess State (N)
KALM KALAC
AEAMMM KALM
KALM KALM
KALM
BI
(a) Task in offline data
(b) Rephrasing goals
(c) Unseen (casy)
(d) Unseen (hard)
Figure 4: Success rate bars of different methods on various levels goals. The x-axis denotes the offline RL algorithm, and the y y-axis denotes the success rate for completing various natural language goals. The success rate is calculated based on the average of the last five checkpoints, and the error bars stand for the half standard deviation over three random seeds. We present the training curves in Fig. g
99 generate 18 different natural language sentence patterns to describe each target configuration, resulting in 1440 different natural language goals. For the Meta-world, the dataset involves 10 different target configurations, and we employ ChatGPT to generate 20 different natural language goals to describe each target configuration, resulting in 200 different goals in total.
When KALM fine-tunes LLM, we construct a training set comprising 300,000 trajectories, each rollout-goal pair in the offline dataset extending three trajectories: dynamics prediction, rollout explanation, and rollout generation (as detailed in Sec. 3. When training policies with offline RL, baseline methods use the offline data (6400 rollout-goal pairs), while KALM generates additional 5600, 72400 and 1680 imaginary rollouts for rephrasing goal, unseen (easy) and unseen (hard) tasks, respectively. For each level of novel goals. KALM trains the policy on the offline dataset and the generated rollouts at the corresponding level, with the same proportion of offline data and imaginary rollout in each training batch. There are 1400 and 240 novel goals for CLEVR- RoRoott and Meta- Meta-world, respectively.
Implementation details. We utilize the Alama-2--7b-catte model 36 the backbone LLM across the experiments. The LLLM undergoes training for epochs with batch size 32 on Meta-world and 10 with batch size 24 on CLEVR-Robot. We implement baseline methods by utilizing d3ripy [54), a well-established code base. We utilize Adam optimizer (55) to optimize policy. All methods train policy for 500,000 gradient steps. The offline RL training is replicated with three different random seeds to ensure the robustness Appendix A.3] We implement all the above modules using 64 AMD EPYC 9374F 32- 22-Core Processor, NVIDIA GeForee RTX 4090 cards and 1TB RAM.
results. Details about hypar-parameters are provided in
4.2 Main Results
Baselines for comparison. We consider representative offline RL methods that train policies on offline data for comparison. We briefly introduce them as follows: (1) PRDC ([566) is an offline RL method regularizing the policy towards the nearest state-action pair in the offline data based on tree-search method. (2) Behavior Cloning (BC) adopts a supervised learning approach to mimic
the actions within the offline dataset. (3) Conservative Q-Learning (CQL) [23] prominent offline RL algorithm, constructs a conservative Q-function that ensures the policy's 's expected value does not exceed its true value. (4) TD3+BC [51) guides the agent stay closer to the demonstrated behavior while benefiting from TD3 (57] algorithm's 's stability and efficiency. (5) AWAC (58) weights the actions according to their estimated advantages to improve policy learning. We use the suffix '*KALM' to denote the method that trains policy on offline datasets and the imaginary rollouts. also consider baseline, LLM policy that takes advantages of both offline data and LLM. (6) COMBO 32 utilizes ensemble environment models achieve conservative policy learning.
To ensure comprehensive evaluation,
Comparison with offline RI methods. Fig. 4ggsents the comparative performance on two CLEVR-
Robot and Meta-worid. In Fig. 4(a) these methods with the 'KALM' suffix denote that the imaginary
7

---

rollouts are generated for rephrasing goals task. Overall, these methods equipped with KALM gain clear improvement over the baseline methods, especially on these tasks with novel goals (see last three columns in the figure). For example, CQL+KALM achieves averaged success rate of 27. 1 % on novel tasks in CLEVR- while the score of CQL is 15.5%. Besides, on task in offline data, KALM outperforms or is comparable to baselines, indicating that the inclusion of generated rollouts can not only preserve the performance but also potentially enhance the performance on these tasks within the real data. This could be attributed to the fact the imaginary rollouts improve the diversity of the dataset. It worth noting that baselines show an clear decrease in performance on the rephrasing goal task compared to the performance on the task in offline data. This indicates that policies trained exclusively on offline data exhibit limited generalizability to rephrasing the goals they have encountered. In contrast, policies incorporating imaginary rollouts exhibit remarkable performance enhancements over baselines (see results on rephrasing goals and unseen (easy) tasks). When the task becomes more novel and complex, the performance improveent brought by KALM gets more sig significant in improving proportion. In comparison, baseline methods hardly progress on unseen (hard) tasks, reinforcing the utilityy of generated rollouts in acquiring complex novel skills. While COMBO gets considerable score on seen tasks and rephrasing goals, it performs poorly on unseen (easy) and unseen (hard). This can be attributed to that their environment models are learned from scratch, lacking the ability to generalize novel goals and leading to the low scores on unseen tasks.
LLM as direct policy. We also consider baseline that utilizes offline data and LLM, which we call 'LLM as policy'. In this experiment, the fine-tuned LLM directly outputs the action given the
KALM
policy
Saccoss Rate: (N3
DT
goal-oriented prompt GOP (refer to Sec. (33 and previous states and
20-
actions: at MMGOP, 50 0o. ... 8z). We also consider DT [59] that utilizes both LLM and offline data, with Llama-2-7b-chate
as
the backbonn policy model. DT treats recisionnmaking as a sequence modeling problem, using a transformer architecture to predict actions based on the desired future return. We evaluate the baseline on CLEVR-Robor with the results presented in Fig, [5] The report KALM results are success rate evaluated with COL+KALM policy. We observe that KALM outperforms both LLM as policy and DT. LLM as policy yields a relatively low success rate. This result may be due to the primary focus of the LLM's fine-tuning process, which is to capture sequence information related to completing the given goal rather than to precisely predict actions, as trajectory transformers do [27]. While the action might be sub-optimal, the generated rollouts can still contribute positively to skill acquisition in offline RL. The comparison results with LLM as policy method suggest that the current approach to simultaneously modelling state and action sequences may lead to inaccurate behaviours. Future work should explore the possibility of separating the modelling of states and actions by employing two distinct LLMs, thereby enhancing the accuracy of behaviour prediction in complex environments.
Replnniig poth Uhocaa (omry) Unacen (hasd)
Figure 5: Comparison with base- line method that directly utilizes LLM as policy.
4.3 Performance of LLM Grounding
The previous section demonstrates that KALM improves policy's performance on unseen novel goals. In this section, we dive into the details of the KALM rumning process and examine why KALM can improve the policy performance. Specifically, we demonstrate the examples of the imaginary rollouts and the accuracy of the rollout explanation, which reflects the performance of LLM grounding.
Analysis on the LLM rollouts. To investigate the quality of the generated rollouts, we showcase illustrative examples of the imaginary rollouts in Fig. 6 We reset the environment to the generated state to obtain the visualization image. In the meta-world environment, the LLM generates a rollout to reach the target point behind a wall. Although the object 'wall' never occurs in the offline data, the LLM can adjust the robotics arm's 's trajectory to avoid collision. This adaptation underscores the LLM's s comprehension of the environment and ability to leverage prior knowledge. For CLEVR- Robot, the goal 'gather all balls close to green' deviates significantly from the goals in the offline dataset. Notably, the LLM correctly identifies the green ball and orchestrates the movement of the remaining balls towards this target. These results support using pre-trained L.M to generate imaginary rollouts for novel goals. We present more examples of the generated rollouts and corresponding analysis in Appendix C.2
8

---

(a) Goal: Employ the gripper tool pick the destred object and move it the intended location, sonwiihstanddnng the hindrance a wall in the patk.
green ball of the arranging
Figure 6: Examples of the imaginary rollouts generated by the fine-tuned LLM.
Task
Rephrasing goals
Unseen (easy)
Unseen (hard)
Seen lhumpt
Method
w/o rollout
83%
39%
8%
explanation
w/o dynamics 83%
50
prediction
42 %
15%
KALM
85%
36%
28 %
Table I Ablation study on training ohjective the L.L. fine- tuning. The values represent the match rate of the generated rollouts with the given goals. Results indicate that the rollout explanation term has deeper impact the generation.
Epoch
Figure 7: Rollout explanation accuracy.
Accuracy of the rollout explanation. We measure the LLM ground in terms of LLM's 's ability to explain environmental rollouts, where the LLM is asked to explain the rollouts with both seen prompt (Appendix A.4) and unseen prompt (Suppose you are playing guming of five balls with different colours. Please explain the following rollout briefly. \n rollout: [ROILOUT] Answer:(ANSWER] The accuracy of the explanation is calculated by the keyword match between the LLM output and the ground-trute label.
depicted Fig. [7] the LLM demonstrates high level of explanatory accuracy, even after only two training epochs. These results suggest that the LLM correctly captures the meaning of rollouts and retains natural language expression capability after fine-tuning, Besides, after supervised fine-tuning, the rollout explanation accuracy exceeds 85% on unseen prompts, demonstrating that the LLaMA2 model retains its language understanding capability.
4.4 Ablation Study
We conduct ablation evaluation KALM with different SFT objectives, i.e., dynamics prediction and rollout explanation (refer to Sec. 3.2) We maintain consistent hyper-parameters across experiments isolate the effects of these training objectives. The match rate between the generated rollouts and the goals measures the effectiveness of the training. Tab. illustrates that the rollout explanation objective more critical than dynamies prediction. This attributed the fact that the rollout explanation assists the I.LMM22 in capturing the temporal consistency between the rollout sequences and corresponding goal descriptions. Besides, integrating all three training objectives has yet to improve on unseen (casy) tasks. The performance degradation is attributed to the specific nature of the unseen (easy) task, whose objective is to predict one-step transitions given unseen language goals. To be more specifically, we would discuss the two components of LLM fine-tuning in KALM (i.e., dynamics prediction and rollout explanation) respectively. For dynamics prediction objective, unseen (easy) task objective (predicting and given and shares similarity, yet diverges from the dynamics prediction objective (predict given and ). This difference introduces a potential conflict the LLM SFT. This is evidenced by the empirical results that KALM w/o dynamics prediction achieves highest rollout accuracy for unseen (easy) task. For rollout explanation objective, focuses on giving explanation long rollout sequences. While this objective enriches the model's capability to

---

provide coherence over temporal sequence, it may inadvertently detract from the model's 's ability to capture the immediate logic of transitions between adjacent two steps.
Conclusion and Limitation
This study investigates the integration of RL and LLMs for low-level control in interactive, physical environments. We introduce a novel method, KALM, which bridges the gap between LLMs and environments, and extracts the knowledge from LLM in the form of imaginary rollouts. Offline RL is then applied to facilitate skill acquisition from the imaginary rollouts. The experiment results on two robotics manipulation tasks validate the effectiveness of KALM and demonstrate the feasibility of acquiring knowledge from language model rollouts. However, there are still some limitations. First, the LLM learns to generate both state and action, while the action is correlated with the behavior policy. This dual responsibility increases the burden on the LLM, requiring it to learn the environment and imitate the behavior policy simultaneously. A solution could be employing state and action modelling separately. Second, current experiments only consider the state in the form of vector. Future work can evaluate KALM on tasks with state other modalities, e.g., image, incorporating a vision encoder to handle visual data. Last, all parameters of LLM trained during the grounding phase, potentially influeneing the embedded knowledge and hindering ability of the LLM to generalize novel goals. Incorporating a general text dataset during fine-tuning could mitigate this issue. We believe these interesting directions are worth further exploration for developing smarter and more robust agents with the support of large pre-trained models.
LLMs to learn
Acknowledgment
This work is supported by Jiangsu Science Foundation (BK20243039), The authors extend their appreciation to Chengxing Jia and Yidi Wang for their detailed discussions the implementation details, and anonymous reviewers for providing valuable comments during the reviewing process.
10

---

References
[1] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schritwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mas- tering the game of go with deep neural networks and tree search. Nature, 529(7587)44444889 2016.
[2] Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang. Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Paine, Çaglar Gülçehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McK- inney, Oliver Smith, Tom Schaul, Timothy Lillicrap. Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft II using multi-agent reinforcement learning. Nature, 575(7782)3350-333 2019.
[3] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-55555 2015.
[4] Romal Thoppilan, Daniel De Freitas, and et al. LaMDA: Language models for dialog applica- tions. arXiv, abs/2201.08239 2022.
[5] Jing-Cheng Pang. Heng-Bo Bo Fan, Pengyuan Wang. Jiahao Xiao, Nan Tang, Si-Hang Yang, Chengxing Jia, Sheng-Jun Huang, and Yang Yu. Empowering language models with active inquiry for deeper understanding. arXiv, abs/2402.03719, 2024.
[6] Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv, abs/2306.1482) 2023.
[7] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of tf-tuuugttt prompting elicits reasoning in large language models. In NeurIPS, 2022.
[8] Jing-Chin ting-Cheng Pang. Pengyuan Wang, Kaiyuan Li, Xiong-Hur Hui Chen, Jiacheng Xu, Zongzhang Zhang, and Yang Yu. Language model self improvement by reinforcement learning contempla- tion. In ICLR, 2024.
[9] OpenAl Team. Chatgpt. https://caat.openai.comm 2023.
[10] Xiong-Hui Chen, Ziyan Wang, Yali Du, Shengyi Jiang, Meng Fang, Yang Yu, and Jun Wang. Policy learning from tutorial books via understanding, rehearsing and introspecting. In NeurIPS, 2024.
[11] Chan Hee Song. Brian M. Sadler, Jiaman Wu, Wei-Lun Chao, Clayton Washington, and Yu Su. Lim-planner: Few-shot grounded planning for embodied agents with large language models. In ICCV, 2023.
[12] Brian Ichter, Anthony Brohan, and et al. Do as can, not as say: Grounding language in robotic affordances. In CoRL, 2022.
[13] Jing-Cheng Pang, Xinyu Yang, Si-Hang Yang, Xiong-Hui Chen, and Yang Yu. Natural language instruction-followingg with task-related language development and translation. In NeurIPS, 2023.
[14] Google Research. Clevr-robot environment, htwp://wwww.......ooooooooooooee 2019.

---

[15] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In CoRI, 2019.
[16] Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley Series in Probability and Statistics. Wiley, 1994.
[17] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. IEEE Trans. Neural Networks, 9(5):1054-100 1054, 1998.
[18] Jing-Cheng Pang, Si-Hang Yang, Xiong-Hui Chen, Xinyu Yang, Yang Yu, Mas Ma, Ziqi Guo, Howard Yang, and Bill Huang. Object-oriented option framework for robotics manipulation in clutter. In IROS, 2023.
[19] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv, abs/2005.01644 2020.
[20] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In ICML, 2019.
[21] Xiong-Hui Chen, Junyin Ye, Hang Zhao, Yi-Chen Li, XuHui Liu, Haoran Shi, Yu-Yan Xu, Zhihao Ye, Si-Hang Yang. Yang Yu, Anqi Huang, Kai Xu, and Zongzhang Zhang. Deep demonstration tracing: Learning generalizable imitator policy for runtime imitation from single demonstration. In ICML, 2024.
[22] Dean Pomerleau. ALVINN: an autonomous land vehicle in neural network. In NeurlPS, 1988.
[23] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In NeurlPS, 2020.
[24] Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M. Susskind, Jian Zhang, Ruslan Salakhut- dinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. In ICML, 2021.
[25] lya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learming. In ICLR, 2022.
[26] Shengyi Jiang, ChngCh Pang. and Yang Yu. Offline imitation learning with misspecified simulator. In NeurlPS, 2020.
[27] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In NeurIPS, 2021.
[28] Fan-Ming Luo, Tian Xu, Hang Lai, Xiong-Hui Chen, Weinan Zhang, and Yang Yu. A survey on model-based reinforcement learning. Science China Information Science, 67(2). 2024.
[29] Fan-Ming Luo, Tian Xu, Xingehen Cao, and Yang Yu. Reward-consistent dynamics models strongly generalizable for offline reinforcement learning. In ICLR, 2024.
[30] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: model-based offline policy optimization. In NeurIPS, 2020.
[31] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Model-based offline reinforcement learning. In NeurIPS, 2020.
Morel:
[32] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBO: conservative offline model-based policy optimization. In NeurIPS, 2021.
[33] Xiong-Hui Chen, Yang Yu, Qingyang Li, Fan-Ming Luo, Zhiwei (Tony) Qin, Wenjie Shang, and lieping Ye. Offline model-based adaptable policy learning. In NeurlPS, 2021.
[34] Chengxing Jia, Chenxiao Gao, Hao Yin, Fuxiang Zhang, Xiong-Hui Chen, Tian Xu, Lei Yuan, Zongzhang Zhang, Zhi-Hua Zhou, and Yang Yu. Policy rehearsing: Training generalizable policies for reinforcement learning. In ICLR, 2024.
12

---

[35] OpenAl. GPT-4 technical report. arXiv, abs/2303.08774, 2023.
[36] Hugo Touvron, Louis Martin, and et al. LLaMA 2: Open foundation and fine-tuned chat models. arXiv, abs/2307.09288, 2023.
[37]
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. ACL, 2022.
[38] Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. ICML, 2024.
[39] Amelia Glaese, Nat McAleese, Maja Trebacz, and et al. Improving alignment of dialogue agents via targeted human judgements. arXiv, abs/2209. 2022.
[40] Yixin Liu, Avi Singh, C. Daniel Freeman, John D. Co-Reyes, and Peter J. Liu. Improving large language model fine-tuning for solving math problems. arXiv, abs/2310. 10047, 2023.
[41] Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre- Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. In ICML, 2023.
[42] Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Walter Talbott, Katherine Metcalf, Natalie Mackraz, R. Devon Hjelm, and Alexander Toshev. Large language models as generalizable policies for embodied tasks. In ICLR, 2024.
[43] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In ICML, 2022.
[44] Eva Eigner and Thorsten Händler. Determinants of Ilm-assisted decision-making. arXiv, abs/2402.17385, 2024.
[45] Wenlong Huang, Fei Xia, and et al. Inner monologue: Embodied reasoning through planning with language models. In CoRL, 2022.
[46] Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. ICLR, 2023.
[47] Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. In ICLR, 2024.
[48] Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Text2reward: Automated dense reward function generation for reinforcement learning. In ICLR, 2024.
[49] Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, and Bo An. True knowledge comes from practice: Aligning 1ims with embodied environments via reinforcement learning. In ICLR, 2024.
[50] Jacob Devlin, Ming-Wes Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
[51] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. In NewrIPS, 2021.
[52] Emanuel Todorov. Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, 2012.
[53] Yiding Jiang, Shixiang Gu, Kevin Murphy, and Chelsea Finn. Language as an abstraction for hierarchical deep reinforcement learning. In NeurIPS, 2019.
[54] Takuma Seno and Michita Imai, d3rlpy: An offline deep reinforcement learning library. Journal of Machine Learning Research, 23(315):1-20 2022.
13

---

[55] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[56] Yuhang Ran, Yi-Chen Li, Fuxiang Zhang, Zongzhang Zhang, and Yang Yu, Policy regularization with dataset constraint for offline reinforcement learning. In /CML, 2023.
[57] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. JCML, 2018.
[58] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv, saas/20000093553 2020.
[59] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In NeurIPS, 2021.
[60] Yizhou Wang, Yue Kang, Can Qin, Huan Wang, Yi Xu, Yulun Zhang, and Yun Fu. Momentum is all you need for data-driven adaptive optimization. In ICDM, 2023.
14

---

Appendix
Table of Contents
A Experiment Details
16
A.1 Detailed Descriptions of Natural Language Goals for Two Environments
16
Novel Goal for CLEVR-Robol
A.3 Hyper-parameters
22
A.4 Prompts for LLM Supervised Fine-tuning
22
B Algorithm Descriptions
23
Additional Results
24
Training Curves of Difterent Methods
24
More Examples Generated Rollouts
................................
24
Examples Rollout explanations
................................
27
C.4
Step Match Rate of the Generated Rollouts CLEVR Robot
..........
27
Quality of the Generated Rollouts
28
Retrievals for Notations and Abbreviations
29
Broader Impact
29
15

---

A Experiment Details
A.1 Detailed Descriptions of Natural Language Goals for Two Environments
Level of novelty
CLEVR-Robot
Meta-world
Tasks in dataset
Move specific goal to one direction of the target ball. Examples: (1) Can you push the red ball to the left of the blue ball? (2) Help me keep the green ball behind the purple ball. (3) Move the cyan ball front of the red ball. (4) Can you help me move the blue ball to the right of the red ball?
Agent completes ten different tasks (provided by Meta-world benchmark), including Reach, Push, Pick-Place, Button-Press, Door-Unlock, Door- Open, Window-Open, Faucet-Open, Coffee-Push,
Coffee-Button-Pree Examples: (1) Reach: Adjust the
gripper to the desired location. (2) Coffee-Button-
Utilize the gripper to deprexs the button on the coffee machine.
Rephrasing Goal
The task goals involve moving specific goal to one direction of the target ball, but in different expressions as in the of- fline dataset. Refer to [A22.] for rephras- ing examples of natural language goals.
Agent completes tasks in the dataset with the natural language expressions rephrasing from the tasks in the dataset. Example: (1) Push: The target ob- ject's '' present placement does not meet preferences; kindly use the grip- per to push it the desired position. (2) Window-Open: The closed window bothers me: could you please operate the gripper to open it?
Unseen (Easy)
Agent moves specific ball in given direction in one step. This task does not exist in the offline dataset but requires the agent's 's deep understanding of the natural language goal. Refer to A.2.2 for unseen-easy examples natural lan- guage goals.
Agent completes eight novel tasks, in- cluding Reach- Wall, Push- Wall, Pick- Place-Wall, Button-Press-Wan Door- Lock, Door-Close, Window-Clon Faucet-Close. Example: (1) Button- Press-Wall: Apply the gripper to ac- tivate the button, yet find its path ob- structed by an unexpected wall. (2) Door-Close: Employ the gripper shut the door.
Unseen (Hard)
Agent completes tasks substantially dif- ferent from original tasks, such as "gather all the balls together", "arrange the balls in a line." and so on. Refer to A.2.3 for unseen-hard examples of natural language goals.
Agent completes four novel tasks: Make-Coffee, Locked- Mooked--Dor-Open. Open, Hammer, and Soccer. Example: (1) Make-Coffee: Engage the gripper to manoeuvre the coffee cup beneath the spout of the coffee machine. (2)
Hammer: Utilize the gripper to grasp the hammer and strike the nail at the designated spot.
Table Detailed descriptions of the training goals and novel goals.
16

---

A.2 Novel Goal List for CLEVR-Robot
A.2.1 Rephrasing Goal
We use 40 different NL. expressions as the novel goals generated by ChatGPT express different target configuration. For example, if we take goal configuration such as "red ball and blue ball", its corresponding NL instructions (i.e., eighteen NL sentence patterns) can be one of the following:
I can't stand the red ball ahead of the blue one. Could you switch the positions of them?
The sight of the red ball ahead of the blue one bothers me. Can we reverse their order?
really dislike how the red ball positioned in front of the blue ball. Could you exchange
their places?
It annoys me to the red ball in front of the blue ball. Can we swap them around?
Seeing the red ball ahead of the blue ball fills me with frustration. Let's switch them.
The placement of the red ball in front the blue ball is something detest. Can you flip them?
can't bear to see the red ball positioned in front of the blue ball. Would you mind interchang- ing them?
It irks me to have the red ball come in front of the blue ball. Could we trade their positions?
The red ball being in front of the blue ball is something can't tolerate. Let's switch them up.
really bothers me that the red ball precedes the blue ball. Can we swap their positions, please?
Move the red ball gently to the left of the blue ball.
Slowly nudge the red ball to the left side of the blue ball.
Push the red ball towards the left of the blue ball at a leisurely pace.
Slide the red ball the left of the blue ball with a gentle touch.
Gradually maneuver the red ball the left side of the blue ball.
Hasten the movement of the red ball the left of the blue ball promptly.
Expeditiously maneuver the red ball to the left side of the blue ball.
Swiftly propel the red ball to the left, positioning it adjacent to the blue ball.
Urgently shift the red ball to the left of the blue ball with rapidity.
Accelerate the motion of the red ball towards the left the blue ball expeditiously.
Push the red sphere ahead of the blue sphere.
Drive the red orb in front of the blue orb.
Launch the red ball forward, preceding the blue one.
Catapult the red sphere ahead of the blue sphere.
Thrust the red sphere in front of the blue sphere.
Propel the red-colored orb forward, leading the blue-colored orb.
Send the red ball ahead of the blue ball.
Fling the red sphere in front of the blue sphere.
Hurl the red ball forward, preceding the blue ball.
Cast the red sphere ahead of the blue sphere.
The sight of the red ball positioned to the right of the blue ball brings me joy.
pleases me to observe the red ball situated on the right side the blue ball.
Seeing the red ball to the right of the blue ball fills me with happiness.
feel delighted witnessing the red ball located to the right of the blue ball.
the red ball positioned the right of the blue ball. I am happy to notice the red ball situated the right-hand side of the blue ball.
It brings satisfaction
17

---

Observing the red ball to the right of the blue ball brings me contentment.
am pleased by the arrangement of the red ball to the right the blue ball.
The red ball being on the right side of the blue ball gives me sense of satisfaction.
find joy in the sight of the red ball being positioned the right of the blue ball.
A.2.2 Unseen (Easy)
In unseen (easy) task, the agent needs to move one ball to a specific direction. The natural language goal can be one of the following:
Move the ball backward, it's red.
Push the red ball in reverse.
Back up the red ball, please.
Shift the red ball backwards.
Can you move the red ball backwards?
Retract the red ball, moving it backwards.
Put the red ball in backward motion.
Move the red ball back, not forward.
Send the red ball backward, if you can.
Maneuver the red ball to the rear.
Drive the red ball backward, please.
Pull the red ball back.
Make the ball move backwards.
Shift the red ball rearward.
Go backwards with the red ball.
Execute backward movement with the red ball.
Make the red ball's position backward.
Pull the red ball towards the back.
Slide the red ball backwards.
Propel the red ball backward, if possible.
Kindly relocate the red sphere towards the left.
Would you mind shifting the red orb to the left?
request that you move the red spherical object to the left.
Could you please transfer the red ball towards the left?
It would be appreciated if you could shift the red ball to the left.
Please adjust the position of the red ball to the left.
Kindly reposition the red ball to the left.
I'd like you to move the red ball to the left, please.
Please ensure the red ball is moved to the left.
Could you relocate the red ball the left?
Please shift the red ball leftwards.
Please slide the red ball over to the left.
need you to move the red ball leftward, please.
Please nudge the red ball towards the left.
Please push the red ball to the left.
Would you kindly push the red ball towards the left?
18

---

Kindly shift the red ball in the leftward direction.
Could you move the red ball to the left side?
It's required that you move the red ball towards the left.
Please execute leftward movement of the red ball.
Kindly shift the red sphere ahead.
Would you mind advancing the red orb?
Can you push the red ball onward?
Please nudge the red-colored sphere ahead.
Kindly relocate the red-colored orb forward.
Would you be so kind as to move the red-colored ball forward?
Can you shift the red-colored sphere ahead?
Please push the red-hued orb onward.
Kindly advance the red-colored ball forward.
Would you mind nudging the red sphere ahead?
Can you move the red-colored ball forward?
Please shift the red-toned orb onward.
Kindly relocate the red-hued sphere forward.
Would you be kind as to push the red-colored ball forward?
Can you nudge the red-colored sphere ahead?
Please move the red-colored orb forward.
Kindly advance the red-hued ball forward.
Would you mind shifting the red sphere ahead?
Can you push the red-colored orb onward?
Please nudge the red-hued ball forward.
Kindly relocate the red sphere to the starboard side.
Move the red orb towards the right.
Could you shift red ball to the right?
request that you move the red ball the right.
Please shift the red ball to the right.
Move the ball, which is red, to the right.
Would you mind moving the ball, which is colored red, to the right?
Kindly relocate the spherical object red hue towards the right.
Can you shift the ball, which happens be red, to the right?
I'd appreciate it if you could move the red ball to the right.
Please adjust the position of the red ball to the right.
Could you possibly move the red ball to the right?
would be great you could move the red ball to the right.
Kindly transfer the red-colored ball the right.
Move the ball that has the color red to the right.
Would you kindly relocate the ball, specifically the red one, to the right?
Please make the red ball move to the right.
Can you shift the ball that's red to the right?
Move the ball with the red hue to the right, please.
Could you adjust the position of the ball, specifically the that's red, to the right?
19

---

A.2.3 Unseen (Hard)
We designed 4 types of tasks for testing KALM's performance on the completed unseen tasks: combination of two simple tasks, combination of three simple tasks, object arrangement task, and object collection task.
NL sentence patterns used in combination of simple tasks (Using "red ball behind blue ball" as goal configuration):
1. Push the red ball behind the blue ball.
2. Move the red ball behind the blue ball.
3. Keep the red ball behind the blue ball.
4. Help me push the red ball behind the blue ball.
5. Help me move the red ball behind the blue ball.
6. Help me keep the red ball behind the blue ball.
Combination of two simple tasks: Push the red ball behind the blue ball and move the green ball behind the purple ball.
Combination of three simple tasks: Push the red ball behind the blue ball and move the green ball the left of the purple ball and keep the cyan ball in front of the red ball.
Object arrangement task
1. Place the balls horizontally, lining them up from left to right, in the order of red, blue, green, purple, and eyan.
2. Set up the balls in row from left right, with red, blue, green, purple, and cyan in sequence.
3. Arrange the balls in a line, moving from left to right, with red, blue, green, purple, and cyan.
4. Position the balls borizontally, organizing them from left to right, following the se- quence of red, blue, green, purple, and cyan.
5. Line up the balls horizontally, sequencing them left to right as follows: red, blue, green, purple, and cyan.
6. Order the balls purple, and cyan.
row from left to right, with the sequence being red, blue, green,
7. Arrange the balls in horizontal line, starting from the left and proceeding to the right, with red, blue, green, purple, and cyan in order.
8. Place the balls in row horizontally, from left to right, in the sequence: red, blue, green, purple, and eyan.
9. Set up the balls horizontally, arranging them in the order of red, blue, green, purple, and cyan from left to right.
10. Line up the balls horizontally, sequencing them from left to right: red, blue, green, purple, and cyan.
11. Position the balls in a horizontal row, ordering them left to right as follows: red, blue, green, purple, and cyan.
12. Organize the balls horizontally, moving from left to right, with the sequence being red, blue, green, purple, and cyan.
13. Place the balls in a line horizontally, arranging them from left to right, in the following order: red, blue, green, purple, and cyan.
14. Set up the balls in row horizontally, starting from the left and proceeding the right, with red, blue, green, purple, and cyan in sequence.
15. Arrange the balls in horizontal line, sequencing them left to right as follows: red, blue, green, purple, and cyan.
16. Order the balls horizontal row from left to right, with the sequence being red, blue, green, purple, and cyan.
17. Position the balls horizontally, organizing them in the order of red, blue, green, purple, and cyan from left right.
18. Line up the balls horizontally, arranging them from left to right: red, blue, green, purple, and cyan.
20

---

19. Place the balls in a horizontal row, ordering them left to right as follows: red, blue, green, purple, and cyan.
20. Set up the balls being red, blue, green, purple, and cyan.
row horizontally, moving from left to right, with the sequence
Object collection task
1. Position all the other balls around the green ball, considering it as the circle's focal point.
2. Use the green ball as nucleus of the circle, arranging the rest around it.
3. Let the green ball be the anchor of the circle, and arrange the others accordingly.
4. Make the green ball the center of attention in the circle and rearrange the others accordingly.
5. Arrange all other balls around the green one, treating the hub of the circle.
6. Centralize the circle around the green ball, repositioning the others accordingly.
7. Focus the circle around the green ball, adjusting the positions of the others.
8. Orient the other balls around the green one, treating it as the central axis of the circle.
9. Use the green ball the reference point for the circle's arrangement, positioning the others around
10. Position all the other balls around the green ball to create the circle.
11. Arrange the other balls around the green ball, making it the center of the circle.
12. Let the green ball dictate the layout of the circle, with the other balls positioned around it.
13. Create the circle with the green ball as the center, arranging the others accordingly.
14. Use the green ball as the pivot for the circle, arranging the other balls around it.
15. Organize the circle around the green ball, adjusting the positions of the other balls.
16. Centralize the arrangement of the circle around the green ball, repositioning the others.
17. Treat the green ball as the central node of the circle and arrange the other balls accordingly.
18. Position all the other balls around the green ball, with it the focal point of the circle.
19. Arrange the circle with the green ball the center and the others positioned around
20. Base the arrangement of the circle on the green ball, repositioning the others accord- ingly.
21

---

A.3 Hlyper-parameters
The nyper-parameters for implementing KALM are presented in Table 3 parameters, offline RL algorithms, and policy network architecture when implementing baseline methods.
We use the same hyper-
Table 3: Hyper-parameters.
Hyper-parameters
Value
Discount Factor
0.99
BC Batch Size
100
BC LR
le-3
BC Imitation Weight
0.5
CQL Batch Size
32
CQL LR
6.25e-5
CQL Conservative Weight
10.0
TD3+BC Batch Size
256
TD3+BC Actor LR
3e-4
TD3+BC Critic LR
3e-4
TD3+BC Alpha
0.5
AWAC Batch Size
1024
AWAC Actor LR
3e-4
AWAC Critic LR
3e-4
AWAC Lambda
1.0
Feature Extractor Net
[[S] + [4], 256, 256), ReLU
A.4 Prompts for LLM Supervised Fine-tuning
Dynamics prediction: You are an expert in identifying environmental dynamics change. Current state is [st], after executing action [ac], get next state: [ANSWER).
Rollout to goal translation: Translate the statelaction rollout to textual goal. 'n Roll- ou::RROLLUUT] Goal: [ANSWER].
Goal to rollout translation: Translate the textual goal to state/action rollout, n Goal:[G]. Rollout: [ANSWER]
Here, [ANSWER] is the content that LLM should generate.
22

---

B Algorithm Descriptions
Algorithm and present the procedures of the LLM grounding, Rollout generation and Offline RL training, respectively.
Algorithm LLM Grounding with Supervised Fine-tuning.
Input: Offline dataset D {G*; (shi ab st a.. M= and pre-trained LLM Mg.
1: Construct the SFT dataset D' based on D following method described in Sec. 3.2
2: while training not complete do
3: Sample batch of data from D'
4: Update to minimize the SFT loss, MSELoss for continuous data, CrossEntropyLoss for discrete data.
5: end while
6: return the fine-tuned L.LM Ma-
Algorithm 2 Imaginary Rollout Generation.
Ad Input: A set of predefined novel goals: D = (
Jm=1 and the fine-tuned LLM Mg.
1: Initialize the dataset for storing imaginary rollouts D = 0-
2: for each goal G in D do
3: Sample an initial state so randomly from the initial state distribution.
4: Generate imaginary rollout (so, do $1, a.,-..) based G and so with M, using goal-oriented
prompt.
5: Append (G, (So. ap, $1 ... }} to D"
6: end for
7: return imaginary rollout dataset D
Algorithm 3 Training procedure of the offline RL policy.
Input: Offline dataset D {G*, (sf at r a rt. ................... K imaginary rollout dataset
Dm (G (sø ap si aT ))) m= and text encoding model Mg.
1: Initialize the complete dataset D (..
2: for {G, (Sp. ap, $1, 11,00.1................ :) }} in D DUD do
3: Encode the natural language goal G, to encoding Gy using Ma.
4: Update each state $, = [st, Gyl-
5: Compute rollout rewards (ro, "1, using with reward function.
6: Append ((50, 4o, ro, 81, a1 Ti, ).}} to Dc.
7: end for
8: Initialize the policy æ.
9: while training not complete do
10: Sample a batch {(*,, a,, r3) from Dc.
Update the policy with offline RL algorithm.
12: end while
return the optimized policy n.
23

---

Additional Results
C.1 Training Curves of Different Methods
Meta- Meta-worid woold
BCHOSCH
U.B
TOH PUHECCIAAN
ESHOCHAAM
OqU OQUEAUR
QE-EALM
DOTEALM
Speck
Spab
CLEVR- CLEVR-Robot Robott
Statooa Tlae (54)
CCC-
OCL
CCL
OLt
Epcc
Ryads
Epeit
Spach
(a) Task in offfine data
(b) Rephrasing goals
(c) Unseen (casy)
(d) Unseen (hard)
Figure 8: Success rate curves of different methods on various levels of goals. The x-axis denotes the training epochs, and the y-axis denotes the success rale for completing various natural language goals. The shaded area stands for the half standard deviation over three random seeds.
C.2 More Examples of Generated Rollouts
The generated imaginary rollouts are important for policy optimization [60] We present additional examples of imaginary rollouts generated by LLM in Fig. [ T0 and II Overall, the generated imaginary rollouts conform to the basic physical rules of the world: LLM can correctly capture the novel goals' objects, directions and meaning.
For CLEVR-Robot environment: On rephrasing goal (Fig. [a)): LLM successfully captures the semantics contained in the goals rather than directly perceives the task goals through character matching. We also find that the performance of LLM might be sensitive to initial observation. As illustrated in Fig. b)), the natural language goal is easy to understand, while the LLM still fails due to the unfamiliar intiill observation. On unseen (easy) tasks (Fig. a)), the LLM adeptly captures the information from the goal, even if the goal is substantially different from those encountered in the training set. However, from Fig. [0]b), we find that while the system demonstrates proficiency in colour recognition, it encounters challenges in accurately discerning the direction of motion. unseen (hard) (Fig. [T Row 1)), although the task requirements are more complex, requiring manipulations multiple objects (whereas the training set of L.M only contains examples about manipulations on a single object), LLM is still able to complete the task. However, as illustrated in Fig. Row 2], as the difficulty of the task continues to increase, that is, when the relative positional relationships multiple objects need to be additionally considered, LLM is no longer effective. Furthermore, although LLM did not accurately arrange the objects in row according to the prescribed colour order, there was clear tendency to align the objects in a row.
For Meta-world environment: On rephrasing goal (Fig. Ta)): LLM can more accurately simulate simple interaction between objects. The grasped object will move with the gripper the imaginary rollouts. As illustrated in Fig. b)), as interactions between objects become more complex, LLM cannot accurately model this process, resulting in the weird interaction between gripper and window. On unseen (casy) tasks (Fig. [)(a),, LLM understands the physical world relatively well. After discovering that the gripper was blocked, it adjusted and finally moved to the target position. As illustrated in Fig. b)),). Although LLM failed to complete the task in compliance with the laws of real physics (because of the dangling faucet), LLM did close the faucet. On unseen (hard) (Fig. IT]. for complex task which is combination of basic tasks (first push the coffee cup, then press the coffee machine button), the LLM can only complete part of the task (pushing the coffee cup).
24

---

Rephrasing Goal
Meta world
(a) Success: Unilize the gripper system to nuvigate the specified object to the desired destination.
(b) Failure: The closed window is bothersome to me; could you please utilize the gripper to open ut?
CLEVR Robot
Success: Propel the red-colored orb forward, leading the blue-colored orb.
(b) Failure: The placement of the red ball in front blue ball something detest. Can you flip them?
Figure Additional examples of the generated rollouts for rephrasing goal tasks.
25

---

Unseen (Easy)
Meta world
(u) Success; Position the gripper to reach the turget area, with avareness of the wall obstructing the path.
(b) Failure: Utilize the gripper to terminate the faucet's fanction.
CLEVR Robot
Suecess: Could you shift the blue ball right?
Success: Please push green ball
(c) Failure: you shift the ball, which happens red, the right?
(d) Failure: Please nudge the blue-hued ball forward.
Figure 10: Additional examples of the generated rollouts for unseen (easy) tasks.
26

---

Unseen (Hard)
Meta world
Employ the gripper to maneuver the coffee mug into place beneath the coffee machine's 's spout, ready for brewing.
CLEVR- otbot
Use the green ball as nucleus of the circle, arranging the rest around it.
Position the balls horizontally, organizing them from left to right, the sequence of red, blue, green, purple, and cyan.
Figure 11: Additional examples of the generated rollouts for unseen (hard) tasks.
C.3 Examples of Rollout explanations
We present few examples of generated rollout explanations used calculate accuracy in Figure The prompt used to generate an explanation of Seen Rollouts is presented in AA44 and the prompt for Unseen Rollouts is Suppose you are playing a gaming of five balls with different colours. Please explain the following trajectory briefly. biTrajectory(ROOLOOU hatp::::eROOOLTTTTTTaeee/e//AAA (Annser:(ANSWER)
Success cases:
Model output: The green ball is being moved left of the cyan ball
Label: The green ball is pushed to the left of the cyan ball
Model output: The cyan ball is being moved in front the purple ball
Label: Push the purple ball in front of the cyan ball
Failure case:
Model output: The red ball is being moved behind the cyan ball
Label: The blue ball is pushed in behind the cyan ball
C.4 Step Match Rate of the Generated Rollouts on CLEVR-Robot
We examine the quality of the generated rollouts on unseen (easy) tasks. At this task level, the agent aims to reposition a ball specified direction in one step. The quality of the generated data quantified by measuring the generation accuracy, which is determined by the match between the generated state/action and the given goal. For instance, given the goal "Move the red ball to the left," " the accuracy is calculated by checking the alignment ratio of the generated state/action with this
27

---

natural language goal. Fig. 12 shows the results of the generation accuracy of five checkpoints during the training. The results show that while action generation accuracy remains constant approximately 30%, state generation accuracy improves as training progresses. These findings suggest that L.M works better generating states than generating actions. Besides, the L.LM can generate imaginary rollouts for novel tasks without being explicitly trained on such tasks.
Action
State
Rates
10
Epochs
Figure 12:
Single step match rate of the generated rollouts, which is assessed by examining the alignment between the states and actions generated by the LLM and the labelled goals unseen (easy) tasks. The horizontal axis represents the training epochs, while the vertical axis is the match rate of the generated states/actions.
C.5 Quality of the Generated Rollouts
We conduct statistics the quality of the generated rollouts. Note that the experiments focus the environment state in numerical vector, where each dimension in the vector has specific semantic. Therefore, the LLM would not generate rollouts containing impossible objects. Here focus on investigating the implausible rollouts as follow topics: 1. the object is out the table; 2. the object floats in the air; implausible robotics joint pose (e.g., of the joint rotation angle bound) and exceeding dynamics limits between two steps. The statistics results are presented in Table and
Table 4: Ratios of various types of unrcalistic transitions of the imaginary rollouts on CLEVR Roboot..
Out of workbench
Exceed dynamics limits
Rephrasing goals
18.4
0.1
Unseen (easy)
0.0
0.2
Unseen (hard)
4.3
0.2
Table 5: Ratios of various types of unrealistic transitions of the imaginary rollouts on Meta-world.
Float in the Air
Out of workbench
Implausible pose
Exceed dynamics limits
Rephrasing goals
39.5
18.3
34.0
44.6
Unseen (easy)
35.5
10.9
39.2
47.9
Unseen (hard)
22.3
64.9
31.2
31.1
The results indicate while there is certain level of hallucination present in the outputs of the LLM, the majority of the imaginary rollouts remain within reasonable scope. Besides, the results on CLEVR- E.EER-t demonstrate lower anomaly ratio. These hallucinations may serve as a form of domain randomization that improves the policy's robustness. Besides, KALM provides possibility for inferring novel task states based on the generalized knowledge of L.LMs, just like the human brain's 's imagination the place they's ve never been
28

---

D Retrievals for Notations and Abbreviations
Tab a list of the notations and abbreviations employed throughout this paper, serving as convenient reference for the reader.
Name Notations
Meaning
D
dataset
M
large language model token embedding layer state embedding layer action embedding layer
Fr
Es
EA G G SST
goal space
goal in the goal space
dimension of observation dimension of language model hidden state
[C]
Abbreviations
KALM
Knowledgeable Agent from Language Model Rollout
LLM
Large Language Model
BC
behavior cloning algorithm
CQL
conservative q q-learning algorithm twin delayed deep deterministic policy gradient algorithm
TD3
AWAC
advantage weighted actor-critic algorithm
MLP
multi-layer perceptrons
Table Notations and abbreviations in this paper.
E
Broader Impact
By integrating LLMs with RL, KALM enables creating Al agents that understand and perform complex tasks with greater efficiency and adaptability. This could enhance industries that rely on automation and intelligent systems, such as manufacturing, where robots could learn to handle and complicated manipulation tasks without extensive reprogramming. In the realm of robotics and automation, the ability of agents understand and perform complex tasks with high degree of success could lead to more efficient and versatile robotic systems. These systems could be deployed in various sectors, including manufacturing, healthcare, and disaster response, where they can perform tasks that are dangerous, tedious, or beyond human capabilities, thereby improving safety, productivity, and efficiency. KALM presents an exciting advancement in the field of RL that could impact society, while we must ensure that such Al systems are bandled responsibly, considering ethical implications such as privacy.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's s contributions and scope?
Answer: [Yes]
Justification:
We introduce the contribution in the abstract and the last paragraph of Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer this question will be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected generalize other settings.
It fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of this work in detail in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specificatio, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Question:
Answer: [NA]
30

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We report important details about the experiments in Sec. We also provide the necessary hyper-paramen parameters for reproducibility: the choice of the pre- ire-traied LLM (Llama-
2 2-7b-chat-hf model the offline RL algorithm (BC, [23 25 [51] and corresponding
library ([54)), the network architecture (A.3). the hyperparameters for LLM grounding.
A.3 and the loss function
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how construct the dataset).
(d) We recognize that reproducibilityy may be tricky in some cases, in which authors are welcome to describe the particular way they provide for repro- ducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
31

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have released the code and data based on the submission guidelines.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We illustrate the details about implementation (pre-trained language model, dataset, optimizer, etc.) Implementation details of We list all the hyperparameters We selected those hyperparameters for their good performances.
A.3
Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The results reported are averaged over three random runs. We use error bars or shaded area to present the statistical significance, e.g., Fig. 4
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: We illustrate the details about computer resources (CPU, GPU and memory) on Implementation details of 4.
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code Ethics https://merrpsscccpppii//hhsttts
Answer: [Yes]
Justification; The research conforms with the NeurlPS Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification; We have discussed the broader impacts in detail in Appendix
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
This paper poses no such risks, as it does not release new pre-trained models or datasets.
Justification:
12. Licenses for existing assets
Question: Are the creators original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
32

---

Justification: The assets/environments used in this paper are properly credited: LLaMA2 [6].
CLEVR-Robot [4]. Meta-world
[5] and d3rlpy [54].
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve erowdsourcing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/revieww based on the requirements of your country or institution) obtained?
Question:
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
33


Reviews:
[{"review_id": "wKR7Rd6UEO", "reviewer": "Reviewer_rv4h", "summary": "To bridge the gap between agents that can act and the vast prior knowledge that language models contain, the authors propose KALM (Knolwedgable Agents from Language Model rollouts), a method for training action-taking agents from language models. KALM is a finetuning method that enables a language model to translate between textual descriptions of goals and environment rollouts. Imaginary rollouts are then used for offline RL training. The paper demonstrates the efficacy and generalizability of the method on robotic manipulation tasks.", "questions": "Why do the authors think that the ablation study for Table 1 shows that adding components of the method actually leads to a degradation of performance on rollout accuracy on the unseen (easy) set? The paper presents an explanation that the current objectives don’t address learning action semantics, but this also applies to the unseen (hard) tasks and does not explain a *degradation* in performance.", "limitations": "The authors discuss the limitations of their work in the paper.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- To the best of my knowledge, the proposed method (translating bidirectionally between environment rollouts and textual goal descriptions, then using generated imaginary rollouts for RL training) is a novel way of grounding language models as agents for low-level control in robotic manipulation. \n- The experiments seem thorough (across two environments, range of tasks) and the evaluation sets are curated to measure the intended effects (generalization). The empirical results generally support the claims from the authors that adding KALM helps with policy learning across multiple RL algorithms, and compare against relevant baselines (directly finetuning the LLM as a policy) and method components are ablated, demonstrating their importance.\n- The problem setting is of significance to the robotics community, where there have been an increase in recent works looking to connect the capabilities of large foundation models with lower level controls.", "weaknesses": "- While the method seems to work in these simple simulated settings, the method itself seems susceptible to language model hallucinations. It would be interesting to see statistics on how often the model imagines objects / obstacles that don’t exist, or implausible rollouts in the simulator. Alternatively, it would be interesting to see that having hallucinations can actually lead to increased robustness. \n- Given that the motivation is to improve generalizability via LLMs, it would be helpful to also include a comparison against RL methods that use prior knowledge from LLMS (e.g. the cited LLaRP). While the LLM as a policy method touches on this, it would support the motivations of the paper more if there were direct comparisons against methods that use LLMs for world knowledge & priors.\n- There are a few grammatical and writing errors in the paper:\n- Line 21: “intelligent agents to acquire such ability” → “intelligent agents to acquire such an ability/such abilities”\n- Line 26: “despite they are highly similar tasks” → “despite them being highly similar tasks”\n- Line 28: “general tasks in text domain” → “general tasks in text domains”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your highlight of the method novelty and the significance of the problem setting. Please find our response to each comment as follow.\n\n> Q1: While the method seems to work in these simple simulated settings, the method itself seems susceptible to language model hallucinations. It would be interesting to see statistics on how often the model imagines objects / obstacles that don’t exist, or implausible rollouts in the simulator. Alternatively, it would be interesting to see that having hallucinations can actually lead to increased robustness.\n\nHallucination is indeed a central issue to consider when we use LLM. The results in Table 1 provide a general result in term of matching rate between generated rollouts and the given goal. In response to your comment, we have conducted more detailed statistics on the quality of the generated rollouts. Note that current experiments focus on the environment state in numerical vector, where each dimension in the vector has specific semantic. Thus the LLM would not generate rollouts containing impossible objects. Here we focus on investigating the implausible rollouts as follow topics: 1. the object is out of the table; 2. the object floats in the air; 3. implausible robotics joint pose (e.g., out of the joint rotation angle bound) and 4. exceeding dynamics limits between two steps. The statistics results are as follow:\n\n|   CLEVR-Robot    | Out of workbench | Exceed dynamics limits |\n| :--------------: | :---------: | :--------------------: |\n| Rephrasing goals |    18.4     |          0.1           |\n|  Unseen (easy)   |     0.0     |          0.2           |\n|  Unseen (hard)   |     4.3     |          0.2           |\n\n|    Meta-world    | Float in the Air | Out of workbench | Implausible pose | Exceed dynamics limits |\n| :--------------: | :--------------: | :---------: | :--------------: | :--------------------: |\n| Rephrasing goals |       39.5       |    18.3     |       34.0       |          44.6          |\n|  Unseen (easy)   |       35.5       |    10.9     |       39.2       |          47.9          |\n|  Unseen (hard)   |       22.3       |    64.9     |       31.2       |          31.1          |\n\nThe results indicate while there is a certain level of hallucination present in the outputs of the LLM, the majority of the imaginary rollouts remain within reasonable scope. Besides, the results on CLEVR-Robot demonstrate lower anomaly ratio. These hallucinations may serve as a form of domain randomization that improves the policy's robustness.\n\n> Q2: Given that the motivation is to improve generalizability via LLMs, it would be helpful to also include a comparison against RL methods that use prior knowledge from LLMS (e.g. the cited LLaRP). While the LLM as a policy method touches on this, it would support the motivations of the paper more if there were direct comparisons against methods that use LLMs for world knowledge & priors.\n\nGood point. We agree with you that an additional baseline utilizing both LLM and offline data more would be a strong supplement to our experiments. In response to your concern, we conduct comparison with an additional baseline, Decision-transformer (DT) [1], with Llama-2-7b-chat-hf as the backbone policy model. DT treats decision-making as a sequence modeling problem, using a transformer architecture to predict actions based on the desired future return. In this experiment, DT trains on the offline data same as other methods, and the results are as follow:\n\n|                      |   KALM   |  DT  |\n| :------------------: | :------: | :--: |\n| Task in offline data | **46.1** | 26.4 |\n|   Rephrasing goals   | **30.8** | 19.1 |\n|    Unseen (easy)     | **12.5** | 11.4 |\n|    Unseen (hard)     | **7.2**  | 4.5  |\n\nThe results show that KALM outperforms DT on all four types of tasks. We would like to update the paper to incorporate the additional result.\n\n> Q3: Why do the authors think that the ablation study for Table 1 shows that adding components of the method actually leads to a degradation of performance on rollout accuracy on the unseen (easy) set? The paper presents an explanation that the current objectives don’t address learning action semantics, but this also applies to the unseen (hard) tasks and does not explain a *degradation* in performance.\n\nAnswer: We suggest that the performance degradation is attributed to the specific nature of the unseen (easy) task, whose objective is to predict one-step transitions given unseen language goals. To be more specifically, we would discuss the two components of LLM fine-tuning in KALM (i.e., dynamics prediction and rollout explanation) respectively. For dynamics prediction objective,  unseen (easy) task objective (predicting $a_t$ and $s_{t+1}$ given $G$ and $s_t$) shares similarity, yet diverges from the dynamics prediction objective (predict $s_{t+1}$ given $s_t$ and $a_t$). This difference introduces a potential conflict in the LLM SFT. This is evidenced by the empirical results that KALM w/o dynamics prediction achieves highest rollout accuracy for unseen (easy) task. For rollout explanation objective, it focuses on giving explanation on a long rollout sequences. While this objective enriches the model's capability to provide coherence over temporal sequence, it may inadvertently detract from the model's ability to capture the immediate logic of transitions between adjacent two steps.\n\nWe would like to update the paper to reflect the above discussion about the experiment results.\n\n> Q4: There are a few grammatical and writing errors in the paper.\n\nWe have fixed your mentioned typos and would make a more thorough check to fix the grammatical and writing errors.\n\n------\nWe believe the experiment and the result analysis have been clearly enhanced based on your c comments. If you had any further concerns, please let us know.\n\n## Reference:\n\n[1] Decision Transformer: Reinforcement Learning via Sequence Modeling. Lili Chen, et al. NeurIPS 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DzHONqposL", "reviewer": "Reviewer_1PB3", "summary": "This paper provides KALM (Knowledgeable Agents from Language Model Rollouts) that fine-tunes a LLM (e.g., Llama-2-7B-Chat) with offline RL (e.g., CQL) for robotic manipulation tasks (e.g., CLEVR-Robot and Meta-world).\n\nKALM mainly consists of three steps: (1) LLM grounding, (2) rollout generation, and (3) offline RL. In the LLM grounding step, KALM fine-tunes a LLM in a supervised manner on an offline dataset collected from the environment. More specifically, supervised fine-tuning involves three tasks: dynamics prediction, rollout explanation, and rollout generation. In the rollout generation step, KALM uses the fine-tuned LLM to generate imaginary rollouts with goal-oriented prompt (GOP). Here, the GOP is generated by paraphrasing the goals in the offline dataset or synthesizing new goals. Finally, in the offline RL step, KALM fine-tunes the LLM on the combination of the offline dataset and the imaginary rollout dataset. \n\nThis paper evaluates KALM on two robotic manipulation benchmarks: CLEVR-Robot and Meta-world. This paper empirically demonstrates that KALM can outperform offline RL algorithms such as CQL and BC.", "questions": "Q1. When generating imaginary rollouts, does KALM interact with the environment?\n\nQ2. Does KALM use rewards from the environment to filter out low-quality rollouts?\n\nQ3. Can we use DPO (Direct Preference Optimization) instead of offline RL algorithms? Are there any pros and cons?", "limitations": "The authors adequately provides limitations of their work in Section 5 (i.e., Conclusion and Limitation).", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "S1. This paper proposes a method that uses a LLM (e.g., Llama-2-7B-Chat) to generate an imaginary rollout data in addition to an offline dataset. This approach can be seen as a kind of data augmentation. And, this approach improves the performance of an agents both in-distribution and out-of-distribution test cases. Especially, this paper demonstrates that KALM performs well in the out-of-distribution cases.", "weaknesses": "W1. This paper mainly compares KALM with offline RL algorithms such as CQL. However, I am not sure that this comparison is fair and reasonable. Since KALM uses imaginary rollout data in addition to an offline dataset, it may not be considered as an offline method. If it true, comparing KALM with pure offline algorithms may not be fair.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and time in reviewing this paper. It appears there may be some confusion regarding the fairness of the experimental comparison. Below, we provide a comprehensive response to address these concerns.\n\n> Q1: Concerns about the fairness of comparison.\n\n**Comment 1:** I am not sure that this comparison is fair and reasonable. Since KALM uses imaginary rollout data in addition to an offline dataset, it may not be considered as an offline method.\n\n**Response 1:** We first clarify the fairness of the comparison: all methods in our experiments are offline methods, which *does not require online interaction* with the environment (we would discuss the environment interaction in detail in response 2 below). KALM fine-tunes a general purpose LLM with the offline data, and generates imaginary rollouts without online interaction, thus adhering to the offline setting and ensuring the fair comparison. In the initial submission, we have made a comparison between KALM and a LLM-related baseline (LLM as policy), as shown in Figure 5. To further justify the proposed method and address your concern, we add an additional baseline for comparison, Decision-transformer (DT) [1], which utilizes Llama-2-7b-chat-hf as the backbone policy model and offline data to training the policy. DT treats decision-making as a sequence modeling problem, using a transformer architecture to predict actions based on the desired future return. In this experiment, DT trains on the offline data same as other methods, and the results are as follow (as well shown in Figure 1 in the attached PDF file):\n\n|                      |   KALM   |  DT  |\n| :------------------: | :------: | :--: |\n| Task in offline data | **46.1** | 26.4 |\n|   Rephrasing goals   | **30.8** | 19.1 |\n|    Unseen (easy)     | **12.5** | 11.4 |\n|    Unseen (hard)     | **7.2**  | 4.5  |\n\nThe results show that KALM outperforms DT on all four types of tasks. We would like to update the paper to incorporate the additional result.\n\n**Comment 2:** When generating imaginary rollouts, does KALM interact with the environment?\n\n**Response 2:** KAML does not require interaction with the environment, and the imaginary rollouts can be generated in pure offline manner. We appologize for any confusion brought by Figure 2, which involves an *optional* online process. For clarity, it should be noted that in our comparative experiments, KALM is implemented ***without the online procedure***. The inclusion of the optional online process in Figure 2 is intended to illustrate the method's extendibility and capability for future integration with real-time environment interactions.\n\n> Q2: Does KALM use rewards from the environment to filter out low-quality rollouts?\n\nKALM does not employs rollout filter. We use all imaginary rollouts generated by LLM. We suggest that offline RL algorithm can also learn useful knowledge from failure experience. Besides, these failure rollouts may contribute to the robustness of the resulting policy by providing a diverse range of scenarios for the model to learn from, although they are not successful rollouts. This insight is aligned with the comment proposed by Reviewer rv4h.\n\n> Q3: Can we use DPO (Direct Preference Optimization) instead of offline RL algorithms? Are there any pros and cons?\n\nDPO can not be directly applied to our problem setting, as it requires preference data annotated by human or AI and is initally designed for aligning LLMs with human value. A potential adaptation of DPO for integration with KALM is treating the offline rollout as the preferred data and the generated rollouts as the less-preferred data. However, this is still unfeasible as we do not have offline rollouts (i.e., preferred data for DPO) that correspond to unseen language goals. Note that the primary motivation of this research is to leverage the knowledge of LLMs to develop agents with enhanced knowledge, this goal is not dependent on the use of any specific offline RL optimization technique.\n\n------\n\nWe hope that our response has addressed your concern and questions satisfactorily. If you had any further concerns, we are glad for discussion.\n\n## Reference:\n\n[1] Decision Transformer: Reinforcement Learning via Sequence Modeling. Lili Chen, et al. NeurIPS 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jJpcdZH4DM", "reviewer": "Reviewer_JwPT", "summary": "This paper presents KALM, a novel approach adopting LLMs to generate imiginary rollouts to augment the offline dataset for offline RL. Structure of LLMs is altered to handle the numeric states and actions in decision-making environments. Experiments demonstrate the imaginary rollouts benefits the offline policy learning and effectively help KALM surpass the baselines.", "questions": "See weaknesses.\n\nI will increase my score to 6 upon seeing comparisons between KALM and model-based offline RL. This comparison directly demonstrates the effectiveness of using LLMs instead of traditional neural networks to generate rollouts.", "limitations": "Yes, limitations are discussed.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Adopting LLMs to generate imaginary rollouts is a promising and interesting direction for decision-making field, as LLMs possess extensive knowledge to help model the environment mechanism.\n\n2. This paper is well-written and clearly describes the proposed method.\n\n3. Empirical results validate the effectiveness of using LLMs to generate rollouts. This may further pave the way for using LLMs as world models.", "weaknesses": "1. Considering the close connection between this paper and model-based offline RL, I'm surprised that they are not even mentioned in the related work. Model-based offline RL methods like MOPO, MoREL and COMBO demonstrate strong performances and are also capable of generating imaginary rollouts  just like KALM. They should be discussed and included in the baselines.\n\n2. I wonder whether the evaluation between baselines with/without KALM is fair. The policy in KALM also conditions on $G$, a hidden vector generated by BERT. I think this introduces additional information to the policy. Does the other baseline also use this information?\n\n3. This is an open question. What do you think are the major differences between KALM and world models? Is there any possibilities to extend KALM to more complicated environments (even real-world cases)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful comments and finding our idea interesting. To address your concern, we have devoted effort to include discussion and experiment on model-based RL. Please find our response below.\n\n> Q1: Model-based offline RL methods like MOPO, MOReL and COMBO demonstrate strong performances and are also capable of generating imaginary rollouts just like KALM. They should be discussed and included in the baselines.\n\n**Comment 1:** Discussion about model-based RL methods.\n\n**Response 1:** We agree with you at this point. In our initial submission, we touch upon model-based RL (MBRL) methods (MAPLE and RedM) in Section 2.2, referring to them as *environment models*. However, we acknowledge the need for a more comprehensive discussion about prevalent MBRL methods:\n\nMBRL algorithms learn a dynamic model from offline data, which can then be used by any policy learning algorithm to recover the policy. MOPO [1] and MOReL [2] use uncertainty quantification to construct a lower bound, aiming to avoid issues like model bias and distribution shift. COMBO [3] employs both the offline dataset and model-generated rollouts to train a value function and regularize it on out-of-support state-action tuples. Despite both MBRL and KALM methods utilized generated rollouts for policy training, they are different from motivation. KALM, in particular, extracts knowledge from pre-trained LLM to build knwoledgeable agent. Leveraging the LLM's general world knowledge, we demonstrate that LLM has the potential to generate rollouts for unseen goals, extending beyond the scope of offline data and enabling the acquisition of novel skills.\n\nWe would like to incorporate the discussion into the revised version.\n\n**Comment 2:** I will increase my score to 6 upon seeing comparisons between KALM and model-based offline RL. This comparison directly demonstrates the effectiveness of using LLMs instead of traditional neural networks to generate rollouts.\n\n**Response 2:**  Thanks for your suggestions. We have conducted experiments to compare KALM and two representative baselines: COMBO and MOPO, as the results in the table:\n\n| |   KALM   | COMBO | MOPO \n| :-: | :-: | :-: | :-: |\n| Task in offline data | 46.1 | 39.3  | 0.2\n|   Rephrasing goals   | 30.8 | 23.3  | 0.2\n|    Unseen (easy)  | 12.5 |  3.4  | 1.0\n| Unseen (hard)  | 7.2  |  6.9  | 1.6\n\nKALM surpasses two compared model-based offline RL methods, demonstrating the effectiveness imaginary of rollouts generated from LLM.\n\n> Q2: The policy in KALM also conditions on G, a hidden vector generated by BERT. I think this introduces additional information to the policy. Does the other baseline also use this information?\n\nIn our experiments, all baseline methods utilize the BERT to convert textual goals into vector representations, ensuring a fair comparison. We appologize for the omission of this detail in the description of experiment setting. To further improve the comprehensiveness of our experiment, we have now included an addition baseline, DT [4], that utilizes both LLM and offline dataset. DT utilizes Llama-2-7b-chat-hf as the backbone policy model and offline data to training the policy. It treats decision-making as a sequence modeling problem, using a transformer architecture to predict actions based on the desired future return. In this experiment, DT trains on the offline data same as other methods, and the results are as follow:\n\n|  | KALM | DT\n| :-: | :-: | :-: |\n| Task in offline data | 46.1 | 26.4\n| Rephrasing goals   | 30.8 | 19.1\n| Unseen (easy) | 12.5 | 11.4\n| Unseen (hard) | 7.2  | 4.5\n\nThe results show that KALM outperforms DT on all four types of tasks. \n\n> Q3: This is an open question. What do you think are the major differences between KALM and world models? Is there any possibilities to extend KALM to more complicated environments?\n\n**Comment 1:** What do you think are the major differences between KALM and world models?\n\n**Response 1:** Both world models and KALM train model to generate environmental rollouts, but their motivations are different: KALM leverages the embedded knowledge within pre-trained LLM, utilizing this pre-existing information to generate rollouts. Conversely, world models begin with no prior knowledge and learn from extensive datasets, constructing knowledge from foundational principles. \n\nBesides, world models and KALM utilize the offline data for different purposes. World models particularly excel in visual tasks, processing observations such as high-dimensional images. They utilize offline datasets to abstract observation features and learn environmental dynamics. On the other hand, KALM utilizes offline data to train policy and applies the offline data to bridge the gap between LLM and the environmental.\n\n**Comment 2:** Is there any possibilities to extend KALM to more complicated environments (even real-world cases)?\n\n**Response 2:** As we discuss in Section 5, we have considered extending KALM to more complex tasks, e.g., tasks with visual input. We make attempt towards this direction on Meta-world environment with visual observation (visual input is closer to the real world). We utilize a randomly initialized ViT as the vision encoder/decoder to process visual input. Figure 3 in the attached PDF file shows the example of the generated rollout, given a language goal *unseen during the training*. The generated rollout successfully captures the core information of the environment, reflecting the reasonable robotic movements tendency. This result provides evidence of the potential for extending KALM to more complicated environments. \n\n## Reference:\n\n[1] MOPO: Model-based Offline Policy Optimization. Tianhe Yu, et al. NeurIPS 2020.\n\n[2] MOReL: Model-Based Offline Reinforcement Learning. Rahul Kidambi, et al. NeurIPS 2020.\n\n[3] COMBO: Conservative Offline Model-Based Policy Optimization. Tianhe Yu, et al. NeurIPS 2021.\n\n[4] Decision Transformer: Reinforcement Learning via Sequence Modeling. Lili Chen, et al. NeurIPS 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "87dFuW2GA4", "reviewer": "Reviewer_bRuq", "summary": "This paper introduces a novel approach, KALM (Knowledgeable Agents from Language Model Rollouts). KALM extracts knowledge from LLMs in the form of imaginary rollouts, which agents can learn through offline RL. KALM fine-tunes the LLM to bridge the semantic gap between LLMs and RL agents. The paper demonstrates the effectiveness of KALM on robotic manipulation tasks, achieving a higher success rate than baseline methods, especially in unseen tasks.", "questions": "- How is the dataset collected? What is the quality of the policy used to collect data?\n- Why use only 6400 rollout-goal pairs to train offline RL, given there are 100,000 rollout-goal pairs available?", "limitations": "The authors mentioned that the generation of both state and action increases the burden on the LLM. Additionally, the current version is limited to state in the form of vector.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of extracting knowledge from LLMs in the form of imaginary rollouts is interesting and useful.\n- The experimental analyses are sufficient and well-conducted.", "weaknesses": "- Constrained by the limited context length, generating a complete trajectory seems unreliable for more complex tasks with longer trajectories. Moreover, there is a lack of details regarding the trajectory length of the experimental tasks.\n- The proposed method requires the environment-built-in reward function to obtain and incorporate rewards, which may not be accessible for some other tasks. What if training the LLM to generate the rewards?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and valuable comments. Below, we address each of your questions and provide detailed responses to your concerns.\n\n> Q1: Constrained by the limited context length, generating a complete trajectory seems unreliable for more complex tasks with longer trajectories. Moreover, there is a lack of details regarding the trajectory length of the experimental tasks.\n\n**Comment 1:** Constrained by the limited context length, generating a complete trajectory seems unreliable for more complex tasks with longer trajectories. \n\n**Response 1:** This is a good point. We understand your concern about the trajectory generation on long-horizon tasks. We clarity that current experiments primarily demonstrate the efficacy of our proposed methodology, which extracts knowledge from LLM to facilitate low-level control. For tasks with longer trajectories, there have been established techniques verified to be effective, e.g., uncertainty estimation and branch rollout in MOPO [1], and long-range coherence in Sora [2], which is capable of generating video exceeding one minute. While these techniques are compatible with KALM, they were not employed in this study since the average trajectory length was approximately 70 timesteps (as shown below), which did not necessitate their integration. It would be interesting for future research to investigate how these techniques could be incorporated to handle longer trajectory tasks within the KALM framework.\n\n**Comment 2:** there is a lack of details regarding the trajectory length of the experimental tasks.\n\n**Response 2:** Thanks for your suggestion. We have included the details regarding the trajectory length of offline data, as shown in the following table. \n\n|     CLEVR-Robot      | Min  | Mean | Max  |\n| :------------------: | :--: | :--: | :--: |\n| Task in offline data | 2.0  | 30.8 | 50.0 |\n|   Rephrasing goals   | 1.0  | 17.2 | 50.0 |\n|    Unseen (easy)     | 1.0  | 1.0  | 1.0  |\n|    Unseen (hard)     | 1.0  | 45.1 | 50.0 |\n\n|      Meta-world      |  Min  | Mean  |  Max  |\n| :------------------: | :---: | :---: | :---: |\n| Task in offline data | 17.0  | 69.5  | 100.0 |\n|   Rephrasing goals   | 14.0  | 73.7  | 100.0 |\n|    Unseen (easy)     | 17.0  | 54.7  | 100.0 |\n|    Unseen (hard)     | 100.0 | 100.0 | 100.0 |\n\n> Q2: The proposed method requires the environment-built-in reward function to obtain and incorporate rewards, which may not be accessible for some other tasks. What if training the LLM to generate the rewards?\n\nWe acknowledge that the availability of a built-in reward function can be a limitation in environments where such functions are not readily accessible or well-defined. However, we would like to emphasize that the main focus of this work is to explore the effective utilization of knowledge in LLM for low-level control, and the proposed method is not inherently dependent on the existence of such a reward function. Considering the potential absence of such functions, training LLM to generate rewards is a potential solution, while it may add burden to the LLM training. As an alternative, there have been other well-validated methods utilizing LLM to generate reward function [3,4] or dealing with sparse reward problems [5,6].\n\n> Q3: Questions about experiment setting.\n\n**Comment 1:** How is the dataset collected? What is the quality of the policy used to collect data?\n\n**Response 1:** We collect the offline dataset by utilizing expert policies specifically trained to complete natural language goal for each task. To simulate the noise in real-world data, we introduce a small portation (10%) of rollouts generated by the random policy. In this way, the dataset reflects a degree of the variability of the environment. We would incorporate these details in the experiment setting in the revised version.\n\n**Comment 2:** Why use only 6400 rollout-goal pairs to train offline RL, given there are 100,000 rollout-goal pairs available?\n\n**Response 2:** We apologize for the confusion brought by the dataset utilization. This is attributed to an issue of computation resources. The generation of imaginary rollouts is computationally expensive and time-consuming, with the capacity to generate approximately 6,400 pairs in a 24-hour period. Therefore, to prevent potential sample bias and maintain a balanced dataset, we choose an equivalent number of offline rollouts from the available 100,000 pairs.\n\nNevertheless, we acknowledge it worth exploring to utilize all the 100,000 offline rollouts, by balancing the imaginary / offline rollout in each training batch. The experiment results on CLEVR-Robot are shown in the table:\n\n|                       | BC [6,400] | BC+KALM [6,400] | BC [100,000] | BC+KALM [100,000] |\n| :-------------------: | :--------: | :-------------: | :----------: | :---------------: |\n| Tasks in offline data |    63.1    |      56.3       |   **70.4**   |       63.4        |\n|   Rephrasing goals    |    26.3    |      44.6       |     26.3     |     **51.1**      |\n|     Unseen (easy)     |    17.5    |    **33.5**     |     16.8     |     **33.7**      |\n|     Unseen (hard)     |    1.4     |       7.2       |     1.0      |      **7.7**      |\n\nThe result suggests that an increase in the amount of offline rollouts can lead to improvements in method performance. Overall, the KALM still outperforms the baseline under this setting.\n\n## Reference\n\n[1] MOPO: Model-based Offline Policy Optimization. Tianhe Yu, et al. NeurIPS 2020.\n\n[2] Video generation models as world simulators. OpenAI team. 2024.\n\n[3] Text2Reward: Reward Shaping with Language Models for Reinforcement Learning. Tianbao Xie, et al. ICLR 2024.\n\n[4] Eureka: Human-Level Reward Design via Coding Large Language Models. Yecheng Jason Ma, et al. ICLR 2024.\n\n[5] Learning by playing solving sparse reward tasks from scratch. Riedmiller, et al. ICML 2018.\n\n[6] Overcoming exploration in reinforcement learning with demonstrations. Ashvin Nair. ICRA 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We express our sincere thanks to the reviewers and chairs for their valuable time and constructive feedback on this paper. We are encouraged to note that reviewers acknowledge this paper's novelty (Reviewer rv4h,JwPT,bRuq), its contribution to community (Reviewer rv4h,JwPT,bRuq) and the sufficiency/performance of the experiments (Reviewer rv4h,1PB3,JwPT,bRuq). Below we summarize the major concerns raised by reviewers and our corresponding response.

1. Requiring additional baselines (Reviewer rv4h,JwPT): we have incorporated additional baseline methods. Specifically, we have included decision-transformer that leverage both LLM and offline datasets, as well as model-based offline RL approaches.
2. Fairness of the comparison (Reviewer 1PB3): we clarify that all methods in the experiments (including KALM and baselines) are under offline setting, where no online interactions are allowed during training, and only a fixed set of offline trajectories are available for learning. Therefore, the comparison is not unfair. We have added more LLM-related baselines (suggested by Reviewer rv4h) to further justify the proposed method.
3. Extending to more complicated tasks (Reviewer JwPT,bRuq): we elaborate on the extendibility of KALM framework to more challenging setting, such as tasks with longer trajectories, absent reward function or visual inputs. We have also verified the extendibility by conducting experiment on more challenging task with visual observation. Please refer to Figure 3 in for the results.

We believe that the experiments are thorough and this paper makes significant contributions to research community, as there have been an increase in connecting the capabilities of large foundation models with lower level controls (as suggested by Reviewer rv4h). We have provided detailed responses to each reviewer's comments below, and we *are eager to* receive the reviewers' feedback! Please let us know if there are any further concerns, and we are available and glad to discuss.

------
Best wishes,

Authors of Submission #502

Meta-review: Reviewers agreed that the method is both new and useful. The method is also easily adapted to other environments and will likely be generally useful for many NeurIPS readers. In particular, its ability to outperform alternatives like MOPO and COMBO which also have rollouts but are not LLM-based is interesting. In the future one might imagine some hybrid between capturing different aspects of the environment.

The authors did a good job addressing reviewer concerns about other baselines, which they should incorporate into the final manuscript. Similarly for the vision-based experiments and the more difficult settings that reviewers asked for. This only makes the paper stronger and engages with a wider literature.
Decision: Accept (poster)