Large Language Models as Generalizable Policies for Embodied Tasks
Authors: Andrew Szot
Max Schwarzer
Harsh Agrawal
Bogdan Mazoure
Rin Metcalf
Walter Talbott
Natalie Mackraz
R Devon Hjelm
Alexander T Toshev
Venue: ICLR 2024 poster (2024)
Keywords: Embodied AI
Reinforcement Learning
Large Language Models
Foundational Models
Primary area: applications to robotics, autonomy, planning
TL;DR: Reinforcement Learned Policies for multi-task embodied AI problems, when initialized from LLMs, demonstrate strong generalization properties across novel task and novel ways of describing tasks.

Abstract: We show that large language models (LLMs) can be adapted to be generalizable policies for embodied visual tasks. Our approach, called Large LAnguage model Reinforcement Learning Policy (LLaRP), adapts a pre-trained frozen LLM to take as input text instructions and visual egocentric observations and output actions directly in the environment. Using reinforcement learning, we train LLaRP to see and act solely through environmental interactions. We show that LLaRP is robust to complex paraphrasings of task instructions and can generalize to new tasks that require novel optimal behavior. In particular, on 1,000 unseen tasks it achieves 42% success rate, 1.7x the success rate of other common learned baselines or zero-shot applications of LLMs. Finally, to aid the community in studying language conditioned, massively multi-task, embodied AI problems we release a novel benchmark, Language Rearrangement, consisting of 150,000 training and 1,000 testing tasks for language-conditioned rearrangement.

Published as a conference paper at ICLR 2024
LARGE LANGUAGE MODELS AS GENERALIZABLE POLICIES FOR EMBODIED TASKS
Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Walter Talbott Katherine Metcalf, Natalie Mackraz, Devon Hjelm, Alexander Toshev Apple
ABSTRACT
We show that large language models (LLMs) can be adapted be generalizable policies for embodied visual tasks. Our approach, called Large LAnguage model Reinforcement Learning Policy (LLaRP), adapts pre-trained frozen LLM take as input text instructions and visual egocentric observations and output actions directly in the environment. Using reinforcement learning, we train LLaRP see and act solely through environmental interactions. We show that LLaRP is robust to complex paraphrasings of task instructions and can generalize to new tasks that require novel optimal behavior. In particular, on 1, 000 unseen tasks it achieves 42% success rate, 1.7x the success rate other common learned base- lines or zero-shot applications of LLMs. Finally, to aid the community in studying language conditioned, massively multi-task, embodied AI problems we release a novel benchmark, Language Rearrangement, consisting of 150, 000 training and 1, 000 testing tasks for language-condition rearrangement. Video examples of LLaRP in Language Rearrangement and the code
https///llm-l-ggihbbl
INTRODUCTION
Large Language Models (LLMs), characterized as billion-paramett parameter models trained on enormous amounts of text data, have demonstrated unprecedented language understanding capabilities. Fur- thermore, L.LMs have demonstrated powerful capabilities beyond core language understanding prob- lems, such as dialog systems (Thoppilan et al., 2022; Glaese et al., 2022), visual understanding prob- lems (Alayrac et al., 2022; Li et al., 2023b; Peng et al., 2023; Koh et al., 2023), reasoning (Wei et al., 2022; Lewkowycz et al., 2022), code generation (Chen et al., 2021b), embodied reasoning (Driess et al., 2023), and robot control (Ahn et al., 2022). These capabilities often emerge in zero-shot fashion, without dedicated training data for each capability, indicating that LLMs contain knowledge general and broad enough to apply to numerous domains. Furthermore, these capabilities emerge despite that the input and output spaces in these domains are oftentimes not naturally expressed in language, e. g. images as inputs, and robot commands as outputs.
key objective in Embodied Al generalizable makiin that can transfer to novel tasks, so is natural ask if the generalization abilities of LLMs can be incorporated into embodied prob- lems. Existing advances in using LLMs for Embodied AI have relied on static expert datasets (Driess al., 2023; Brohan al., 2023), which requires prohibitively large and expensive amounts of di- verse expert data. Conversely, Embodied AI simulators enable agents to learn from an environment through direct interaction, exploration, and reward feedback (Kolve et al., 2019; Szot et al., 2021; Li et al., 2023a). However, the generalization capabilities of such agents to a large number of new embodied tasks are not on par with the aforementioned domains.
LLMSs have been shown to be applicable in online settings when the control domain is that of natural language, e.g., Reinforcement Learning from Human Feedback (RLHF) for multi-turn dialog appli- cations (Ouyang et al., 2022). In this work, we successfully show that LLMs can be adapted via reinforcement learning as a vision-language policy for problems in embodied AI, using a method call Large LAnguage model Reinforcement learning Policy (LLaRP). We demonstrate advanced capabilities on a diverse set of rearrangement tasks, where the input and output domains aren't 't just language (see Fig. 1) In particular, demonstrate the following three contributions:

---

Published as a conference paper at ICLR 2024
Reinforcement Learning of LLM-based Palicy for Many Tasks
Generalization Across Novel Tasks
Mi Piple Onjects
LLaRP
the bidgo
Pinta peir and Rway frrttge
Objects
Act on novel instructions
Belering Expreodonss
LLaRP
Enstrommen
Pur aooii for nen the tala
Eppcertte Poroustion
Sparse Rewars
Sparial and
ognn kitttin atee
Conditiona instructions
Frite tinge rppnn (P 2:00 lppln inside
Figure I: We demonstrate that by utilizing Reinforcement Learning together with a pre-trained LLM and maximizing only sparse rewards, we can learn policy that generalizes to novel language rearrangement tasks. The method robustly generalizes over unseen objects and scenes, novel ways of referring objects, either by description or explanation of an activity; and even novel descriptions of tasks, including variable number of rearrangements, spatial descriptions, and conditional statements.
First, we show that using a pre-trained and frozen LLM as a Vision-Language Model (VLM) policy with learned input and output adapter layers results in policy that exhibits strong generalization capabilities. We train this policy using online RL and measure generalization along two axes:
Paraphrastic Robustness (PR): the agent produces the same optimal behavior under linguistic variations of an instruction where the "intention" of the instruction does not change. This includes novel ways of describing the same behavior or novel ways of referring to seen object.
Behavior Generalization (BG): the agent solves tasks that require novel optimal behavior. This means the desired behavior outcome is distinet from those seen during training. For example, act on new types of or combinations of objects, new types of combined behaviors (e.g., finding "all" of something) or new logical conditions (e.g.. "if" statements).
LLLRP is thoroughly evaluated on over 1, 0,00 unseen tasks spanning the above axes and attains 42% success rate, compared to 25% for an -SSed policy and 22% for zero-shot applications of LLMs. Our approach outperforms all baselines both when it is instructed in novel ways as well as when tasked to perform unseen behaviors. Further, we demonstrate that the LLaRP LLM-based policy gives non-trivial boost another domain, Atari, compared to Transformer baseline.
We demonstrate that, when the agent has access to the world knowledge encoded an LLM, RL exhibits various forms of training efficiencies. For one, LLM-bbseed models exhibit better sample efficiency than other common architectures in both basic PPO RL and continual learning settings (training the model downstream tasks beyond the training tasks). Further, we show that LLakP is more efficient with what supervision is needed than commonly used imitation learning.
Finally, in order facilitate the above contributions and promote future work studying general- ization in Embodied AI, we introduce the Language Rearrangement task which includes 150,000 distinct language instructions, each with automatically generated rewards. The large number of di- verse tasks brings the system closer to wall-wrrdd setups where agents should be able to do anything and everything, and pushes the limits on performance. For generalization evaluation, we define splits that stress test the system on PR- and BG-types of generalization.
RELATED WORK
Prior work has demonstrated large language models (LLMs) can be used as zero-shot policies for interactive preision-making tasks, without task-specific training. in settings where the states and ac- tion spaces are both text-based (Zeng et al., 2022; Shah et al., 2023; Huang et al., 2022; Liang et al., 2023; Huang et al., 2023b; Wu et al., 2023a; Silver et al., 2023; Wang et al., 2023). Namely, Code as Policies (Liang et al., 2023) relies more, LLMs can be adapted to text-based decision making by fine-tuning with a standard language modeling objective (Chalvatzaki et al., 2023). In addition. has been shown that LLMs can be used to provide rewards or useful high-level goals for learning policies (Du et al., 2023; Hu & Sadigh, 2023; Wu et al., 2023b) in text-based human-interaction settings.
perception modules not available in our setting. Further-
2

---

Published as a conference paper at ICLR 2024
a-
at
Trained with Online Reinforcement learning
Action Head
Value Head
Action Head
Value Head
MLP
MLP
Large Language Model
Embed
Embed
Tokenize:
Vision Encoder
Vision Encoder
Task Instruction:
"Move the fruit the fridge"
Frozen
Egocentric Visual Observations
Trained from scratch with RLL
Figure 2: LLaRP architecture. The bottom of the figure shows the policy inputs including the task instruction and the egocentric visual RGB frames from the current time step to the start of the episode. These are encoded using the LLM embeddings or a vision encoder. The embeddings are input to pre-trained LLM. The hidden outputs then projected to action and value predictions. The entire system learns from online RL, where the action output module and observation encoder ML.P are the only trained components and all other components are frozen.
has been shown that agents can learn tasks specified by natural language. Shridhar et al. (2022); Team et al. (2021); Abramson et al. (2022); Yu & Mooney (2022); Lynch al. (2023); Liu et al. (2022); Shridhar et al. (2023); Brohan et al. (2022); Ha et al. (2023) learn language conditioned policies through imitation learning Perceiver-actor Shridhar et al. (2023) requires offline data for imitation learning while LLaRP learns from interaction through RL. Xiao al. (2022); Yu et al. (2023); Myers al. (2023) augment the paired dataset to increase the quantity and diversity of data for imitation learning. Open-vocabulary mobile manipulation (OVMM, Yenamandra et al., 2023) demonstrated open-vocab rearrangement capabilities in pick-and-place settings, while other works demonstrate open-vocab object manipu- lation capabilities navigation settings (Gadre al., 2023; Shah et al., 2023; Huang et al., 2023a; Bolte et al., 2023).
a dataset of expert trajectory and language pairs. Namely,
Works in Vision Language Models (VLMs) have combined pre-trained LLMs with visual reason- ing (Alayrac et al., 2022). Works have imbued VLMs with 3D spatial information (Hong et al., 2023; Jatavallabhula ct al., 2023; Ha Song. 2022), usually in a static setting without environment inter- action. Like work, PaLM-E (Driess et al., 2023) and RT-2 (Brohan et al., 2023) extend VLMs to interactive decision-making in visual embodied environments, but using high-quality expert data.
To our knowledge, prior work demonstrates that LLMs can be used vision-language policies in online RL problems to improve generalization. Like our work, Carta et al. (2023) also adapts an LLM with online RL, but does in environments with textual observations and text actions, while we adapt LLMs for lasion-language policies with online RL. Furthermore, we demonstrate these capabilities over diverse set of evaluations, both in terms of linguistic (paraphrastic gener- alization) and target behavior (behavior generalization) variations. Our Language Rearrangement extends beyond the scope of prior work using LLMs in embodied settings by requiring complex manipulation, navigation, and exploration. The tasks contain 18x more instructions involving more complex rearrangement concepts and object interactions than the benchmark ALFRED (Shridhar et al., 2020), one of the more common condiiio-- Embodied AI benchmarks.
3 METHOD
Our method, Large LAnguage model Reinforcement learning Policy (LLaRPP), adapts pre-trained LLMs to operate in embodied multi-modal recision-making settings. We show how modify existing LLMs for embodied settings and train policy for embodied tasks, leading to an agent with significantly improved generalization capabilities to new language instructions.
3

---

Published as a conference paper at ICLR 2024
Our problem can be formulated fined as a tuple O, P. R, 7)) for underlying state space S, observation space O action space A, transition distribution P. reward function R, initial state distribution Po and discount fac- tor 7. In our setting, O are egocentric high-dimensional visual observations, such as a robot's RGB camera which only observes part of the scene. We consider the extension of including goal dis- tribution G and the case where the reward is formulated as R(s,g) (sgg for E S and g E G. We seek to learn a goal-conditioned policy (alo, g) mapping from observation o and goal g an action that maximizes the sum of discounted rewards Beo--p.gg~G Σ. r (s(,,,, g). Furthermore, we seek to learn policy that generalizes and achieves high rewards for goal distributions G' not seen during training. Specifically, we consider goals specified as natural language instructions and want policies to generalize to new distributions of natural language instructions.
Partiall--Observan Markov Decision Process (POMDP), de-
We utilize large language models (LLMss, which are large auto-regressive text prediction models. Given text represented as a sequence of tokens I, the LLM trained to predict each token in that sequence conditioned on all prior tokens <L 44,..., Since an embodied agent policy needs to consume visual observations O and predict actions A, both of which are not language tokens, we strip away the input and output layers of the LLM. In particular, the LLM input layer encodes the text tokens producing vector embeddings e₁ = E (1x E RR,, while the output layer classifies words.
After we strip away the input and output layers, we call the resulting network an LLM backbone and denote it by (e1,Mee,....... E TRP This backbone consumes sequence of D-dim. token embeddings and produces a D-dim. token embedding.
3.1 LARGE LANGUAGE MODEL REINFORCEMENT LEARNING POLICY ARCHITECTURE
LLaRP, visualized in Figure 2, has two input types. First, it is conditioned on a goal g (4) E G expressed as language. This goal can be embedded using the language encoder Fyg into a sequence of D-dim. vectors. Second, it consumes visual observations 01,...,00 Ot dur- ing the policy rollout which are encoded using separate learnable observation encoder module The observation encoder module consists of a vision encoder that produces a visual embedding and an MLP network that projects the visual embedding to the language model token embedding dimension. The encoded text and visual observations create a k + t length sequence of D-dimension embeddings, which are input to the LLM backbone viLLM. as defined above.
E :6 R.D
To decode action as output, we employ a learnable action output module D : RD D(A) that converts the output of the LLM backbone to distribution over actions from A. With the two additional adapter modules D. and EY we are able to adapt the LLM take as input goal specification and visual observations up time f in order output action time t:
n. LaRRP (at
.r.g) g) = (w (EE (1),...,, EG (1x), EX (01). (o)).
The action output module is an MLP that predicts a distribution over environment actions. We exclude the first outputs from wLLLM that correspond to the language task specification tokens. The hidden outputs corresponding to observations at each time step are fed through the action modeling MLP produce the distribution over actions. The action output module also predicts the value estimate used for the reinforcement learning update. More details about the visual encoder and action output module are Appendix C.I.
3.2 LARGE LANGUAGE MODEL REINFORCEMENT LEARNING POLICY TRAINING
We train LLaRP using only reinforcement learning (Sutton & Barto, 2018). Specifically, we train with DD-PPO (Wijmans et al., 2019), an adaptation of PPO (Schulman et al., 2017) for multi- GPU training. LLaRP collects experience interactively by rolling out its policy suto-regressively to generate actions to take in the environment. For the PPO update, we sample minibatches from the collected data and compute action log probabilities in parallel for the PPO update.
During training, we freeze the LLM backbone and the visual encoder. A frozen visual encoder helps maintain visual features that can generalize different environments (Majumdar et al., 2023). A frozen LLM backbone helps maintain language reasoning capabilities, which can be lost during fine tuning (Alayrac et al., 2022). For more details about the method, refer to Appendix C.I.
4

---

Published as a conference paper at ICLR 2024
Dataset Name
Instruction Example
Description
Train
Find apple and put away the fridge.
Instructions used train the agent.
Same instructions, but new scenes. All other datasets are new scenes.
New Scenes
Find an apple and put it away in the fridge
Paraphrasic Robustness
Instruction
the fridge, stow an apple.
Swup the order that nouns appear in the in- struction and substitute synonyms for verbs.
Rephrasing
Referring Expressions
Find the round red fruit and pit it in the fridge
Refer to objects by their visual appearance.
Spatial Relationships
Find on apple and put
receptacle
Refer to receptacles indirectly by their location relative to other receplacles.
the right of the kitchen counter.
I am hungry for something sweet and healthy. snack for me the table.
Describe a situation where particular object fits the context.
Cuntext
Irrelevant Instruction Text
There is pear on the counter. Find an apple and put away in the fridge.
Instructions that include irrelevant context.
Multiple
Find an apple, pear, and banana and put them away the fridge.
Rearrange objects (2 max during training).
Rearrangements
Novel Objects
Find an orange and put it away in the fridge
New entity instruction pairs.
Multiple
Behavior
Objects
Put all the apples in the fridge.
Find all of object.
Conditional Instructions
If the fridge is open, find apple and it there. Otherwise put an apple on the table.
Adjust behavior based on if the conditional statement is true.
Table 1: Evaluation datasets. The datasets with unseen instructions are divided into two categories: paraphrastic robustness which tests if the agent can produce the same behavior under linguistic variation and behavior generalization where the agent has to demonstrate a new type of behavior. The red text highlights the concept in the instruction distinct from the training distribution that emphasizes the dataset evaluation purpose. See Appendix B.5 for more details.
LANGUAGE REARRANGEMENT PROBLEM
To study generalization properties across a large number of language conditioned tasks we introduce a novel problem called "Language Rearrangement". Language Rearrangement strives to cover a large number of tasks of home environment tasks, such as, "Bring a mug to the couch.", "Store all the fruit in the fridge.", or "I am hungry, bring something from the kitchen to the table.". This problem space extends the Rearrangement task (Batra et al., 2020) by defining 150,000 training and 1,000 testing tasks, and providing a textual instruction for each one. The tasks require agent to generalize to a variety of unseen instruction concepts requiring multiple object interactions (picking, placing, opening, closing), searching for objects and logical reasoning (e.g. "if" statements).
4.1 TASK DEFINITION
In Language Rearrangement, agent starts in an unseen house and is tasked to execute a com- mon household activity, that reduces to moving objects from specified start positions to desired goal positions. The agent provided with a natural language instruction specifying the desired goal state. We generate instructions at scale using instruction templates, instruction template rephras- ings, sceneegrounded entity enumeration, and solver feasibility validation check. A sparse reward is provided for successfully completing the entire task or a subtask. Completing the instructions requires the agent to explore. For example, to "Put all the dishes in the sink if the sink is empty" requires the agent to explore to find all the dishes in the house. While exploring, the agent needs detect when sees plate and then pick it up. The instructions vary in what information is re- vealed to the agent (such as the starting positions of objects), how many objects to rearrange, and logical concepts such as "for all". "exists", conditionals, swapping, and removals. See complete task details in Appendix B. We compare Language Rearrangement to other benchmarks in detail in Appendix B.7.
The agent has to perform the tasks entirely from onboard sensing capabilities and without any pre- built maps, object 3D models, or other privileged information. A simulated Fetch robot (Fetch Robotics, 2020) senses the world through a 256 x 2556-ixel head-mounted RGB camera, robot joint positions, an indicator whether the robot is holding an object or not, and base egomotion giving the relative position of the robot from the start of the episode. The agent interacts with the world
5

---

Published as a conference paper at ICLR 2024
via mobile base, 7DoF arm, and suction grip attached to the arm. Language Rearrangement is implemented in Habitat 2.0 (Szot et al., 2021). When the policy selects a valid low-level skill policy that can be executed in the current state, the simulation is kinematically updated with the skill post conditions.
We supplement all methods with low-level skills and focus high-level decision-making, Lan- guage Rearrangement poses the challenge of long-horizon tasks spanning tens of thousands of low- level control. Even single-object mobile pick and place tasks are challenging for end-to-end meth- ods (Berges et al., 2023; Huang et al., 2023c). Yet hierarchical methods where a high-level policy selects from fixed skills is effective for rearrangement tasks (Gu et al., 2022). Therefore like other rearrangement works (Szot et al., 2023), we train high-level policy to select low-level policies. There are 70 skills for the high-level policy to choose between at every step. These skills include picking objects by name, placing on receptacles by name, navigating to receptacles, and opening and closing receptacles by name. Example skills include pick(apple), place(sink), navigate(left counter), and open(fridge), See Appendix B.3 for details.
4.2 GENERALIZATION TEST DATASETS
Language Rearrangement seeks to evaluate how well agents can complete language instructions during evaluation that are different from those seen during training. To facilitate this, we generate dataset of 150kk distinct training instructions covering basic rearrangement concepts, from single- object interactions to finding and moving two objects. The train instructions include several phras- ings of the same rearrangement concept and cover interacting with various objects and receptacles. For evaluation, we construct an evaluation dataset that tests the generalization capabilities with re- spect to different rearrangement concepts expressed with language (see Table and Appendix B.5). All evaluation datasets are in unseen houses.
The ultimate goal of training models capable of solving tasks from natural language instruction is to allow humans to easily provide instructions to embodied agents. However, humans exhibit high variability how they describe a task (Schreitter & Krenn, 2014), so it is necessary that agents robust to paraphrasing. Paraphrasic Robustness (PR) evaluates if the agent can produce the same behavior under linguistic variation. Such variations include new ways of saying the instruction and referring objects indirectly rather than by name. The underlying goal of these instructions are contained in the training dataset.
Paraphrastic Robustness (PR):
Behavior Generalization (BG): The second category of datasets test Behavior Generalization (BG) in which the agent has to demonstrate a new type of behavior specified by the language instructions not present in the training dataset. This involves a new logical expression. For example, during training the agent was told how many objects to find. But BG includes the Multiple Rearrangements task, which requires an agent to find "all" of a particular object.
We believe these two axes of generalization have large coverage in terms of realistic semantic sit- uations that robot could encounter. PR and BG roughly align with generalization concepts in psychology; PR is similar to what is referred to as "stimulus generalization" in psychology, while BG can be thought of as type of "response generalization" (Shepard, 1957), though in the latter we do not keep the linguistic instruction the "stimulus" fixed.
5 EXPERIMENTS
5.1 BASELINES
We compare LLaRP to the following baselines. For all methods in Language Rearrangement, we use the pre-trained VisualCortex model (VC1, Majumdar et al., 2023) a ViT backbone designed for egocentric visual tasks. We represent the visual observations using the ViT (CLS) token from VC1. We freeze the VCI visual encoder module. All RL methods are trained with PPO. Further details for all methods are in Appendix C.
ZS-ChatGPPT/ZS ZS- Input the instruction to the LLM and zero-shot (ZS) plan sequence of bigh-level actions. This policy is blind and plans based only on the language instruction. The prompt lists all the constraints (e.g., only one object can be picked receptacles and objects, along with examples of successful behavior. We compare against an instruction-tuned LLLMA-65B (ZS-LLaMA) (Touvron al., 2023) which generates plan only
time), possible actions,

---

Published as a conference paper at ICLR 2024
100
LLaRP
ZS-ChatGPT
LLaRP Scratch
Z5 LLaMA
80
LSTM-F Fan
ZS Flamingo
Success (95)
LSTM-LLaMA
60
40
20
melevant
Rotemiing
Spattal
Rephrasing
Rearrange
Dejects
cibjects
Instructs
Paraphrastic Robustness
Behavior Oeneralization
Figure 3: Success Rates across all evaluation tasks, aggregated in groups per Table Across all unseen task groups, LLaRP generalizes better than the respective baselines, apart from multiple rearrangement, in which all models perform poorly. For exact numbers, see Table
once at the beginning. We also compare against ChatGPT (ZS-ChatGPT) that performs multi-step reasoning to iteratively refine the plan based on proprioceptive feedback from the environment. For details about the ZS baselines, prompt, and environment feedback see Appendix C.4.
ZS-Flamingo: Multimodal (text image) version of the ZS-LLaMA bascline. Uses IDEFICS, an open source 80B parameter VLM model (Laurençon et al., 2023) based on Flamingo (Alayrac al., 2022). Given a prompt similar ZS-L.LMMA, the image observation and the textual in- struction, FS-Flamingo outputs a single plan for the agent to follow.
LLaRP-Scratch: Same architecture as LLaMA-7B but with 2 billion parameters. The entire transformer is trained from scratch and the pretrained visual encoder is frozen.
LSTM-Flan/ L.STM-LLLMA: The instruction is encoded
fixed-length vector that LSTM takes as input along with the observation encoding. The action is predicted from the LSTT hidden state. LSTM-Flan uses Flan-T5 (Chung al., 2022) to encode the instruction while LSTM- LLaMA uses LLaMA and Perciever Resampler network (Jaegle et al., 2021).
5.2 EMPIRICAL ANALYSIS
In this section, we analyze the empirical performance of LLaRP and the baselines on Language Rearrangement. We show LLaRP has better zero-shot generalization capabilities than the baselines on most of the unseen Language Rearrangement evaluation datasets. LLaRP also learns efficiently, learning faster during training and comparing favorably to training with expert demonstrations.
LLaRP improves generalization across all generalization types. We report aggregate success rates across all generalization datasets in Table We report the mean and standard deviation across 3 random seeds for all RL-based methods. We that LLaRP is almost 1.7x better than the next best performing baseline, 42% vs. 25%. This trend is true for both Paraphrastic and Behavior generalizations, which shows that the of LLM allows for a model that can better understand natural language and execute novel tasks. Althought this is expected for paraphrastic robustnes as LLMs are known for their language understanding capabilities, is somewhat surprising see that the LLM helps even for novel behaviors, achieving 45% vs. 28% from LSTM-Flan.
Results across individual generalization test sets are shown in Figure 3. LLaRP displays superior generalization capabilities across all settings. Most learned methods achieve near 100% success rates on the train split and almost 100% when varies the scene, but uses the same instructions as in train.
Paraphrastic Robusiness
Behavior
Method
Total
Generalization
LLaRP
42 2
38
45 3
LLaRP-Scratch
17 4
16
18 1
1.STM-Flan
23
28 11
LSTM-LLaMA
2
3
0
ZS-ChatGPP
22
21
23
ZS-LLaMA
12
10
14
Flaminnnoo
LLaRP performs 7% and 12% better when we rephrase instructions or talk about novel objects compared to the next best performing model. However, the boost LLaRP gets from using a LLM is substantial for more complex novel tasks, g. when context, have conditional statement, multiple chained rearrangements, etc.
Table 2: Combined zero-shot success rate. LLaRP outperforms all baselines in all categories.

---

Published as a conference paper at ICLR 2024
100
RL E--LaRR
Success Rate (%)
Success Patee (N)
60
Success
SaRP--cratch Llan-Flan
LLaRP
LSTM-Flan
10
LSTM LLaMA
Environment
Environment Steps
Instruction Count
(a)
(b)
(c)
Figure 4: Success rates (SR) for different stages training (left and middle) or number of episodes / demonstrations (right). (a) SR vs number of steps during learning. (b) SR vs number of steps during
continual learning on "Multiple Rearrangments" tasks. (c) LLaRP with RL vs LLaRP with IL with the same number of episodes/ demonstrations. See text for further discussion.
The second best performing model is LSTM-Flan, which like LLaRP uses billion parameter lan- guage model (Flan-T5). However, LSTM-Flan uses to comprehend instructions through encoding the instruction fixed size vector, rather than using the LLM as decoder like LLaRP. LSTM- Flan generalizes some scenarios such as to new phrasings and objects, indicating that the LSTM policy learned to interpret these aspects of the FLAN embedding. Whereas unlike LLaRP, LSTM- Flan performs worse on Behavior Generalization than Paraphrasic Robustness, indicating that using the LLM directly as decoder is important for generalization to new behaviors. We also compare to LSTM-LLaMA which uses the same LLaMA hidden state activations as LLaRP. However, it performs generally worse, perhaps because LLaMA was not trained as an encoder model.
The zero-shot application of LLM, ZS-ChatGPT, requires no training and cannot receive images as inputs. Nevertheless, it achieves hetter-than-random performance across many of the evaluation splits, which demonstrates that the L.M contains relevant information for embodied tasks.
LLMs lead faster learning convergence. As shown Figure 4a, LLaRP is the most sample efficient model, converging in around 50M-100M steps before LSTM-Flan, despite being a sub- stantially larger model. Further, LLaRP-Scratch takes 10x more environment samples to converge than LLaRP (50M versus 500M) despite both having the same architecture, showing that pre-trained LLMs are a good fit for the problem space.
Note that only few prior works have used transformers for online RL (Parisotto et al., 2020). In Appendix D.I, we further explain the training architecture and settings that were critical to stably train LLaRP and baseline transformer policy training with PPO.
LLMS leads to faster continual learning convergence. To evaluate the continual learning efficiency, continue learn- ing the model on downstream tasks beyond the training tasks and analyze the training convergence. Specifically, we take the "Multiple Rearrangements" dataset, generate 10K episodes, and train until convergence. The results, shown in Fig- ure 4b, show that LLaRP is 3x more efficient than LSTM-Flan, achieving near perfect performance 500k vs 1.5M environ- ment steps. Hence, LLMS can lead to faster learning of addi- tional tasks.
LLaRP-7b
LLB
Success Rare 05a
50
35
LLaRP with RI performs better and utilizes supervision more efficiently than II., A common paradigm of endow- ing LLMs with novel capabilities is is Imitation Learning (IL) with decision making data. More recently, works have en- dowed LLMswith decision making capabilities in embodied settings (Driess et al., 2023; Brohan et al., 2023).
Agg Eval
Para Robust
Behav Gen
Figure 5:
Average success rates
across our evaluation datasets for LLaRP scaled from Llama-7b to Llama-13b
Hence, natural question is compare LLaRP trained with RL vs IL. IL uses demonstrations generated by a fully trained LLaRP policy. For each data point, we use the same number of episodes for RI. as demonstrations for II., both denoted by instruction count, and train until convergence.
8

---

Published as a conference paper at ICLR 2024
Locate tool to fix loose screw in the sofa and bring it there.
navigate(brown_tabbl (ttbbll navigate(dark_tabl
3
navigata(blue__tn -table)
4
pick(screwcrriver)
5 navigate(sofa)
6
place(screwdriver, sofa)
Figure 6: Success trajectory in the Context dataset. Arrows indicate navigation receptacles. Right shows actions selected by LLaRP. Upper-left show egocentric observations for some steps.
In Figure 4c, we that for any number of episodes (or demonstrations) RI. outperforms II., with a larger margin in the low regime of data. This shows that in RL settings, an LLM-based LLaRP is able to collect episodes that are useful for policy improvement. For RL we merely need reward definition, while in IL settings full demonstration trajectories are required, thus making RL less costly than IL for each instruction count.
Larger LLMs lead to better results. In Figure 5, we show the effect of scaling the size of the LLM LLaRP. We compare using LLaMA- 13B and LLaMA-7B in LLaRP, We find that LLaMA- 13B gives a 4% boost in total evaluation performance. This indicates that larger, more capable LLMs, can translate to more capable embodied reasoning.
Qualitative Result. In Fig. 6, we show a success example of LLaRP on the Context dataset. The agent explores for first actions. Then it finds the screwdriver implied by the phrase "fix a loose screw" in the instruction. It then successfully brings that screwdriver to the couch. For more qualitative examples in the other datasets, see Appendix and videos at https://llm-a githhuuu..
LLMs boost performance with tasks beyond Language Rearrangement. We evaluate LLaRP across Atari 2600 games using the Arcade Learning Environment (Bellemare et al., 2013), configured following the recommendations of (Machado al., 2018). We train LLaRP and LLaRP- Scratch as in Habitat, with the exception of using a fully- trainable ResNet-18 visual encoder, Atari is visually dis- tinct from the training data used for VC-1. To take advan- tage of the diversity available Atari environments, we train LLaRP and LLaRP-scrater individually each game of 55 Atari games for 100M environment steps, and report human- normalized scores (Mnih et al., 2013).
2.5
LLaRP
2.0
1.5
1.0
0.5
0.0
IQM
Median
Mean
Figure 7: LLaRP and LLaRP-
Scratch performance over 55 Atari games (two seeds per game).
We find that LLaRP outperforms LLaRP-Scratch by a large margin, with LLaRP achieving a higher average score 43 out of 55. When aggregated, LLaRP's mean, median, and in- between 3x and 4x higher than LLaRP-
terquartile mean (Agarwal et al., 2021) performance Scratch (see Figure 7). Additional details about our Atari experiments can be found in Appendix C.6.
Further Experiments. Additional analyses in Appendix D show the impact of batch size and LLM weights for RL with transformers, unfreezing LLM weights for LSTM-Flan Flan,,,n, LLaRP efficiency, and that our findings hold in harder setting with no invalid actions and a termination action.
6 CONCLUSION
We introduce LLARP,, a scheme for employing pretrained LLMs on embodied tasks with reinforce- ment learning. To aid in our research, we introduce dataset of rearrangement tasks (consisting of 150k training instructions and 10 challenging evaluation datasets). LLaRP outperforms non- pretrained transformer- and LSTM-based models on both sample efficiency and generalization. Limitations to be addressed in the future include the significantly larger size of LLMs than typi- cal RL models. Training the action decoder module may also hinder the policy's ability leverage the world knowledge from the LLM. Future work can explore how LLaRP can directly interact with the environment via the language head of the L.M,, without the need for action decoder module.

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGEMENTS
The authors would like to thank Eduord Grave, Mathias Muller, Josh Susskind, Barry Theobald, Yizhe Zhang for valuable comments and discussion.
REFERENCES
Josh Abramson, Arun Ahuja, Federico Carnevale, Petko Georgiev, Alex Goldin, Alden Hung, Jes- sica Landon, Timothy Lillicrap, Alistair Muldal, Blake Richards, et al. Evaluating multimodal interactive agents. arXiv preprint arXiv: 13277 2022.
Constructions Aeronautiques, Adele Howe, Craig Knoblock, ISI Drew McDermott, Ashwin Ram, Manuela Veloso, Daniel Weld, David Wilkins SRI, Anthony Barrett, Dave Christianson, et al. Pddl- the planning domain definition language. Technical Report, Tech. Rep., 1998. 17
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. NeurIPS, 2021. 9
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01699, 2022.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-
23736, 2022. 1, 3, 4, 24
Dhruv Batra, Angel X Chang, Sonia Chernova, Andrew Davison, Jia Deng, Vladlen Koltun, Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, et al. Rearrangement: challenge for embodied ai. arXiv preprint arXiv:2011.01975 2020. 5
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal Artificial Intelligence Research, 47, 2013. 9
Vincent-Pierrs Berges, Andrew Szot, Devendra Singh Chaplot, Aaron Gokaslan, Roozbeh Mottaghi, Dhruv Batra, and Eric Undersander. Galactic: Scaling end-to-end reinforcement learning for rear- rangement at 100k pees--per-secon. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13767-13777, 2023. 6
Benjamin Bolte, Austin Wang. Jimmy Yang, Mustafa Mukadam, Mrinal Kalakrishnan, and Chris Paxton. Usa-net: Unified semantie and affordance representations for robot memory. arXiv preprint arXiv:2304.12164 2023. 3
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for sal-world control at scale. arXiv preprint arXiv:22 22.0//2../.... 2022. 3
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choroman- ski, Tianli Ding, Danny Dricss, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint 1581v 2023. 1,3,
Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. The yeb object and model set: Towards common benchmarks for manipulation research. In 2015 international conference on advanced robotics (ICAR), pp- 510-517. IEBE, 2015. 20
Thomas Carta. Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre- Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement
learning. arXiv preprint arXiv:2302.02662. 2023. 3,
10

---

Published as a conference paper at ICLR 2024
Georgia Chalvatzaki, Ali Younes, Daljeet Nandha, An T. Le, Leonardo F. R. Ribeiro, and Iryna Gurevych. Learning to reason over scene graphs: a case study of finetuning gpt-2 into a robot language model for grounded task planning. Frontiers in Robotics and AI, 2023. 2
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 14:15584-15997, 2021a. 27
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.033: 0333444 2021b. 1
Maxime Chevalier-Boisvert,, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. AabyAl: First steps towards grounded language learning with a human in the loop. In ICLR, 2019. URL https: Btttttt:////peeeeett// //tp///ppppeeweoeee/ee net forum?id- IJeXCoOcYX. 22
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. Scaling mnstruction-finetunee language mod- cls. arXiv preprint arXiv:2210.114100 2022. 21
Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied ai using procedural generation. Advances in Neural Information Processing Systems, 35:5982- 5994, 2022. 22
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. / 1929, 2020. 23
Danny Driess, Fei Xia, Mehdi SM Sajadi,, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multi- modal language model. arXiv preprint arXiv:2303.0333 2023. 3,
Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding pretraining in reinforcement learning with large language models. arXiv preprint arXiv:2302.066 02669, 2023. 2
Kevin Esslinger, Robert Platt, and Christopher Amato. Deep transformer q-networks for partially observable reinforcement learning. arXiv preprint arXiv:2206.01077 2022. 28
Fetch Robotics. Fetch. http: /ttt:://feteee http:///fcccctttoooooo cs. com 2020. 5
Richard Fikes and Nils Nilsson. Strips: A new approach to the application of theorem proving
to problem solving, Artificial intelligence, 288-4):189-208, 1971. 19
Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, and Shuran Song. Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
23171-23181, 2023. 3
Chuang Gan, Jeremy Schwartz, Seth Alter, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, et al. Threedworld: A platform for interactive multi-modal physical simulation. arXiv preprint XXiv:2007.04954 2020. 22
Amelia Glaese, Nat McAleese, Maja Trkebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Mari- beth Rauh. Laura Weidinger, Martin Chadwick, Phoebe Thacker, al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14333 2022.
Ran Gong. Jiangyong Huang, Yizhou Zhao, Haoran Geng, Xiaofeng Gao, Qingyang Wu, Wensi Ai, Ziheng Zhou, Demetri Terzopoulos, Song- Chag-Chun Zhu, et al. Amold: A benchmark for Linguage-groundee task learning with continuous states in realistic 3d scenes. arXiv preprint arXiv:2304.0432 04321,, 2023. 22
11

---

Published as a conference paper at ICLR 2024
Jiayuan Gu, Devendra Singh Chaplot, Hao Su, and Jitendra Malik. Multi-skill mobile manipulation for object rearrangement. arXiv preprint arXiv:2209.02778, 2022. 6
Huy Ha and Shuran Song. Semantic abstraction: Open-world 3d scene understanding from 2d vision-language models. In 6th Annual Conference on Robot Learning, 2022.
Huy Ha, Pete Florence, and Shuran Song. Scaling up and distilling down: Language-guinn guidded robot
skill acquisition. arXiv preprint arXiv:2307.14553 2023. 3
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-lIm: Injecting the 3d world into large language models. arXiv preprint arXiv:2307. 230.::2937..288... /298/, 2023. 3
Hengyuan Hu and Dorsa Sadigh. Language instructed reinforcement learning for human-ai coordi- nation. arXiv preprint arXiv:2304.07297, 2023. 2
Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual language maps for robot navigation. In 2023 IEEE International Conference Robotics and Automation (ICRA), pp- 10608-10615. IEEE, 2023a. 3
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint (iti::2227.050000 2022.
Wenlong Huang, Fei Xia, Dhruv Shah, Danny Driess, Andy Zeng, Yao Lu, Pete Florence, Igor Mordatch, Sergey Levine, Karol Hausman, et al. Grounded decoding: Guiding text generation with grounded models for robot control. arXiv preprint arXiv:2303. 2003000050000. 2023b.
Xiaoyu Huang, Dhruv Batra, Akshara Rai, and Andrew Szot. Skill transformer: A monolithic policy for mobile manipulation. arXiv preprint arXiv:2308.0998 09873, 2023c. 6
Andrew Jaegle, Sebastian Borgeaud, Jean- Baptiste Alayrac, Carl Doersch, Catalin lonescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv: 1rX 107.14795, 2021. 7, 23
Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, et al. Conceptfusion: Open-set multimodal 3d mapping. arXiv preprint arXiv:2302.000 330.00222, 2023. 3
Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal generation. arXiv preprint arXiv:2301.13825 2023.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12-THOR: an interactive 3d environment for visual AL. arXiv preprint arXhv:1712.05474, 2019.
Hugo Laurengon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang. Siddharth Karamcheti, Alexander M Rush, Douwe Kiela, al. Obelisc: An open web-scale filtered dataset of interleaved image-text documents. arXiv preprint arXiv:2306.1633 16527, 2023. 7, 24
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Sola et al. Solving quantitative reasoning problems with language models. Advances Neural Information Processing Systems, 35:3843-3857, 2022. 1
Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. Behavior- lk: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference Robot Learning, pp. 80-93. PMLR, 2023a. 1, 22
12

---

Published as a conference paper at ICLR 2024
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-
image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.1259/ 2023b.
Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong. and Teknium. Openorea: An open dataset of gpt augmented flan reasoning traces. https//httpss/muggggggae...oo htwp://aaaaaaa.pppppooonaaraeaaa
2023. 24
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference Robotics and Automation (ICRA), pp. 9493-9500. IEEE, 2023. 2
Hao Liu, Lisa Lee, Kimin Lee, and Pieter Abbeel. Instruction-followingg agents with jointly pre- trained vision-language models. arXiv preprint arXiv:2210. 13431, 2022.
Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots in real time. IEEE Robofics and Automation Letters, 2023. 3
Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents. Journal of Artificial Intelligence Research, 61:523-562, 2018. 9
Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Pieter Abbeel, Jitendra Malik, Dhruv Batra, Yixin Lin, Olek- sandr Maksymets, Aravind Rajeswaran, and Franziska Meier. Where are we in the search for an artificial visual cortex for embodied intelligence? arXiv preprint arXiv:2303.1 18240, 2023. 4, 6
Oier Mees, Lukas Hermann, Erick Rosete-Beas. and Wolfram Burgard. Calvin: A benchmark for language-conditionee policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 7(3):7327-7333 2022." 22
Volodymyr Mnih, Koray Kavukcuoglu. David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 1312v:1312.5602. 2013. 9,
Steven Morad, Ryan Kortvelesy, Matteo Bettini, Stephan Liwicki, and Amanda Prorok. Popgym: Benchmarking partially observable reinforcement learning. arXiv preprint arXiv:2303.01859 2023. 28
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and
Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4, 2023.
24
Vivek Myers, Andre He, Kuan Fang, Homer Walke, Philippe Hansen Esttruh,, Ching-An Cheng, Mihai Jalobeanu, Andrey Kolobov, Anca Dragan, and Sergey Levine. Goal representations for instruction following: A semi-supervised language interface to control. arXiv preprint
arXiv:2307.00117, 2023. 3
Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free rl can be strong baseline for many pomdps. arXiv preprint arXiv:2110.05038, 2021. 27
Tianwei Ni, Michel Ma, Benjamin Eysenbach, and Pierre-Lue Bacon. When do transformers shine in rl? decoupling memory from credit assignment. arXiv preprint arXiv:2307.03888 2023. 28
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for reinforcement learning. In International conference on machine learning, pp. 7487-74998
PMLR, 2020. 8, 27, 28
13

---

Published as a conference paper at ICLR 2024
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint 1.XYv:2306.14824. 2023. 1
Marco Pleines, Matthias Pallasch, Frank Zimmer, and Mike Preuss. Memory gym: Partially observ- able challenges memory- based agents. In The Eleventh International Conference on Learning Representations, 2022. 28
Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Πlya Sutskever. Learning transferable visual models from natural language supervision. In ICMI,, 2021. 24
Stephanie Schreitter and Brigitte Krenn. Exploring inter-and intra-speaker variability in multi-modna task descriptions. In The 23rd IEEE International symposium robot and human interactive communication, pp. 43.48.147 IEEE, 2014.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. 24
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 4
Dhruv Shah, Btažej Osiński, Sergey Levine, et al. Lm-nav: Robotic navigation with large pre- trained models of language, vision, and action. In Conference on Robot Learning, pp. 492-504. PMLR, 2023. 3.3
Roger N Shepard. Stimulus and response generalization: A stochastic model relating generalization to distance psychological space. Psychometrika, 22(4):325-345, 1957, 6
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi,
Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions
for everyday tasks. In CVPR, 2020. 3, 22
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning, pp. 894-906. PMLR, 2022. 3, 22
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pp. 785-799. PMLR, 2023.
Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B Tenenbaum, Leslie Pack Kaelbling, and Michael Katz. Generalized planning in pddl domains with pretrained large language models. arXiv preprint arXiv:2305.11014 2023. 2
Vitalii Sopov and Ilya Makarov. Transformer- based deep reinforcement learning in vizdoom. In International Conference Analysis Images. Social Networks and Texts, pp. 96-110. Springer, 2021. 28
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, et al. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in Neural Information Processing Systems, 34, 2021. 1, 6, 19, 20, 21, 22
Andrew Szot, Karmesh Yadav, Alex Clegg, Vincent-Piees Pierre Berges, Aaron Gokaslan, Angel Chang, Manolis Savva, Zsolt Kira, and Dhruv Batra. Habitat rearrangement challenge 2022. https: / habitat org/ chal weelllengee2222 2tap:/2222222_n22ee reerrerrneee 2022. 22
14

---

Published as a conference paper at ICLR 2024
Andrew Szot, Unnat Jain, Dhruv Batra, Zsolt Kira, Ruta Desai, and Akshara Rai. Adaptive coordi- nation in social embodied rearrangement. arXiv preprint wwXiv:2306.00087, 2023.
DeepMind Interactive Agents Team, Josh Abramson, Arun Ahuja, Arthur Brussee, Federico Camevale, Mary Cassin, Felix Fischer, Petko Georgiev, Alex Goldin, Mansi Gupta, al. Cre- ating multimodal interactive agents with imitation and self-supervised learning. arXiv preprint urXiv:2/12.03777 2021. 3
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze -ze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint (tt::22011002.:33 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302. 13971, 2023. 6, 23, 24
Aaron Walsman, Muru Zhang, Sanjiban Choudhury, Dieter Fox, and Ali Farhadi. Impossibly good experts and how to follow them. In The Eleventh International Conference on Learning Repre- sentations, 2022. 20
Guanzhi Wang. Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305. 16291, 2023. 2
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models, Advances in Neural Information Processing Systems, 35:24824-24837, 2022. 1
Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames. arXiv preprint arXiv: 10X.::1911.00355 2019. 4, 23
Jimmy WU, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song, Jeannette Bohg, Szymon Rusinkiewicz, and Thomas Funkhouser. Tidybot: Personalized robot assistance with large language models. arXiv preprint arXiv:2305.05668, 2023a.
Yue Wu, So Yeon Min, Yonatan Bisk, Ruslan Salakhutdinov, Amos Azaria, Yuanzhi Li, Tom Mitchell, and Shrimai Prabhumoye. Plan, climinate, and track-language models are good teachers for embodied agents. arXiv preprint arXiv:2305.02412. 2023b. 2
Ted Xiao, Harris Chan, Pierre Sermanet, Ayzaan Wahid, Anthony Brohan, Karol Hausman, Sergey Levine, and Jonathan Tompson. Robotic skill acquisition via instruction augmentation with vision-language models. arXiv preprint 2rXiv:2211.117250 2022.
Mengdi Xu. Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In International Con- PMLR, 2022. 27
ference Machine Learning, pp.
Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin Wang, Mukul Khanna, Theophile Gervet, Yen Yang, Vidhi Jain, Alexander William Clegg, John Turner, et al. Homerobot: Open-vocabulary mobile manipulation. arXiv preprint arXiv:2306.1 //565, 2023.
Albert Yu and Raymond J Mooney. Using both demonstrations and language instructions to effi-
ciently learn robotic tasks. arXiv preprint arXiv:2210.04476. 2022. 3
Tianhe Yu, Ted Xiao, Austin Stone, Jonathan Tompson, Anthony Brohan, Su Wang, Jaspiar Singh, Clayton Tan, Jodilyn Peralta, Brian Ichter, et al. Scaling robot learning with semantically imag- ined experience. arXiv preprint arXiv:2302. 11550, 2023.
Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani. et al. Transporter networks: Re- arranging the visual world for robotic manipulation. In Conference on Robot Learning, pp. 726- 747. PMLR, 2021. 22
15

---

Published as a conference paper at ICLR 2024
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong. Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, et al. Socratic models: Com- posing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00558, 2022.
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. arXiv preprint arXiv:2202.05607, 2022, 27
16

---

Published as a conference paper at ICLR 2024
The code and video examples for the Language Rearrangement benchmark and LLaRP can be found at https://llm-rl. gl..guub...o.. The appendix is structured as follows:
A Author Contributions.
B Additional details about Language Rearrangement and dataset analysis.
C LLakP and baseline details.
D Additional experiments including:
D.1 Analyzing important aspects of training transformers with RL.
D.2 Additional experiments including full Language Rearrangement zero-shot generalization numbers, comparing efficiency results, and the choice of unfreezing the Flan encoder in LSTM-Flan.
D.3 Comparing performance a harder version of Language Rearrangement where invalid ac- tions terminate the episode and stop action is required.
E Visualizations of success trajectories of LLaRP on the Language Rearrangement evaluation datasets.
A CONTRIBUTIONS
Andrew Szot co-initiated the project, design both the benchmark and co-designed and implemented the algorithmic solution, conducted all RL and IL experiments on Habitat and their analysis, and co-wrote the paper.
Max Schwarzer ran experiments on Atari and co-wrote the paper.
Harsh Agrawal implemented, improved, and analyzed the zero-shot baselines, worked on dataset and evaluation methodology. and co-wrote the paper,
Bogdan Mazoure help with debugging training setup and co-wrote the paper.
Rin Metcalf Susa investigated non-LLaMA LLMs and edited the paper.
Walter Talbott ran model probing analysis and contributed to discussions.
Natalie Mackraz worked RL policy text input and provided feedback for the paper.
Devon Hjelm advised on experiment setup, helped shape up the overall story of the work, and co-wrote the paper.
Alexander Toshev co-initiated the project, managed and advised throughout the project, co- designed the algorithmic solution, and co-wrote the paper.
B LANGUAGE REARRANGEMENT DETAILS
B. PLANNING DOMAIN DEFINITION LANGUAGE (PDDL) DETAILS
Language Rearrangement is represented with Planning Domain Definition Language (PDDL) specification (Aeronautiques et al., 1998). Specifically, each episode is linked to PDDL problem specification. This specification consists of the following components:
Entity types: Each entity in the scene is associated with an entity type. This entity type is used determine which predicates are applicable. The type system hierarchical, and derived types also apply to higher level types. We define core types such as pickable object and receptacle. We automatically derive types from the object dataset based on object high-level categories such as fruit and object classes such apple.
Entities: For each object and receptacle in the scene, the PDDL associates it with a symbolic entity. Each entity has an associated entity type. The entities automatically populated for each rearrangement episode based on the loaded objects and receptacles.
Predicates: These binary expressions that are evaluated based the underlying simulator state. For example, on top(X: pickable object. Y: receptacle) corresponds to if object on top of receptacle Y.
17

---

Published as a conference paper at ICLR 2024
Between episodes, the only component that changes is the "Entities" category since each episode will have different objects. The same entity types and predicates apply between episodes.
A predicate expression refers to a boolcan expression stated in first order logic involving the PDDL components. Predicate expressions involve the predicates, entities, entity types, logical connectives ("and", "or", "not"" and quantifier symbols ("for all", "exists"). For example, the predicate expres- sion 3 x is Jype (x, "apple"), on onpop( "table") ") will be evaluated to true, when any apple is placed on top of the table, and false otherwise. As described in the next section, we use predicate expressions to describe success criteria for instructions.
B.2 INSTRUCTION GENERATION DETAILS
We create scalable pipeline for generating large number of satisfiable, plausible instructions from a small number of instruction templates. Each instruction is associated with predicate expression defining the success criteria for that instruction. These components are described in detail below.
Instruction Template: This refers to a particular outcome in the environment and corresponding ways of expressing this outcome in language where nouns are replaced with placeholder variables. Specifically, an instruction template consists of:
Template Goal Condition: This describes the desired outcome for the instruction template represented as an ungrounded predicate expression (predicate expressions are described in Ap- pendix B) It is ungrounded because there placeholder variables for entities rather than ac- tual entities in the scene to refer to multiple outcomes depending on what entities are substituted into the placeholders. For example, for the ungrounded desired outcome of object going receptacle, the goal condition would be the predicate expression 3 x is type (x, "OBJECT"), ), on_lop(x, "RECEPTACLE" where OBJECT and RECEPTACLE Placeholder Constraints: The template goal condition used a set of placeholder variables. But in the predicate expression 3 is type (x, "OBJECT"), on top(x, "RECEPTACLE") we need OBJECT to be a pickable entity type and RECEPTACLE to be a receptacle entity type. The instruction template thus contains constraints on the entities that can be sampled for the template goal condition.
placeholders.
Each instruction template has a set of N ways expressing the template goal condition in ungrounded natural language. It is ungrounded because the lan- guage will contain placeholder variables that will be later substituted for entities in the scene. For example, "Move an 'OBJECT' to the "RECEPTACLE" is language template for the template goal condition 3x: : is type (x, "OBJECT"), on top(x, "RECEPTACIE") We set N to 11, mean- ing each template goal has 11 different language templates, each expressing the same template goal condition with different language.
Instruction Language Templates:
We define set of instruction templates for each of the datasets from Section 4.2. We show examples of the language templates from these instruction templates for all the datasets in Table 7.
The next step of our pipeline uses these instruction templates to generate dataset of rearrangement episodes. When generating a particular dataset, we start by randomly sampling a template from that dataset. We then substitute random entities into the template placeholders, taking into account the placeholder constraints. We then randomly sample a scene that is compatible from the associated scene set. A scene is compatible with an instruction if all the receptacles the instruction refers to are present in the scene. We then populate the scene with objects. We constrain the object sampling for the template. We include all template substituted entities and constraints in the object sampling process. Thus episode consists of:
Scene: The empty scene specifying the house floor plan.
Entity Locations: The transformations of all receptacles and objects.
(Grounded) Goal Condition: A predicate expression describing the desired outcome without any placeholder variables.
(Grounded) Instruction Language: An instruction in natural language specifying the goal con-
dition without any placeholder variables.
Note that the robot starting transformation is not included in the above, and is instead randomly generated at the start of every episode.
18

---

Published as a conference paper at ICLR 2024
30-
Thannig tnatrcctios
25
Count 12
Make:
Indegel
Celrral
Dayn
Trans
-02
(a) Instruction embedding T-SNE.
(b) Avg Tokens per Instruct
Figure 8: Left: Visualization of the evaluation and training dataset instructions. We embed the instruction using Flan. Only the instructions depicted in red are seen during training. Right: The average number of language tokens per instruction each dataset.
We then check that each generated episode is solvable. Random object placement and physics simulation can result objects falling and other instabilities. We check the episode solvable by an oracle planner. We run a STRIPS based planner Fikes & Nilsson (1971) that operates from the PDDL. problem instantiated by the episode, If the planner times out or the solution is longer than 32 high-level steps (a high-level corresponds to a single skill, defined in Appendix B.3), we consider the episode unsolvable and remove it from the training set.
B.3 SKILL DETAILS
We train the policies in abstracted, high-level action space. Each high-level action corresponds to particular skill invocation, for example, pick(apple)". We consider the following skills:
Navigation:
Parameterized by the name of the receptacle to navigate to. So long the receptacle is present in the scene, this skill is always valid
Pick: Parameterized by the name of the object to pick. Only valid if the robot is close to the
object, not holding another object, and the object is not inside a closed receptacle.
Place: Parameterized by the name of the receptacle place the object on. Only valid if the robot
is close to the receptacle and is holding an object.
Open: Parameterized by the name of the receptacle open. Only valid if the receptacle is closed
and the robot is close to the receptacle.
Close: Parameterized by the name of the receptacle to close. Only valid the receptacle is open and the robot is close to the receptacle.
The skill is only executed is valid the current state. The action space consists all possible skill parameterizations given all possible objects, giving 70 total skills meaning 70 actions for the policy to select from. Note that some these skills may be applied to objects that are not present in the current scene, in which case selecting that skill would be invalid action. Furthermore, the actions are fixed at every step, and thus a majority of the actions will be invalid at a given step. We treat invalid actions no-ops, but we compare where the robot is not allowed to take invalid actions in Appendix D.3. If the action is valid, we execute it and instantaneously transform the simulator state based the effect of the skill. For example, if the skill "pick(apple)" is selected and the robot is near the apple, not holding anything, and the apple is not in a receptacle, then the apple will teleport to the robot's gripper.
B.4 ADDITIONAL TASK DETAILS
Language Rearrangement is simulated in Habitat 2.0 (Szot et al., 2021). The task is simulated with kinematic dynamics. As described in Appendix B.3, after the agent selects a valid skill, the simu- lator state is instantaneously transformed the skill post-condition. Only valid skills executed. Invalid skills that result in collisions, impossible interactions (such as the agent grabbing an object out of reach) counted invalid actions.
19

---

Published as a conference paper at ICLR 2024
Hyperparameter
LLaRP
LLaRP-Scratch
LSTM-Flan
LSTM-LLaMA
LR
3e-t
3e-1
3e-1
3e
Optimizer
AdamW
A
Adam
Adam
Number of Mini Batches
6
6
4
4
Environments Per GPU
18
18
18
Entropy Coefficient
0.01
0.01
0.01
0.01
Value Loss Coefficient
0.5
0.5
0.5
0.5
Number of Rollout Steps
32
32
32
32
Number of PPO Epochs
1
2
Batch Size Per Update
768
768
1152
1152
Table 3: Hyperparameters for all RL methods.
The episode is considered successful if the goal condition evaluates to true. The episode is failure if the agent doesn't achieve success in under 32 high-level policy steps. We consider requiring the agent to call separate stop action in Appendix D.3. We use the objects from the YCB (Calli et al., 2015) and ReplicaCAD (Szot al., 2021) object datasets. The object categories used are: "ball, clamp, hammer, screwdriver, padlock, scissors, block, drill, spatula, knife, spoon, plate, sponge, cleanser, plum, pear, peach, apple, lemon, can, box, banana, strawberry, lego, rubriks cube, book, bowl, cup, fork". We hold out "mug, orange, lid, toy airplane, wrench" for the Novel Objects evaluation split.
The Language Rearrangement reward function consists of sparse reward for completing the task, subgoal rewards for completing individual parts of the task, and a slack penalty for completing the task faster. The reward at step is:
rt 10 succoss subgoal 0.1 Livalidd 0.01
Where luccess indicates if the PDDL goal expression is evaluated as true, meaning the episode was successfully solved. inwalid indicates if the agent called an invalid action the current step. subgoal indicates if the agent achieved any subgoal necessary to achieve the overall goal. For example, for the instruction "Find apple and put it away in the fridge", the agent needs to first pick up the apple and potentially open the fridge. When running the STRIPS planner in the episode validation process described in Appendix B.2, compute the optimal action sequence and extract subgoals from this action sequence. We then reward the agent for reaching any of these high-level subgoals. Note we cannot directly imitate the optimal action sequence because is computed with oracle state information and is an "impossibly good" expert (Walsman et al., 2022).
B.5 ADDITIONAL DATASET DETAILS
Further details the Paraphrasic Robustness datasets:
The same underlying instruction goals, but stated in a different way. For example, rephrasing of the training instruction. The order that nouns appear in the instruction is permuted and synonyms for verbs are substituted.
Instruction Rephrasing:
Referring Expressions: Refer objects by their visual appearance rather than directly as their entity name. For example, an "apple" is referred to by its visual appearance as "round red fruit". During training, objects are only referred to by single name. The referring expression was never seen during training.
This refers to receptacles indirectly by their location relative to other receptacles. All receptacles positioned against the walls, there is spatial ambiguity depending on the agent viewpoint. There are no spatial concepts in the training data.
Spatial Relationships:
Context: Describe situation where particular object fits the context. Irrelevant Instruction Text: Instructions that include irrelevant context.
Further details the Behavior Generalization datasets:
Multiple Rearrangements: Generalize to rearranging 3 objects when the agent only rearranges
2 objects during training.
20

---

Published as a conference paper at ICLR 2024
100
Success Rate (%)
80
100
Success Rate (96)
60
LLaRP
LLaRP-Scratch
40
LSTM-Flan
LLaRP-13b
20
LSTM-LLaMA
LLaRP-7b
Lb-FP-7b-FT
Environment Steps
168
Environment Steps
le7
(a) Main task.
(b) Scaling Learning Curves.
Figure 9: Left: Learning curves for all baselines the main Language Rearrangement task from Figure 3. Note that we train all methods for 200M steps except for LLaRP-Scrated as we observed that it had not converged and was still making learning progress at that point. Right: Learning curves for the scaling results from Figure 5.
Novel Objects: New entity, instruction pairs. We holdout particular objects from single object pick and place instructions during training. This split evaluates on the single object pick and place instructions with this holdout object.
Multiple Objects Different rearrangement concepts. The agent has never seen the concept of "for all" in the training dataset. This requires the agent to rearrange variable number of objects depending on how many entities are in the scene. The agent may need to rearrange to objects in the scene since the number of each object is randomized during scene generation. Therefore, the
agent must search to detect that all the objects belonging to the specified type have been moved. Conditional Instructions Adjust behavior based if the conditional statement is true. In
Language Rearrangement we consider changing the behavior based on if the fridge is open or closed. We split this dataset into the fridge being open half of the time. The agent has find and move one of two objects depending on if the fridge is closed or not. For the object that doesn't t need be moved, the agent must also not displace this object.
B.6 DATASET ANALYSIS
In this section, we study how the evaluation datasets differ from the training dataset and each other in more detail. We note the high-level differences below:
Each dataset consists of different instruction templates (instruction templates are described in Appendix B.2). Different instruction templates mean the underlying goal conditions and language instructions will be different.
All non-train datasets are on unseen scenes. These unseen scenes are from the evaluation split from ReplicaCAD (Szot al., 2021). The location of the kitchen and furniture differs from training. Objects are placed in new locations. In every episode, objects start in new position. Thus we always evaluate on unseen object positions, even for the train split.
We qualitatively analyze the differences between datasets by visualizing language embeddings of the instructions between the different instructions Figure 8a. Specifically, we take 100 random instructions from each dataset and embed them using the Flan-T5-X0 model (Chung et al., 2022). These embeddings are then visualized as a T-SNE plot. The training instructions, visualized as red points, only have minimal overlap with the evaluation instructions.
Additionally, we display the average number of tokens per instruction (see Figure 8b). From here that evaluation splits have variability at the token-count level. which in the largest case (for conditional instruction datasets) have approximately double the length of prompts than the median task.
21

---

Published as a conference paper at ICLR 2024
Diving into this analysis further, in Figure 14, display the per-word frequencies in the instruc- tions. As expected, we see a long-tailed distribution across the board, favoring receptacles and relative indicators for nouns (e.g. counter, table, left) and instructions for verbs (e.g. move, place, and swap). It also is no surprise that when looking at all words in the combined plot, articles are the most common. Overall, this shows the diverse language across our instructions, and aids in verifying that our instruction generation is acting as expected.
B.7 COMPARING LANGUAGE REARRANGEMENT TO OTHER BENCHMARKS
In Appendix B.7, we compare Language Rearrangement to related embodied Al and interactive in- struction following benchmarks. The primary difference from Language Rearrangement and other benchmarks, is that Language Rearrangement has more instructions (almost 19x more than the clos- est work ALFRED (Shridhar et al., 2020)), generated by more instruction templates (8x more than CALVIN Mees et al. (2022)). Also unique about Language Rearrangement is that has dense re- wards for the language specified tasks (the dense reward is described in Appendix B.4). All other tasks with language instructions only define sparse success-based rewards. Since Language Rear- rangement is implemented with Habitat 2.0 (Szot et al., 2021), it also inherits the same fast simula- tion speeds.
While Language Rearrangement supports low-level skill exccution since is built on Habitat 2.0 which supports dynamic physics, we do not evaluate with dynamically simulated low-level skills and instead kinematically update the simulator state to the skill post condition. This is similar to the high-level interaction space in ALFRED (Shridhar et al., 2020). Other tasks focus on low-level control, and thus simulate the full robot control, but are constrained to the more limited setting of a robot arm fixed on tabletop table top (Mees et al., 2022; Gong et al., 2023; Shridhar et al., 2022; Zeng et al., 2021).
Benchmark
Instractions (Tasss))
Instruction Templates
Observation Type
Reward Type
Generalization Type
Sim Speed stepwhecond
Scenes
Ohjects
Language Rearrangement
Casezn Instructions [30
151,000
Visual
Deuse Reward
datasets
14001
105
82
ALFRED
8,055
Sparse Success
Random Split
120
CALVIN Mees
et al E20221
400
Visual
Spune Success
Unseen Instrucions
NA
ARNOLD Googg
al. (2023)
32
Visual
Spome Success
Random Split
200-400
20
40
CLIPort (Raveus) Shridhar et al. (2033) 7ang al. (2021)
(-Procolurat)
10
10
Visual
Spasse Staccess
Unsern Ohjects [colors. shapes, types]
NA
56
al, (20199
ChevalierBBoisvert
Procedural
Procedural
Spume Success
Unseon Instructions [composition,, objects, synouyms, dialcsts]
NA
dtal. (2023)
Hibital
Rearrangement
NA
NA
Visual
Dense Reward
Usseen Scenes
1400
105
20
Soot et (2022)
Behaviorlk La al. (2023a)
NA
NA
Visual
Syume Success
NA
60
50
5215
TDW Gian et
(200))
NA
NA
Visual
Spusse Stacessi
Utseen Scenes
15
5-168
112
ProTTIOR Deiike
NA
Visual
Donse Reward
Unseen Scenes
90-180
10k
118
Table 4: Comparing Language Rearrangement to similar benchmarks. "NA" under the simulator speed indicates the benchmark is used for offline learning and not for RL where simulation speed is a concern. "NA" under instructions indicates the task does not provide language instructions and instead features task specification such as target object name to navigate to or rearrange. "NA" under generalization type means the task does not evaluate trained policies in unseen task variations, and instead focuses within train task distribution performance. BabyAl uses an instruction language grammar instead of instruction templates. This grammar has 4 clauses, 10 attributes (locations and colors), and 4 nouns.
C METHOD DETAILS
In this section, we describe the architectures, training procedure of LLaRP, and baselines more detail. Unless specified otherwise, every method is trained using full node of 8 A100-80GB GPUs
22

---

Published as a conference paper at ICLR 2024
Aggregiasl Beluriie
Por Dazee Rookiens
Parsprratic
Trais
Scame
Inataat Rephrmnngg
Nord
Mastippe
Referitng
Crenat
Irrdenent
Mantple
Spaital
Conditinal
LLabe
42.3
45
36
99
95
47
16.2
0:1
Bas
Serathh
1.1
ISTIRALMAA
0.14
12 g.
1:51
0.1
0.1
0.1
0.e
2.4
00
50
25-F 75-Elanitgo
24
18
24
ELRRP-FT
LLaug-7
11
---1
10 07
15
47
26-3
36-33
33
35
LLaHe (SL)
02
18
15
47
26-2
34.2
35
Mal
19
LLaR iHladds
LSTM-Fla (flander)
Table 5: Zero-shot results on Language Rearrangement for all baselines and settings. This includes the numbers from the bar plots in Figure 3, Figure 5 and Figure 12a. All numbers except for nLn methods, LLaRP-FT, LLaRP-13b, and the harder task setting are mean and standard deviation over 3 random seeds.
and 96 Intel(R) RR) CPUs 2.20GHz. The base models are able fit on a single GPU and we use data parallelism. Each GPU runs policy and set of N environment workers. During the PPO rollout phase, the policy acts in parallel in all N environments. Each GPU then computes the PPO update and synchronizes gradients. We use DD-PPO (Wijmans et al., 2019) to handle straggler environment workers and speed up synchronization between GPUs.
Hyperparameters for all reinforcement learning based methods are summarized in Table 3. Next, we detail specific method architecture choices.
C.I LLARP DETAILS
In general, the visual encoder can produce M embeddings per observation ON, which consists of high-dimensional visual component (the robot's RGB camera) and a low-dimensional state com- ponent (the robot joint angles). The visual component is projected to a of M tokens. For example, using a VIT (Dosovitskiy et al., 2020) for Ey produces an embedding per image patch which are projected to M - 1 embeddings using Perceiver Resampler Jaegle et al. (2021) network. The state components are concatenated and projected with an MLP produce another embedding, giving M total embeddings. Since there are now M tokens per observation or, to produce a distri- bution over actions, we skip the first tokens corresponding to instruction tokens, and sub-sample every th hidden output to extract just one action per time step.
However, in this work we just use the visual RGB observation and set M = All methods use the frozen VCI visual encoder whose weights are represented in b6loat16. We then take the /CLS] token of the VCI encoder for the image observation. We then input this embedding to linear projection layer that produces an embedding the same dimension as the LLM text tokens. For the action output module, we use a 2-layer MLP with ReLU activations, LayerNorm, and a hidden dimension size of 512.
By default, we use the base LLaMA-7B VI (Touvron et al., 2023) for the LLM in LLaRP. We con- vert the LLaMA weights bfloat16. The observation encoder and action output modules represent their weights in float32. The context window in Language Rearrangement is the maximum episode horizon of 32 steps. We compute the attention mask during training so the transformer only attends to inputs from the current episode. Despite running such a large policy with RL, we find that total training throughput is 700-800 steps-per-second on full compute node of 8 GPUs. This timing includes policy inference for data collection, policy updates, and environment stepping including rendering and physics. The biggest bottleneck during training policy inference.
C.2 LLARP-SCRATCH DETAILS
For the transformer network, we use the same architecture and details LLaMA. We restrict the transformer network size to around 2 billion parameters. As with LLaRP, we represent the trans- former decoder network with bfloat16 data type. We update all parameters. Since the policy is smaller than regular LLaRP, the training throughput is even faster at 800-900 steps-per-second on a full compute node, despite updating all 2 billion policy parameters.
23

---

Published as a conference paper at ICLR 2024
C.3 LSTM-FLAV/LLAMA DETAILS
For LSTM-Flan, we only use the Flan-T5-XL encoder, and remove the decoder. The encoder is used summarize the instruction. Specifically, we take the hidden activation of the final token as the instruction representation. We fine tune the weights of the Flan encoder which we show is necessary for good training performance
Appendix D.2. We represent the Flan weights bfloat We represent the image as the /CI.S] token from VC1 and process this embedding with a linear layer before concatenating it with the language representation and inputting it to the LSTM.
For LSTM-LLaMA, we use LLaMA-7B. The LLaMA weights are frozen and in bíloat16 format. For the instruction, we summarize the hidden outputs in a single instruction representation using a Perciever network.
C.4 HERO-SHOT BASELINE DETAILS
ZS-LLaMA uses LLaMA-65B V1 (Touvron et al., 2023) model that was instruction tuned on ORCA style data (Mukherjee et al., 2023; Lian et al., 2023). Since, LLaMA-65B is language-only model, this bascline is blind plans actions based only the language instruction. To help the baseline reason about the available objects and actions, the prompt lists all receptacles and objects along with examples of successful behavior. For multimodal baseline ZS ZS-Flamingo, open-source vision-and -an-aauuaee model IDEFICS (Laurençon et al., 2023) which is reproduction of the closed-source Flamingo (Alayrac al., 2022) model. IDEFICS uses LLaMA-65B VI as its language model backbone. Its vision encoder is vision transformer (ViT-H/14) trained using OpenCLIP (Radford al., 2021) on the LAION-2B English subset LAION-5B (Schuhmann et al., 2022) dataset. For both ZS LL-MLAMA and ZS ZS-Flamingo, we use the following prompt:
You are a home robot assistant that can take actions the house. Remember the following guidelines:
1. Your possible actions are: pick (ebject), placo_on_recop (receptacic), navigate (rcceptacle), open_fridge(), flose_fridget! open_cabinet (cabinet), STOP.
2. Possible objects are: ball, clamp, hammer, screwdriver, padlock,
scissors, block, drill, spatula, knife, spoon, plate, sponse, cleanser, plum, pear, peach, apple, lemon, can, box, banana, strawberry, lego, rubrik's cube, book, boxl, cup, mug, orange, lia, toy airplane, wrench.
3. You can only pick one object at a time.
4. If you place an object you must have previously picked it.
5. You must always place the object that you have picked.
6. open fridge, you have to navigate to the fridge.
7. To pick an object from the cabinet, you need to open first. is not valid for cabinets like 'cabinet drawer 7' or
8. [lace_cn_recees 'cabinet drawer 6',
9. There are
than objects.
10. When explering, select randomly from possible receptacles: [cabinet drawer 7, cabinet drawer 6, fridge, chair, black table, brown table, TV stand, sink, right counter, left counter]
11. when you are done output STOP.
12. If the instruction doesn't specify where the object is located, you should explore by navigating to previously unvisited receptacle.
13. If the instruction asks to pick up more than one object, you should attempt multiple pick and place. Look for each object by visiting all the receptacles to find the objects mentioned in the instruction.
14. Don't get stuck in loop by picking from and placing receptacle on the same receptacle.
To help you understand, here's s are twos example: User: Instruction: Move the screwdriver from the left counter the sofa.
Assistant: navigate (left counter), pick (screwdriver), navigate (sofa), place_on_recep (sofa), SToP.
24

---

Published as a conference paper at ICLR 2024
User: Instruction: Find apple and put it the brown table.
Assistant: navigate open_fridge (), pick (apple), navigate (brown table), placo_on_rocop (brown table), STOP.
The 7.S--aaCiee OS-CChtGPPT baseline can perform multi-step reasoning to generate plan. Unlike 7S-LL is not limited to generating the whole plan in a single step. Instead, it can continuously refine the plan based on environment feedback. We use GPT-3.5-Turbo which is trained for chat applications for these experiments. The environment provides feedback in natural language consisting of agent's location (e.g. You are at black table), failed action (e.g. Couldn' 't execute pick("apple") Object wasn't found), and asks for the new plan. We update the prompt to also contain examples that require multiple steps of reasoning. The prompt is follows:
#System: You are a home robot assistant that can take actions in the house. Remember the following guidelines:
1. Your possible actions are: pick (cbject), place_on_recep (receptacic), navigate (receptacle), open_fridge(), close_fridge apen_cabines (cabinet), STOP.
2. Possible objects are: ball, clamp, hammer, screwdriver, padlock,
scissors, block, dri--, spatula, knife, spoon, plate, sponse, cleanser, plum, pear, peach, apple, Lemon, can, banana, strawberry, lego, rubrik's cabe, book, bowi, cup, mug, orange, lia, toy airplane, wrench.
3. You can only pick one object at a time.
4. If you place an object you must have previously picked it.
5. You must always place the object that you have picked.
6. To open fridge, you have to navigate to the fridge.
7. To pick an object from the cabinet, you need to open it first.
8. place_en_rcccp() is not valid for cabinets like 'cabinet drawor 7' or 'cabinet drawer 6'.
9. There are no more than objects.
10. When exploring, select randomly from possible receptacles: [cabinet drawer 7, cabinet drawer 6, fridge, chair, black table, brown table, TV stand, sink, right counter, 1eft counter]
11. when you
done output STOP.
12. If the instruction doesn't specify where the object located, you should explore by navigating to previously unvisited receptacle.
13. If the instruction asks to pick up more than one object, you should attempt multiple pick and place. Look for each object visiting all the receptacles
find the objects mentioned the instruction. 14. Don't t get stuck in a loop by picking from and placing receptacle on the same receptacle.
help you understand, here's are example: #User: Move the screwdriver from the left counter to the sefa. #Assistant:
plan - navigate ("left counter"), pick ("screwdriver"), navigate ( ("sofa"), place_on on lal__on__ee__eeeee recep ("scfa"), STOP
..
Find an apple and banana and put it the left counter #Assistant: 111
#User:
plan =
navigate ("capinet drawer 7"), # exploring an unvisited receptacle open cabinet ("cabinet draver ?"), opening the cabinet drawer for the first time.
25

---

Published as a conference paper at ICLR 2024
pick ("apple"), navigate ( (*left counter"), place_on_recop (*left counter"), STOP
#User: You are now at cabinet drawer. Couldn't execute pick ("apple") Object wasn't found. Randomly select an unexplored receptacle in Icabinet drawer 7, cabinet drawer 6, fridge, chair, black table, brown table, TV stand, sink, right counter, left counter] #Assistant: ...
plan - navigate ("Eridge"), # exploring unvisited receptacle open sen_frride((),
pick
navigate("led counter"), placo_on_reccp( counter"), STOP
...
#User:
You are now at fridge. Couldn't execute pick ("applee) Object wasn't found. RandomLy select an unexplored receptacle in Icabinet drawer cabinet drawer 6, fridge, chair, black table, brown table, TV stand, sink, right counter, left counter) VAssistant:
plan =
navigate ("brown table"), exploring unvisited receptacle picp("ap"""",,
havigate("lef" counter"), placo_on_recop("Iee) counter"), STOP
...
#User: Found an apple.
#Assistant:
111
plan I
navigate ( ("cabinet drawer 7"), open_cabis canicaabnetttt ("cabinet draver 7") pick ("banana"), havigate counter"), place_on_recop ("left counter"), STOP
#User: You are now at cabinet drawer 7. Couldn't execute pick ("banana"). Object wasn't found. Randomly select an unexplored receptacle in Icabinet drawer ", cabinet drawer 6, fridge, chair, black table, brown table, TV stand, sink, right counter, left counter] #Assistant:
...
plan =
Dovigate("chaire pick ("banana"""" navigato counter"), place_on_recop ("1eft counter"), STOP
#User: Found the banana. Thanks!
26

---

Published as a conference paper at ICLR 2024
C.5 MODIFICATIONS TO SCALE AND TRAIN LLAMA
For the LLaMA models larger than 7B parameters or with trainable parameters (unfrozen), training longer fits single GPU. To train these LLaRP variants, we use model parallelism. Specifi- cally, we distribute the model weights between 4 GPUs and scale training to nodes (8 GPUs each) match the batch size.
C.6 MODIFICATIONS FOR ATART
When training LLaRP Atari games, found that it was vital to allow the visual encoder be fully trainable. Initial experiments with a frozen VC-I visual encoder demonstrated some limited learning, but were unable to reliably solve Pong within 100M steps, our threshold for success. Even with this change, we observed frequent instabilities in our early experiments. Generally speaking, instabilities propagated from value learning the actor over the course of 1-2 gradient steps. We were able to fix this by using Huber loss for value learning, as well as by applying gradient norm clipping (with max gradient norm 0.5). We also apply reward clipping (Mnih et al., 2013) stabilize our predicted values.
We also found that training at relatively large batch sizes compared to language rearrangement was beneficial. By default we used a per-device batch size of 64, with context length 32, leading to a total 16,384 states being seen at once when training across 8 GPUs.
100
Success Rate (%)
Transformer 8x Batch
60
Transformer 2x Batch
Transformer 1x Batch
8x Batch
LSTM 2x Batch
20-
Fine-Tune Flan
LSTM 1x Batch
Freeze Flan
1.5
175
Environment Steps
107
Environment Steps
168
(a) Batch size analysis.
(b) Encoder freezing.
Figure 10: Left: Success rate policies on the Train evaluation dataset. Right: Impact of freezing vs unfreezing the FLAN encoder in LSTM-Flan.
D FURTHER RESULTS
this section, we show further results and analysis on LLaRP and baselines.
D.1 TRANSFORMER POLICY EXPERIMENTS
primary challenge of LLaRP was implementing PPO training for transformer-basee policies. By a "transformer-basee policy", we mean policy architecture that uses a transformer to attend to previous observations. Transformers online RL algorithms, such as PPO, are rare. RNN-based policies are more typically used for problems where the history is important Ni et al. (2021). l). In this section, we include empirical analysis into important components for stable transformer-based policy training.
Prior work has demonstrated transformers working for offline RL (Xu et al., 2022; Chen et al., 202 but such works rely supervised losses from static datasets. Other works explore using the transformer-basee policy to collect data (Zheng et al., 2022) but use a similar offline-RL RL training process. Parisotto et al. (2020) transformers for online-RL, but shows that additional trans- former architecture changes are necessary for stabilizing RL training. We note that these changes incompatible with LLM architectures, precluding the of LLMs in their framework. We
27

---

Published as a conference paper at ICLR 2024
Agpoppol
Pei Duzzut Breaidons
Total
Belarker
Pamplrasti
Train
New Sceees
Instr entra
Nasel
Mulligle
Refennig
Contest
Indesast
Muliiple
Spatial
Conditioall
1daRp
8:35
0.79
6/65
0.87
18136-Faa
11:11
0:31
0.30
0:81
0.79
8.76
11:8
61:1
1636
1629
216
0.00
0101
0.16
Sentch
25-ChuiCPP
0:21
0.45
6.00
0.6c
ZSLLLMA
0.12
8:12
0131
0.90
0.36
0.23
1141
816
1601
0622
008
0.00
019
0101
25-7anitar
5077
0.13
0.6
1020
0600
0181
0.0
Table 6: Numerical results for Language Rearrangement efficcien from Figure 11. Numbers are for random seed.
able to train transformer-based policies with PPO, despite not using any of the modifications from Parisotto et al. (2020). Other works combine various transformer architectures with long-horizon RL tasks (Ni et al., 2023; Morad al., 2023; Esslinger et al., 2022; Sopov & Makarov, 2021; Pleines et al., 2022). Unlike these works, we use billion parameter transformer models in tasks with high-dimensional visual observations.
Effeet of Batch Size on Stability: In Figure 10a, we illustrate the importance of large batch size for transformer-basee policy training. We compare learning curves of a LSTM and transformer- based policy 100 training episodes from the overall training dataset. Both policies be 40M parameters and neither policy has any pre-trained LLM. The policy only takes as input the RGB visual observations and learned embedding of the current instruction. We only analyze training performance, thus this learned embedding is sufficient for distinguishing the instructions during training. As seen from Figure 10a, the transformer runs are more unstable at lower batch sizes, with more jagged learning curves indicating drops in performance. The RNN-based policy the same batch does not suffer from this issue and has smooth learning for every batch size setting. Note that smaller batch sizes converge faster because they update the policy more for fixed number of environment interactions. The transformer-base the RNN-based policy for every batch size setting. We note this efficiency finding is reversed when comparing L.aRP to RNN-based approaches in Language Rearrangement.
fixed to
policy is also less sample efficient than
Effeet of Pretrained LLM Weights: We found that using pre-trained and frozen LLM weights are important for stable and fast convergence, Figure 9a, training LLaRP--cratch required 500M samples to converge and exhibited unstable training demonstrated by the dips in training perfor- mance. LLaRP, using the same architecture, but frozen LLM weights, learned in under 100M steps and did not have the same dips in performance during training.
D.2 ADDITIONAL LANGUAGE REARRANGEMENT EXPERIMENTS AND ANALYSES
Full Generalization Numbers: In Table 5 show the numerical results for all numbers from the paper. These include the main zero-shot generalization results from Figure 3, the scaling results from Figure and the task setting results from Figure 12a.
ChaiGPT
Scratch
LLaMA
0.6
LSa
Fl.Flainngo
LSTM--LLMA
0.6
O01
Language Rearrangement efficiency. Numbers are for 1 random seed. Higher numbers indicate more efficient solutions. Numbers are rescaled in [0, 1] where is the least efficient and the most efficient.
Figure 11:
Efficiency Results: In Figure 11, compare the efficiency of all methods in the Language Rear- rangement zero-shot evaluation. We measure efficiency by the number of steps agents take to solve the tasks. Specifically compute the efficiency of episode as mtops
where n, is the number
28

---

Published as a conference paper at ICLR 2024
Success Rate (%6
100
LLRP DEL
LlaPP (itander)
100
LSTM (HL)
Fla (Harder)
60
20
20
LLaRP
10
LSTM-Flan
Eval
Robust
Gan
Environment Steps
les
(a) Generalization on Harder setting.
(b) Training on Harder Setting.
Figure 12: Left: The zero-shot evaluation performance on Language Rearrangement where invalid actions result in immediate failure and the stop action is required. Right: Training curves under this harder setting. We enforce these termination conditions during training as well, resulting in slower training. Results in the harder task are just for seed since training is slower.
of steps needed complete episode i. ny is assigned max steps the episode unsuccessful. We report the average efficiency over all episodes in the evaluation datasets. LLaRP is more efficient than baselines, and over 1.5x as efficient the best performing baseline.
Importance of Unfreezing LLM Weights in LSTM-Flan: In Figure 10b we analyze the effect of freezing or unfreezing the Flan-T5 encoder weights during training. Unlike for LLaRP, keeping the LLM weights frozen results in the policy only learning the easy instructions even after 200M steps of training. We found fine-tuning the language encoder to be effective learning the harder instructions and converging much faster to the maximum training performance.
LLaRP Full Finetuning (LLaRP-FT): Table we compare the effect of not freezing the LLM in LLaRP and fine tuning it during training (LLaRP-FT), Like with training LLaRP-13b, model parallelism and scale training to 4 nodes (8 GPUs each). However, as seen Figure 9b, training performance is poor. We were only able to train for 15M steps due to the slow training speeds. A larger batch size and longer training could result in LLaRP-FT working better.
D.3 HARDER TASK VARIANT
We consider harder variant of Language Rearrangement where invalid actions immediately end the episode and the agent must call termination action at the end of the episode. Calling the termination action before the task is successfully completed results in a failed episode. We train LLaRP and the best performing baseline, LSTM-Flan, in this setting. We train the same training datasets and evaluate on the same holdout datasets as in the main Language Rearrangement experiments.
We find training policies to be more difficult in this setting. As from the learning curves in Figure 12b, even after 700M steps of training, policies are not yet converged and still cannot solve the hardest instructions. In Figure 12a, analyze the zero-shot performance Rearrangement evaluation datasets. Both LLaRP and LSTM-Flan suffer a drop in performance on this harder task. LLaRP still greatly outperforms LSTM-Flan in this harder setting.
the Language
E QUALITATIVE RESULTS
In Figure 13, we visualize success examples for LLaRP. See the figure caption for a breakdown of each of the success examples.
Below we also describe common failure modes LLaRP on the datasets where LLaRP has less than 90% success rate.
Context: For the instructions involving "playing sport" and "clean spill", the agent never picks up the desired object and instead aimlessly navigates between receptacles.
29

---

Published as a conference paper at ICLR 2024
Referring Expressions: Like in context, the agent randomly moves around for some of the instruc-
tion types like "yellow round fruit" (lemon) and "puple fruit" (plum).
Irrelevant Instruction Text: The agent will sometimes move a wrong object never described in the instruction or distractor text.
Multiple Rearrangements: The agent moves 2 of the 3 objects.
Multiple Objects: The agent often fails to pick the 2nd object when exists in the scene.
Conditional Instructions: The agent fails to check the fridge is open before doing the rearrange-
ment.
30

---

Published as a conference paper at ICLR 2024
Moue box the right the counter
Swep the wrench and peach from the sink and brown table
nacigatatena
picijpon)
pioeteeenchi
redkgatetaaio
necigalafrronn
placaboss airo)
placetarend, brown_1ibid
pkrepcactt)
nzvigatztarkt
placoppcach,
(a) Spatial Relationships success example.
(b) Novel Objects success example.
Locate tool to fix loose sarew in and bring
Deposit apples
Powggatefrron took))
redggteejaakk
nut gategaaioooal
reddgsteiiton table)
picklagpit kt
revigalebsckj
paaeepplle Mice Mon_Jsttee
new www.ppeeightttoom courter)
pickizppki
narigataabuu Jabkc)
(ntle)
(c) Context success example.
(d) Multiple Rearrungements success example.
otherwise
Tike yellow curved fruit and place on the blue table Fevigulizein
pear sink
nedigraajeoee
novigano (dari, tabl))
raddgstajourk tnbrle
rautgstefrrren, batley
nowlganobllue table)
radigsteligtt.com
pickkpaart
plvobarend)
navkgnebiine)
Hovigatofooue, sook)
placeipaww
pdecefparans,, latall
(e) Referring Expressions success example.
(f) Conditional Instructions success example.
Figure 13: Successful LLaRP trajectories in some of the Language Rearrangement evaluation datasets. Fig. 13a: the agent infers the sink is the receptacle to the right of the left counter. Fig. 13b: the agent has interact with the wrench not seen before during training. It correctly picks the wrench despite not sceing this object in the context this instruction during training. Fig. 13c: the agent infers screwdriver is needed to satisfy the context of "a loose screw".. It explores until finds the screwdriver and then brings it to the couch. Fig. 13d: The agent explores until finds an apple which point it places it the receptacle. The agent continues explore find if there are other apples, finds one on the right counter, and then rearranges it to the blue table. Fig. 13e: The agent explores until it
banana which it infers is "yellow curved fruit". The agent then moves the banana to the blue table. Fig. 13f: The fridge is closed, so the agent is correct to move the pear to the sink. However, the agent didn't 't explicitly check to see if the fridge is open. This likely prevents LLaRP from achieving higher success on the Conditional Instructions dataset.
31

---

Published as a conference paper at ICLR 2024
Train
Move "aarget.objec_name from the soource.eceeptaclc.nam to the "tar- getrrocptacle._amm
Move "'arget.objec__nan to prrget.recettcke name"
Move the "target object name off the 'target receptacle name".
Place "object1" and 'object2" on the 'targ recep'.
Can you swap the 'objectl' and the 'object2" in the 'receptaclel and 'receptacle2"? accidently left the fridge open, can you close Can you open the fridge for me?
Go to "target receetrclleeeaaamee. Find objjett name".
Instruction Rephrasing
On the wwwreeerccepaall..ant nnmme there is target_object.namee move it to the "tar- ret.recepacccc _maeee
On the receptaccee _naeee need you put surget.object_nee out a "plate" one person
receptaccce saaaaacleemme....
The
name" should he devoid of any fanget_ohject_nmme
On the 'targ recep', need 'objectl' and 'object2'. I misplaced the 'object!" the 'receptacle1" and the 'ohject2' the 'receptacle2.. Can you swap their positions?
When putting away groceries, forgot to shut the fridge. Can you help?
Referring Expressions
Bring green to "target receptacle name*
Bring yellow round fruit the 'target. receptacle name Bring yellow curved fruit to the 'target receptacle name" Bring round red fruit to the 'target receptacle name'
Multiple Objects
Put the "object.nnne name" on the 'targ_reeep"
'object.name namee from the 'receptaclel" on the 'receptacle2'. the fridge is open wise move 'target object name2t to receptaaee name'.
Conditional Instructions
"larget.object name to the 'target "larget receptacle name" other-
the fridge is open move 'target object namel' to the 'target receptacle name', other- wise move suggt_.bject.name? to the 'aarget.eceeptacle.nna namee
Spatial Relationships
Move a "target object name' from the 'source receptacle name' to the left of the right counter.
Move 'aarget.object.naee from the 'source.receptacle.nne to the right of the left counter.
Move a "aageet.object.naa name" from the Source_receptiee name" to the right the TV stand.
Context
I want to play sport, bring something to play with to the reapeetrcceeacee...e name screw loose in the ttarget.receptacle.ne I need cut a piece paper the "target receptacc name". can you bring something to help?
bring something fix it.
1 spilt something and need to clean it. Can you bring something to the 'tar- get_rcceptacle_nage to help?
Bring something to pour hot coffee into the rarget.receptacle__ama
Multiple Rearrangements
Move the 'objl" to the 'targ-recepl, the 'obj2' to the 'targ_rece22', and the 'obj3" the 'targ3r
Irrelevant Instruction Text
There's apple the sofa, but the 'aagge_receptacle.nnac need you put "barget.object.nnn
Table 7: Sampling of the instruction templates for each of the task datasets. Note that for each template we include multiple phrasings. Names backtics indicate template placeholders. Our pipeline described Appendix B.2 automatically grounds these placeholders with feasible entity names.
32

---

Published as a conference paper at ICLR 2024
25000
20000
15000
10000
5000
Budh
(a) Noun frequency for instructions.
10000
4000
2000
Ploce
Pick
Hest letee
Lob The
Thy Extra
Nive
Aspreciie
Azsipned
These
Lay
Interdhangl beatre Sechonee Suuure teach
bequeniin
Eorpot
Sugar de
Di
Seck
Locks
Preccod
trigg
Enouls
Leate
cuitingg
(b) Verb frequency for all instructions.
120000
100000
80000
60000
40000
20000
Rof Becooii Nani tie Baa bal
sor
(c) Word frequency for all instructions
Figure 14: Distribution of word counts for all the instructions.
33


Reviews:
[{"review_id": "tonG85Ji6w", "reviewer": "Reviewer_vC8Q", "summary": "This paper introduces Large Language model Reinforcement Learning Policy, which adapts a pre-trained LLM to take text instruction and egocentric observations as inputs, and output actions. Fixing the body of LLM, adapters are trained using the standard reinforcement learning algorithm. The paper is also provided a new benchmark called language rearrangement, consisting of 150,000 training sets. They empirically show the benefits of the proposed method, comparing with several simple baselines including zero-shot LLM, zero-shot VLM, and LSTM policy in conjunction with T5 and LLaMA.", "questions": "See Weakness section.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) The paper is well written and easy to follow. \n\n(2) The method is simple and straightforward extension of prior studies but seems to provide solid performance gain.", "weaknesses": "(1) Technical novelty is not high. Using VLMs to control a task is not new, e.g., PaLM-E and RT-2 as discussed in this paper. It is true these two method does not employ reinforcement learning, but architectural or technical difference is slight and is not well discussed in the paper. \n\n(2) While the one of the main claim or implication of the paper might be linguistic knowledge in LLMs can be used in online RL, but the statement itself is already validated in prior studies, e.g. [1]. While it is true that [1] focus on the textual environment, but there are no discussion how this paper extends the prior understanding on what kind of knowledge is encoded in the LLM and what is not. \n\n[1] Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning\n\n(3) The paper lacks in depth analysis of the failure case, which might be important to dig in the internal knowledge of LLM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gSqUIRICjk", "reviewer": "Reviewer_Daer", "summary": "The paper introduces an approach that adapts pre-trained LLMs to embodied visual tasks, by leveraging the world knowledge encoded in LLMs to enhance training efficiency. The paper also introduces a new benchmark, Language Rearrangement, with 150,000 training and 1,000 testing tasks for studying language-conditioned, which contains a diverse set of language-conditioned rearrangement tasks, such as complex manipulation, navigation, and exploration tasks.", "questions": "- How are the current baselines chosen, e.g., what would be the reason for not comparing with an LSTM-Flamingo baseline or using other VLMs such as InstructBLIP, LLaVA, miniGPT?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- In the context of contemporary works, such as emdodiedGTP, Eureka, RoboCat, Open X-Embodiment, etc., this work demonstrates that the LLM-based LLaRP model exhibits strong generalization capabilities. It can handle complex paraphrasing of task instructions and generalize to new tasks.\n- LLaRP shows faster convergence during training compared to other baselines, indicating its sample efficiency. Scaling up the size of the underlying LLM (from 7B to 13B parameters) leads to better results, suggesting that larger LLMs enhance embodied reasoning capabilities.\n- The paper provides comparisons with zero-shot baselines, showing LLaRP outperforms models that rely solely on language understanding without training. It also shows that LLaRP trained with reinforcement learning (RL) outperforms LLaRP trained with imitation learning (IL), highlighting the effectiveness of RL in this context.", "weaknesses": "- What distinguishes this work from [1,2, 3, 4], which also appear to emphasize the evaluation of LLMs' generalization abilities in embodied settings? \n- In the current way results are presented, it is very difficult to understand the differences in model capacity across baselines and LLaRP. Please consider including a direct comparison of model capacity in Fig. 3, as well as exact numbers in the bar plots (it seems that there is sufficient space to do so given the white space surrounding the bar plots). Same for Figures 5, 7, and so on.\n- The paper presents successful results but does not thoroughly explore or discuss failure cases. It would be great to have more qualitative examples that enable readers to understand the limitations of employing pretrained LLMs on embodied tasks.\n- It would be valuable to gain insights into the performance of simpler baselines, such as embedding-based models like embCLIP, or baselines from the embodied rearrangement literature such as [5], in comparison to the proposed approach. Currently, all existing baselines rely on pretrained Language Models (LLMs), with varying prompts and inputs. While this dependence on LLMs is shown to yield strong results, it would be good to understand the necessity and efficiency of LLMs for the tasks at hand. In other words, it is difficult to ground the work in existing embodied literature, since it introduces a new task (language rearrangement) and a new model (LLM-based LLaRP model trained with online RL). \n\n[1] Li, Shuang, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen et. al. \"Pre-trained language models for interactive decision-making.\" Advances in Neural Information Processing Systems 35 (2022): 31199-31212.\n\n[2] Carta, Thomas, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Yves Oudeyer. \"Grounding large language models in interactive environments with online reinforcement learning.\" ICML (2023).\n\n[3] Xiang, Jiannan, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang, Zichao Yang, and Zhiting Hu. \"Language Models Meet World Models: Embodied Experiences Enhance Language Models.\" NeurIPS (2023).\n\n[4] Huang, Wenlong, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. \"Language models as zero-shot planners: Extracting actionable knowledge for embodied agents.\" In International Conference on Machine Learning, pp. 9118-9147. PMLR, 2022.\n\n[5] Wijmans, Erik, Irfan Essa, and Dhruv Batra. \"VER: Scaling On-Policy RL Leads to the Emergence of Navigation in Embodied Rearrangement.\" Advances in Neural Information Processing Systems 35 (2022): 7727-7740.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xjEgsOP46P", "reviewer": "Reviewer_DZLf", "summary": "This paper presents a method for embodied AI for object rearrangement that leverages Vision-Language Models for learning embodied policies. The paper also presents an evaluation benchmark based on a simulated environment (likely Habitat) that significantly extends the number of scenarios and natural language instructions, additionally including novel evaluation axes such as robustness to paraphrasing and robustness to alternative behaviors. The paper reports significant generalization improvements compared to baseline methods and reveals the weaknesses of alternative approaches when dealing with linguistic ambiguities and behavioral differences.\n\n**Update after rebuttal**\n\nAfter the rebuttal, and with taking the additional clarity provided by the authors during the rebuttal into account, I raise my score, since they have clarified several points on the details of their work, and its limitation. Nevertheless, I still believe that the scope of their work can be improved if they provide additional evidence for LLaRP's performance outside their own benchmark, where previous competitive work exists.", "questions": "- extend related work and discuss relation to existing work that has been detailed in the weaknesses section.\n- provide details that have been denoted in the weaknesses section.\n- provide comparison to existing benchmarks that has been pointed out in the weaknesses section.\n- extend empirical evaluations and comparison to existing work that has been described in the weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- despite existing similar benchmarks such as Habitat rearrangement task and Alfred, the proposed large-scale evaluation benchmark and their perspectives on generalization is useful for comparing embodied agents that can leverage both perception and language instructions, while working in an interactive simulation.\n- extending habitat rearrangement with large number of natural language instructions enables research in this directions to be able to move beyond zero-shot or fewshot in-context agents.\n- the paper compares the proposed approach that leverages finetuning the policy and image-to-language adaptive layers, to zeroshot methods and alternative sequence models, and reports significant improvements in generalisation of the aforementioned axes of evaluation.", "weaknesses": "- **incorrect claims and missing related work:** the paper states \"To our knowledge, there are no prior work which demonstrate that the linguistic knowledge in LLMs can be used in online RL problems to improve generalization in embodied AI settings\". There are many examples in the literature that actually have demonstrated that, some of which have been cited (such as ELLM, PALM-E, etc) and many were not discussed (e.g, SayCan, CodeAsPolicy, PercieverActor, HELM, ProgPrompt, EmbodiedGPT,...), though there are many examples of such methods. I recommend correcting the inaccuracies in such statements.\n- **lack of details:** the main paper lacks many details such as what simulation were used, how the agent executes the skills, how the skills are defined, and what is the contribution of the defined PDDL. Although some details are provided in the supplementary material, clarifications are needed to be presented in the main paper.\n- **lack of comparison to existing benchmarks:** The paper does not provide a comprehensive comparison to available benchmarks (some examples includes but not limited to Habitat rearrangement, CortexBench, AI2Thor, Behavior1k, Procthor, ALFRED). Hence, it is not clear how the new provided axes and extending instructions stand against existing work. It is quite common in the literature to dedicate a section and a table to compare various aspects of a newly proposed benchmark to existing ones from various aspects such as scale, generalization aspects, number of samples, kinds of provided data (such as language instructions) etc.\n- **limited comparison to existing work:** although this paper compares to relevant approaches that have proven to be effective such as zeroshot in-context text-only, as well as encoder+LSTM, there are many seminal works that are applicable to this environment and it would bring more value to comparisons if the paper actually leverages some pre-existing methods. Examples of such methods include CodeAsPolicy (Code Gen), PercieverActor (BC), and SemanticHELM (pretrained vision and LLM+LSTM). Atari is a well-established benchmark, which has many competitive baselines, which has not been provided for comparison. Providing additional evidence on other embodied AI benchmarks where previous established baselines exists provides better comparison datapoints to compare the proposed method to the existing work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zJ18pVWMrn", "reviewer": "Reviewer_92Kq", "summary": "This paper proposed a new method to adapt LLMs to embodied visual tasks, to leverage the world knowledge of LLM and achieve better generalization on new tasks.\nIn this method, a pre-trained frozen LLM is used to take text instructions as input. A frozen vision encoder is used to encode visual observations. A trainable MLP is used to map the output of visual encoder to the LLM input space. Finally, a few MLPs are used to map LLM output to a policy's action spaces.", "questions": "Why not use a VLM backbone but instead use a language-only backbone?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors proposed a nice way to leverage LLM in the embodied ai tasks with visual inputs. LLM in the design are essentially producing the state representation for the action decoder, given a visual input adaptor. The total number of trainable parameters are constraints so that RL can be efficiently applied.\nEmpirically, the authors showed that the learned policy generalizes well to large diverse tasks. The unseen tasks are divided into two categories: paraphrastic robustness and behavior generalization to evaluate the generalization of the policy. The empirical evaluation method itself has its own value.", "weaknesses": "The proposed method uses MLP to map LLM output to discrete actions. Therefore, the actions and LLM outputs are not in the same space. Due to this, I am not sure if the policy can fully leverage the world knowledge from LLM. It would be better if the actions are directly produced by LLM. Of course, in that case, we may need to unfreeze the LLM weights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)