SPRINT: Scalable Semantic Policy Pre-training via Language Instruction Relabeling
Authors: Jesse Zhang
Karl Pertsch
Jiahui Zhang
Taewook Nam
Sung Ju Hwang
Xiang Ren
Joseph J Lim
Venue: Submitted to ICLR 2023 (2023)
Keywords: reinforcement learning
language-guided RL
offline RL
policy pre-training

Abstract: We propose SPRINT, an approach for scalable offline policy pre-training based on natural language instructions. SPRINT pre-trains an agent’s policy to execute a diverse set of semantically meaningful skills that it can leverage to learn new tasks faster. Prior work on offline pre-training required tedious manual definition of pre-training tasks or learned semantically meaningless skills via random goal-reaching. Instead, our approach SPRINT (Scalable Pre-training via Relabeling Language INsTructions) leverages natural language instruction labels on offline agent experience, collected at scale (e.g., via crowd-sourcing), to define a rich set of tasks with minimal human effort. Furthermore, by using natural language to define tasks, SPRINT can use pre-trained large language models to automatically expand the initial task set. By relabeling and aggregating task instructions, even across multiple training trajectories, we can learn a large set of new skills during pre-training. In experiments using a realistic household simulator, we show that agents pre-trained with SPRINT learn new long-horizon household tasks substantially faster than with previous pre-training approaches.

Under review as a conference paper at ICLR 2023
SPRINT: SCALABLE SEMANTIC POLICY PRE- TRAINING VIA LANGUAGE INSTRUCTION RELABELING
Anonymous authors Paper under double-blind review
ABSTRACT
We propose SPRINT, a scalable offline policy pre-training approach based on nat- ural language instructions. SPRINT pre-trains an agent's policy to execute di- verse set of semantically meaningful skills that it can leverage to learn new tasks faster. Prior work on offline pre-training required tedious manual definition of pre-training tasks learned semantically meaningless skills via random goal- reaching. Instead, our approach SPRINT (Scalable Pre-training via Relabeling Language INsTructions) leverages natural language instruction labels offline agent experience, collected at scale (e.g.. via crowd-sourcing), to define a rich set of tasks with minimal human effort. Furthermore, by using natural language define tasks, SPRINT can use large language models to automatically expand the initial task set. As result, we can learn an extensive collection of new skills via offline RL during pre- pre-training by relabeling and aggregating task instructions, even across multiple trajectories. Experiments in ALFRED, a realistic house- hold simulator, show that agents pre-trained with SPRINT learn new hong-horizon household tasks substantially faster than with previous pre-training approaches.
INTRODUCTION
When humans learn a new task, e.g., how cook a new dish, we rely on large repertoire of previously learned skills, like "chopping vegetables" or "boiling pasta", that make learning more efficient. Improving learning efficiency is crucial for practical deployment of artificial agents; thus, many works in reinforcement learning (RL) aim to equip agents with a similar set of skills. To autonomously acquire such skills, recent works optimize for diverse agent behaviors (Fysenbach eet al., 2019 Sharma et al., 2020 Mendonca et al.) (021). imitate short action sequences (Lynch ict al. 2020 Pertsch et al. 2020 or reach randomly sampled goal states (Chebotar et al. 2021) Trom pre-collected experience. However, such objectives may result in the agent learning skills that
are not semantically plausible in practice, e.g., "placing a knife in the microwave" or "half- "half-closing the microwave door." To focus pre-training on plausible skills, one could instead manually curate set of pre-training tasks for the policy, but this requires tedious reward function design and does not scale well beyond few dozen tasks (Yu et al., 2019). Yet, defining large set of pre-training tasks is crucial: only a policy with wide range of skills can accelerate learning on many downstream tasks. How can we detine a large set of meaningful pre-training tasks scalable manner?
In this paper, we propose to leverage natural language instructions to define a large number of semantically meaningful tasks for policy pre-training. Natural language has recently been used to allow humans to effectively interact with agents (Lynch & Sermanet 2021) or to generate long- horizon plans (Ahn et al.) 2022). In the context of defnnng pre-training tasks, using natural language has two important benefits: (1) language is natural and expressive interface for humans to specify tasks (in contrast to, e-g., numerical reward functions) as is the primary way to communicate tasks in our everyday lives. Thus, even non-experts can define tasks casily via language instructions. (2) By specifying pre-training tasks via natural language, we can leverage the knowledge captured in large language models to automatically generate more tasks through instruction relabeling.
To combine both benefits we introduce SPRINT (Scalable Pre-training via Relabeling Language I scalable pre-training approach that equips policies with repertoire of seman- tically meaningful skills (see Figure [ for an illustration). SPRINT has three core components: (1) langunge-conditioned offline RI., (2) L.M--based skill aggregation and (3) cross-trajectory skill

---

Under review as a conference paper at ICLR 2023
4
2
Target Task
LLM Relabeling
Human Instruction Annotator
"Maka coffee"
Relabeled Language Instructions
Policy
"Get shett
coffee machino"
Finetune
Pre-train language instructions
"Maka breakfast"
"Place bread the microwavee
Cross-
Trajectory Chaining
*Put broed table"
"Gat mug from the tridge"
*Pick
Agent Experience w/ Language Instructions
Cross- cooss-Trajectory Chained Instructions
We propose SPRINT, a scalable approach for policy pre-training with semantic skills. We assume access an offline dataset of agent experience with natural language instruction labels of the performed skills, e.g., provided by human annotators (1). We use the instructions to pre-train semantic skill policy via inaruction-ccoditionndd offline RL. To increase pre-training task diversity, instruction relabeling,
Figure I:
we automatically generate new instructions via (2) language-model- and (3) cross-trajectory skill chaining. We demonstrate that an agent pre-trained with SPRINT can leverage the diverse set of learned semantic skills to finetune efficiently unseen target tasks (4).
chaining. SPRINT assumes access to an offline dataset state-action trajectories, each of which performs more skills. We assume that the data has corresponding natural language instruction labels for the performed skills, such as "place mug in coffee machine" or "press brew button". Such labels can be crowd-sourced from non-expert human annotators scale. We use these annotations as task instructions and train a policy with language-conditione
offline RL to solve them. Crucially, SPRINT uses two techniques to expand this initial task set: firstly, use pre-trained large lan- guage model to relabel the language instructions, thereby creating new tasks. For example, the tasks "place mug coffee machine" and "press brew button" can be combined into new task: "make coffee." Secondly, we chain behaviors across multiple trajectories from the training data; starting with a skill like "pick up bread" from one trajectory and ending with "place bread on table" from another. This allows the policy to learn semantic skills completely unseen in the training data.
SPRINT trains a policy on the combined set of task instructions, thereby equipping the agent with policy that can execute wide range of semantically meaningful skills. Our experiments demonstrate that this allows for substantially more sample-efficient learning and better zero-shot execution of new downstream tasks like "prepare breakfast" than prior pre-training approaches.
In summary, our contributions are threefold: (I) we propose SPRINT, which leverages natural lan- guage instructions for scalable policy pre-training via expand the set of pre-training tasks via LLM--based skill relabeling and cross-trajectory chaining, efficiently learn long-horizon household tasks in the ALFRED simulator (Shridhar et al.) 2020) than prior pre-training approaches.
cnnncttionconnttionnne offline RL, (2)
(3) demonstrate that SPRINT enables agents
RELATED WORK
Language in RL. There is mtansinaning interest in leveraging natural language during behavior
learning, e.g., to structure agent's 's internal representations (Andreas et al. 2017b), learn to interact with text-based games (Narasimhan et al., 2013) Küttter et al., 2020), or guide long-horizon task
learning via recipe-like plans (Branavan al. 2009 Andreas et al. 2017a). The recent progress
2

---

Under review as a conference paper at ICLR 2023
in training large, general-purpose language models has enabled approaches that directly generate and execute such plans solely from a high-level task description (Huang et al. 2022a) Ahn et al.) 2022 Huang et al., 2022b). Others have shown how language in combination with viston imputs can be used to learn state representations (Nair et al. 2022) or reward functions (Fan et al. 2022). Finally, there is a rich body of work that explores using language
intuitive agent interface for humans: given a free-form natural language command they aim to train language-condittone policies that can execute the instruction the environment (Lynch & Sermanet 2021). p. In contrast, our work aims to leverage language for task-agnostic agent pre-training. Instead of traming a policy to execute human language commands, we aim to equip an agent with a set of semantic skills that improves the efficiency of reinforcement learning
downstream tasks. In similar vein, Colas et al. 2020 explored using language to define tasks for semantically meaningful exploration RL, but they required a hand-defined grammar for task instruction generation, while we leverage offline data and large language models for automatic generation of new training tasks.
Pre-training Policies for RL. Developing policy pre-training approaches for faster downstream
learning has been investigated for many years (ljspeert et al. 2002] Theodorou et al. 2010 Hester
al. 2018). Recent advances in offline reinforcement learning (Levine et al.j 2020) enabled ap- proaches that can pre-train agents offline and effectively finetune them on online tasks (Peng et al.)
2019 [ingh et al. 2020 Nair et al. 2020 Kostrikov al.. 2022). However, these approaches require the pre-training data to have target-task reward annotations and the resulting pre-trained policies are trained only to solve the target task. In contrast, meta-RL. approaches pre-train on a range of tasks and allow fast adaptation to unseen downstream tasks (Duan et al. 2016 Finn et al. 2017 Rakelly (et al.) 2019 Nam et al. 2022), yet require tedious manual definition of pre-training tasks by experts, making them less scalable. To avoid manual task design, other works have explored unsupervised
pre-training approaches based on behavior diversification (Achiam et al.] 2018 EEsenbach et al.]
2019 Sharma et al.] (019) extraction of behavior priors from offline agent experience (Pertsch ct al. 2020) Ajay et al. [0200] Singh et al., (002]. Closest to ours is the approach of [Cebotar et al. (2021). which performs unsupervised pre-training by learning to reach randomly sampled states from offline agent experience. Yet, unsupervised pre-training approaches learn semantically mean- ingless skills which, we demonstrate in Section [3] lead worse transfer to downstream tasks. In contrast, we introduce a scalable pre-trat ve-training approach based on natural language instructions that equips agents with semantically meaningful skills and allows for efficient transfer unseen tasks,
PRELIMINARIES
Offiine RL. Offline reinforcement learning (RL) methods assume access to datasets of trajectories composed of ordered (s,a,8',1) s',r) tuples of current state, action, next state, and reward labels, from a Markov Decision Process M = AS, P. R. 7). S and A are state and action spaces. P S X A x S - R4, represents the transition probability distribution. R. S x A x S + I denotes the reward function, and y is the discount factor. These methods learn offline, with no environment interactions, and aim train policy ((ass) that maximizes the return per episode. Cemporal-different (TD) learning methods also train a critic, Q= a), representing the discounted future sum of rewards using policy æ after taking action a from the state s (Sutton & Barto 2018).
4 SPRINT: SCALABLE SEMANTIC POLICY TRE-TRGINING WITH LANGUAGE INSTRUCTIONS
In this work, we propose SPRINT (Scalable Pre-training via Relabeling Language INsTructions), an approach for pre-training a policy solve wide range of semantic skills in order to enable efficient finetuning on unseen tasks. SPRINT has three core technical components which we will describe in this section: (1) offline instruction-condtiionnn
RL with crowd-sourced language instructions, skill aggregation and (3) cross-traeectory skill chaining. But first, we will detail the data we use for pre-training and give an intuitive overview of our approach.
(2) soetaage-moee--baae
Pre-training Data. Following prior work agent pre-training, we assume access to large of- fline dataset D of agent experience (Gupta et al.] 2019; Lynch et al., 2020 Pertsch et al., 2020) Chebotar al. 2021 Ebert et al. 2022; Pertsch al. 2021). Such data can be collected at scale,
3

---

Under review as a conference paper at ICLR 2023
Hindsight Language Annotation
Language Model Skill Aggregation
Cross- Cross-Trajectory Skill Chaining
2
3
D
F
ZB
1
2
"Place mug coffee machina"
"Press start button"
"Place mug in coffee machine"
*Press start button"
D
1
*Clean mug in sinkk"
"Place mug in shell"
LLM Relabeling
*Place mug in coffee machine"
*Press start button"
2
2
&
mug
mug shell
"Cleen mug in sink. Place mug coffee machine.
"Make coffee"
Figure 2: SPRINT overview. Left: Hindsight annotation of agent experience with language instruc- tions is scalable approach for defining pre-training tasks. We pre-train via instruction-condiionn offline RL with a sparse goal-reaching reward (Section (.I). Middle: We expand the set of pre- training tasks by aggregating language instructions with an LLM and adding the relabeled trajecto- ries back into the pre-training dataset (Section (.2).. Right: We perform cross-trajectory chaining of skills enable pre-training of skills that are unseen in the offline agent experience (Section 4.3)
e.g., from prior RL runs, via teleoperation, through autonomous agent exploration or any combi- nation thereof. We further assume that the data contains natural language descriptions of the skills performed by the agent, e-g., "pul a mug in the coffee machine" or "push the brew button." Such de- scriptions can be collected scale by non-experts by annotating sequences from our offline dataset 'D in hindsight, e.g., via soorrinu through platforms like Amazon Mechanical Turk (Lynch & Sermanet 2021 Shridhar al., 2020). Given randomly sampled sequence from the dataset D, annotators can label sub-trajectories 71 [sg, 40. $1,..-), 1, T2
with free-form language de- scriptions 21, ...... of the tasks performed in the respective sub-trajectories, for example:
[T Tayl
[[(50' ag =1),.., a (
21)]
[( ag" 22).
(I)
"Pul a mug in the coffee muchine"
"Push the brow button"
This results in dataset DL (Ta, Taas ... of diverse trajectories with natural language descrip- tions (see Figure 2 left).
Approach Overview. Our approach leverages skill descriptions in DL as instructions for the pol- icy during pre-training: each description defines a task and the policy is rewarded for successfully executing the instruction. Intuitively, the more diverse the set of language instructions during pre- training, the more semantic skills the policy will Iearn and the more downstream tasks can it fine- tune on efficiently. Thus, SPRINT introduces two approaches for increasing the diversity of the pre-training instructions without additional human inputs. Firstly, SPRINT leverages trained language models to aggregate the human annotated instructions (Figure 2 propose an approach for cross-trajectory skill-chaining, that allows us to combine behaviors across multiple training trajectories to generate completely unseen instructions (Figure 2 proaches expand the set of pre- pre-training tasks, leading to more effective pre-training. Once the policy is trained, we can transfer it to a new task for finetuning, either by providing a language descrip- tion of the target task by replacing the language input with vector of trainable parameters and finetuning it jointly on the target task, like prior work (Chebotar et al. 2021
middle). Secondly, we
right). Both ap-
Laskin et al.) 2021).
4.1 INSTRUCTION-CONDITTOIN OFFLINE RL.
To pre-train our policy with the natural language instruction dataset D'. we take inspiration from goal-conditioned RL approaches (introduced in Section B): instead of conditioning the policy goal states and rewarding it for reaching these states, we condition our policy π ((a)s, 2) on language instructions from DL and provide sparse reward to the agent for reaching the end-state sT of the
4

---

Under review as a conference paper at ICLR 2023
respective sub-trajectory. Formally, we define the reward as:
f1, for =
a()
(2)
0, otherwise.
We train policy x[a|ss 2) and critic ((,, a, this reward function with offline RI.. In this
work, we use Implicit Q-Learning (Kostrikov et al. 2022) since we found it to be easy to tune.
4.2 SAAGUAGE-MODEEL -BASED INSTRUCTION AGGREGATION
Large language models (LLMs), trained massive cor- pora of internet text data, have been shown to be effective at performing variety of tasks from question answer-
LLM Prompt Example
Summarize the following steps.
ing to program synthesis when prompted with relevant
1: Pick up the tomato slice.
text (Devlin et al., 2018: Brown al. 20200 Wang Ko-
2: Heat it up in the microwave.
matsuzaki 202 Rae et al. [021] | Hoffmann et al.. [022]
Summary: Microwave a tomato slice.
Zhang al.)
2022] Chowdhery et al. 2022). Here we
use LLMs to aggregate, i.e., paraphrase, the existing lan-
1: [SKILL 1]
guage annotations in DL (see Figure 2 middle, for an il-
2: [SKILL 2]
lustration). Given trajectory that contains multiple sub- trajectories, we can aggregate adjacent sub-trajectories into longer trajectory and relabel its natural language annotation with a summary of the individual instructions generated by the LLM, thereby generating new higher- instructions from multiple sub-trajectories[] We use a
Summary:
Figure 3: A shortened example of the LLM prompt. Full prompt in Section B
level pre-training task that encompasses simple summarization prompt to instruct the language model (see Figure 3 for an example, Sec- tion B for the full prompt). Specifically, we aggregate with OPT-13B (Zhang et al.) [2022), an open-source 13 billion parameter LLM comparable with the second-largess GPT-3 text completion model, text-curttrrttee 001 (Brown et al. (020) El Like before, the reward for this new aggregated sub-trajectory at the last transition and O otherwise. Taking Eq. [ as an example, we prompt the LLM to summarize the two skills (=1 "Put a mug in the coffee machine," 22 "Push the brew button") ) resulting in new semantic annotation 2 describing both skills (e.g.. "Make coffee"). We then add the new trajectory back to our dataset DL, Using this technique, we generate new language annotations for all (2)) tuples of consecutive sub-trajectories in our dataset. In practice, this allows us to increase the number pre- pre-training task instructions by 2.5x without additional human effort.
4.3 CROSS TRAJECTORY CHAINING
Agents trained with offline RL can combine behaviors from multiple trajectories via value propa- gation, i.e., "stitch" them (evinee et al., 2020). For example, if trajectory (A) shows cleaning the mug in the sink and then placing it on the shelf, while trajectory (B) starts with placing the mug in the coffee machine, offline RL algorithms are able to learn to clean the mug in the sink and then place it in the coffee machine (see Figure 2 right). In our case of instruction-condiionel offline RL, enabling such stitching behavior requires special care. Due to the different language instruction conditionings for the critic Q(s,, a, 2A) and Q(s,, a, ZB), values do not naturally propagate from tra- jectory (B) back to trajectory (A). Instead, we must actively add "chaining examples" to our training dataset (Chebotar et al.) 2021). For this, we randomly sample sub-trajectory T= [su.e. Gu:t A from the training dataset D=, e.g., part of trajectory (A), and replace its language instruction with the instruction n. from another trujectory Tag e.g., "place mug in coffee machine." Note that we do not need to sample the full trajectory (A). Instead, can be any state from trajectory (A) from which we try to execute skill (B). Crucially, we cannot use the same reward function as before: since the last state st of the sampled sub tujjtrajcc,ory e.g., 2. a mug in the sink, does not solve the instruction sB, "put mug in coffee machine," " we cannot set its reward to 1. Which reward should we use instead?
Let's recap that when using iemporal-differenc (TD) learning (Sutton & Barto 2018). functions intuitively represent a value that is proportional to
for the sparse reward definition from Equation] 2
'Other relaheling operations, such splitting an instruction into lower-level instructions, can also be per- formed by the I.I.M However, such operations require grounding the L.MM to the agent's observations to determine sub-trajectory split points. We leave investigating this to future work.
"See appendix, Sections F.1 and R.2 for example aggregated skills and comparisons of different L.I.Ms.
5

---

Under review as a conference paper at ICLR 2023
the probability of reaching the goal at time T (Eysenbach al. 2022 Chebotar et al. 2021):
Q 0z, =) E
Σ of Riss ar, x) = E [---- [sr s=l] x P (88 = 92/0e, 0t).
(3)
Similarly, we want ((s, a, BB) to represent the probability of reaching the goal Ixp+ the mug being in the coffee machine, from state s. Thus, we need set the reward of the last state 5t in the sampled sub-trajectory to the probability reaching the goal Ian from 8t, i.., ((5e, dt zB):
R(s, a, =B) = (s, EB); for 8 = 8T
(4)
00
otherwise.
Finally, we can apply the skill aggregation approach from Section 4.2 in the cross-trajectory case. Instead of sampling two sub-trajectories from the same training trajectory as in skill aggregation, we can now sample Tæa and Tag from different training trajectories and chain them together. The aggregate instruction implies that the agent first finishes skill (A) and then finishes skill (B), e-g., "clean the coffee mug and place in the coffee machine." Thus, following the logic above, we need set the reward for the final state in trajectory (B). STy. to since it solves the task and the reward in the final state of trajectory (A), STA, to the probability of solving skill (B) from there:
1,
for 8Tn for STA otherwise.
a)
a(s,a, 0,
(5)
Note that here, unlike in Section 4.2 do not construct combined trajectory from Tz Tags as we do not know the states and actions required to transition from the last state of (A) to states in (B). See appendix Section C.4 for a more detailed discussion chaining. To generate the aggregate instruction we can use the LLM summarization from Section 4.2 Yet, in practice we found the resulting summaries are often meaningless, since randomly paired instructions from different trajectories can rarely be summarized meaningfully. Instead, we saw better performance by simply concatenating the natural language instructions from TA and Tn. Finally, since Q changes over the course of training, we compute the rewards in Eqs. and 5 online during training. We perform relabeling trajectories at equal proportions for both rewards.
EXPERIMENTS
In our experiments, we investigate how well an agent pre-trained with SPRINT performs on unseen, semantically meaningful tasks. Specifically, we are interested in answering the following questions: (1) Does pre-training with SPRINT lead to more efficient finetuning on unseen target tasks than previous pre-training approaches?
(2) Can agents pre-trained with SPRINT execute unseen language instructions zero-shot? (3) Does pre-trate pre-training with semantic skills lead to better generalization to unseen environments than unsupervised pre-traine pre-training approaches?
5.1 EXPERIMENTAL SETUP
ALFRED Benchmark. We choose the ALFRED benchmark (Shridhar et al. 2020) for evaluation (see Figure since in contrast existing multi-task RL environments (Yu
2019 it allows us to test our pre-trained agents on semantically meaningful, long- ainghooizonn tasks in a realistic household setting. Additionally, ALFRED provides dataset of 6.6k episodes of offline agent experience with crowd-sourced natural language instruction annotations (e.g., "pick up the dirly mug", "wash it in the sink" see Figure 4. The observation space in AL.RRED consists of 300 x 300 egocentric RGB observations and the action space consists of 12 discrete action choices (c.g. turn left. look up, pick up object), along with 82 discrete object types (Pashevich et al. 2021). We leverage 73k natural language instructions and their associated observation-acee action trajectories from the dataset for pre-training. For more environment and dataset details, see Section D
Evaluation Tasks. We create set of 100 varied-hor horizon evaluation tasks (EVAL_100), by sampling sequences to 7 instructions from the ALFRED dataset and asking human annotators to provide a high-level language instruction that describes the sequence. For example, the 5-instruction sequence

---

Under review as a conference paper at ICLR 2023
Number of Completed Subtasks
1.4
1.0
AM
L-BC
SPRINT
EVAL_100
EVAL_CHAIN
Proportion Taak Completees peted
1.2
0.8
1.0
0.8
0.6
0.8
0.4
0.4
0.2
0.2
Eval1000 Overall
Task Length (Number of Subtasks)
Figure 5: Zero shot evaluation performance on the EVAL_ and EVAL.CHAIN task sets. The left figure plots the overall number of completed subtasks on EVAL_J00 (i.e., average return), with our method in green. The right figure plots the proportion of task completed (number of subtasks suc- cessfully executed divided by task length), split by task length, in both the task sets. See appendix, Table 3 for exact numerical results.
"pick up the knife", "slice the potato", "put the knife in the fridge", "pick up a potato slice", "put the slice in the fridge" was summarized as "Cut and refrigerate the potato on the counter" (for more examples, see appendix, Section D.2 The task is solved if an agent completes all 5 instructions, in order, conditioned on the human task description. We ensure that EVAL_ has an equal distribution of length 1, 2, 3, 4, and 5+ tasks (20 of each category). Additionally, we create a set of 20 evaluation commands that test agent's long-horizon chaining capabilities (EVAL_CHAIN): we withhold trajectories and instruction summaries of sequences with 7 to 8 instructions from the ALFRED dataset, making sure that the respective skill chains are not present any single training trajectory. Thus, the agent needs to learn to chain behaviors across multiple trajectories solve these tasks. Finally, to test environment generalization capabilities, we also create set of 10 tasks in household floor plans seen in any trajectory in the training dataset UULE,N,,E,N), consisting in equal parts of I to 5-instruction sequences.
Comparisons. We compare SPRINT against mon policy pre-training approaches: behavioral cloning and offline goal-conditioned RL. Specifi- cally, we compare to the following prior works:
Goal: "Rinse a mug and place it in the coffee maker"
Colle Fraler
WORK BY Mea?
Trih ts Barcelay
Language-conditioned BC (L-BC) (Jang et al. [021 Lynch & Sermanet 2021 Be- havior cloning (BC) conditioned on the in- dividual ALFRED language instructions.
Tout twr
Episodic Transformers (ET) Pashevich
al. 2021 A transformer BC architecture
that conditions on full sequences of lan- guage instructions-SOn on certain AL-
FRED tasks.
Figure from Shridhar et 22020), with permission.
4:
ALFRED.
Figure drawn
Actionable Models (AM) (Chebotar al.
202T]: Goal-conditioned offline RI. with randomly sampled goal observations from the ALFRED training set.
We implement all methods with the same architecture and hyperparameters where possible, and pre- train them for the same number steps with the same language tokens.. For more implementation details, appendix Section C All results reported means and standard deviations over 3 seeds.
5.1.1 SPRINT SOLVES
TASKS ZERO-SHOT
We first test the effectiveness of SPRINT's pre-training by analyzing zero-shot performance across the 100 tasks in the EVAL 100 evaluation set. We report results Figure 5 (left). Our approach, SPRINT, achieves 3x higher zero-shot task performance than prior pre-training approaches AM

---

Under review as a conference paper at ICLR 2023
and L-BC. Even though ET is designed for ALFRED evaluations and benefits from transformer architecture, SPRINT still outperforms it by 1.5x. To better understand the differences between the methods, we report the breakdown of returns by length of the evaluation task in Figure [ right. We find that L.-C achieves good performance on short-horizon tasks. However, on long-horizon tasks, SPRINT achieves much higher returns, since it leverage the language-model automatically generate longer-horizon pre-training tasks. In contrast, standard L-BC approaches train only on the human-provided, shorter-horizon annotations and thus cannot zero-shot perform long-horizon tasks. This trend holds even when evaluating with detailed step-by-step instructions (see appendix Section E.3) Similar to our approach, AM trains to reach long-horizen goals during pre-training but the results in Figure 5 right, show that AM's pre-training with goal-state conditioning is less effective than our language-conditioned pre-training-
To evaluate the pre-training approaches' ability to perform unseen skill sequences, we evaluate zero- shot performance on the EVAL_CHAIN task set in Figure 5 Unsurprisingly, both L-BC and AM largely fail to generate returns these long-horizon tasks, ET does slightly better as it is trained to attend to long-horizon instruction sequences, but it still performs poorly on these tasks as cannot generalize to the higher-level unseen long-horizon tasks with up to subtasks (see Figure / for an example trajectory, appendix Section E.S for more qualitative comparisons). This shows that, with the help of cross-traeectory chaining, SPRINT is able to go beyond the skills in the training data and execute unseen tasks.
wwitten task descriptions. In contrast, SPRINT can solve
5.1.2 SPRINT AGENTS FINETUNE EFFECTIVELY IN UNSEEN ENVIRONMENTS
To test downstream task performance, we finetune the pre-trained agents on the EVAL.NNSEEN task set in unseen household floor plans. To implement finetuning for SPRINT and AM, we condition the policy on a language instruction or goal image from
EVAL_UNSEEN Average Retum vs Env Steps
SPRINT
AM
Average Retur
LBCC
0.4
the target task respectively and then run IQL with
0.2
online data collection. For L-BC, we first pre-train language-condttioned Q-function with IQL on the pre-training dataset and then finetune policy and Q- function with online IQL. We do not finetune ET as it is transformer architecture that is nontrivial to implement for RL.
OK
20k
40k
60k
Environment Steps (1K)
Figure 6: Finetuning results floorplans not seen during pre- training.
We report finetuning results in Figure 6 with qual-
Our approach, SPRINT, able to achieve 2x higher downstream task return than the best prior work. Specifically, find L-BC converges quickly to a low return as it is not pre-trained for longer horizon tasks. Meanwhile, AM struggles to transfer the learned skills the new environment, possibly since the goal states from the new environment are unseen. In contrast, our method's 's pre-training with language conditioning allows for effective transfer even unseen environments since the semanties of the tasks transfer well: the language description "place cup in coffee machine" transfers to many environments while the goal image for the same task might look very different. Thus, pre-training with semantic language instructions can enable better transfer for learning tasks in new environments than pre-training to reach goal states.
itative examples in appendix, Section E.5
5.1.3 ABLATION STUDIES
We verify the effectiveness of the components of our approach, with the following ablations:
SPRINT w/o chain: removes cross-trajectory chaining (Section 4.3 instead trains only on -rihin-trajectory human-provided and LLM-aggregated tasks
SPRINT w/o LLM-agg: additionally removes the L.M aggregation (Section 4.2 thus trains offline RL agent only on the puoviedr task annotations.
We report zero-shot evaluation results in Table T The results show the importance of the different components of our approach. Without cross-trajectory chaining, performance drops by ~20-30% Interestingly, we find that performance also degrades on the EVAL_100 task set, indicating that cross-trajectory chaining can improve performance even on long-horizon tasks the training set.
8

---

Under review as a conference paper at ICLR 2023
Task: "Warm a piece of apple"
"Pick up apple"
"Place apple on table'
"Pick up knife"
"Slice apple"
*Place knife table"
up apple slice"
"Heat apple
back out"
*Place slice the table"
Figure 7: Example task execution of our pre-traine SPRINT agent for the task "Warm a piece of apple". Successful execution this task requires solving 8 subtasks in sequence and total of 50 steps. This exact sequence of subtasks was never observed in the training data. SPRINT leverages cross-trajectory stitching and LLM aggregation to learn how to execute unseen tasks.
We observe an even larger performance loss when removing the language model aggregation of pre-training tasks: the re- sulting agent is solely trained on the shorter horizon human task annotations, and thus struggles to solve long-horizon evaluation tasks, similar the L-BC ap- proach in Section 1.1.1 We provide the per-task-length breakdown of the ablation performances appendix, Section B.4 This section also presents additional ablation experiments that use simple concatenation instead of LLM summariza- tion for task aggregation, resulting in the same longer-horizon pre-training trajectories but different task labels, further supporting the importance of the usage of L.LMs in our pre-training approach.
Table 1: EVAL 100 and EVAL CHAIN returns.
EVAL.100
EVAL.CHAIN
SPRINT (ours)
1.27 1: 0.13
2.59 0.66
SPRINT w/o chain
0.91 + 0.03
2.04 0.04
SPRINT w/o LLM-agg
0.38 + 0.05
0.10 0.04
DISCUSSION, LIMITATIONS, AND FUTURE WORK
In this paper, presented SPRINT, approach for scalable agent pre-training with offline iestructio-ccondiiioned RI. We demonstrated how we can leverage easy-to-collect natural language instructions offline agent experience for effective pre-training. Further, we introduced approaches for using pre-trained large language models and cross-trajectory skill chaining to automatically expand the set of pre-training tasks. In our experimental evaluations on the ALFRED household task benchmark (Shridhar et al. 2020) we demonstrated that SPRINT pre pre-trainen training allows for sub- stantially more efficient finetuning on downstream tasks unseen environments than prior works.
SPRINT represents a step towards scalable approaches for pre-training agents with set of seman- tically meaningful skills.
Naturally, pre-training with semantic skills will be most effective long-horizon, semantically meaningful tasks. Many of the common RL benchmarks like MetaWorld (Yu et al.] 2019) and D4RL (Fu et al. 2020) do not evaluate agents semantically meaningful or long-horizon tasks and thus we do not expect our method to show significant pre-training benefits there. However, realistic, semantically meaningful tasks,
with the recent trend towards evaluating agents like BEHAVIOR (Srivastava et al. 2022) or MineDojo (Fan et al. 2022), we believe semantic pre- training approaches like SPRINT will be beneficial. Another limitation of current approach is that SPRINT can only leverage language-annotater data. While we can collect such annotations scale crowd-soureing, there will always be lot of agent experience data without language annotations. Investigating approaches that can combine language-annotated and un-annotated data for effective pre-training is an interesting direction for future work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv, 2018.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang. Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huel Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do i can and not as say: Grounding language in robotic affordances. In arXiv preprint arXiv:2204.01691, 2022.
Anurag Ajay. Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline prim- itive discovery for accelerating offline reinforcement learning. arXiv preprint urXiv:2010. 13611, 2020.
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In International Conference on Machine Learning, 2017a.
Jacob Andreas, Dan Klein, and Sergey Levine. Learning with latent language. In North American Chapter of the Association for Computational Linguistics, 2017b.
Satchuthananthavale RK Branavan, Harr Chen, Luke Zettlemoyer, and Regina Barzilay. Reinforce- ment learning for mapping instructions to actions. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pp. 82 02-90,22 2009.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https:
wttgabss/////oooooor//oo/o/ssaaa 2005. 14165
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jacob Varley, Alex Irpan, Benjamin Eysenbach, Ryan C Julian, Chelsea Finn, and Sergey Levine. Actionable mod- els: Unsupervised offline reinforcement learning of robotic skills. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research. pp. 1518-1528. PMLR, 18-24 Jul 2021. .tmll
URL http:://pppoceeciigg.......ee http:////11///chcttta2a22
mlr
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov. Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022. URL.tttss: :/ //arxivvvvvv///// www. abs 2204 02311
Douglas
Cédric Colas, Tristan Karch, Nicolas Lair, Jean-Michel Dussoux, Clément Moulin-Frier. Peter Dominey, and Pierre- Yves Oudeyer. Language as a cognitive tool to imagine goals in curios- ity driven exploration. Advances in Neural Information Processing Systems, 2020.
10

---

Under review as a conference paper at ICLR 2023
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2018. URL https://arxiv.or abs/ 1818 0. 04805
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, llya Sutskever, and Picter Abbeel. RL 2: Fast reinforcement learning via slow reinforcement learning, arXiv preprint arXiv:1611.0277 2016.
Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniilidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. In RSS, 2022.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function. In ICLR, 2019.
Benjamin Eysenbach, Tianjun Zhang, Ruslan Salakhutdinov, and Sergey Levine. Contrastive learn- ing goal-conditioned reinforcement learning, 2022. URL https://arxiss www.worrg/ooo abs/ 2206. 07568
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. arXiv preprint arXiv: Arxiv-2206.08833 08853, 2022.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. ICMI., 2017.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072/9, 2020.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy
learning: Solving long- morizoo tasks via imitation and reinforcement learning. CoRL,, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Association for the Advancement of Artificial Intelligence, 2018.
Jordan Hoffimann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen- nigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022. URL htttps:///arttt. org.vv.rrrg///////
2203. 15556
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv:2201.07207, 2022a.
arXiv preprint
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models. In arXiv preprint arXiv:2207.05666 05608 2022b.
Auke Jan ljspeert, Jun Nakanishi, and Stefan Schaal. Movement imitation with nonlinear dynamical systems in humamoid robots. In Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No. 02CH37292), volume 2, pp. 1398-140X. IEEE, 2002.

---

Under review as a conference paper at ICLR 2023
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. BC-z: Zero-shot task generalization with robotic imitation learning. In 5th Annual Conference Robot Learning, 2021. URL id=8kbp23tsGYV
www.:/eee/ooooo forum?
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In International Conference on Learning Representations, 2022. URL https: //
wpwnreview.neeeeeeeeee... net/ wow.e/oooor.ddmmmmmm ppeettrrreee.nfefffoorn////////n
Heinrich Küttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment, 2020. URL https:
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. Urlb: Unsupervised reinforcement learning benchmark. NeurIPS, 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.016433 2020.
Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. Robotics: Science and Systems, 2021.
07648
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In CoRL, 2020.
Russell Mendonca, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak Pathak. Discover- ing and achieving goals via world models, 2021.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 2020.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A uni- versal visual representation for robot manipulation, 2022. URL https:://aaivv..rg/////o///////e
22C3.12601
Taewook Nam, Shao-Hua Sun, Karl Pertsch, Sung Ju Hwang, and Joseph Lim. Skill-based meta- reinforcement learning. In International Conference on Learning Representations (ICLR). 2022.
Karthik Narasimhan, Tejas Kulkarni, and Regina Barzilay. Language understanding for text- based games using deep reinforcement learning. In Proceedings of the 2015 Conference Empirical Methods in Natural Language Processing, pp. 1-11, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10. 10.18653/1/DD15-000. URL https: //aclanthology, orq/b15-1001 1at.laathoooooyooor/11111-1111
Alexander Pashevich, Cordelia Schmid, and Chen Sun. Episodic Transformer for Vision-and-
Language Navigation. ICCV, 2021.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weightte regressssoon Simple and scalable off-policy reinforcement learning, 2019. URL htt.hhtss///arxiv.ooom
abs 19100.0177
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual Reasoning with General Conditioning Layer. In Association for the Advancement of Artificial Intelligence, 2018.
Karl Pertsch, Youngwoon Lee, and Joseph J. Lim. Accelerating reinforcement learning with learned skill priors. In CoRL, 2020.
Karl Pertsch, Youngwoon Lee, Yue Wu, and Joseph J. Lim. Demonstration guidedd reinforcement learning with learned skills. In CoRI, 2021.
12

---

Under review as a conference paper at ICLR 2023
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young. Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Pagannni, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kun- coro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Men- sch, Jean- -aaantBpttitte Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong. Daniel Toyama, Cyprien de Masson d' Autume, Yu- jia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Au- relia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, lason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training go- pher, 2021. URL htt.http///// htt.tpps//aaxll.oog///a// wwtpv/...rggabb//11.... 11446
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reiiffrrcement learning via probabilistic context variables. In JCML, 2019.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference Empiricul Methods in Natural Language Processing. Association for Computational Linguistics, 2019. URL htttps://arxv..
erg/abs/ 1908. 10084
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. arXiv, abs/1907.01657, 2019..
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. ICLR, 2020.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyrr and Dieter Fox. ALFRED: A Benchmark for Interpreting Grounded Instruc- tions for Everyday Tasks. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2020. URL httpt::: http:://z/x:w org/ abs 11...9111111
Avi Singh, Albert Yu, Jonathan Yang, Jesse Zhang, Aviral Kumar, and Sergey Levine. Cog: Con- necting new skills to past experience with offline reinforcement learning. Conference on Robot Learning, november 2020.
Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine, Parrot: Data-driven behavioral priors for reinforcement learning. ICLR, 2021.
Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Martin-Martin, Fei Xia, Kent Elliott Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, Karen Liu, et al. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. Conference on Robot Learning, pp. 477-490. PMLR, 2022.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Evangelos Theodorou, Jonas Buchli, and Stefan Schaal. Reinforcement learning of motor skills in high dimensions: A path integral approach. In 2010 IEEE International Conference on Robotics and Automation, pp. 2397-2403. IEEE, 2010.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https::/giithb.. com www.co//w/iiooooooocoooco http:///tttttttttoooooeeeeee jax May 2021.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning, 2019.
13

---

Under review as a conference paper at ICLR 2023
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettle- moyer. Opt: Open pre-trained transformer language models, 2022. URL https: :ttpppss//rxiiv
prg/abs/2205. 2205 01068
14

---

Under review as a conference paper at ICLR 2023
APPENDIX
Table of Contents
SPRINT Pscudocode
16
Large Language Model Prompt
Baselines and Implementation
17
C.I
Language-conin conditioned Behavior Cloning
19
Episodic Transtormers
19
Actionable Models (AM)
19
C.4 SPRINT
20
Dataset and Environment Details
21
D.J Dataset Details
21
D22 Evaluation Tasks
23
E
Extended Experiments, Results, and Analysis
27
E,1
LLM Summary Examples
................................
27
E.2
Comparing Summaries from Different LLMS
................................
29
E.3
Step by Step Zero-shot evaluations
29
E,4
Extended Ablation Study
30
Qualitative Comparison Results
30
15

---

Under review as a conference paper at ICLR 2023
SETUDOCODE 
A SPRINT PSEUDOCODE
Algorithmi 1 SPRINT Algorithm
Require: Dataset D w/ language instruction labels, LLM
1: AGGREGATESKILLSS(D* LLM)
Automated LLM skill relabeling (Sec.
4.2
2: while not converged do
3: To 4- DP: Sample an annotated skill (sub-)trajectory
4: Train offline RL on Tz
5:
Cross Cross-trajectonn trajectory skill chaining (Sec.
6:
chain
-CROSSCHAINSKILLSSD)
4.3
Train offline RL on 13 Jaan
7:
8:
9: Tags 7 Fasz, <-
CROSSACCRRGGATTSSSIIISSSSSS LLM)
> Cross-traj. aggregation (Sec.
10:
Train offline RI. on Tagey Tageg
11:
end while
procedure ACCREEAATESKILLLN LLM) for composite trajectory Ts in D do
13:
14:
15:
Separate into language annotated sub-trajectories (Ta Taye for all adjacent sub- cab-trajectories [T., do
16:
17:
Assign name from LLM: (L. Beg
18:
Taus Concat [Tacsrn um Tay_ and relabel with &ag
T + length of Taus
20:
R(sT, or, 21:z) =
Label last reward with 1 (Eq. 2
D4 DP (Taeg}
22:
end for
23:
end for end procedure
24:
procedure CROSSCHAINSKKLLLSSER C
25:
27:
Sample random (sub)trajectories Tai Tep ~ DL Sample random endpoint in Tax chain
28:
29:
T22
[[50, Go, 0, 22) (8y,
b Relabel reward Eq
30:
return T=2
end procedure
32:
procedure CROSS CAOSSAAACRREGGAATSKIILLSSSS (LLM)
33:
34:
Sample random (sub)traecctories T.y Tag D with lengths 7i,7b
Assign new name: 1 [=2)-
36:
Tages
(S0, ao, 0, .) (ST, aT, Q* (ST, arl2),,)
P Relabel reward w/ Eq
37:
Taga,
(A0, 0a, 0, 2) (sT,, or,, 2)]
b. Relahel reward w/ Eq.
38:
return Tagsi Tapez
39:
end procedure
B LARGE LANGUAGE MODEL PROMPT
We list the full large language model summarization prompt in Figure 8 The examples in the prompt are fixed for all summarization queries. These examples are selected from the ALFRED validation dataset (which is not otherwise used our work) random: We spell out the primitive v
skill annotations in the "Task Steps:" part of each prompt prompexxample Then, the "Summary" for each of these is the high-level, human-written inooationn for that trajectory from ALFRED. We repeatedly samma these trajectories until each example mentioned a different object to prevent biasing the LLM towards certain types objects.
exampe
We note that the "Look at the vase under the light" example is important to make the LLM give reasonable summaries for similar tasks in ALFRED where the agent picks something up and turns on light. This is because most of the human labels for turning the lamp do not mention the object in the previous step, making it difficult for the LLM to realize that the task has to do with h looking at the held object under lamp.
16

---

Under review as a conference paper at ICLR 2023
Instructions: summarize the following ordered steps describing common household tasks.
Task Steps: 1:
Pick up the smaller knife on the counter to the left of the stove. 2: Slice the tomato with the smaller knife. Put the knife in the sink. 4: Pick up a slice of tomato from the countertop. 5: Heat up the slice of tomato in the microwave, removing it afterwards. Summary: Microwave the tomato slice after slicing it with the smaller knife on the counter.
Task Steps: I:
Pick up the vase. 2: Turn on the lamp.
Summary: Look at the vase under the light.
Task Steps: I: Grab the pencil off of the desk. 2: Put the pencil in the bowl. 3: Grab the container off of the desk. 4: Put the container down at the back of the desk. Summary: Put bowl with pencil in it on the desk.
Task Steps: I: Pick up the bar of soap from the back of the toilet. 2: Put the bar of soap in the sink, turn on the faucet to rinse off the soap, pick up the soap out of the sink. 3: Put the soap in the cabinet under the sink and on the left. Summary: Put a rinsed bar of soap in the cabinet under the sink.
Task Steps: I: [SKILL 1J. 2: [SKILL 2]. 3: [SKILL 3]. N: [SKILL NJ. Summary:
Figure 8: The full prompt that we use for summarization. Following the suggestions of Ahn et al. (022) for prompt design, we explicitly number each step. The L.LM completion task begins after "Summary:". For brevity, we omit the new line characters between all numbered steps.
C BASELINES AND IMPLEMENTATION
We implement IQL (Kostrikov et al. 2022 as the base offline RL algorithm for all goal-conditioned offline RL pretraining baselines and ablations due to its strong offline and finetuning performance on a variety of dense and sparse reward environments. At a high level, IQL trains on in in-distrbbution (s,a, a, 8 rr,a') tuples from the dataset rather than sampling policy for a' ensure that the Q and value functions represent accurate estimated returns constrained to actions in the dataset. The value function is trained with an expectile regression loss controlled by a hyperparameter т, where 7 0.5 results in standard mean squared eгror loss and 7 + 1 approximates the max operator, resulting in more optimistic value function that can better "stitch" together trajectories to obtain distant reward in sparse reward settings. The IQL policy is trained to maximize the following objective:
((((s.)) log Toga((6 (als),
which performs advantage- advantage-weighted regression (Peng et al. 2019) with an inverse temperature term 3. In practice, the exponential advantage overflow issues. We detail shared training and implementation details below, with method-specific information and hyperparameters in the following subsections.
limited to maximum value avoid numerical
Observation space. The state space of the ALFRED environment consists of 300 x 300 RGB images. Following the baseline method in ALFRED (Shridhar al.) 2020), we preprocess these images by sending them through frozen ResNet-18 encoder (He et al., 2016) pretrained on Ima- geNet (Deng et al. 2009). This results a 512 input to all networks. Furthermore, as ALFRED is partially observable, egocentric navigation en- vironment, we concatenate the last 5 frames as the full observation, resulting in an observation that of the shape (512 5) x 7 x 7.
7 feature map that we use as the observation
Action space. The agent chooses from 12 discrete low-level actions. There are 5 navigation ac- tions: MoveAhead, RotateRight RotateLeft, LookUp, and LookDown and 7 interaction actions: P Pickup, Put, Open, Close, ToggleOn, ToggleOff, and Slice. For interaction actions the agent additionally seleets of 82 object types to interact with, as defined by Pashevich jet al. (202T). In total, the action space consists of 5 + * 82 = 579 discrete action choices. Note that this action space definition different from the action space in Shridhar et al. (222)). which
17

---

Under review as a conference paper at ICLR 2023
used pixel-wise mask output determine the object to interact with. In contrast to Shridhar al. 2020) we aim to train agents with reinforcement learning instead of imitation learning and found the discrete action parametrization more amenable to RL training than dense mask outputs. For all methods, due to the large discrete action space, we perform some basic action masking to prevent agents from taking actions that are not possible. For example, we do not allow the agent Close objects that aren't I closeable nor can they ToggleOn objects that can't be turned on.
Language embedding. We embed language instructions with the all all-mpnet basebaaeeeeee.ee pretrained sentence embedding language model from the SentenceTrransormee python pack- age (Reimers & Gurevych 2019). This produces 768-dimensional language embedding which is used as input for language-conditionee conditioned policy and critic functions, as detailed below.
Policy and critic networks. We train a discrete policies with two output heads of size 12 and 82 for the action and interaction object outputs respectively. Critic networks are conditioned on both the observation and the discrete action output of the policy. all policy and critic networks, we process the ResNet feature observation inputs with 4 convolutional layers. In networks with language input, we flatten the output of the convolutional layers and concatenate the observation features with the 768-dim language embedding, before passing the concatenated image-language features through a series of fully connected layers. Additionally, we use FiLM (Perez et al. 2018) to condition convolutional layers on the language embeddings.
Pre-training hyperparameters. A hyperparameter search was performed first on the language- conditioned BC-baseline to optimize for training accuracy. These hyperparameters were carried over to the IQL implementationn,, and another search for IQL-specific hyperpameters were performed on baseline IQL policy conditioned semantic instructions. With these parameters fixed, performed one more hyperparameter search specific to Actionable Models but for the final imple- mentation of SPRINT we re-used the same hyperparameters and only selected SPRINT-specific parameters heuristically. Hyperparameters for each method are detailed in a separate table. Shared hyperparameters for all methods (where applicable) are listed below:
Param
Value
Batch Size # Training Batches
1024
140k
Learning Rate
2e-3
Optimizer
AdamW
Dropout Rate
0.2
Weight Decay
0.05
Discount ?
0.98
Q Update Polyak Averaging Coefficient
0.005
Q-Network Discrete Action Embedding Size
48
Q-Network Discrete Object Selection Action Embedding Size
24
Policy and Q Update Period
Once every training iteration
Batch Norm
True
Nonlinearity
ReLU
IQL Advantage Clipping
[0. 100]
IQL Advantage Inverse Temperature 3
10
IQI. Quantile
0.9
Finetuning details and hyperparameters. We perform finetuning experiments for Language- conditioned BC, Actionable Models, and SPRINT. For all models, we finetune the model on only newly collected task data by running online IQL (without any of the chaining or aggregation steps). Each method is finetuned on every task in the EVAL.UNSEEN task set individually; that is, we pre-train once and then finetune ten times, turss over all tasks, then report metrics averaged over all random seeds. For each task, we define a maximum rollout time horizon of timesteps per environment action required by the expert planner.
for every task in the task set. We then average
When not specified, finetuning parameters are identical to pre- pre-tranning parameters. Finetuning hy- perparameters specified below:
18

---

Under review as a conference paper at ICLR 2023
Param
Value
Dropout Rate
# Initial Rollouts
50
Training to Env Step Ratio
0.5
e in e-greedy action sampling
0.25: annealed down to 0.05
# Parallel Rollout Samplers
4
C.I LANGUAGE-CONDTTINNE BELIAVIOR CLONING
Our condutioee behavior cloning (L-BC) comparison method is inspired by and replicates BC-Zero (Jang et al. 2021) and La (Lnnch & Sermanet (02T). BC-Zero performs FiLM- conditioned semantic imitation learning (Perez et al. 2018) 8) and both BC-7ero and IfngI.PP have an additional image/video- languag alignment objective. In BC-Zero, their video alignment objective aligns language embeddings with videos of humans performing tasks related to those the BC- robot agent trains on. LangLiP's image-language alignment objective allows their policy to accept both image and natural language goals input due to only having subset of their data labeled with hindsight language labels. As we don't 't have human videos of these tasks and our entire dataset is labeled with language labels, do not add video or image alignment objective.
Therefore, we implement L-BC by using the same architecture as described above with just a single policy network that trains maximize the log-likelihood of actions in the dataset. As entire dataset consists of expert trajectories, this baseline idcally learns optimal actions for the instructions.
Hyperparameters for the L-BC baseline are identical to the shared parameters above, where appli- cable.
C.2 EPISODIC TRANSFORMERS
Episodic Transformers (ET) Pashevich al. 2021 trains transformer architecture full sequences of ALFRED instructions with a behavior cloning objective. This is currently state of the art in the "Seen Path-Length Weighted Success Rate" evaluation metric the ALFRED leaderboard. We used the ET implementation from the official code repository and used the
pre-tuned hyperparameters.
For fair comparison, we make few modifications make it as close as possible SPRINT and the baselines: 1) we train it on the same dataset as all basclines, so we do not generate new synthetic training data like Pashevich all 202T does in the original implementation, frames with Resnet-18 18 instead of Resnet-50 backbone, 3) we use a context window of 5 frames just like for other policies, 4) we remove the high-level goal specification from the input text tokens as we do not assume access to those, and 5) we train the model for longer to match the number of training steps for all methods.
we encode visual
C.3 ACTIONABLE MODELS (AM)
Actionable Models Chebotar et al. 2021 pretrains goal-conditioned Q function conditioned on randomly sampled image goals and also performs goal-chaining procedure very similar our semantic skill chaining procedure. We implement AM by modifying the base IQL policy and critic networks take in image goals instead of natural language embeddings as goals. These goals are provided in the same way the observations, i.e., as a concatenated stack of 5 frames (the last 5 frames in the trajectory) processed by a frozen ResNet-18. Therefore, goals are the same shape as observations: (512 * 5) x
To allow for fair comparison between our approach and AM, we implement AM with the same powerful offline RL algorithm, IQL (Kostrikov al. 2022). used in our method. IQL ensures that the policy does not choose distribbittiiitiion actions by using advantage- weighted regression on in-distribution actions for policy extraction. With this, found the conservative auxiliary loss AM adds to push down Q-values for out-of-distribution actions to be unnecessary and even hurtful to its overall performance, omit this additional loss term.
weightee
19

---

Under review as a conference paper at ICLR 2023
We also pre-train AM the same long-horizon trajectories those generated by SPRINT during LLM-based skill aggregation. This ensures a fair comparison in terms of the types and lengths of tasks seen during pre-training.
Finally, after consulting the authors of AM, we tried varying maximum trajectory lengths when sampling random goals. We found that allowing random goals to be sampled from anywhere within trajectory resulted in the best zero-shot evaluation performance for AM, so our numbers are reported with this implementation detail.
C.4 SPRINT
The implementation details of SPRINT follow from the discussion about implementing IQL the top of this section. The key differences are in (1) language model skill aggregation and (2) cross- trajectory skill chaining, detailed below.
LLM Skill Aggregation. We perform LLM skill aggregation fully offline by iterating through every ALFRED trajectory and aggregating sequences of adjacent primitive skill sub-trajectories. Assuming trajectory with N primitive skills, we select all (N) pairs of start and end skills and aggregate all instructions from start to end with the LLM. With 73k original language-annotee andottaed trajectories, this procedure allows us to generate an additional 1 110k aggregated trajectories. We then add these trajectories to the original dataset and train on the entire set.
Cross-trajectory skill chaining implementation. We perform cross-trajectory skill chaining in- batch. Instead of sampling a second trajectory to perform chaining on, we simply permute the batch indicies to generate a set of randomly sampled second trajectories. Then, we perform second loss function update, in addition to the original update on the sampled trajectories, with equal loss weighting, to apply the skill-chaining update. We apply both chaining procedures from Eq, + and Eq. 5 in-batch with equal weight. Empirically, we found that cross-trajectory skill chaining works slightly better with the on-policy Value function obtained through IQL, therefore we use state values at the chaining targets instead of state-action Q-values.
SPRINT-specific hyperparameters follow:
Param
Value
Large Language Model for Relabeling OPT-13B (Zhang et al. 2022
LLM Token Filtering Top-r[]
0.9
LLM Token Sampling Temperature
0.8
I At each token generation step, only the highest probability tokens with total probability mass that add up to the top-p are considered.
Cross-trajectory chaining extended discussion. When performing cross-trajectory chaining, particularly using Eq. 5 special care must be taken preserve the dynamics of the original MDP. When chaining together two trajectories TA and TR. we concatenate the two sentences of each trajectory together and relabel their rewards with Eq. 5 The new language annotation used chain together these trajectories is the concatenation of the two sentences, implying that the agent finishes skill (A) and then skill (B). However, we cannot concatenate the two trajectories together into one longer trajectory, as doing so would imply that the agent can instantancously jump from the last state of skill (A) to the first state of skill (B), which may not be possible. Therefore, we instead treat the relabeled trajectories at separate trajectories with the same language annotation (lines 36 and 37
of Algorithm
However, this introduces two possible complications: 1) Language annotations differing in structure from those in the original dataset, and 2) Possible instruction ambiguity:
20

---

Under review as a conference paper at ICLR 2023
1. Language annotations differing in structure. Language annotations produced by the chaining procedure will result in annotations that implicitly skip certain steps. For example, when chaining skill (A), "make the bed," and skill (B), "make a cup of coffee," the resulting chained annotation will be "Make the bed. Make cup of coffee." However to perform skill (B) the agent needs first move to the kitchen from the hedroom to make the cup of coffee, which is skipped in this annotation. LLM-based skill aggregation (Section 4.2) helps bridge this gap by summarizing long-horizon sequences while skipping certain implied steps. For example, one real LLM summary (listed in Figure [4] summarized the sequence: "I: Pick up the plaid pillow that on the left end of the couch. 2: Place the pillow on the ottoman," which skipped
the otloman" into the instruction "Place a plaid pillow the step of picking up the pillow as it is implied that you must do so before placing the pillow down. Using the LLM augments our original dataset such that, in ALFRED, we have 2.5X more data after performing offline skill aggregation. Therefore after performing LLM aggregation, there are many examples of similar instructions to those used for chained trajectories that imply certain steps without mentioning them explicitly.
2. Instruction ambiguity. When chaining trajectories, there will be some ambiguity intro- duced as we do not have intermediate instructions for going from the last state of A to the initial state of B (obtaining these instructions requires additional human effort). This ambiguity is only present in the states of trajectory A, as in trajectory B we presume that the instructions for trajectory A are finished and the agent then can just follow the instruc- tions relevant for trajectory B. We believe that the effects of the ambiguity on pre-training performance depends greatly on the given dataset. In complex and diverse environments like ALFRED, hindsight-labeled annotations will contain details specific to certain scenes, resolving this ambiguity. In ALFRED, the annotations usually contain information about the specific objects that the agent must interact with or locations that the agent must go to. For example, annotations for rinsing mugs typically are of the form "clean the MUG in the sink," annotations for picking up a candle will often say something like "pick up the YELLOW CANDLE on the COUNTER," highlighting specific details regarding what the agent is supposed to do to complete the trajectory.
D DATASET AND ENVIRONMENT DETAILS
D. DATASET DETAILS
For training and evaluation we leverage the ALFRED benchmark and dataset (Shridhar et al. 2020
The ALFRED training dataset contains ~6.6k trajectories collected by an optimal planner following a set of 7 high-level tasks with randomly sampled objects (e.g.. pick up an object and heat it). Each trajectory has at least three growd-sourced sets of language instruction annotations. Each trajectory consists of sequence of 3-19 individually annotated skills (see Figure 9 left). This results in a total of 141k language- annotated skill trajectories.
However, nearly half of the language instructions in the ALFRED dataset are navigation skill in- structions like "turn left, then look up and walk to the counter on the right". To get a more balanced skill annotation dataset, we merge all navigation skills with the skill that immediately follows them, using only the annotation of the next skill. After this processing step, the resulting dataset contains 73k language-and -ndttttedd primitive skill trajectories. After merge the navigation skills, the aver- age number of skills in each trajectory is 3.5 skills per trajectory (Figure 9 middle), and the average number of actions in each skill is 14.3 (Figure 9 right).
21

---

Under review as a conference paper at ICLR 2023
2000
2000
1292
1750
1990
1200
The
200
1000
Numbe
730
205
18
Number
Number
Number
(a) Skills trajectory in the origi- nal ALFRED dataset.
(b) Skills per trajectory in the (c) Actions per skill in the merged dataset.
merged dataset.
Figure 9: Left: distribution of the number of skills in each trajectory in the original ALFRED dataset. Middle: distribution of skills per trajectory the "merged" dataset with merged navigation skills. Right: distribution of number of actions per skill in the "merged" dataset.
22

---

Under review as a conference paper at ICLR 2023
D.2 EVALUATION TASKS
Data Collection Overview
you partolpsting summaraation willbe sentencs instuctionn deseribing household tosks. bry unoure une sentence surrr my you witt other suliiiyy explcily usscrioes the instuctions
For seample givan the folloingg centences:
You could witt "Bel pot water from the Hisgo."
fow exprples
1. Cec Lactuce to the indt the wisk.
2. pec
down store.
3- aliie lattoce.
acanaryy cut bettuce on the lade the wink and take allce.
up the glans of waser
ziqut spanis, on the laft the shele.
nour glane of wacer tnto plancie cotl..
3. 2111 glase with watar
cink.
d- Pour glass of waune into the plant.
5- men ikon the glose water.
drararys sater the plass with twn glanne wwine..
will
inst
from instructions
Earing th arceess you lik given instn weadly furned into cre-sarianca igh-insel luurraary rarr get eeriances by prexing the 'Sep' button Oncw you underntard thase Instn chooe, feel continue the noxt tegitt
Annotation Task (40 summaries)
Intersettprogronn
toaiin cnroreeed shiclE soene bypee trale"
program/train,
cype)
summittec
Please serbence descipton that then folloing inttrcctions
Pick up horts bean.
Place the bottle bahind aoop courter
Sko
Submii Summery
Figure 10: Data collection jupyter notebook page. Note that there is a "Skip" button so that human annotators can skip an instruction sequence they do not feel it is semantically meaningful or easy to summarize.
We evaluate agents through zero-shot policy evaluation and finetuning on three sets of evaluation tasks in the ALFRED environment: (1) EVAL_100 to measure the ability of pre-trained agents to exccute semantically meaningful instructions at varied levels of abstraction. (2) EVAL-CHAIN to measure the ability of agents chain behaviors across multiple trajectories to solve long tasks, and (3) ENSEEN to evaluate generalization performance when finetuning to unseen household floor plans.
Collecting evaluation task data. The ALFRED dataset provides high-level language annotations for each of the trajectories in the dataset. We could use these annotations as unseen aaskiintructtins to evaluate our agents. However, we found that the different skills are not equally distributed across trajectories different skill lengths, e.g., most 2-skill trajectories perform pick-and-place tasks while tasks involving heating skills only appear in length 7+ trajectories. To allow evaluation with a less biased skill distribution, we create the EVAL_100 task set by randomly choosing trajectory from the ALFRED dataset and then randomly sampling subsequence of skills of certain length from this trajectory. To obtain a high-level language instruction that summarizes this new subse- quence, crowd-source labels from human annotators. For labeling, each annotator is presented
23

---

Under review as a conference paper at ICLR 2023
Whenever we by chance sample a full ALFRED trajectory for annotation, we directly used the existing high-level annotation from the ALFRED dataset. We annotate 80 trajectories with human annotators and combine them with 20 randomly sampled single-skill trajectories, resulting in a total of 100 evaluation tasks (see Figure for example instructions). This results in 20 tasks of length skills, 20 tasks of length 2 skills, 20 tasks of length 3 skills, 20 tasks of length 4 skills, and 20 tasks of lengths 5+ (5-7) skills.
with remotely hosted Jupyter notebook interface (see Figure 10
For CVAI, we randomly sampled 20 full trajectories from the ALFRED dataset that had sequences of 7 or 8 skills (I0 of length 7, 10 of length 8) and removed these trajectories from the post-LLM aggregated training dataset. We did not remove any of the LLM-aggregated trajectories made up of subsequences of skills within that trajectory. This allows AM and SPRINT to perform skill chaining to solve these tasks by ensuring that there were valid sequences of skills to chain together to be able to solve these removed tasks. For example, assume (shortened for clarity) sampled skill sequence is "pick up apple," then "put apple in microwave", then "slice the apple." Then, either Actionable Models or SPRINT can chain together the sub-trajectory associated with "pick up apple" then "put apple in microwave" with the "slice the apple" sub-trajectory to solve this task. These trajectories all had annotations from ALFRED annotators, directly (see Figure 13 for example instructions).
used those annotations
Finally, for EVAL UNSEEN, collected set of 10 full-length trajectories from the ALFRED "valid-unseen" dataset consisting of validation tasks unseen floor plans. We collected 2 of each length from 1 through 5 for total of 10 tasks by sampling random full-length trajectories from this dataset, with the exception of length I tasks (we just sample random skills to create length tasks). As these are full trajectories, they already have human annotations from ALFRED, which we directly use as the task description (see Figure 12 for example instructions).
We list additional details about the tasks in each evaluation set in Table
Table 2: Evaluation Task Specifics. Note that the "number of env actions per task" corresponds to the number of environment actions the ALFRED expert planner required to complete that task.
EVAL
EVAL-CHAIN
EVAL Unseen
Number of Tasks
100
20
10
Task Lengths (# primitive skills)
[1, 2, 3, 6, 7]
[7.8)
[1, 3, 4, 5]
Min Number of Eny Actions per Task
34
2
Avg Number of Env Actions per Task
39.1
60.9
46.6
Max Number of Eny Actions per Task
113
104
124
Finally, we display 5 randomly sampled tasks, along with their human annotations, from each of our task sets in Figures [1 12 and 13
Online finetuning environment setup. During online- finetuning we initialize the agent in the same house floor plan as the trajectory the task was extracted from to ensure executability. During finetuning, we give each episode a time horizon of 2x the number of environment actions needed by the expert planner solve the task. We give sparse rewards for each skill solved by the agent during the episode. Therefore for length tasks, the agent can only be rewarded once before the episode ends, while for length 5 tasks, the episode terminates on the fifth reward signal. We give a reward of sotal skills for each skill the agent successfully executes so that the return sums to 1. We found that this helped to finetune all comparison methods more stably, possibly due to the fact that giving larger rewards (e.g., I for each skill) results in out-of-distribution critic values (when compared to pre-training) that de-stabilize online reinforcement learning.
24

---

Under review as a conference paper at ICLR 2023
Skills to Summarize: 1: Grab the knife on the counter. 2: Place the knife in the sink then turn the faucet on so water fills the sink. Turn the faucet off and pick up the knife again. Place the knife on the table to the left of the wooden bowl.
Annotator Summary: Wash the knife from the counter, put in on the table.
Skills Summarize: Pick up the blue book closest to your and the phone from the bed. Turn on the lamp to take a look at the book in the light. Annotator Summary: Examine the book by the light of a lamp.
Skills to Summarize: Pick up yellow candle on counter. Open cabinet, put candle cabinet, close cabinet 3: Pick up yellow candle from toilet.
Annotator Summary: Move the candle from the sink to the cabinet under the sink, close and and then pick the candle from the top of the toilet in front of you.
Skills Summarize: I: Pick the pot on the left side up from the stove. 2: Set the bowl and knife on the table next to the tomato. Annotator Summary: Put the bowl with the knife in it next the tomato.
Skills to Summarize: 1: Pick up the pen that's front of you that's under the mug. Put the pencil in the mug that was above it. 3: Pick up the mug with the pencil in it.
Annotator Summary: Put the pen into the mug and pick up the mug.
Figure 11: Randomly sampled, human language instruction annotations from the EVAL_/00 task
Skills to Summarize: 1: Pick up the lettuce on the counter. 2: Chill the lettuce in the fridge. 3: Put the chilled lettuce the counter, in front of the bread. Annotator Summary: Put chilled lettuce on the counter.
Skills to Summarize: I: Pick up an egg from off of the kitchen counter. 2: Open the fridge, put the egg in chill for few seconds and then take it back out. Place the cold egg in the sink. Annotator Summary: Chill an egg and put it in the sink.
Skills to Summarize: 1: Pick up the butter knife off of the right side of the kitchen island. 2: Put the knife handle down the frying pan that is on the front left burner the stove. 3: Pick up the frying pan with the knife in it off of the stove. 4: Put the frying pan with the knife in it into the sink basin to the right of the potato. Annotator Summary: Put frying pan with knife in it into the sink.
Skills Summarize: 1:
Take the pencil from the desk. 2: Put the pencil the desk. Annotator Summary: Take the pencil from the desk, put it on the other side of the desk.
Skills to Summarize: 1: Pick the left pillow on the chair. 2: Put the pillow on the sofa right
of the newspaper. 3: Pick up the pillow on the chair. 4: Put the pillow the sofa left of the newspaper.
Annotator Summary: Place two pillows on sofa.
Figure 12: Randomly sampled, human language instruction annotations from the EVAL UNSEEN task set.
25

---

Under review as a conference paper at ICLR 2023
Skills Summarize: 1: Pick up the knife in front the lettuce. 2: Slice the apple in the sink
with the knife. 3: Place the knife into the sink. 4: Pick up the sliced apple from the sink. 5: Place the apple slice into the pot on the stove. Pick up the pot from the stove. 7: Pick up the pot from the stove.
Annotator Summary: Slice an apple for the pot on the stove and put the pot on the counter to the right of the door.
Skills to Summarize: 1: Take the apple from the counter in front of you. 2: Place the apple
in the sink in front of you. 3: Take the knife by the sink in front of you. 4: Cut the apple in
the sink in front of you. 5: Place the knife in the sink in front of you. 6: Take apple slice
from the sink in front of you. 7: Heat the apple in the microwave, take it out and close the microwave. 8: Place the apple slice in the sink in front of you. Annotator Summary: Place warm apple slice in the sink.
Skills to Summarize: 1: Pick up the loaf of bread. 2: Put the bread on the counter above the spatula. Pick up the knife that's above and the right of the loaf of bread. 4: Cut the top half of the loaf of bread into slices. 5: Put the knife on the edge of the counter in front of you horizontally, 6: Pick up a slice of bread from the middle of the loaf. Cook the bread in the microwave then take it out and close the microwave door. 8: Throw the cooked slice of bread away. Annotator Summary: Put microwaved slice of bread in the oven.
Pick the knife up from off of the table. 2: Open the microwave, slice the potato. and close the microwave. 3: Open the microwave, place the knife inside of it, and close the microwave. 4: Open the microwave, pick up the potato slice inside, close the microwave. 5: Place the potato slice in the pan on the stove. 6: Pick up the pan from the stove. 7: Open the refrigerator, place the pan inside, and close the refrigerator. Annotator Summary: Move the pan from the stove top to inside the black refrigerator.
Skills Summarize: I:
Skills to Summarize: 1: Pick up the red tomato the counter to the right of the stove. 2: Put the tomato onto the island below the butter knife. Pick up the butter knife off of the kitchen island. 4: Slice up the tomato the kitchen island. 5: Place the butter knife onto the island to the right of the sliced tomato. Pick up tomato slice off the kitchen island. 7: Open the fridge and put the tomato slice the bottom shelf, then close the door, after couple seconds open the fridge and remove the tomato slice then close the door. 8: Open the microwave door and place the tomato slice inside the microwave in front of the egg. Annotator Summary: Put a chilled tomato slice into the microwave.
Figure 13: Randomly sampled, human language instruction annotations from the EVAL_CHAIN task set.
26

---

Under review as a conference paper at ICLR 2023
E EXTENDED EXPERIMENTS, RESULTS, AND ANALYSIS
Table 3: EVAL 100 and EVAL Chain eval dataset per-length and overall skill completion rates. See
Section 5 for experiment setup.
AM
L.BC
SPRINT
ET
Number of Completed Subtasks Overall
0.46 + 0.05
0.41 0.02
1.27 - 0.13
0.84 + 0.01
Length Progress
0.23 0.01
0.77 0.04
0.53 0.07
0.78 -
Length Progress
0,22 0.03
0.24 - 0.03
0.49 = 0.03
0.53 - 0.06
EVAL
Length Progress
0.17 0.03
0.16 1 0.04
0.48 0.03
0.33 1: 0.12
Length Progress
0,10 0.00
0.06 0.00
0.35
0.06
0.24 1 0.07
Length Progress
0.16 0.08
0.02 0.01
0.46 = 0.06
0.09 + 0.01
Length Progress
0.03
Length Progress
0.00 0.00
0.00 1 0.01
0.01 :: 0.02
0.00 + 0.00
Number of Completed Subtasks Overall
0.67 0.09
0.04 0.05
2.59 = 0,66
1.04 + 0.35
EVAL.CHAIN
Length Progress
0.04 2
0.00 - 0.00
0.25 = 0.10
0.09 + 0.00
Length Progress
0.13
0.01 +
0.43 0.08
Here, we present additional results complementary to the experiments in the main paper in Section We present and analyze LLM annotation examples in Section E.1 zero-shot evaluations with step- by-step task instructions in Section E.3 and an extended ablation analysis Section E.4
E.1 LLM SUMMARY EXAMPLES
We randomly sample 12 OPT-13B task summaries produced while performing skill aggregation (explained in Section 4.2) using the prompt Figure display them in Figure 14 Because sample the tokens sequentially during text generation, some of the summaries mostly describe one or few of the skills but a non-sensical manner (e.g.. LLM(("ic up the plastic bowl the left. Turn on the lamp.") ") "Look at the plastic bowl on the left") "). The summaries also likely can be improved by including more examples the prompt. However, after analyzing many more examples, we see that OPT-13B generally provides fitting high-level summaries for most sequences by skipping over implied sub-tasks (although sometimes also skips over important sub-tasks, likely due to the prompt). For example, it summarizes the task "Pick up the plaid pillow that is the left end of the couch. Place the pillow on the ottoman" as "Place a plaid pillow on the ottoman," in which picking up the plaid pillow is implied. It also often demonstrates object relation understanding, e.g., by stating "Place plaid pillow" " when only the first sub-task in the sequence mentions that the pillow is plaid.
27

---

Under review as a conference paper at ICLR 2023
Skills to Summarize: 1: Pick up the plaid pillow that on the left end of the couch. Place the pillow on the ottoman. LLM-generated Summary: Place a plaid pillow on the ottoman.
Pick up the piece of potato furthest to the right. 2: Open the microwave, place the potato inside, close the door, microwave for two seconds, then remove the potato. 3: Place the cooked potato piece to the right of the salt shaker. L.MM-generaed Summary: Place a cooked potato piece on the table.
Skills to Summarize: 1:
Skills to Summarize: 1: Put the spoon in the pot on the left. 2: Pick up the pot with the spoon. LLM--ennrrted Summary: Put the pot with the spoon the left.
Open the microwave above the stove and slice the apple inside with the knife. 2: Place the knife standing up on the blue dish. 3: Remove slice of apple from the microwave. 4: Place the apple slice the blue plate.
Skills Summarize: 1:
LLM-generated Summary: Put an apple slice on the blue plate.
Pick up the butter knife that is next to the fork. 2: Open the refrigerator. Open the refrigerator and place the knife on the bottom shelf the refrigerator then close the door.
Skills to Summarize: 1:
-Lneggeeeated Summary: Open the refrigerator and place the butter knife on the bottom shelf in the refrigerator then close the door.
Skills to Summarize: 1: Open the fridge and put the knife inside. 2: Take piece of the apple out of the microwave.
I.LM-generaed Summary: Take slice of apple out of the microwave.
Skills to Summarize: I: Pick up the remote that on the shelf. 2: Put the remote the couch, LLM LLM-generated Summary: Put the remote on the couch.
Skills to Summarize: 1: Pick up the dirty green towel to the right of the bar of soap on the vanity. 2: Put the towel into the sink, rinse for a few seconds pick it back up. Open the far left bottom drawer to the left of the sink, put the rinsed green towel inside and close the drawer.
LLM-generatee Summary: Put the towel in the drawer.
Skills Summarize: 1: Pick up the plastic bowl the left. 2: Turn the lamp. LLM-generated Summary: Look at the plastic bowl on the left.
Skills to Summarize: 1: Slice up the fruit that is in the sink. 2: Place the knife in the second row of the fridge. 3: Grab slice of the fruit. 4: Place the fruit slice on the back right pan. 5: Grab the pan that has the fruit in it. --ne-redd Summary: Put slice of fruit on a pan in the fridge
Skills to Summarize: Pick up the pot on the front right burner. 2: Chill the pot in the fridge then get it back out and close the door. LLM-generated Summary: Put a pot on the stove, put in the fridge, get it out, close the door.
Pick up the knife on the counter. 2: Cut the potato in the sink into slices. 3: Put the knife in the sink with the potato. 4. Pick up slice of potato from in the sink. 5. Warm the potato slice in the microwave and then remove -enereeee Summary: Warm the potato slice in the microwave.
Skills to Summarize: 1:
Figure 14: LLM Summary examples. We randomly sample 12 language combinations generated by the OPT-13B text generation model (Zhang al. 2022) and display them without cherry-picking specific examples. The LLM often creates high level description by skipping implied steps (like humans do), e.g., "Put the remote on the couch" implies the need to pick up the remote.
28

---

Under review as a conference paper at ICLR 2023
E.2 COMPARING SUMMARIES FROM DIFFERENT LLMS
In this section, justify the use of OPT-13b our summarization model by comparing its sum- maries of three different skill sequences to OPT-350m (350 million parameters) and OPT-1.3b (1.3 billion parameters). The summaries and sequences displayed in Figure [5 Although these 3 randomly sampled examples, they highlight differences between the models that hold true across many examples that examined.
As the language model size increases, it understands more the relationships between sub-tasks in each sequence, therefore generating better summaries. An example of this is the second example in Figure [5] which highlights sequence in which both OPT-1.3b and OPT-13b understand the high-level task of putting warmed plate in the cabinet, as opposed to OPT-350m which essentially repeats the second sub-task in the sequence. Yet sometimes, only OPT-13b produces a reasonable summary, In the last example, OPT-13b produces summary that implies that an agent must wash the potato ("Put rinsed potato on the counter..."". the other two models' summaries only mention the last step of putting the potato down.
Skills to summarize: 1. Pick the clock from the desk. 2. Turn the lamp on. OPT-350m Summary: Look at the clock.
OPT-1.3b Summary: Turn on the lamp. OPT-13b Summary: Look the clock.
Skills summarize: 1. Take the plate from the cabinet to the left of the stove. 2. Warm the plate in the microwave, above the stove. 3. Put the plate in the cabinet.
OPT-350m Summary: Place the plate in the microwave. OPT-1.3b Summary: Put a warmed plate in the cabinet. OPT-13b Summary: Put warmed plate in the cabinet.
Open the microwave, take out the potato and shut the door. 2. Put the potato in the sink, turn the water on and off again. Take the potato out of the sink. 3. Place the potato on the counter, to the right of the sink. OPT-350m Summary:
Skills to summarize: 1.
the potato on the counter, to the left the sink. OPT-1.3b Summary: Place the potato on the counter, to the right of the sink. OPT-13b Summary: Put rinsed potato on the counter to the right of the sink.
Comparison of OPT-350m, 1.3b, and 13b summaries 3 randomly sampled sequences from our dataset. In general, as the model size increases, the summary becomes better. However, there are some sequences all three on which all 3 models do not generate good summaries for, such as the first sequence in these examples.
Figure 15:
E.3 STEP BY STEP ZERO-SHOT EVALUATIONS
Humans may sometimes prefer to give more detailed instructions, e.g., if they do not trust house- hold robot successfully execute a high-level instruction. In that case, they are likely give detailed, step-by-step instructions instead. In this section, we demonstrate the effectiveness of our pre-training strategy in enabling execution of longer-horizon tasks when specified by such step-by- step instructions. To generate these step-by-step instructions, into one paragraph to condition agents on instead of the high-level human annotation. For example, task with 2 subtasks, "Pick up the knife" and "Slice the potato," is given the simple task annotation "Pick up the knife. Slice the potato."
combine all subtask instructions
We compare SPRINT and IL both EVALJ00 and EVAL-CHAIN tasks with the combined, step- by-step task annotations in Table 4 L-BC and SPRINT both perform similarly on the step-by- step EVAL 100 tasks compared to with the original annotations, while both perform better
29

---

Under review as a conference paper at ICLR 2023
EVAL.CHAIN tasks thatn with the original annotations. This is likely due to how EVAL.CHAIN tasks are on average much longer (7 or 8 tasks). Therefore, instructions spelling out exactly what to do allow agents to more easily understand what and in which order subtasks should be performed.
Table 4: Step-by-step evaluation number of completed sub-tasks (number of completed sub-tasks).
L-BC
SPRINT
EVAL_100 Completed Subtasks
0,46 + 0.01
1.24 = 0.19
EVAL_CHAIN Completed Subtasks
0.22 + 0.03
3.11 + 0.08
E.4 EXTENDED ABLATION STUDY
In this section, we examine series of additional questions regarding specific design choices of our method, manner complementary to Section 5.1.3 We also visualize all ablations' zero-shot policy evaluation performance in on both EVAL__00 and EVAL_CHAIN task sets in Table
How much does the LLM contribute skill aggregation? To answer this question, we com- pare using SPRINT with LLM aggregation on adjacent sub-trajectory sequences but chaining (SPRINT w/o chain) to SPRINT with skill aggregation, but where the skills are relabeled by naively concatenating the sentences together (SPRINT w/o chain, w/ concat-agg). Across both task sets, SPRINT w/o chain, w/ concat-agg is outperformed by SPRINT w/o chain, especially in EVAL CHAIN. This signifies that using the LLM helps with understanding very-long horizon, high- level semantic instructions as the LLM generates relevant task summaries for consecutive skills in a trajectory.
What if we relabel with the LLM during chaining? We examine also using the LLM to label skills during cross-trajectory skill aggregation (SPRINT w/ LLM-chain), rather than concatenating the skill annotations together (SPRINT). Overall, SPRINT w/ LLM--hhin performs slightly worse in average return and success rates. When analyzing the summaries generated by the LLM, we found that randomly paired instructions can rarely be summarized meaningfully, thereby resulting in noisy and sometimes meaningless instructions. Therefore, we implemented SPRINT w/ L.LM- chain by only utilizing the top 1% (top 10 candidates with our batch size of 1024) of in-batch chaining candidates (ranked by the LLM's prediction of what the next skill should be). Even so, the performance is not better than just simply concatenating the instructions, even these top 1% candidates still have a high chance of not being sensible sentences to summarize.
Does LLM skill aggregation help L-BC? We perform LLM skill aggregation with the L-BC base- line in order to examine whether LLM skill aggregation alone can help with pure imitation learning methods (L-BC w/ LLM-agg). This change along doubles the average number of subtasks com- pleted against regular L-BC in EVAL 100 and causes it to jump from 0.04 subtasks completed in EVAL_CHAIN to 1.86 (Table for original L-BC results). These results demonstrate the significant impact LLM aggregation alone can make.
Table Zero-shot Ablation Subtasks Completion Table
EVAL 100 Subtasks Completed
EVAL CHAIN Subtasks Completed
SPRINT (ours)
1.27 0.13
2.59 + 0.66
SPRINT w/ LLM-chain
1.15 + 0.01
2.46 + 0.21
SPRINT w/o chain
0.91 + 0.03
2.04 + 0.04
SPRINT w/o chain, w/ concat-agg
0.77 1 0.06
0.67 1 0.20
L-BC w/ LLM-agg
0,93 1 0.04
1.86 1 0.12
SPRINT w/o LLM-agg
0.38 0.05
0.10 1 0.04
E.5 QUALITATIVE COMPARISON RESULTS
Zero-shot evaluation. We compare SPRINT, AM, and L-BC zero-shot evaluation results on long EVAL CHAIN tasks in Figure 16 In general, SPRINT is able to make substantially more progress
30

---

Under review as a conference paper at ICLR 2023
EVAL.CHAIN tasks leverages the large language model to generate longer-horizon, seman- tically meaningful pre-training tasks and performs cross-trajectory chaining to learn to chain its existing dataset tasks. In the visualized examples, SPRINT is able to understand and successfully execute many of the sub-tasks implied but not directly stated by the natural language task instruc- tion. L.-BC makes very little progress these tasks, not even understanding what the first sub-task to complete should be as the task annotation is out of distribution from what it saw while training. Finally, AM is able make some progress on some of these tasks due its long-horizon goal pre- training objective. However, this is less effective than our language-condinggg conditioned pre- vre-training in such zero-shot evaluations.
Finetuning. We finetune SPRINT, AM, and L-BC on EVAL.UNSEEN tasks, in houschold floor- plans that were never seen while training, and visualize qualitative policy rollout examples after finetuning in Figure 17 In general, SPRINT is able finetune longer-horizon tasks while AM and L-BC both struggle with making progress on longer-horizon tasks despite receiving rewards for every completed sub-task. SPRINT's ability complete more sub-tasks
many of the longer- horizon tasks is demonstrated in Figure 17a. while a case in which both SPRINT and AM make partial progress throughout finetuning is demonstrated in Figure 17b We believe that AM has more trouble finetuning these tasks than SPRINT because the task specification for AM (goal images) is of distribution; pre-training on semantic tasks with SPRINT allows agents to more easily learn longer-horizon behaviors as the task specifications may still be in-distribution of the pre-training tasks that LLM skill- aggreggagggtno and skill chaining produce.
31

---

Under review as a conference paper at ICLR 2023
Task: Throw away a microwaved slice of potato.
Completed Subtasks
SPRINT
8/8
AM
L-BC
SPRINT successfully solves this task, while AM fails slice the potato and repetitively iterates between putting the potato the fridge and microwave. L-BC fails even pick up the potato, the task annotation does not directly describe picking up potato.
Task: Place cooked potato slice irside the refrigerator.
Completed Subtasks
SPRINT
7/8
0/8
L-BC
SPRINT nearly this task, while AM picks up an egg instead a potato. L-BC picks up random objects not related to the annotation.
Task: Put chilled tomato slice into the microwave.
Completed Subtasies
SPRINT
8/8
1/8
L-BC
(c) SPRINT completes entire AM picks up the tomato but fails put down onto the counter and slice it L-BC aimlessly wanders and picks up random objects.
Figure 16: Visualizations of zero-shot policy rollouts three tasks in the EVAL.CHAIN task set.
32

---

Under review as a conference paper at ICLR 2023
Task:: Place piloos sofa.
Completed Subtasks
SPRINT
SESO
AM
L-BC
1/4
(a) SPRINT picks up and places one of the pillows on the sofa, and picks up the second but does not manage place the second on the sofa, thus completing 3/4 subtasks. AM and L-BC both learn to pick up pillow but never learned to place in the correct spot.
Task: pencil desk, put
side
Completed Subtasks
SPRINT
1/2
AM
L-BC
(b) SPRINT and AM both learn to pick up pencil from the desk, although neither manage put pencil down the correct place the other side of the desk." Meanwhile, L-BC never picks up the pencil.
Figure 17:
Visualizations of policy rollouts tasks in the tuning each method. These floor plans were originally unseen to all agents until finetuning.
task set, after fine-
33


Reviews:
[{"review_id": "spuSfP1thkD", "reviewer": "Reviewer_QEPp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a scalable offline policy pre-training approach based on natural language instructions.\n\nIt enables automatic augmentation of pre-training data with large language models to relabel and chain across trajectories.\n\nIt empirically shows that the proposed approach outperforms prior pre-training approaches.", "summary_of_the_review": "This paper proposes a scalable offline policy pre-training approach that outperforms conventional ones.\nIt combines sub-trajectories to augment data without extra human annotation.\n\nHowever, it still has significant concerns, as mentioned earlier.\nMore explanations might help.", "strength_and_weaknesses": "***Strength***\n\n- It can populate pre-training data without additional human labeling. It only requires labeling sub-trajectories (maybe with crowd-sourcing), and the approach will combine the trajectories for new data.\n\n- The evaluation results show its advantage over prior methods.\n\n***Weakness***\n\n**The weakness is mainly in cross-trajectory chaining.**\n\n**1. The primary concern is the change of MDP**\n\nThe cross-trajectories chaining changes the original Markov Decision Process in the transition probability distribution P. More precisely, the value of chaining transition s_TA x a_TA x s_0B is changed.\nFor example, if sub-trajectory A ends in a conference room, and sub-trajectory B starts in a kitchen, then the last action of A warps the agent in the new MDP.\nIt encourages the agent to try warp when it wants to go to another room (after finishing a sub-trajectory).\nIt seems the reward of Q(s, a, z_B) is not designed to correct the problem of MDP change.\n\nIt might need explanations if the change of MDP is supposed not to have significant influences in general.\n\n**2. The online change of Q**\n\nAs mentioned in the paper, the reward Q(s, a, z_B) changes during training and is updated online.\nIt requires the RL algorithm to accept online changes in rewards.\nSo, the paper may need more arguments on what RL algorithms are compatible (it says \"any offline RL algorithm\" in Section 4.1).\n\n**3. The novelty**\n\n- The instruction aggregation and using LLM for it might be novel. However, existing LLMs are used.\n\n- The use of consecutive sub-trajectories seems straightforward.\n\n- Cross-trajectory chaining has novelty, though many of its techniques are inspired by goal-conditioned RL approaches (e.g., offline goal chaining [Chebotar et al., 2021]), as mentioned in the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "THrT3e72pRH", "reviewer": "Reviewer_pfEp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes two data augmentation schemes for instruction following training via offline RL:\n\n(1) **same-trajectory instruction aggregation**: Given a trajectory with multiple sub-trajectories, aggregate adjacent sub-trajectories into a longer trajectory and relabel its natural language annotation with a summary of the individual instructions. This summary is generated via a large language model (LLM) prompted with few-shot examples.\n\n(2) **cross-trajectory chaining**: concatenate two random trajectories, and their instructions via “and”. \n\nThe proposed approach, SPRINT (Scalable Pre-training via Relabeling Language INsTructions), is evaluated on ALFRED benchmark, where authors create a set of 100 unseen\nlong-horizon evaluation instructions (EVAL 100), and a set of 20 evaluation commands that test the agent’s chaining capabilities (EVAL CHAIN). On these sets, SPRINT is shown to outperform imitation learning (L-BC) and offline RL (AM) baselines in zero-shot and finetuning setups.", "summary_of_the_review": "The approach is simple, interesting, and potentially useful for more instruction following tasks, but more evaluations and explanations of setups would help better justify it.\n\n==\n\nAfter rebuttal, authors addressed many clarification questions and I raise my score from 5 to 6 in light of these.", "strength_and_weaknesses": "**Strength**: The proposed approach is simple and effective on authors’ test sets of ALFRED, with potential to apply to more instruction following tasks with long-horizon challenges.\n\n**Weakness**: My main concern is that experiments are only done in one benchmark with test sets specifically created by authors that would intuitively benefit from such compositional data augmentations. Is it also possible to also provide results on standard ALFRED evaluation/test sets, which should be more than 100 selected task instances? \n\n\nSome other suggestions/questions:\n- Maybe mention the domain (ALFRED) and learning method (offline RL, instead of more vague “policy pre-training”) in abstract? \n- Can benchmark setups be better explained? The first paragraph of Experiment sections talks about unseen task/language/environment, but then EVAL100 is said to be taken from the training set (why not eval set?), and EVALCHAIN unclear. A better explanation of task splits in original ALFRED and author created test sets is needed.\n- Learning setups: Is comparison to baselines fair - do they have similar architecture (size), training steps, number of training language tokens? Are two data augmentation subschemes mixed 1:1 or other proportions? I tried to check some appendix but they are not clear, and maybe including them in main paper would be nice.\n- The cross-trajectory scheme seems only applicable to offline RL, while the same-trajectory one also applies to imitation learning. Would be interesting how the latter alone might benefit IL, if authors want to claim usefulness beyond offline RL but for general policy pre-training.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Fhis7esceJ", "reviewer": "Reviewer_QcsA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a data-augmentation strategy to learn from demonstrations in an embodied agent setting. Two types of augmentation strategies are proposed - relabeling language instructions with a language model and changing together different tasks/trajectories to obtain a new task/trajectory. The proposed method claims to be able to learn a diverse set of skills with these augmentation strategies. The method outperforms two baselines on an evaluation set constructed by the authors based on the ALFRED dataset. ", "summary_of_the_review": "Raising score to 5 post-rebuttal.\n\nThe paper suffers from lack of strong motivations for the problem setting considered and several aspects of the experimental results are lacking. ", "strength_and_weaknesses": "Pros\n* Paper is generally easy to follow\n* Interesting use of language models for instruction re-labeling and summarization\n\nCons\n* Limited technical novelty\n* The motivations for the paper are not clear \n* Details about experimental setup are vague/missing and the results do not look particularly convincing\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "98Btwa-o1U", "reviewer": "Reviewer_bhG4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The proposed method is a offline language-based goal-conditioned reinforcement learning. It needs human annotator for labeling basic tasks with languages. It implements large languages models for relabeling and cross-trajectory chaining, in order to increase the diversity of the task set. Experiments are conducted to show the proposed method exceeds previous approaches in terms of task completion rate, more efficient finetuning and 0-shot generalization.", "summary_of_the_review": "It is a well-written paper with high completeness. The method leverages language models for efficient pretraining. ", "strength_and_weaknesses": "> Strength\n\n1. Generally well writen, structure of the article is well organized. Illustrative figures are intuitive and do a good job to help reader understand the main ideas. Important details are included in the appendix. Every claims of the authors are well-supported by their experiments\n2. Experiments are well conducted on ALFRED household task benchmark. The results support the claimed points of the proposed. Comparisons are provided to show the superior performance of proposed methods. Ablation study is conducted to show all parts of the proposed method are useful.\n3. This work has good motivation. It pretrains generalization agent with interpretable skills. The proposed method leverages easy-to-collect natural language instructions to generate unseen tasks by taking advantage of language semantics. The learned agents have skills that are semantically meaningful. \n\n> Weaknesses\n\n1. The idea of generating unseen language-based tasks is not novel. Previous work has already used the idea of generating new language-based tasks. The paper IMAGINE used Construction Grammar Heuristic for this job while this paper uses large language model. Therefore, the technical innovation is limited. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject