Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition
Authors: Huy Ha
Pete Florence
Shuran Song
Venue: CoRL 2023 Poster (2023)
Keywords: skill learning
language
diffusion

Abstract: We present a framework for robot skill acquisition, which 1) efficiently scale up data generation of language-labelled robot data and 2) effectively distills this data down into a robust multi-task language-conditioned visuo-motor policy. For (1), we use a large language model (LLM) to guide high-level planning, and sampling-based robot planners (e.g. motion or grasp samplers) for generating diverse and rich manipulation trajectories. To robustify this data-collection process, the LLM also infers a code-snippet for the success condition of each task, simultaneously enabling the data-collection process to detect failure and retry as well as the automatic labeling of trajectories with success/failure. For (2), we extend the diffusion policy single-task behavior-cloning approach to multi-task settings with language conditioning. Finally, we propose a new multi-task benchmark with 18 tasks across five domains to test long-horizon behavior, common-sense reasoning, tool-use, and intuitive physics. We find that our distilled policy successfully learned the robust retrying behavior in its data collection procedure, while improving absolute success rates by 33.2% on average across five domains. Code, data, and additional qualitative results are available on https://www.cs.columbia.edu/~huy/scalingup/.

Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition
Huy Ha', Pete Florence", and Shuran Song'
Coolmbia University "Google DeepMind
We present framework for robot skill acquisition, which 1) efficiently scale up data generation of language- labelle robot data and 2) effectively distills this data down into a robust multi-task language-conditvng conditiond visuo-motor policy. For (1), we use large language model (LLM) to guide high-level planning, and sampling- -based robot planners (e.g. motion or grasp samplers) for generating diverse and rich manipulation trajectories. process, the LLM also infers code-snippet for the success condition of each task, simultaneously enabling the data-collection process detect failure and retry as well as the automatic labeling of trajectories with success/failure For (2), extend the diffusion policy single-task behavior-clome setrings with language conditioning. Finally, we propose a new multi-tast benchmark with 18 tasks across five domains to test long-horizon hehavior, common-sense reasoning, tool-use, and intuitive physics. We find that our distilled policy successfully learned the robust retrying behavior in its data collection procedure, while improving absolute success rates by 33.2% on average across five domains. All code, data, and qualitative policy results available our project website.
Abstract:
To robustify this data-collectioe
approach to multi-task
Cocaaoo
taskc task..
Scale up Language-labetled robot data generation
Distill down to Language- conditioned policy
Notion Planner
package
Sresp Sampler
Policy
return
iin
Inout Tasl: Desc
Privileged
Generation
Multi-task
Real world Visuo-notoz Palicy
Language-guided Skill Acquisition enables scalable robot learning. In the data generation stage, a LLM takes as input task descriptions (a) and uses sampling- based robotic planners and privileged simulation information (b) to perform task- directed exploration. This enables the scaling up of language and task success labeled dataset generation (c). In the second stage, the dataset is filtered for success and distilled down into closed-loop linguageconddtioned visuomotor policy for real world deployment (d).
Figure I:
Introduction
How scalably acquire robust, reusable, real-world manipulation skills? This question has been the driv- ing force behind extensive research robot learning. Attempts in the field have focused on two primary aspects: First, how to scale up the data collection for a diverse range of manipulation skills, which involves efforts such as improving the hardware [1, 2] and software [3, 4] which support demonstration collection, utilization of non-robotics datasets [5, 6), or trial-and-error explorations [7]. The second aspect of this question concerns effective learning from the collected data, which delves into exploring effective action representations [8-10] and policy formulations [11, 12] that robustly model the training data and generalize to novel scenarios.
This paper proposes a new framework that provides comprehensive solution for both aspects by leveraging language guidance, while using no expert demonstrations or reward specification//egineeinng We contribute two key components with our framework:
Our data-collection policy is a large language model (LLM) which has access to suite of 6DoF exploration primitives (i.e., sampling-based robot planners and utilities). Given an input task description, this policy first simplifies the task by recursively decomposing it into subtasks, resulting in hierarchical plan (i.e., task tree). Next, this plan is grounded into sequence
Scaling Up Language-Guided Data Generation:
7th Conference Robot Learning (CoRL 2023), Atlanta, USA.

---

of 6DoF exploration primitives, which generates diverse robot trajectories for the task. Finally, the data collection policy verifies the trajectories' success with inferred success function and retries the task until it succeeds. This verify & retry step not only improves the data-collection policy's 's success, but also adds robot experience on how to recover from failure, an important trait for downstream policy distillation. This data generation approach is scalable, enabling significantly more efficient autonomous task-directed exploration than unguided altematties (i.e., reinforcement learning) while not being limited by the lack of low-level understanding of the LLM-only solution.
We distill these robot experiences into a visue-linguo-motor policy that infers control sequences from visual observations and natural language e
Distilling Down to Language-Conditiones Conditioned Visuomotor Policy:
task description. To enable effective learning of high entropy, diverse robot trajectories, we extend the diffusion policy [12] to handle language- based conditioning for multi-task learning. This allows the leamee policy to be reused and recomposed through language-basee successfully learned the robust retrying behavior from its data collection policy, while improving upon its absolute success rate across five domains by 33. 33.2%. Further, we demonstrate that our policy directly transfers to the real-world without
planners. We found that our distilled policy
using domain randomization.
Our framework combines these two components get the best of both worlds leverage LLM's common-seeee sense reasoning abilities for efficient exploration while learning robust and re-usable 6DoF skills for real-world deployment. In summary, the key contribution of this paper is a new framework for visuo-linguo-motor policy learning that is enabled by three novel components:
A new language-guiin guiidggeggddedddded data collection framework that combines aagguge--based task planner with 6DoF robot utilities (e.g. motion planning, grasp sampling).
New formulation of diffusion-based policy that effectively learns multi-task language-conditioned closed-loop control policies.
In addition to our algorithmic contributions, we also contribute a new multi-task benchmark that includes 18 tasks across five domains, requiring long-horizon (s= 800 control cycles), common sense, tool-use, and intuitive physics understanding
Sannsaannnnnnnn caapabiittiss lacking in existing manipulation benchmarks.
2 Related Works
Scaling risuo-linguo-mon -ootrr data. In learning vision-andd language-conditinnn real-world deployment [9, 10, 13-18), one of the most important questions is how to scale up "robot-complete data" data that has robot sensory inputs (e.g. vision), action labels (e-g- target end-effector & gripper commands), and task labels (e.g. language description, success). The most prevalent paradigm is to use humans to annotate both actions (e.g. teleoperation) and language [9, 10, 13-18). When providing action labels, humans can either provide task-specific [9. 10, 15, 18], or task-agnostic ("play") data [13, 14, 16, 19]. A primary limitation, however, is that data scalability is human- limitedd.
motor policies for
data. To scale language models [20, 21]. or procedurally post-hoc provided simulation [19]. To scale action labels, methods study how to use autonomous sub-optimal policies from random [7] to learned [22] policies. Human egocentric videos [6. 23, 24] has also been shown to be relevant to robot learning [5, 25). but not robot-complete (lacks action labels), and requires emboddmetd transfer. Towards unsupervised exploration, prior works have also investigated evolving environments [26, 27] and embodiments [28], automatic task generation [29], leveraging language guidance [30, 31] and moded error [32], but have not been demonstrated to scale to 6 DoF robotic skill learning. While these approaches reduce human efforts, they are still limited in optimality, generality, and/or completeness of robot data labels.
Other prior works have proposed strategies to enable more-autonomouully-ccllaa annotation, prior works study using
Another option for the autonomous data collection policy is to use a model- policy, e.g. task and motion planning (TAMP) [33]. Our approach extends such methods in terms of flexibility and task generality by leveraging LLM's common-sense knowledge. However, in contrast to recent works which use LLMs as the final policy [34-00], we use the LL.M-based planner as a suboptimal data-colledn coll-collctton policy. We then distill only successful trajectories into an observable-infommtionn [41-43] policy, allowing the distilled policy to improve upon its LLM data collection policy's performance.
Policy Representations and Multi-task Policy Distillation. One primary question in visuo-motor learning [44] has been how to represent the policy for effective learning, i.c. to enable high precision, multi-modal robot behavior [2, 11, 12, 45, 46). Another related question has been how best train multi-task policies [47, 48), including those conditioned on language [9, 10, 13, 15, 16, 18]. Our work presents the novel formulation of bringing diffusion- based [49, 50] policies [12] into the language-coniee contuuigioodiiionnndeceeeeeeelee [51, 52] visuomotor domain. Additionally, prior works in multi-task Language-conditione
typically focus on cloning policies from experts, meanwhile we study distilling data from a success-filtered suboptimal policy. Success- Success-filtering [11, 53] can be viewed as the simplest forn of offline RL [54].

---

catapult the yellou block into the turthest bin
put the horse toy into the botton drawer
balance the bus the block
put the vicanin battle into the drawer
put toy Left
package
Figure 2: Benchmark. We validate our approach new multi-task benchmark addressing challenging long-horizon tasks (Le., control cycles) requiring language understanding (e-g.. put [object] to [top] drawer), common sense knowledge (e.g. send package for return requires raising the mailbox flag). tool-use (e.g., catapult). and intuitive physics (egg,, balance the bus). The tasks are best viewed on our our project website.
3 Approach
We propose a new framework for robot learning that performs automatic data collection and policy learning from only a task description. Our design grounded on four key observations:
We recognize the importance of random exploration in reinforcement learning, but aim to not be constrained by inefficiency for long-horizon, sparse reward tasks.
We acknowledge the usefulness of LLM's common and zero-shot capabilities, but believe language is not by üseef the ideal representation for robust, rich, and precise robotic manipulation.
We are inspired by the effectiveness of robotic planning methods, c.g. TAMP, but wish to be flexible novel tasks and domains and non-reliant on ground truth state during policy inference.
We aim achieve the simplicity and effectiveness of behavior cloning in distilling collected robot experience into a policy for real-world deployment, while side-stepping the requirement for costly human demonstrations or play data collection.
Using no human demonstration or manually specified reward, our framework combines the strengths of these four areas into unified framework for both efficient task-directed exploration and multi-task lesuo-lingu-motee policy learning.
Method Overview, In the data generation phase, we use an LLM to recursively decompose ($3.1) tasks into hierachical plan (i.e., task tree) for exploration and ground the plan into sampling saasplin--based robot utilities and motion primitives ($3.2). Next, the LLM infers success success-detection functions for each task the plan ($3.3), providing success-labeling. This autonomous data generation process outputs a replay buffer of bask-directed exploration experience, labeled with language descriptions and success labels. In the training phase ($3.4), we filter this data for success according to the LLM inferred success condition and distill into multi task vision- and and-aandddannn langllln language-conditioned conditioondd diffusion policy [12].
3.1 Simplity: Task Planning and Decomposition
Given task description, the first step is to generate a high-level task plan. To improve the flexibility to work with any tasks and 3D assets, opted for LLM--baee planner to leverage their common-sense and zero-shot reasoning skills. Unlike classical TAMP planners, our framework does not require domain-specifie engineering and transition function design to work with new tasks.
Concretely, our recursive LLM planner takes as input the task description, the simulation state, and outputs plan in the form of a task tree (Fig. 3a). To do so, the LLM first checks whether the task description involves the robot interacting with multiple or only one object. For instance, "move the package into the mailbox" involves opening the mailbox before picking up the package and putting the mailbox in, and should be considered a multi-object task. Meanwhile, "with the mailbox opened, move the package into the mailbox" should be single-object task. For the base case of single-object tasks, we prompt the LLM to which object part name to interact. For the case of multi-object tasks, we prompt the LLM to decompose the task into subtasks, and recurse down each subtask.
3

---

a) simplify (53.1.1)
b) ground ($3.1.2)
Anout task dese "sond package
LLM
for retuzn"
input
kinenatic struct.
put the package inside the nai
2.1. pick up the package
graze w
canolaa be
place serpter
nocion planner
Link nanes contact info
place package into nailbox
2. raise the maii flag
...
grasp the mailbox flag
3.2. tuzn the naftbox flas
DoF Exploration Prinicives
output replay buffez
c) verify retry (53.1.3)
sis state contact info
exploratior plan rollout
s:lppe: dose & 5:1p command
gennetzy
2.2
Takk 20 canT
LLM
out of time?
deso
yes
Llld
successful?
FuNsess concitine
labelled with sacoms/fall
no
Langunge
retcy
Figure 3: Language-Drrie Driven Robot Data Generation takes as input the task description and simulation state, and outputs a replay buffer, labelled with language descriptions and success. It starts by using an L.M to simplify tasks recursively (a) until the task involves only one object, resulting in a hicrarchical exploration plan. Next, the plan is grounded (b) into sequence of 6 DOF exploration primitives (e.g. grasp samplers, motion planners, etc.) and rolled out in simulation to give an unlabelled robot trajectory. Finally, an LLM infers success function code sode-snippet, and uses it to verify (c) and label it with succeeded or failed. If the trajectory failed, the LLM retries the exploration plan with a different random seed (e.g. different grasp pose from the grasp sampler). If the rohot succeeds or run out of time, the labeled trajectory is returned.
3.2 Ground: Compiling a Plan into Robot Utilities
With the generated task tree $3.1, the next step is to ground the high-level plan into physical actions. Here, the choice of the low-level robot API critically defines the system's capability and, therefore, becomes a key differentiating factor between different systems. In principle, there are three desired properties we want see in the action space design:
Flexibiliiyy. Planar actions [10, 37] aren't flexible enough to manipulate prismatic and revolute joints.
Scalable. Namely, actions should not require human demonstrations to acquire [9, 10, 13-16, 35).
Language-friendll, While joint sequences can encode any action, it is not language-friendly,
We propose to ground the LLM's plan with API calls into set of robot utility functions, which include sampling-based motion planner, a geometry- grasp and placement sampler, and motion primitives for ar- ticulated manipulation. We refer to these utilities as DOF Exploration Primitives (Fig 3b) because, by virtue of being neeudorandom, the sampling- based utilities generate diverse robot trajectories, enabling effective explo- ration for rich 6 DoF' manipulation settings. For instance, our grasp and placement samplers samples uniformly amongst all points in the object part's point cloud to find good grasps and placements poses, respectively, which are used as input into rapidly-exploring random trees [55] motion planner that samples uniformly in joint space. This results in diverse grasps, placements, and motion trajectories connecting grasps and placements.
For each leaf node in the inferred task tree (S 3.1). the grounding process takes as input the node's 's task de- scription (e.g. "open the mailbox"),, its associated object part name (e.g. "mailbox lid"", and the simulation state, and outputs a sequence of 6 DoF Exploration Primitive API calls. Using the object part name, we can parse the object's 's kinematic structure from the simulation state and handle articulated and non-articulated (i.e., rigid, deformable) objects separately. For non-articulated objects, the LLM is prompted to choose the pick & place object names, used to sample grasp and placement pose candidates. For articulated objects (with either revolute or prismatic joints), the leaf node's s associated object part name is used to sample a grasp candidate followed by a rotation or translation primitive conditioned on its joint parameters (i.e., joint type, axis, and origin).
Exploration Plan Rollout. Each node in the exploration plan is grounded only when it is being executed, where the order of execution follows a pre-order tree traversal. By keeping track of the subtask's 's state, sub-segments of robot trajectory can be labelled with the subtask's 's description, thereby providing dense and automatic text labels for the trajectory. For instance, all actions taken during the inferred subtask "open the mailbox" can be labeled with both the subtask's 's description "open the mailbox" and the root task description "move the package into the mailbox""
Since grounding happens only when task node is visited, each node's grounding process is independent of the other leaf nodes, depending only on the simulation state when is evaluated. While this simplifies planning significantly. it also means that failed execution can occur. For instance, a grasp candidate may render all placement candidates infeasible.

---

3.3 Verify & Retry: Robustifying the Data Collection Policy
Recall, the planning and grounding step can fail, especially when we consider long-horizon tasks. To address this, we propose a verify & retry (Fig. 3c) scheme, which uses environment feedback to detect failed execution.
Verify. For each task, the LLM infers a success function code snippet given the task description, simulation state, and API functions to for query simulation state (e.g. checking contact or joint values. etc). This amounts to prompting the LLM to complete a task success function definition that outputs a boolean value, indicating task success. For instance, given the task "raise the mailbox flag", the LLM's inferred code snippet should check whether the mailbox's 's flag hinge is raised (Fig. 3c, highlighted green).
Retry. When a trajectory is labeled failed, the robot retries the same sequence of robot utilities with a different random seed (i.e., for the sampling-base until the task succeeds. For instance, in the bus balance task (Fig. top left), the robot would repeatedly try different grasp and place candidates until the bus is balanced. In the tree traversal process 3.2, nodes only yield execution to its parent task when the node's inferred success condition returns true. This design not only leads to higher success rates in data generation but also provides useful demonstrations how to recover from failure. In the output replay buffer, the only failed trajectories are ones which timed-out or led to invalid states (e.g. object dropped on the floor).
robotic utilities) without resetting the simulation state
3.4 Longuage-conditooned Policy Distillation
task desc "send the package foz return"
CLIP
We extend diffusion policy [12], a state-of-the-art ap- proach for single-task behavior cloning, to the multi- task domain by adding policy takes as input a task description CLIP [56] feature, proprioception history, and visual observa- tions, and outputs sequence of end effector control commands. Following Robomimic [4]'s findings, we use a wrist-mounted view in addition to global (workspace) view to help with tasks requiring precise manipulation. We use their ResNet 1 b-baee [57] vision encoders, one for each view. We found that using only the latest visual observation along with the full observation horizon of proprioception maintains the policy's high performance while reducing training time. When used in conjunction with the DDIM [58] noise scheduler, we found that we could use a 10x shorter diffusion process at inference (5 timesteps at inference, 50 timesteps at training) while retaining comparable performance. Quantitatively, when using
vision
Resnet18
This
Resnet18
Stoba
proprioception
Diffusion Policy
gripper pose & command
a) Inputs
b) Networks
c) Action Sequence
Figure 4: Language-Conditione Conditioned Policy Distillation. The policy takes as input task description, two RGB cam- era views, and gripper proprioception data, and outputs a sequence of gripper poses and closing command.
10 dimensional action space", our policy can be run at se 35H22 on an NVIDIA RTX3080.
Complex Artic- Common geometry ulation sense
Multi- Long use task horizon
4 Evaluation
Domain
Our experiments try to validate two questions: 1) Can our data generation approach efficiently perform task-directed explo- ration? 2) Can our policy learning approach effectively distill a modti-modal, multi-task dataset into a generalizable and robust visuo-linguo-mokee policy?
Balance
X
x
x
Catapult
Transport
x
Mailbox
x
Drawer
Table
Benchmark Suite.
Our Benchmark contains 18 tasks across 5 domains (Fig. 2 Tab. 1), with the following properties:
6DoF& & articulated manipulation, for deadling with complex object geometry and articulation.
Geometry Generalization. In our bin transport domain, the robot must generalize its bin transport skill to unseen object instances, with novel shapes, sizes, and colors.
Intuitive physics. Robots should understand the physical properties of the world and use this knowledge to perform tasks. In the bus balance domain, the robot needs to learn the precise grasping and placement to balance large bus toy on a small block. In the catapult domain, where the block is placed along a catapult arm determines how far the block will be launched, and, thus, which bin (if any) the block will land in.
Common-senss reasoning & Tool-use. Natural language task description user-friendly but often under-specifies the task. Common-senss can help to fill in the gaps. In the mailbox domain, given the task "send the package for return", the robot should understand that it not only needs put the package inside, but also raise the mailbox flag to indicate that the package is ready for pickup. In the catapult domain, the robot needs to understand that pressing the catapult's button will activate the catapult, and that the block needs to be placed on the catapult arm to be launched.
*3 for position, 6 for rotation using the upper rows of the rotation matrix, and gripper close command
5

---

a)
b C)
d)
e)
f
Figure 5: High Entropy yet Precise Language-Guide Guided Action Sequences. Running the pseudorandom language- conditioned diffusion process with different seeds on the same observations yields language-consisten consistent (a-c, different colors for different task descriptions), high entropy actions when possible (a-f, object grasping, transports, & placements) and precise actions when necessary (d, narrow mailbox with large package). Further, domain randomization enables simulation trained policy (e) to generalize to the real world (f).
Multi-task conditioning, Given the same visual observations but different task description, the robot should perform different and task-relevant actions. The catapult domain has 3 tasks for three target bins, and the drawer domain has 12 tasks.
Long horizon behaviour. Our longest horizon domain, mailbox, takes at least 4 subtasks to complete (open the mailbox, put the package in the mailbox while opened, close the mailbox, then raise the mailbox flag) which can require up to 800 control cycles. In the drawer domain, the robot needs to open the drawer, move the object into the drawer, then close it, which takes about 300 control cycles.
The benchmark is built top of the MuJoCo [3] simulator, using assets from the Google Scanned dataset [59, 60). We use a table-top manipulation set-up with a 6DoF robot arm. The task success evaluation is a manually designed function, instead of LLM generated function used for data collection.
Metrics. We report the success rates (%) averaged over 200 episodes in Table 2, a task completion efficiency plot in Fig. and qualitative results Fig. 5. If domain has multiple tasks then we report the average performance of all tasks. We also compare different LLMs in Table 4 (10 samples per task) and investigate the sources of error in system for the mailbox domain in Table 3 (200 trials per execution).
Data Generation Baselines. Code-as-Policy [37] is ofateoofthheaar approach for using LLM directly as robot policy by making state (e.g. query present objects) and action primitive API calls to a robot. Given an M-infrrreded code string, they execute the snippet in an open-loop fashion. Crucially, in their table top manipulation setting, they assume access planar action primitives. Thus, we introduce the following baselines, which build on top of Code-as-Polits Policy and each other as follows:
LLM-as-Police (2D): Similar code-as-policy using planar pick-and-place, but use ground truth object segmentation instead of their off-the- object detectors [61, 62).
(+) DOF robot utils: Builds on top of the previous baseline by adding access to 6 DOF robot utilities for grasping, placement, motion planning, and articulated manipulation.
(+) Verify & Retry: Adding to the previous baselines, this baseline uses the LLM's predicted success condition to label trajectories and retry failed ones. Since the robot utilities involve pseudo-random samplers (e.g. RRT, grasp sampling), retrying the task means running these samplers again using the pseudo-random state and environment state from where failed trajectory left Since we use this approach as our data generation policy, it also serves as ablation of our approach.
Policy Distillation Ablations. We compare against BC-Z [15]'s single-task policies which does not use FiLM conditioning (used in their bin emptying and door opening tasks). To understand the effects of our policy learning design decisions in the single-task regime, we fix training time and dataset size (2 days using at least 500 successful trajectories), and provide the following ablations:
Action Generation: Instead of using diffusion processes conditioned on the policy input embedding to decode actions, it is typical use multi-layer perceptrons. Following Jang et al. [15], we use one MLP with two hidden layers and ReLU activations for end effector position, one for the orientation, and another for

---

gnpper command. This standard policy architecture is deterministic, and is trained with mean-squared error loss for pose and binary cross entropy loss for gripper command.
Action Space: Besides our absolute end effector pose action space, Delta-Action and velocity control
spaces is another popular action space choice [4. 15, 63-65]. We also ablate BC-Z's execution action horizon (Exec) while keeping their original prediction horizon (Pred).
All approaches encode images using R [57] architecture. Although the original architecture was designed with an average pooling layer, its typical for robotic policies to use spatial softmax pooling [44] layer instead.
Observation Encoder:
Data usage: No-Retry trains on successful trajectories generated from the data generation approach without Verify & Retry, so it does not observe any recovery behavior.
4.1 Data Collection Policy Evaluation
6DoF exploration critical. First, we verify different approach's ability to perform and ex- plore in 6DoF, which is crucial for general manip- ulation. When 6DoF exploration
Approach
Planar
6Dol
Average
Balance Catapoli Transport Mailbox Drawer
I.I Policy (2D)
28.0
31,3
21.5
0.0
0.0
27.6
introduced,
(+) 6DeF Robat Utils
5.5
2.5
35.0
(+) Verily & Retry
45,0
7.3
82.0
3.0
318
33.8
we first observe a drop in the average success rate for simple tasks that could be accomplished with planar actions (Balance, Transport, Tab. 2).
Distill No Retry
67.5
38.5
32.5
0.0
22.7
32.2
Distill Ours
80.0
62.0
55.8
67.0
However, this ability is critical for exploring com- plex tasks, providing data to improve upon in the later distilling stage. In particular, we observed that 6DoF actions are important for grasping diverse objects with complex geometry (Transport, Tab. 2), and manipulating articulated objects (Drawer, Mailbox, Tab. 2).
Table 2: Success Rates (%) for data generation (top) and distillation approaches (bottom) over 200 trials.
Moreover, 6DoF exploration also helps in diversifying the data collection strategy, which provides the possibility to improve upon in the later distilling stage. For example in the catapult domain, LLM-as-Police
Subtask
Planning Verify Execution
Open mailbox
100
100
43.5
Pur package in mailbox
100
100
28.5
(2D) is only able to solve one of three pos- sible goals (the closest bin) using a deterministic strategy. However, it provides no useful data for learning the other two goals, making it poor data-collection policy. In contrast, incorporating 6 DOF robot utilities achieves lower but non-zero average success rates in all bins (16.3%, 3.3%, and 2.2%. full table in appendix), which provide much better exploration data for distillation.
Ruise mailbox flag
100
100
62.0
Close muilbox
100
100
94.2
Table 3: Sources & Propagation of
Error. Accuracy (%) of planning, veri- fication, and execution success rate (%) for each mailbox subtask.
Verify & Retry always helps. In the verify & retry step, the LLM retries all tasks until they are successful. This simple addition improves performance in all domains, with 2x, 3x, 8x, and 13x in transport, catapult, balance, and drawer domains. Without this crucial step, we observe 0.0% success rate in the mailbox domain. underscoring the difficulty of flawlessly executing long sequences of 6 DOF actions, and the importance of recovery after failure.
Model
Planning
Success
Language Model Scaling, In addition to the final task success, we provide more detailed analysis of planning and success condition inference accuracy Tab, 4. We evaluate on the proprietary GPT3 [66] (175B text-davinci-003) and the open LLAMA2 [67] (7B and 13B). We found that Llama models struggles in complex planning domains because they Table 4: LLM Evaluation. do not follow instructions provided in the prompts, For instance, in the drawer domain, both models fail to account for drawer opening and closing. However, we observe an upwards trend with respect to Llama model size, with the 13B model outperforming the 7B model by +20.0% and +38.3% in planning and success verification accuracy respectively.
LLAMA2
7B
42.0
10.0
13B
62.0
48.3
GPT3
175B
82.0
91.1
4.2 Distilled Policy Evaluation
Robustness In, Robustness Out. By filtering trajectories with L.LM's inferred success condition, distilled policies inherit the robustness of their data collection policies while improving upon success rates (-23.4% and +33.2% for no-retry and ours, Tab. 2). Since our distilled policy learned from robust data collection policy, it also recovers from failures (e.g. failed grasps or placements) and continuously retries a task until it succeeds. Mcanwhile, since the no-retry distilled policy leamed from a data collection policy which did not retry upon failure, it is sensitive and brittle, leading to 34.8% lower average success rate across all domains compared to ours (Tab. 2).
High Performance From Diverse Retry Attempts. Plotting how long policies take to solve the bal- ance task (Fig, 6), we observed that our policy and its data collection policy continuously tries diverse
7

---

set of grasps and placements after each failed attempt until it succeeds. This results in higher success rates the policy is given more time, and is reflected in their monotonically increasing success rates. In contrast, baselines plateau after their first grasp/plate- ment attempts. This highlights the synergy of two design decisions. First, the verify & retry step ($ 3.3) is crucial for demonstrating retrying behavior, but is by itself insuffi- cient if each retrying action is the identical as the previous one. Instead, opting for diffusion policy ($ 3.4) for learning from and generating high-entropy, diverse retry attempts (Fig 5) is also essential for high performance.
88
OuXS 79.0%
Disti LL 6% (No Retzy)
60
success by
LLM Po Pocicy 45.0% (6Dof+
48
Policy Learning Baselines. We investigate policy learning design decisions on the single-task balance do- main, and remove language conditioning. While BC-Z found spatial softmax hurt their performance and opted for a mean pool, we observed using spatial softmax improved performance by +5.0%. Further, we found that switching from delta absolute action spaces improved success rates +6.5% and +9.5% when using the MLP action decoder and our diffusion action decoder, respectively,
LLM-Policy 28.0% (28)
28
LLM Pol. Policy (6DoF) 5.5%
20
48
68
89
106
time (seconds)
Figure 6: Distilled Robustness, Our policy inherits robust recovery from failure behavior from its data collection policy, while improving upon success rate. confirming Chi et al. [12]'s findings. Lastly, we find that using our pseudo-random diffusion- bbased action encoder consistently outperforms a deterministic MLP action mappings, regardless of other design decisions.
We evaluated a policy trained on do- main randomized synthetic data in real world transport task with five novel objects (Fig. Se). Averaging across ten episodes per object, our policy achieved 76% success rate, demonstrat- ing the effectiveness of our approach in Sim2Real transfer.
Rim2Real Transfer.
Output
Input
Success
Method
Genention
Rep.
Exec Pred Pool
Proprio
(%)
BC-Z FoodForward Delta
1
10 Avg
X
0.0
FeedFoorward
Avg
X
15.0
FeedFoorwadd Delta
8 10 Avg
X
18.5
Ours FeodForward
Delta
8
16
Spotial
V
29.0
FeedFForward
Sputial
4.3 Limitations
Dillinsio
Delia
8
16 Sputial
68.5
Diffision
Aug
76.5
Diflissio
Abs
8 16
Sputial
79.0
By using priviledged simulation state information, the LLM can infer success conditions which uses ground truth contact, joint information, and object poses. This means our imple- mentation of the data generation phase is limited to simulation environments, and our policy requires sim2real transfer. Fur- ther, Our data generation method relies on existing 3D assets and environments, which presents further opportunity for scaling up with assets from 3D generative models or procedural generation. Finally, while our approach's 's dataset contains text labels and success labels for all subtasks, we have only evaluated its effectiveness in learning the root task. Learning from all subtasks and growing robot's set of learned, reusable sub-skills over time enable compositional generalization is left for future work.
Table 5: Policy Learning Ablations. Ac-
tion generation using diffusion models [50] robustly outperforms feed-forward models across other policy design decisions.
5 Conclusion
We proposed "Scaling Up and Distilling Down",, framework that combines the strengths of LLMs, sampling- based planners, and policy learning into a single system that automatically generates, labels, and distills diverse robot-complete exploration experience into multi-task visuo-linguo-men motor policy. The distilled policy inherits honghorizon behaviour, rich low-level manipulation skills, and robustness from its data collection policy while improving upon performance beyond its training distribution. We believe that this integrated approach is step towards putting robotics on the same scaling trend as that of LLM development while not compromising on the rich low-level control.
Acknowledgments
We would like to thank Cheng Chi, Zeyi Liu, Samir Yitzhak Gadre, Mengda Xu, Zhenjia Xu, Mandi Zhao and Dominik Bauer for their helpful feedback and fruitful discussions. This work was supported in part by Google Research Award, NSF Award #2143601, and #2132519. We would like to thank Google for the URS robot hardware. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors.
8

---

References
[1] S. Song, A. Zeng. Lee, and T. Funkhouser. Grasping the wild: Leaming 6dof closed-loop grasping from low-cost demonstrations. IEFE Robotics and Automation Letters, 5(3):4978-4985, 2020.
[2] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprini 13Xx:23334.1303 2023.
[3] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 20/2 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033, IEEE, 2012. 10.: (010/00O...0...660000 2.6386109.
[4] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang. R. Kulkami, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martin. What matters in learning from offline human demonstrations for robot manipulation.
In arXiv preprint arXiv:2108 03298, 2021.
[5] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203. 2022.
[6] K. Grauman, A. Westbury, E. Byrne Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu,
X. Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995-19012, 2022.
[7] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning arXiv preprint arXiv:2004.07219, 2020.
[8] J. Wu, X. Sun, A. Zeng, S. Song, J. Lee, S. Rusinkiewicz, and T. Funkhouser. Spatial action maps for mobile manipulation. arXiv preprint arXiv:2004.09141, 2020.
[9] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Proceedings of the 6th Conference on Robot Learning (CoRL), 2022.
[10] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipulation. In Proceedings of the 5th Conference on Robot Learning (CoRL), 2021.
[11] P. Florence, C. Lynch, A. Zeng, O. Ramirez, A. Wahid, L. Downs, A. Wong, Lee, I. Mordatch, and
J. Tompson. Implicit behavioral cloning. Conference on Robot Learning (CoRL), November 2021.
[12] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousincau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[13] C. Lynch and P. Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprini 2774540000004488 2020.
[14] S. Stepputtis, J. Campbell, M. Phielipp, S. Lee, C. Baral, and H. Ben Amor. Language-condditonne imitation learning for robot manipulation tasks. Advances in Neural Information Processing Systems, 33:13139-13150 15, 2020.
[15] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Be-z Zero-shot task generalization with robotic imitation learning. In A. Faust, D. Hsu, and G. Neumann, editors, Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceediiggs of Machine Learning Research, pages 991-1002. PMLR, 08-11 11 Nov 2022. URL https//procceedings http://ppooeedinngs...psssss mlr.sssssvv 164/jang22a. html.
[16] C. Lynch, A. Wahid, J. Tompson, T. Ding, J. Betker, R. Baruch, T. Armstrong, and P. Florence. Interactive language: Talking to robots in real time. arXiv preprint arXiv:2210.06400) 2022.
[17] O. Mees, Hermann, and W. Burgard. What matters language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters, 7(4): 1205-1 2022.
[18] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint axXiv:2212.068/7, 2022.
[19] O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard. Calvin: A benchmark for langunge-conditooned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 7(3): 7327-7334, 2022.

---

[20] T. Xiao, H. Chan, P. Sermanet, A. Wahid, A. Brohan, K. Hausman, S. Levine, and J. Tompson. Robotic skill acquisition via instruction augmentation with vision-language models. arXiv preprint arXiv:2211.11736 2022.
[21] J. ZhZan K. Pertsch, J. Zhang, and J. Lim. Sprint: Scalable policy pre-training via language instruction relabeling. arXiv preprint arXiv:2306. 1886, 2023.
[22] S. Nair, E. Mitchell, K. Chen, S. Savarese, C. Finn, et al. Learning language-condiiioned robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning, pages 1303-1315. PMLR, 2022.
[23] R. Goyal, S. Ebrahimi Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pages 5842-5850, 2017.
[24] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European Conference on Computer Vision (ECCV), pages 720-736, 2018.
[25] A. S. Chen, S. Nair, and C. Finn. Learning generalizable robotic reward functions fron"" in-the-wild" human videos. arXiv preprint arXiv:2103.16817, 2021.
[26] R. Wang, J. Lehman, J. Clune, and K. O. Stanley. Paired open-ended trailblazer (poet); Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint 1901.00900011755 2019.
[27] M. Jiang, M. Dennis, J. Parker-Holder, J. Foerster, E. Grefenstette, and T. Rocktäschel. Replay-guided adversarial environment design. Advances in Neural Information Processing Systems, 34:1884-1899 2021.
[28] J.BB. Mouret and J. Clune. Illmminating search spaces by mapping elites. arXiv preprint arXiv: 150.009999499 2015.
[29] K. Fang. T. Migimatsu, A. Mandlekar, I.. Fei-Fei, and J. Bohg. Active task randomization: Learning visuomotor skills for sequential manipulation by proposing feasible and novel tasks. arXiv preprint arXiv2211.006334 2022.
[30] Y. Du, O. Watkins, Z. Wang. C. Colas, T. Darrell, Abbeel, Gupta, and J. Andreas. Guiding pretraining in reinforcement learning with large language models. arXiv preprinl 2rXiv:2302.06692, 2023.
[31] S. Mirchandani, S. Karamcheti, and D. Sadigh, Ella: Exploration through learned language abstraction.
Advances Neural Information Processing Systems, 34:29529 295400, 2021.
[32] Mendonca, S. Bahl, and D. Pathak. Alan: Autonomously exploring robotic agents in the real world. arXiv preprint arXiv:2302.0666 2023.
[33] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-Pérez. Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4:265-299 293,, 2021.
[34] W. Huang, P. Abbeel, D. Pathak, and Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages PMLR, 2022.
[35] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al. Do as can, not as i say: Grounding language in robotic affordances. arXiv preprini arXiv: arXiv:2204.01699
2022.
[36] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang. P. Florence, A. Zeng. J. Tompson, I. Mordatch, Y. Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. In 6th Annual Conference on Robot Learning.
[37] J. Liang, W. Huang. F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. 7eng. Code as policies: Language model programs for embodied control. In arXiv preprint arXiv:2209.00 07753, 2022.
10

---

[38] D. Driess, F. Xia, M. S. Sajjadi. C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong.
T. Yu, et al. Palm-e: An embodied multimodal language model. JCML, 2023.
[39] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and Bohg. Tex22otion: From natural language instructions to feasible plans. arXiv preprint arXiv:2330. 12/53, 2023.
[40] 1. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 1-53--1150. IEEE, 2023.
[41] A. Agarwal, A. Kumar, J. Malik, and D. Pathak. Legged locomotion challenging terrains using egocentric vision, 2022.
[42] D. Seita, A. Ganapathi, R. Hoque, M. Hwang, E, Cen, A. K. Tanwani, A. Balakrishna, B. Thananjeyan.
J. Ichnowski, N. Jamali, K. Yamane, S. Iha, J. F. Canny, and K. Goldberg. Deep imitation learning of sequential fabric smoothing policies. CoRR, abs/1910.04854, 2019. URL http://arxiv.org/ab/// 10. 04854.
[43] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter: Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics, 7662)eabb2222,, 2022.
[44] S. Levine, C. Finn, T. Darrell, and Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1)11334-133, 2016.
[45] K. Hausman, Y. Chebotar, S. Schaal, G. Sukhatme, and J. J. Lim. Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets, Advances in neural information processing systems, 30, 2017.
[46] M. M. Shafiullah, Z. J. Cui, A. Altanzaya, and L. Pinto. Behavior transformers: Cloning k modes with one stone, 2022.
[47] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Morl-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094- 1100. PMLR, 2020.
[48] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXiv:2104.08212, 2021.
[49] J. Dock-Dicctt,in, Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
[50] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[51] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36499 2022.
[52] Rombach, A. Blattmann, D. Lorenz, Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[53] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and L. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances neural information processing systems, 14844-97-55077, 2021.
[54] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint 2tl::2005001643 2020.
[55] S. M. La et al. Rapidly-exploring random trees: new tool for path planning.
[56] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin,
J. Clark, et al. Leaming transferable visual models from natural language supervision. In International Conference on Machine Learning, pages
PMLR, 2021.
11

---

[57] K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and patlern recognition, pages 770-778, 2016.
[58] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. In International Conference Learning Representations.
[59] Downs, A. Francis, N. Koenig, Kinman, Hickman, K. Reymann, McHugh, and
V. Vanhoucke, Google scanned objects: A high-quality dataset of 3d scanned household items, 2022. URL
11918.
[60] K. Zakka. Scanned Objects MuJoCo Models, 7 2022. URL http:///gghhab.ccmm/eeiii/a///o objects..
[61] Kamath. M. Singh. Y. LeCun, G. Synnaeve, I. Misra, and N. Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEFJCVF International Conference on Computer Vision, pages 1780-1790, 2021.
[62] X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.339) 13921, 2021.
[63] T. Zhang. 7. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldherg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality telcoperation. In 2018 IEEE International Conference Robotics and Automation (ICRA), pages
IEEF, 2018.
[64] P. Florence, L. Manuelli, and R. Tedrake. Self superviee correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters, 5(2):492 499, 2019.
[65] A. Mandlekar, F. Ramos, B. Boots, S. Savarese, L. Fei-Fei, A. Garg, and D. Fox. Iris: Implicit reinforcement without interaction scale for learning control from offline robot manipulation data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4414-4420. IEEE, 2020.
[66] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, Sastry, A. Askell, et al. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901, 2020.
[67] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhar- gava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0922 09288, 2023.
12


Reviews:
[{"review_id": "lzcrdMRqTL", "reviewer": "Reviewer_kQcE", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper aims to learn language-conditioned control policies by collecting data in simulation. Using Large Language Models (LLMs) and predefined motion primitives, the proposed method can generate diverse tasks in simulation to train a language-conditioned diffusion policy to solve instruction following tasks. The learned policy is compared with a variant of Code-as-Policies on a set of simulated tasks and achieves better performance. ", "summary_of_recommendation": "The paper proposes a reasonable method for learning language-conditioned policies by generating data in simulation. My main concerns are the lack of justification of the contributions regarding the diffusion policy part and insufficient discussions about prior work. If these concerns are addressed, I will consider raising my rating.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": " ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths: \n\n- The paper studies how to use LLMs to guide data collection, which is interesting and novel. This work can potentially pave the way for exciting new directions. \n\n- An ablation study is included in Table 2 with detailed discussions, which provides useful insights about how each component contributes to the performance. \n\n- The paper is mostly well-written and easy to follow. The data generation pipeline is clearly explained.\n\n- I like the design of the testing tasks. These tasks are sufficiently complex while grounded in real-world applications. \n\nWeaknesses:\n\n- It is hard to justify the contributions regarding the diffusion policy (distillation party) since no comparisons or ablations are provided in that regard. I would suggest the authors run ablations using policy networks adapted from BC-Z (Jang et al.), Language-Play (Lynch et al.), and other prior work. \n\n- It would be better to include discussions about prior work on unsupervised task generation such as POET (Wang et al.), MAP-Elites (Fontaine and Nikolaidis), ATR(Fang and Migimatsu et al.), PLR (Jiang and Dennis et al.); and language-guided exploration such as ELLA (Mirchandani et al.) and ELLM (Du and Watkins et al.).\n\n- It is unclear how such methods can be applied to the real world. \n\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "hGxHK0Evh8", "reviewer": "Reviewer_bJic", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work introduces an iterative procedure for data augmentation where task plans are generated, executed, and success verified.  The approach is simple but builds on several trained components to ensure the accuracy of the resulting data.  I am in agreement with the philosophical claims of collecting and learning alignments to motion.\n\n\nNote: There are some rendering issues with figures that requires they be opened in adobe reader, other (Mac preview, Browser, etc) are not showing Fig 3 or lines in Fig 5. The figures seem to be very heavy so CPU jumped to 100% trying to render them.", "summary_of_recommendation": "Fundamentally, the idea is simple, appears to work well, but I am having trouble understanding the likelihood that other researchers would be able to leverage the approach or what the boundaries are of where it is most useful (e.g. Sim vs Real assumptions, MP quality, ...).  The single biggest gain seems to be related to distillation which is mentioned throughout but details are sparse, so it's difficult to understand what's unique versus retraining.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "My primary concern with is understanding the sources of and propagation of error -- which helps understand what would be necessary to extend the work:\n1. Error from the motion profiles\n2. Error from LM success detection?  How do we trust the verification? What's the process for encoding the scene? and is verification only possible because the work is done in simulation?\n3. Error from incorrect Task Plans generations?\n4. How do the accuracy of the examples stored align with human perception of success?\n5. How expensive is the full process? (e.g. lines 267...) What percent of trials are failures (line 149...) and how computationally expensive is the trial process? --> Edit: This is partially addressed by Appendix 3.1 \n6. How many samples are being evaluated in Table 2?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Fundamentally, the insight is that the data an agent can get via imitation is limited and expensive to collect but that sampling from existing components allows for testing boundaries and compositions that are still valid. This is nicely emphasized in the catapult case where the motion planner allows for atypical but successful placements.  \n\nSee questions before for better understanding the effects of the composition of components\n\nThis is tied to the limitations of this work -- where do we already have robust components? is there a way to use the proposed verification loop to learn new skills? ", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "Xo4LKU6RZsR", "reviewer": "Reviewer_t1nk", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a robust framework for using LLMs to call planners to execute long-horizon manipulation tasks to generate data for training a policy.", "summary_of_recommendation": "I enjoyed this paper. It is a weak accept mainly due to the lack of ablation studies that would be useful for the research community (LLM ablations) and some questions I have regarding the motivations.\n\nIf these are addressed during the rebuttal, I would easily recommend this paper with a strong accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please see above for questions, I interspersed them throughout the review.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, I enjoyed reading this paper and I think it should be accepted. There are a few issues that I found while reading that I think the authors could help address in the rebuttal.\n\n### Strengths\n\n**************************************Writing and Clarity**************************************: This paper is very well written and clear. The method, at a high level, is also very intuitive and simply (though pretty complicated at a lower-level with respect to the LLM) and the authors did a good job separating details from intuition. Figures are also excellent throughout the paper.\n\n**************************Experiments:************************** Overall, experiments are solid and the baselines are sensible. Results demonstrate the method is necessary under this problem formulation compared to the baselines. The environment is also very diverse in task distribution and demonstrates many axes of comparison.\n\n************************Motivation:************************ Upon reading this paper at first, I was thinking “why not just use the planners if they are already good enough to *************generate data************* by using the LLMs to call the planners when a new task arrives?*************”************* However, the authors resolve this problem by using difficult environments in which even good planners struggle to solve the task. This does lead to the question of “can these failed trajectories be used to improve the planners?” but that is likely less scalable than just training a policy.\n\n### Weaknesses\n\n************************Motivation:************************ This work claims to be very general, but does seem to require quite a bit of domain-specific tuning. Robot pick-and-place is the task that the authors focus on but to make this work generally (e.g., navigation + manipulation for general robots) requires difficult design of the robot utility primitives and likely also difficult design of the “verify + retry” mechanism. Of course, to be fair, the authors don’t claim that this will work on those more general cases so this is somewhat of a nitpick.\n\nAlso given how big the LLM prompt is in the supplementary, it likely requires careful tuning of the prompt mechanism along with a very expensive/expressive/large language model, AND extensive scene descriptions to work. Is this approach very scalable to real-world environments? How robust can these success detectors be without perfect object-state information and the like? The authors experiment only in sim.\n\n****************************************Method explanation:**************************************** How do the authors use the failed trajectories? L182: “In the output replay buffer, the only failed trajectories are ones which timed-out or led to invalid states (e.g. object dropped on the floor).” Are these failed trajectories used to train the policy? If so then how does behavior cloning differentiate between successful and failed trajectories? If not, then please clarify this in the text.\n\n************************Experiments:************************\n\n- Baseline explanations: L245 \"ground truth segmentation,” segmentation of what exactly?\n- How much data is generated for each task? This is included in the supp (500 successful trajs) but should be in the main paper for context.\n- It would be beneficial to the academic community, especially among those who want to reproduce this type of work without the ability to work with industry-level language models or paying for GPT-3/3.5/4/4++ credits, to have an ablation study with some other open-source LLMs. VICUNA, LLAMA, ALPACA, and many other animal-inspired open-source LLMs are now flourishing while sometimes claiming to match GPT-3.5/4 performance on certain tasks. Please include some ablation study of this — this is important to the academic community and is the main reason for my “weak” accept.\n- 50 seeds per task but no error bars on anything?\n- Not asking for this to be included because this is asking too much, but did the authors try VLMs that can caption images? E.g., LLAVA-13B, GPT4, PALM (Bard), or more traditional captioning models? These models could help detect success conditions *********possibly********* more robustly than generating success checkers.\n- No robot experiments, but at this point these simulated environments are very similar to real-world robot tabletop manipulation so the application is clear for CoRL.\n\n### Minor Points\n\n- Related work: you can add this recent work to the **************scaling visuo-linguo-motor data************** paragraph for scaling up pre-annotated data: https://arxiv.org/abs/2306.11886\n- Related work: please add this work to the same section for work very similar to code-as-policies: https://arxiv.org/abs/2209.11302 (L78)\n- L142: function → functions\n- L238: an → a", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper combines ideas to scale up the collection of vision+language+motor data with novel approaches to train language conditioned policies (based on Diffusion Policy). The general approach is highly relevant to bringing strongly generalizing LLM to robotic manipulation.

The reviewers appreciate the novel ideas. The rebuttal addressed many technical points raised by the reviewers. (The 2nd reviewer indicated potentially raising the score.) A limitation is the evaluation in simulation only. The authors mention 'the Sim2Real route as a good way for our method to be applied to the real world'.
Decision: Accept (Poster)