BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning
Authors: Eric Jang
Alex Irpan
Mohi Khansari
Daniel Kappler
Frederik Ebert
Corey Lynch
Sergey Levine
Chelsea Finn
Venue: CoRL2021 Poster (2021)
Keywords: Zero-Shot Imitation Learning
One-Shot Imitation Learning
Multi-Task Imitation
Deep Learning

Abstract: In this paper, we study the problem of enabling a vision-based robotic manipulation system to generalize to novel tasks, a long-standing challenge in robot learning. We approach the challenge from an imitation learning perspective, aiming to study how scaling and broadening the data collected can facilitate such generalization. To that end, we develop an interactive and flexible imitation learning system that can learn from both demonstrations and interventions and can be conditioned on different forms of information that convey the task, including pre-trained embeddings of natural language or videos of humans performing the task. When scaling data collection on a real robot to more than 100 distinct tasks, we find that this system can perform 24 unseen manipulation tasks with an average success rate of 44%, without any robot demonstrations for those tasks.

BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning
Eric Jang", Alex Irpan" Mohi Khansari", Daniel Kappler", Frederik Ebert Corey Lynch', Sergey Levine Chelsea Finn!" 4.4
RRbbotics at Google 2x, The Moonshot Factory UCC Berkeley
"Stanford University
https://siter: http:////stee...ogee.... com/viien/bc-z/hoee
Abstract: this paper, we study the problem of enabling a vision-based robotic manipulation system generalize to novel tasks, long-standing challenge in robot learning. We approach the challenge from an imitation learning perspective, aiming to study how scaling and broadening the data collected can facilitate such generalization. To that end, we develop an interactive and flexible imitation learn- ing system that can learn from both demonstrations and interventions and can be conditioned on different forms of information that convey the task, including pre- trained embeddings of natural language or videos of humans performing the task. When scaling data collection on real robot to more than 100 distinct tasks, we find that this system can perform 24 unseen manipulation tasks with an average success rate of 44% without any robot demonstrations for those tasks.
Keywords: Zero-Shot Imitation Learning, Multi-Task Imitation, Deep Learning
Introduction
One of the grand challenges in robotics is to create general- purpose robot capable of perform- ing a multitude of tasks unstructured environments based on arbitrary user commands. The key challenge in this endeavour is generalization: the robot must handle new environments, recognize and manipulate objects it has not seen before, and understand the intent been asked to execute. End-to-end learning from pixels is flexible choice for modeling the be- havior of such generalist robots, as it has minimal assumptions about the state representation of the world. With sufficient real-world data, these methods should in principle enable robots to generalize across new tasks, objects, and scenes without requiring hand-coded, task-specific representations. However, realizing this goal has generally remained elusive. In this paper, study the problem of enabling a robot to generalize zero-shot or few-shot to new vision-based manipulation tasks.
command has never
We study this problem using the framework of imitation learning. Prior works on imitation learning have shown one-shot or zero-shot generalization to new objects [1, 3, 4, 5] and to new object goal configurations [6, 7]. However, zero-shot generalization to new tasks remains challenge, particularly when considering vision-based manipulation tasks that cover a breadth of skills (c.g., wiping, pushing, pick-and-place) with diverse objects. Achieving such generalization depends on solving challenges relating to scaling up data collection and learning algorithms for diverse data.
We develop an interactive imitation learning system with two key properties that enable high-quality data collection and generalization entirely new tasks. First, our system incorporates shared auton- omy into teleoperation to allow us to collect both raw demonstration data and human interventions to correct the robot's current policy. Second, our system flexibly conditions the policy on different forms of task specification, including language instruction or video of a person performing the task. Unlike discrete one-hot task identifiers [8], these continuous forms of task specification can in principle enable the robot to generalize zero-shot or few-shot new tasks by providing language or video command of the new task at test time. These properties have been explored previously; our aim is to empirically study whether these ideas scale to a broad range of real-world tasks.
Equal Contribution
Work done while author was at Google
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Video
Shaned Autonomy Data Collection
Diverse Multi Task Dataset
Policy Training
Goneralization to Unseen Tasks
Overview of BC-Z. We collect large-scale dataset (25,877 episodes) of 100 diverse manipulation tasks, and train 7-DoF multi-task policy that conditions on task language strings or human video. We show this system produces a policy that is capable of generalizing zero-shot to new unseen tasks.
Figure 1:
Our main contribution is an empirical study of large-scale interactive imitation learning system that solves a breadth of tasks, including zero-shot and few-shot generalization to tasks not seen during training. Using this system, we collect large dataset of 100 robotic manipulation tasks, through a combination of expert teleoperation and a shared autonomy process where the human operator "coaches" the learned policy by fixing its mistakes. Across 12 robots, 7 different operators collected 25,877 robot demonstrations that totaled 125 hours of robot time, as well as 18,726 human videos of the same tasks. At test time, the system is capable of performing 24 unseen manipulation tasks between objects that have never previously appeared together in the same scene. These closed-loop visuomotor policies perform asynchronous inference and control at 10Hz, amounting to well over 100 decisions per episode.
2 Related Work
Imitation learning has been successful in learning grasping and pick-place tasks from low- dimensional state [9, 10, 11, 12, 13, 14, 15]. Deep learning has enabled imitation learning directly from raw image observations [8, 16, 17]. In this work, we focus on enabling zero-shot and few-shot generalization to tasks in an imitation learning framework.
Multiple prior imitation learning works have achieved different forms of generalization, includ- ing one-shot generalization to novel objects [1, 2, 3, 4, 18], to novel object configurations [19], and to novel goal configurations [6, 7, 20], as well as zero-shot generalization to new objects [5], scenes [21]. and goal configurations [22]. Many of these works adapt to the new scenario by condi- tioning on a robot demonstration [1, 2]. a video of human [3, 4], language instruction [23, 24), goal image [21]. Our system flexibly conditions on either video of a human or language instruction, and we focus on achieving zero-shot (language) and few-shot (video) generalization to entirely new 7-DoF manipulation tasks real robot, including scenarios without goal images and where task-relevant objects are never encountered together in the training data.
It is standard to collect demonstrations via teleoperation [25] or kinesthetic teaching [10], and active learning methods such as DAgger [26] help reduce distribution shift for the learner. Unfortunately, DAgger and some of its variants [27, 28] are notoriously difficult to apply to robotic manipulation because they necessitate an interface where the expert must annotate the correct action when not in control of the robot policy. Inspired by recent works in autonomous driving, HG-DAgger [29] and EIL [30], our system instead only requires the expert to intervene when they believe the policy is likely to make eтror and allows the expert to temporarily take full control put the policy back on track. The resulting data collection scheme is easy to use and helps address distribution shift. Furthermore, the rate of expert interventions during data collection can be used as a live evaluation metric, which we empirically find correlates with policy success.
Beyond imitation learning, generalization has been studied in a number of other robot learning works. This includes works that generalize skills to novel objects [31, 32, 33, 34, 35], to novel environments [36], from simulation to reality [37, 38, 39, 40, 41J, and to new manipulation skills and objects [42, 43, 44, 45). We focus on the last case of generalizing to novel tasks, but unlike these prior works, tackle large suite of 100 challenging tasks that involve 7 DoF control at 10 Hz and involve more than 100 decisions within an episode to solve the task.
3 Problem Setup and Method Overview
overview of our imitation learning system shown Figure 1. Our goal is to train conditional policy that can interpret RGB images, denoted E S, together with task command W,

---

Wiocbot coarni
Brocktlic enser ove
Cupe carber with
Poot puppa nosal
SoOc Dow Buy
Train Tasks
UM
Jon a stfe
Mag poises polk
place voole in Super
pick Jowel
placo foraar 91 coorm bowl
px looe aCO0S
Mege Day
gornge
Unseen Tasks
bontl
in pe-obre suther
brock sperrpo
up gropes
purple boye
subset of training tasks (top row), and subset of held-out tasks (bottom two rows) used for evaluating zero shot task generalization. Top left: Given pretrained task embedding computed from human vidcos or text, BC-Z acts as an "action decoder" for the task embedding.
Figure 2:
which might correspond to a language string or a video of a person. Different tasks correspond completing distinct objectives; some example tasks and corresponding commands shown in Figure 2. The policy is a mapping from images and commands to actions, and can be written as u S W + A. where the action space A consists of the 6-DoF pose of the end effector as well as a 7th degree of freedom for continuous control of the parallel jaw gripper.
The policy is trained using large-scale dataset collected via a VR-based teleoperation rig (see Fig- ure 1, left) through a combination of direct demonstration and human-in -on--ii-tee-oopp shared autonomy. In the latter, trained policies deployed the robot, and the human operator intervenes to provide corrections when the robot makes a mistake. This procedure resembles the coman-gated DAgger (HG-DAgger) algorithm [26, 29], and provides iterative improvement for the learned policy, well as a continuous signal that can be used to track the policy's performance.
The policy architecture is divided into encoder g(2 which processes the command w into an embedding z E Z, and a control layer т, which processes (s,z) to produce the action a, i.c. S Z A. This decomposition is illustrated in Figure 2. with further details in Section 5. provides our method with the ability to incorporate auxiliary supervision, such as pretrained lan- guage embeddings, which help to structure the latent task space and facilitate generalization. In our experiments, we will show that this enables generalization to tasks that were not seen during training. including novel compositions of verbs and objects.
Data Collection and Workflow
In order for imitation learning system generalize task, we must be able to easily collect a diverse dataset, provide corrective feedback, and evaluate many tasks at scale. In this section, we discuss these components our system.
tasks with zero demonstrations of said
System Setup. Our teleoperation system uses an Oculus VR headset which is attached to the robot's onboard computer via USB cable and tracks two handheld controllers. The teleoperator stands behind the robot and uses the controllers to operate the robot with a ofne-of-sight 3rd-person view. The robot responds to the operator's 's movement in 10 Hz non-realtime control loop. The relatively fast closed-loop control allows the operator to demonstrate a wide range of tasks with ease and quickly intervene if the robot about enter unsafe state during autonomous execution. Further details on the user interface and data collection are in Appendices A and B.
Environment and Tasks. We place each robot in front of table with anywhere from 6 to 15 household objects with randomized poses. We collect demonstrations and videos of humans for 100 pre-specified tasks (listed in Tables 7 and 8), which span 9 underlying skills such as pushing and lock-and-laccs The model is then evaluated on 29 new tasks using a new language deseription or video of that task. For the method to perform well on these held-out tasks, it must both correctly interpret the new task command and output actions that are consistent with that task.
Shared Autonomy Data Collection. Data collection begins with an initial expert-only phase, where the human provides the demonstration of the task from start-to-innish After an initial multi-rask pol- icy is learned from expert-only data, continue collecting demonstrations in "shared autonomy"

---

mode, where the current policy attempts the task while the human supervises. At any point the hu- man may take over by gripping an "override" switch, which allows them briefly take full control of the robot and perform necessary corrections when the policy is about to enter unsafe state, or they believe the current policy will not successfully complete the task. This setup enables HG- DAgger [29], where intervention data is then aggregated with the existing data and used to re-train the policy. For the multi-task manipulation tasks, we collect 11,108 expert-only demonstrations for the initial policies, then collected additional 14,769 HG-DAgger demonstrations covering 16 iterations of policy deployment, where each iteration deploys the most recent policy trained on the aggregated dataset. This gives total of 25,877 robot demos. We find in Table 4 that when controlling for the same number of total episodes, HG-DAgger improves performance substantially.
Shared Autonomy Evaluation. When success rates are low, resources are best spent on collecting more data to improve the policy; but evaluation is also important to debug problems the work- flow. As the expected degree of generalization increases, we need more trials to evaluate the extent of policy generalization. This creates a resource trade-off: how should robot time be allocated between measuring policy success rates and collecting additional demonstrations to improve the policy? Fortunately, shared autonomy data collection confers an additional benefit: the intervention rate, measured as the average number of interventions required per episode, can be used as an indi- cation for policy performance. In Figure 5, we find that the intervention rate correlates negatively with overall policy success rate.
5 Learning Algorithm
The data collection procedure above results in a large multi-task dataset. For each task i, this dataset contains expert data (s, a) E D human video data wh E Dj, and one language command w/. We now discuss how we use this data to train the encoder g(2[w) and the control layer π (a)s,
5.1 Language and Video Encoders
Our encoder ((2/w) takes either a language command wi or a video of a human Wy as input and use a pretrained multi- produces task embedding 2. If the command is language command, lingual sentence encoder [46]' as our encoder, producing a 512-dim language vector for each task. Despite the simplicity, we find that these encoders work well in our experiments.
When task commands are instead a video of a human performing the task, we use a convolutional neural network to produce 2, specifically ResNet-18 based model. Inspired by recent works [2. 3], we train this network in an end-to-end manner. We collected a dataset of 18,726 videos of humans doing each training task, in a variety of home and office locations, camera viewpoints, and object configurations. Using paired examples of human video wi, and corresponding demonstration demo {(s, })}; we encode the human video z ~ q(- | wi,)), then pass the embedding to the control layer Taa ls, 2²), and then backpropagate gradient of the behavior cloning loss to both the policy and encoder parameters.
Visualizations of learned embeddings in Appendix E indicate that by itself, this end-to-end approach tends to overfit to initial object scenes, learn poor embeddings, and show poor task generalization. To help align the video embeddings more semantically, we therefore further introduce an auxiliary language regression loss. Concretely, this auxiliary loss trains the video encoder to predict the embedding of the task's language command with a cosine loss. The resulting video encoder objective is as follows:
min Σ
Σ (s.a)~D( m),Dij D
-log (ogls()),
(2 2
where 2// ~ g(-ww)) t6 g(-l((
task
behavior cloning
language regression
video encoder
language encoder
where Daus denotes the cosine distance. Since robot demos double as videos of the task, also train encoded robot videos to match to the language vector. This language loss is critical to learning more organized embedding space. Additional architecture and training details are in Appendix E.
5.2 Policy Training
Given fixed task embedding, we train (aass, 2) via Huber loss on XYZ and axis-angle predictions, and log loss for the gripper angle. During training, images are randomly cropped, downsampled,
Checkpoont from https::/tffie http:://///tt...e//ggogg//////rr sonpreeeeeettttccccccoooeeeeer-e mult tilguual//

---

Bil RALU (SS
coreacted
Brai ONN
Idber Los
Renkas Commansk petemane debeiic
Viauo Nutvon Plcsid-11:
Mial
Aillvarlee
comenrted
signad
Video Cionder
I Ce dtonnc
Log
supenvibbe agrall
"Place bottle the caramic bowl
Figure 3: BC-Z network architecture. A monocular RGB image from the head mounted camera is passed through a R encoder, then through a two-layer MLP to predict each action modality (delta XYZ, delta axis-angle, and gripper angle). FiLM layers [47] condition the architecture on a task embedding z computed from language ti or video Wi..
and subjected to standard photometric augmentations. Below we describe two additional design choices that we found to be helpful. Additional training details such as learning rates, batch sizes, pseudocode, and further hyperparameters are discussed Appendix D.
Open-Loop Auxiliary Predictions. The policy predicts the action the robot would take, as well as an open-loop trajectory of the next 10 actions the policy would take it were operating in an open- loop manner. At inference time, the policy operates closed-loop, only executing the first action based on the current image. The open-loop prediction confers auxiliary training objective, and provides a way to visually inspect the quality of a closed-loop plan in an offline manner (see Figure 1, right).
State Differences as Actions. In standard imitation learning implementations, actions taken demonstration time are used directly as target labels to be predicted from states. However, cloning expert actions 10Hz resulted in the policy learning very small actions, well as dithering behav- ior. To address this, we define actions as state differences to target poses N > 1 steps in the future, using adaptive algorithm to choose N based on how much the arm and gripper move. We provide ablation studies for this design choice in Section 6.3 and further details in Appendix
5.3 Network Architecture
We model the policy using a deep neural network, shown in Figure 3. The policy network processes the camera image with a ResNet18 "torso" [48], which branches from the last mean-pool layer into multiple "action heads". Each head is a multilayer perceptron with two hidden layers of size 256 each and ReLU activations, and models part of the end-effector action, specifically the delta XYZ, delta axis-angle, and normalized gripper angle. The policy is conditioned on a 512-dim task embedding through FiLM layers (47]. Following Perez et al. [47], the task conditioning is linearly projected to thannel-wise scales and shifts for each channel each of the 4 ResNet blocks.
Experimental Results
Our experiments aim to evaluate BC-Z in large-scale imitation learning settings. We start with an initial validation of BC-Z on single-task visual imitation learning. Then, our experiments will aim to answer the following questions: (1) Can BC-Z enable zero-shot and few-shot generalization to new tasks from command in the form of language
video of human? (2) Is the performance of BC-Z bottlenecked by the task embedding or by the policy? (3) How important are different components of BC-Z, including HG-DAgger data collection and adaptive state diffs? We present experiments aimed at these questions in this section.
6.1 BC-Z on Single-Task Imitation Learning
We first aim to verify that BC-Z can learn individual vision-based tasks before considering the more challenging multi-task setting. We choose two tasks: a bin-emptying task where the robot must grasp objects from a bin and drop them into an adjacent bin, and a door opening task where the robot must push open a door while avoiding collisions. Both tasks use the architecture in Figure 3, except that the door opening task involves predicting the forward and yaw velocity of the base instead of controlling the arm. The bin-emptying dataset has 2,759 demonstrations, while the door opening
5

---

place apple in paper cup
Place bottle in table surface
Figure 4: Qualitative examples of BC-Z successfully performing held-out tasks.
dataset has 12,000 demonstrations collected across 24 meeting rooms and 36,000 demonstrations across 5 meeting rooms in simulation. Further task and dataset details are in Appendix I.
In Table 1, see that the BC-Z model is able to reach a pick-rate of 3,4 picks per minute, over half the speed of a human teleoperator. Further, we see that BC-Z reaches a success rate of 87% on the training door scenes and 94% on held-out door scenes. These results validate that the BC-Z model and data collect system can achieve good performance on both training and held-out scenes in the single-task setting. Additional analysis is provided in Ap- pendix
Table 1:
Single-task bin and door performance, aver- age and standard deviation across runs.
Bin-Emptying
Picks Minute
# Runs
Human Expert BC-Z (2759 demos)
6.3 (2.1)
2759
3.4 (1.2)
9
Door Opening
Success Rate
# Runs
BC-Z (24 Train Doors) BC-Z (4 Holdout Doors)
87% (2.2)
480
94% (2.7)
80
6.2 Evaluating Zero-Shot and Few-Shot Task Generalization
Next, we aim to test whether BC-Z can achieve generalization to new tasks. Demonstrations are collected across 100 different manipulation tasks, comprising two disjoint sets objects. Using disjoint sets of objects allows us specifically test generalization to combinations of object-object pairs and object-action pairs that are not seen together during training. For the first set of objects, demonstrations are collected across 21 different tasks, listed in Table 7, which cover wide range of skills, from pick-and-plan tasks to skills that require positioning the object in a certain way, like "stand the bottle upright". For the second set of objects, demonstrations are collected for 79 different tasks, including pick-and-ples variety of manipulation behaviors, but is defined over larger object with sets are shown in Appendix B and a full list of train task sentences are in Appendix J.
surface wiping, and object stacking. The latter family has a smaller clutter. Object
We evaluate BC-Z on 29 held-out tasks. Language conditioned policies are given novel sentence, while vidco conditioned policies are given the average embedding of few human videos of the new task. Four held-out tasks use objects in the 79-task family, whereas 25 tasks generated by mixing objects between the 21-task family and 79-task family. Thus, the first held-out tasks do not require cross-object set generalization, these 4 tasks are sufficiently challenging that training single-task policies on 300+ held-out demos with DAgger interventions completely fails, achieving 0% task success. This provides degree of calibration on the difficulty of these tasks. We hypothesize that a major contributing factor to this challenge is the wide range of locations, objects, and distractors that the skills must generalize to in our settings, as well as the wide range of these factors in the training data.
they easier generalize to. Even so, we find that each of
In Table 2, we see that language-condittooed BC-Z is able to generalize zero-shot to both kinds of held-out tasks, averaging at 32% success and showing non-zero success on 24 held-out tasks. Among the 24 hold-out tasks with non-zero success rates, BC-Z achieves an average success of 44% when conditioned on language embeddings it has never seen. When conditioning on videos of humans. we find that generalization is much more difficult, but that BC-Z is still able to generalize to nine novel tasks with a non-zeru success rate, particularly when the task does not involve novel object combinations. Qualitatively, towards the correct objects, clearly indicating that the task embedding is reflective of the correct task, we further illustrate in the supplementary video. The most common source of failures "last-centimeter" errors: failing to close the gripper, failing to let go of objects, or near miss of the target object when letting go of an object in the gripper.
observe that the sngguage-conditione policy usually moves
Performance Bottlenecked on the Encoder or the Policy? Now that we see that BC-Z can gen- eralize to substantial number of held-out tasks to some degree, ask whether the performance is

---

Table 2: Success rates for zero-shot (language) and few-shot (video) generalization to tasks not in the training dataset. The first 4 tasks only use objects from the 79-task family. The remaining tasks mix objects between the 21 21-task and 79-task families, requiring further generalization. Numbers in parentheses are I unit standard deviation. The language conditioning generalizes to several holdout tasks, whereas the video conditioning shows promise on tasks that do not mix objects between task families. Overall performance improves slightly with fewer distractor objects.
Skäll
Held-out Lasks
Lang-conditoned (1 distractor)
Lang-conditioned (4-5 distractors)
Video-conditooed (4.5 distractors)
(no demos during training)
"place sponge in Iray'
83% (6.8)
82% (9.2)
22% (2.2)
pick-place
placee grapes in red howl
87% (6.2)
75% (10.8)
12% (7.8)
placee apple in paper cup'
30% (8.4)
33% (12.2)
14% (7.8)
pick wipe
'wipe tray with sponge'
40% (8.9)
0% (0)
28% (10.6)
"place banana ceramic bowl' "place bottle in red howl"
50% (15.8)
75% (9.7)
7.5% (4.2)
50% (15.8)
75% (9.7)
0% (0)
'place grapes in ceramic bowl
70% (14.5)
70% (10.3)
0% (0)
"place bottle in table surface'
50% (11.2)
5% (3.5)
'place white sponge in purple howl"
70% (14.9)
45% (11.2)
(0)
pick-place
'place white sponge in tray"
50% (15.8)
40% (11.0)
0% (0)
"place apple in ceramic bowl'
30% (14.5)
20% (8.9)
0% (0)
"place bottle in purple bowl"
30% (14.5)
20% (8.9)
0% (0)
"place banana in ceramic cup' 'place bamana white sponge' "place metal cup in red bowl"
10% (9.5)
0% (0)
0% (0)
40% (15.5)
(0)
0% (0)
0% (0)
0% (0)
0% (0)
'pick up grapes'
70% (14.5)
65% (10.7)
0% (0)
"pick up apple
20% (12.7)
55% (11.2)
(3.5)
"pick up lowel"
50% (15.8)
42,8% (18.7)
0% (0)
grasp
"pick up pepper"
50% (15.8)
35% (10.7)
12.5% (5.2)
pick up bottle"
40% (15.5)
30% (10.3)
17.5% (6.0)
"pick up the red bowl'
30% (14.5)
0% (0)
0% (0)
pick-dmg
'drng grapes across the table'
0% (0)
14% (13.2)
0% (0)
'wipe table surface with banana'
0% (0)
10% (6.7)
0% (0)
pick-wipe
"wipe tray with white sponge'
20% (12.7)
0% (0)
0% (0)
"wipe coramic bowl with brush
10% (9:49)
(0)
(0)
"push parple
across the table'
50% (15.8)
30% (10.3)
(0)
push
"push tray across the table'
30% (14.5)
25% (9.7)
0% (0)
"push red bowl across the table"
60% (15.5)
0% (0)
0% (0)
Holdon Task Overall
38%
32%
4%
limited more by the generalization of the encoder q( ww) the control layer I (a]s,, 2), or both. To disentangle these factors, we measure the policy success rate on the train- ing tasks conditioned in three ways: a one-hot task identi- fier, language embeddings of the training task commands, and video embeddings of held-out human videos of the training tasks. This comparison is in Table 3. The sim- ilar performance between one-hot and language suggests the latent language space is sufficient, and that language- conditioned performance on held-out tasks is bottlenecked on the control layer more than the embedding. The more significant drop in performance of iideo-conditioned policies suggests inferring tasks from videos is much more difficult, particularly for held-out tasks.
Table 3: Training vs. generalization per- formance, averaged across 21 of the train- ing tasks and all 28 held-out tasks.
Setting
Task Conditioning
Success
One-hot
42%
Train
Language
40%
Video
24%
Language Video
32%
Held-Out
4%
6.3 Ablation Studies and Comparisons
We validate the importance of several BC-Z design decisions using the (training) 21 family. Our first set of ablations evaluate on the "place the bottle in ceramic bowl" command, which has the most demos (1000) of any task. We first test whether multi-task training is helpful for performance: we compare the multi-task system trained on 25,877 demos across all tasks, to a single-task policy trained on just the 1000 demos for the target task. In Table 4 (left), the single-task baseline achieves just 5% success. The low number is consistent with the low single-task performance on holdout tasks from Section 6.2: collecting data over several robots and operators likely makes the task harder to learn. Only when pooling data across many tasks does BC-7 learn to solve the task. We ablate the adaptive state diff scheme described in Section 5.3 and find that it is important; when naively
7

---

Table 4: Ablation Studies. Left: Multi-task vs. single task models on the 'place the bottle in the ceramic bowl' task. Training across tasks and with adaptive state-diffs is important for good training performance. Right: DAgger comparison "place the bottle in the ceramic bowl" (1-Task) and the 8- 8-Tak subset from Table 7. Controlled for the same amount of data, DAgger reaches higher success numbers significantly more quickly.
Method
1-Task
Dataset
1-Task
8-Task
Multi-task, language conditioned
52% (6.3)
100%
Manual
27% (5.2)
23% (4.2)
Multi-task, one-hot conditioned Single-task baseline (1000 demos) Multi-task, one-hot, no adaptive state-diff
45% (5.3)
50% Manual
53% (5.8)
47% (5.2)
5% (2.8)
50% HG-DAgger
3% (2.3)
choosing the N future state compute the expert actions, the policy fits the noise and moves too slowly, resulting in state drift away from good trajectories.
We next ablate the use of HG-DAgger while keeping the total amount of data fixed. Specifically, we com- pare performance of policies trained using 50% ex- pert demos and 50% HG-DAgger interventions, ver- using 100% expert demos. In Table 4 (right),
task
multitask manipulation door opening
Success Rate
0.8
0.6
we find that HG-DAgger significantly improves task performance over cloning expert demonstrations on
0.4
both the 'place bottle in ceramic bowl' task and 7 other training tasks. Further details on this compari- son are in Appendix K. Finally, in Figure 5, we evalu- ate whether measuring HG-DAgger interventions can give us live proxy of policy performance. We see that intervention frequency is inversely correlated with policy success, as measured by the fraction of successful episodes not requiring intervention. This result suggests that we can indeed use this metric with HG-DAgger for development purposes.
02
Mean # Interventions
Figure 5: Mean number of interventions vs. task success rate. Each point represents a pol- icy evaluated during HG-DAgger data collection. There is a clear correlation between the mean number of interventions and success rate, sug- gesting that interventions can be used as live proxy for performance.
Discussion
We presented a multi-task imitation learning system that combines flexible task embeddings with large-scale training on 100-task demonstration dataset, enabling it to generalize to entirely new tasks that were not seen in training based on user-provide providdd language or video commands. Our eval- uation covered 29 unseen vision-based manipulation tasks with a variety of objects and scenes. The key conclusion of our empirical study is that simple imitation learning approaches can be scaled in a way that facilitates generalization to new tasks with zero additional robot data of those tasks. That is, we learn that we do not need more complex approaches to attain task-level generalization. Through the experiments,
also learn that 100 training tasks is sufficient for enabling generalization to new tasks, that HG-DAgger is important for good performance, and that frozen, pre-traine traineine language embeddings make for excellent task conditioners without any additional training.
Our system does have number of limitations. First, the performance on novel tasks varies signifi- cantly. However, even for tasks that are less successful, the robot often exhibits behavior suggesting that it understands at least part of the task, reaching for the right object or performing a semantically related motion. This suggests that exciting direction for future work is to use our policies general-purport initialization for finetuning of downstream tasks, where additional training, perhaps with autonomous RL, could lead to significantly better performance. The structure of our language commands follows a simple "(verb) (noun)" structure. A direction to address this limitation is to re- label the dataset with variety of dumman-poovidee annotations [24], which could enable the system to handle more variability in the language structure. Another limitation is the lower performance of the video-conditioned policy, which encourages future research on improving the generalization of video-based task representations and enhancing the performance of imitation learning algorithms as whole, low-level control errors are also major bottleneck.
8

---

Acknowledgments
Eric Jang, Alex Irpan, and Frederik Ebert ran experiments on different forms of task conditioning in the task generalization setup. Mohi Khansari helped build the HG-DAgger interface and ran experiments in the bin-em bin-emptying and door opening tasks. Daniel Kappler built the data annotation visualizer. Corey Lynch advised Frederik's internship and gave pointers on language models. Sergey Levine and Chelsea Finn supervised the project.
We would like to give special thanks Noah Brown, Omar Cortes, Armando Fuentes, Kyle Jeffrey, Linda Luu, Sphurti Kirit More, Jornell Quiambao, Jarek Rettinghouse, Diego Reyes, Rosario Jau- regui Ruano, and Clayton Tan for overseeing robot operations and collecting human videos of the tasks, as well as Jeffrey Bingham and Kanishka Rao for valuable discussions.
References
[1] C. Finn, т. Yu, T. Zhang, P. Abbeel, and S. Levine. One-shot visual imitation learning via meta-learning. In Conference on Robot Learning, pages 357-368. PMLR, 2017.
[2] James, M. Bloesch, and A. Davison. Task-embedded control networks for few-shot imi- tation learning. In Conference on Robot Learning, pages 783-795. PMLR, 2018.
[3] T. Yu, C. Finn, A. Xie, S. Dasari, T. Zhang. P. Abbeel, and S. Levine. One-shot imitation from observing humans via domain-adaptive meta-learning. Robotics: Science and Systems, 2018.
[4] A. Bonardi, S. James, and A. J. Davison. Learning one-shot imitation from humans without humans. IEEE Robotics and Automation Letters, 5(2):3533 3539, 2020.
[5] S. Young, D. Gandhi, S. Tulsiani, A. Gupta, P. Abbeel, and L. Pinto. Visual imitation made easy. arXiv e-prints, pages arXiv-2008, 2020.
[6] Y. Duan, M. Andrychowicz. B. C. Stadie, J. Ho, J. Schneider, I. Sutskever, P. Abbeel, and
W. Zaremba. One-shot imitation learning. arXiv preprint 1rXiv:1703.07326 2017.
[7] Dasari and A. Gupta. Transformers for one-shot visual imitation. Conference on Robot Learning (CoRI.), 2020.
[8] R. Rahmatizadeh, P. Abolghasemi, L. Bölöni, and S. Levine. Vision-based multi-task manip- ulation for inexpensive robots using end-to-end learning from demonstration. 2018 IEEE international conference robotics and automation (ICRA), pages 3758-3766 IEEE, 2018.
[9] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469 483, 2009.
[10] S. Khansari-Zadeh and A. Billard. Learning stable non-linear dynamical systems with gaussian mixture models. IEEE Transaction on Robotics, 27(5):943-957, 2011.
[11] A. Billard, Y. Epars, Calinon, Schaal, and G. Cheng. Discovering optimal imitation strategies. Robotics and autonomous systems, 47(2-3):69-77, 2004.
[12] S. Schaal, J. Peters, J. Nakanishi, and A. Ijspeert. Learning movement primitives. In Robotics research. the eleventh international symposium, pages 561-572. Springer, 2005.
[13] R. Chalodhorn, B. Grimes, K. Grochow, and R. P. Rao. Learning to walk through imitation. In IJCAI, volume 7, pages 2084-2090, 2007.
[14] P. Pastor, H. Hoffmann, T. Asfour, and S. Schaal. Learning and generalization of motor skills by learning from demonstration. In 2009 IEEE International Conference on Robotics and Automation, pages 763-768. IEEE, 2009.
[15] K. Mülling, J. Kober, O. Kroemer, and J. Peters. Learning to select and generalize striking movements in robot table tennis. The International Journal of Robotics Research, 32(3):263- 279, 2013.
[16] D. A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network, Technical report,
Calleic-Mellon University, 1989.

---

[17] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imita- tion learning for complex manipulation tasks from virtual reality teleoperation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 5628-5635. IEEE, 2018.
[18] A. Zhou, E. Jang, D. Kappler, A. Herzog, M. Khansari, P. Wohlhart, Y. Bai, M. Kalakrishnan, S. Levine, and C. Finn. Watch, try, learn: Meta-learning from demonstrations and reward. International Conference on Learning Representations (ICLR), 2020.
[19] T. L.. Paine, S. G. Colmenarejo. Z. Wang. S. Reed, Y. Aytar, T. Pfaff, M. W. Hoffman, G. Barth- Maron, S. Cabi, D. Budden, et al. One-shot high-fidelity imitation: Training large-scale deep 2018.
nets with rl. Xrviv preprint
[20] D.-A. Huang, S. Nair, D. Xu, Y. Zhu, A. Garg, L. Fei-Fei, S. Savarese, and J. C. Niebles. Neural task graphs: Generalizing to unseen tasks from a single video demonstration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8565-8574, 2019.
[21] D. Pathak, P. Mahmoudieh, G. Luo, P. Agrawal, D. Chen, Y. Shentu, E. Shelhamer, J. Malik, the IEEE conference on computer vision and pattern recognition workshops, pages 2050-2053, 2018.
A. A. Efros, and T. Darrell. Zero-shot visual imitation. In Proceedings the
[22] P. Goyal, R. J. Mooney, and S. Niekum. Zero-shot task adaptation using natural language. arXiv preprint (tti::20060299777 2021.
[23] S. Stepputtis, J. Camphell, M. Phielipp. S. Lee, C. Baral, and H. B. Amor Language- conditioned imitation learning for robot manipulation tasks. arXiv preprint arXiv:2010. /2083, 2020.
[24] C. Lynch and P. Sermanet. Grounding language in play. arXiv preprint arXiv:2005.07648, 2020.
[25] S. Calinon, Evrard, F. Gribovskaya, A. Billard, and A. Kheddar. Learning collaborative ma- nipulation tasks by demonstration using a haptic interface. In 2009 International Conference on Advanced Robotics, pages 1-6. IEEE, 2009.
[26] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured predic- tion to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Pro- ceedings, 2011.
[27] S. Ross and J. A. Bagnell. Reinforcement and imitation learning via interactive no-regret learning. arXiv preprini wrxiv:1406.599) 2014.
[28] M. Laskey, J. Lee, R. Fox, A. Dragan, and K. Goldberg. Dart: Noise injection for robust imitation learning. In Conference on robot learning, pages 143-156. PMLR, 2017.
[29] M. Kelly, C. Sidrane, K. Driggs-Camphell, and M. J. Kochenderfer. Hg-dagger: Interactive imitation learning with human experts. In 2019 International Conference on Robotics and Automation (ICRA), pages 8077-8083. IEEE, 2019.
[30] Spencer, S. Choudhury, M. Barnes, M. Schmittle, M. Chiang. P. Ramadge, and S. Srinivasa. Learning from interventions: Human-robot interaction as both explicit and implicit feedback. 07 2020. doi: 1......7.......... 000.:0..660///8S2200000000
http:///RSSS........2 2020.XV1.055.
[31] M. Khansari, D. Kappler, J. Luo, J. Bingham, and M. Kalakrishnan. Action image repre- sentation: Learning scalable deep grasping policies with zero real world data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 3597-3603, 2020.
[32] L. Pinto and A. Gupta. Supersizing self-supervision: Learning grasp from 50k tries and 700 robot hours. In 2016 IEEE international conference on robotics and automation (ICRA), pages 3406 3413. IEEE, 2016.
[33] J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg. Dex- net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. arXiv preprint (703.0:7770033000 2017.
10

---

[34] D. Kalashnikov, A. Irpan. P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, pages 651-673. PMLR, 2018.
[35] Hatori, Y. Kikuchi, S. Kobayashi, K. Takahashi, Y. Tsuboi, Y. Unno, W. Ko, and Tan. Interactively picking real-world objects with unconstrained spoken language instructions. In IEEE International Conference on Robotics and Automation (ICRA), 2018.
[36] A. Gupta, A. Murali, D. Gandhi, and L.. Pinto. Robot learning in homes: Improving general- ization and reducing dataset bias. arXiv preprint arXiv: 11Xiv::807007449 2018.
[37] F. Sadeghi, A. Toshev, E. Jang, and S. Levine. Sim2real viewpoint invariant visual servoing by recurrent control. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4691-4699, 2018.
[38] J. Tobin, L. Biewald, R. Duan, M. Andrychowicz, A. Handa, V. Kumar, B. McGrew, A. Ray, J. Schneider, P. Welinder, et al. Domain randomization and generative models for robotic grasp- ing. 2018 JEEE/RSJ International Conference on Intellieent Robots and Systems (IROS), pages 3482 -489. IEEE, 2018.
[39] B. Mchta, M. Diaz, F. Golemo, C. J. Pal, and L. Paull. Active domain randomization. In Conference Robot Learning, pages 1162-1 1176. PMLR, 2020.
[40] S. James, P. Wohlhart, M. Kalakrishnan, D. Kalashnikov, A. Irpan, J. Ibarz, S. Levine, R. Hadsell, and K. Bousmalis. Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized- mandoizzedto-ccaoniice ta-connnical adaptation networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12627-12631 2019.
[41] J. Zhang, L. Tai, P. Yun, Y. Xiong, M. Liu, J. Boedecker, and W. Burgard. Vr-goggles for robots: Real-to-sim domain adaptation for visual control. IEEE Robotics and Automation Letters, 4(2): 141111881155 1155, 2019.
[42] C. Finn and S. Levine. Deep visual foresight for planning robot motion. In 2017 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 2786-2793. IEEE, 2017.
[43] S. Dasari, Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, S. Singh, Levine, and
C. Finn. Robonet: Large-scale multi-robot learning. arXiv preprint arXiv:1910. 1910:/191..115, 2019.
[44] Y. Chebotar, K. Hausman, Y. Lu, T. Xiao, D. Kalashnikov, Varley, A. Irpan, B. Eysenbach, R. Julian, C. Fino, et al. Actionable models: Unsupervised offline reinforcement learning of robotic skills. arXiv preprint 0rXiv:2104.07749 2021.
[45] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXiv:2104.08212 2021.
[46] Y. Yang. D. Cer, Ahmad, M. Guo, J. Law, N. Constant, G. H. Abrego. S. Yuan, C. Tar, Y.-H. Sung, et al. Multilingual universal sentence encoder for semantic retrieval. arXiv preprint D:Xiv:1907, 1907.04377 2019.
[47] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
[48] K. He, X. Zhang. S. Ren, and Sun. Identity mappings deep residual networks. In European conference on computer vision, pages 630-645. Springer, 2016.
[49] Huber. Robust estimation of a location parameter. In Breakthroughs in statistics, pages 492-518. Springer, 1992.
[50] A. v. d. Oord, L.,, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint
2018

---

[51] D. Ho. K. Rao, Z. Xu, E. Jang, M. Khansari, and Y. Bai. Retinagan: An object-aware approach sim-to-real transfer. 2021 International Conference Robotics and Automation (ICRA), 2021.
[52] P. de Haan, D. Jayaraman, and S. Levine. Causal confusion in imitation learning. arXiv preprint arXiv: 199.,999 2019.
[53] H. Zhang, M. Cissé, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk mini- mization. In ICLR (Poster), 2018. URL httctps://openreview...ee.
Rb.
[54] A. Miech, J.-B. Alayrac, L. Smaira, I. Laptev, J. Sivic, and A. Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9879-9889, 2020.
12


Reviews:
[{"review_id": "pfuI_LcXRd3", "reviewer": "Reviewer_7L8N", "summary": "In the paper, authors have worked on enabling vision-based robotic systems to carry out novel manipulation tasks. With this goal in mind, authors have collected large robot imitation data with corresponding sentence instruction and human demonstration. Authors have used this data to train imitation learning agent to perform tasks in the real world. To make it possible to either use language instruction or video demonstration as input to define the task, authors conditioned policy on embedding learned from these inputs. They have also built infrastructure to collect data in an online fashion where human experts only jumped in when the policy went in the wrong direction for executing the task. They have shown that data collected in this fashion helped in executing tasks better compared to passive data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* Collected a large amount of robot imitation data with vision and language instructions\n* Successfully able to learn policy conditioned on either vision or sentence instruction, which will generalize to novel tasks \n* Showed that language embedding pre-trained on some other task work well  in conditioning imitation learning policy\n* Showed that system able to generalize to different objects as well as tasks based on language embedding\n\nWeakness\n* Incremental algorithmic improvement: Lots of research has already been done in imitation learning based on human video demonstration or language instruction. This work mainly combine both the input modality, language & vision, for conditioning the imitation learning policy.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "* Collected big dateset for robot imitation learning with human video and language instructions. This dataset can be helpful to other researchers in the field of imitation learning.\n* Showed that pre-trained language embedding is good enough to encode task for imitation learning.\n* Able to show the learned system generalize well with the unseen task and objects based on language instructions\n* Able to show data collected in active manner, by involving expert human when the learned policy goes wrong, compared to passive data collection, helps in learning better policy.  \n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "* Author have showed that language embedding trained on some other task can be useful in conditioning the imitation learning policy, however for videos, authors have tried to learn embedding from collected video demonstration. It would interesting to know how the video embedding learned form other large video datasets works for conditioning imitation learning policy . Also large drop in the performance for unseen task based on video embedding, shown in table-3, shows that video embedding over-fitted to training videos."}}, {"review_id": "lxESELNAat", "reviewer": "Reviewer_KYDU", "summary": "The main contribution of the paper is a VR based interactive imitation learning system for providing demonstrations (and feedback) to a robot. This system is used to collect a very large (> 25K) set of demonstrations across a variety of manipulation tasks. The authors then use this large set of demonstrations to train an imitation learning agent to perform these tasks (and generalize to new variants of these tasks). An important part of the system is that fact that it is interactive (the HG-Dagger component) which allows for providing feedback to the policy.\n\nOverall the paper demonstrates that with a sufficient number of demonstrations the agent is able to generalize to novel tasks with moderate success.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall I think that this paper can be considered as a “systems paper” where the main contribution is the interactive imitation learning system. The comments below should be considered in this context.\n\nStrengths:\nThe problem considered (language-conditioned robotic manipulation in 7-DOF) is both interesting, relevant and very challenging.\nUsing language (and to a lesser extent video) is an interesting modality for conditioning robot tasks.\nThe experimental evaluation is extensive\nThe authors are very honest about the strengths/weaknesses of their approach.\n\n\nWeaknesses:\n\nI am not sure what I have learned and/or what conclusions I should draw having read the paper. Overall I think it is accepted in the robotics community that with enough demonstrations (and a few other tricks, such as HG-Dagger, state differences as actions, etc.) you can achieve reasonable performance with an imitation learning agent. This seems to take that premise and test it out with a very large set of demonstrations on a variety of tasks. This is an interesting endeavor, but the conclusions aren’t particularly clear to me. Overall it seems that the method presented requires a very significant number of demonstrations (25K demonstrations collected over several months) to achieve moderate levels of performance (success rates are below 50% on the held-out tasks). Should we consider this as a negative result and discontinue this line of investigation? Do the authors learning/control approach might need to be reformulated instead of just “scaling up” and collecting another 25K demonstrations?\n\nThe “place the bottle in the ceramic bowl” task is an example of the very large number of demonstrations that this system requires. The single-task model trained with 1000 demonstrations was only able to achieve a 5% success rate. I think that 1000 demonstrations is quite a lot.  so a system that isn’t able to learn with this number of demonstrations is a bit worrying. I think it is interesting that the multi-task model is able to achieve 50% on this task, but that relies on 25K demonstrations that took months to collect. Given the number of demonstrations required are we going to be able to scale this to get the robots to accomplish useful tasks in industrial or home settings?\n\n\nThe network architecture for the policy is simple and straightforward, which is a positive. However, coming back to the earlier point, the method is very data hungry and requires lots of demonstrations to achieve even moderate performance. The authors might consider other more data efficient approaches given the challenge of collecting robot demonstrations in the real world.\n\nIn Table 3 we see that the performance of the model conditioned on the one-hot encoding is effectively identical (40% vs. 42%) to the model conditioned on the language encoding. This makes it seem that what the language encoder is doing (at least on the train tasks) is effectively acting as a one-hot encoder. Thus is this architecture getting any value from using language? I agree with the authors that language is a flexible and general way of specifying tasks, but I would have maybe expected language to impart other benefits that would transfer across tasks in ways that one-hot encodings wouldn’t (e.g. an understanding of colors such as in the “red bowl” and “purple bowl” language instructions). I would be curious to hear the author’s thoughts on this.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Recommendation: weak reject\n\nOverall I think that the paper is well written and executed but that the underlying idea is not particularly novel and the performance of the method is not particularly good. I am not sure what exactly I learned from reading the paper. As such I am recommending weak reject, although the true rating is more somewhere in between weak reject and weak accept.\n\n\n**Updates post-rebuttal**\n\nI am updating my review to a **weak accept** (as selected in the Recommendation section) with my true rating being slightly higher than weak accept but below strong accept.\n\nThe authors have done an excellent and thorough job responding to reviewer concerns and updating the paper accordingly. I think the main improvement has been in clarifying the contribution of the work, namely a large scale empirical study. Given this I think that there is something to learn from the paper and think it merits the updated rating.\n\nThat being said I think that the actual method in question still has significant drawbacks/issues/challenges that culminate in low success rates. The approach is not the one you would necessarily take to get high reliability on a specific task, and the authors acknowledge this. The point of this paper is not to use task-specific tricks (like TransporterNetworks) to accomplish specific tasks effectively but rather to do an empirical study of a general purpose system. I also agree with the authors that the tasks and approach they are taking (e.g. continuous 7-DOF control) is very challenging. I want to make it clear that I understand this and am not trying to hold it against them for taking on this challenge and being honest about the performance of the method.\n\n\nInvestigating the relative merits of general purpose approaches (like the one presented) and more tailored/engineered solutions (e.g. TransporterNetworks) is an interesting direction for future work that I would like to see taken up and debated by the community.\n", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "When discussing demonstrations collected using HG-Dagger the authors should expand a bit on which policy was being executed when these HG-Dagger interventions were done. Did interventions from one-policy help a different policy to improve, or did different trained policies make different types of “mistakes”.\n"}}, {"review_id": "Rbh1WmqLcOu", "reviewer": "Reviewer_aF8M", "summary": "The paper presents BC-0: a *one-shot* imitation learning system for vision-based robot manipulation; I will elaborate on the *one-shot* label (rather than the titles zero-shot) later in the review. As this is an imitation learning method, data collection is key in order to help elevate covariate shift. The data collection has two phases: (1) expert-only, where the agent is trained on full human-provided demos (from a VR system), and (2) a \"shared-autonomy\" mode, which uses HG-Dagger (where the human intervenes and temporarily take full control when they believe the policy is likely to make an error). Policy learning is done similarly to TECNets [2-3] where the paired examples of a human video/text and corresponding robot demo are trained to embed close together, while also training for a behaviour cloning loss. The action mode for the arm is 7 DoF: 6 DoF for the pose, and another for the gripper. Results show that when training on a 100-task demonstration dataset, the method generalises to 29 unseen tasks when given a single user-provided language or video commands.\n\nTLDR: It is a one-shot imitation learning system that is conditioned on either videos or natural language, where the data is collected via shared autonomy.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- Heavy use of real world experiments (with a caveat, see weaknesses).\n- Obvious that a lot of time and effort went into the data collection process.\n- The combination of language and video is a logical and appropriate next step from [1,2,3,4].\n- The shared autonomy data collection is a nice touch.\n\n**Weaknesses**\n- Although having real world results is great, it is also important for the sake of reproducibility, for the paper to supply some for of simulation results, perhaps even on a standard set of tasks, e.g. RLBench, MetaWorld, Robosuite, etc. This is very important for robotics, where comparing methods is difficult. \n- How does this method compare to others? The only baselines provided were ablations of their own method; the paper rightly point to a few works already in this area, and so comparison to these seems important. If someone wanted to reimplement this idea, then they could use your reported simulation results to ensure they had implemented your method correctly. \n- Following the president set by previous work [1-4], this is not zero-shot learning, and seems wrong to label the work as such. It is one-shot learning; as a demonstration must be given to give intent to the control agent. Of course, the lines become blurred when considering *only* text input to convey intent, but as the paper features an equal weighting of video intent, then it is misleading to identify the work as zero-shot. \n- The experimental results when comparing to other methods could be improved. The paper does not compare to other methods expect for ablations of itself. The paper rightly point to a few works already in this area, and so comparison to these seems important. How does this work compare to [3, 4] when trained on the same dataset (without the language component)?\n- The contributions seem minor; the work heavily uses methods from [2, 3, 4] and HG-Dagger [29]. I.e there is not too much to be gained/learned by reading this paper. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The contributions do seem minor (the work heavily uses methods from [2, 3, 4] and HG-Dagger [29]), however the work is still the first I am aware of to have the option to give either video or language intent (most use video). This, in addition to the poor reproducibility via no simulation results is the main reason for my weak accept. \n\n--------------\n\n***Post rebuttal***\n\nOverall, I am happy with the author response. After rereading, it is now much clearer what lessons the community can take away, and indeed, a study of this scale (in real world) has not been done before. I am happy to up my rating from a weak accept to a strong accept; I think many at CoRL would find this study interesting.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "- Some form of simulation results; this will make reproducibility of the method easier, especially as the robot used in this paper seem like a custom robot. If someone wanted to reimplement this idea, then they could use your reported simulation results to ensure they had implemented your method correctly. \n\n**Questions for the authors:**\n- The success rate for using just video and just language were interesting. I wonder, what would be the results of supplying both at test time? I.e. receiving two embeddings z, taking the mean, and then conditioning on the control net?"}}]

Author rebuttal:

Meta-review: Reviewers found the paper to have strengths in the significant effort put into the large-scale real-world evaluation, but weaknesses in the novelty of the proposed method. There is a consensus that the novelty is somewhat limited, and the proposed method is primarily a simple combination of existing methods brought together in a full system, rather than a significantly novel method itself. I would like to hear from the authors their opinion on how novel their proposed method is, beyond the overall system itself, or whether the authors believe the paper is primarily an empirical study, rather than a paper proposing a novel method. The large-scale real-world data collection and experiments are certainly encouraging, but Reviewer KYDU still argues that there is not a huge amount we can actually learn from this, since the proposed method still requires a very large number of demonstrations, and so results do not show any significant leap in data efficiency. Comparisons to existing alternative frameworks (rather than just ablations) on data efficiency, could be important in highlighting how this method sits with respect to alternative frameworks. So overall, the reviews are mixed. The system and data collection are impressive, but I would now like to see a discussion focussing on what the authors believe the key novelties are in the proposed method, and what the authors believe we, as a community, can learn from the method itself. I would also like to hear from the authors about whether they believe their method can truly be considered "zero-shot" when a demonstration still needs to be provided for a novel task, following the point raised by Reviewer aF8M.

------

The review process led to some lengthy and interesting discussions between the authors and reviewers. There is consensus, amongst both the authors and the reviewers, that the contribution of the paper is not in the novelty of the method (which is primarily an integration of a number of existing ideas into a system), but the implementation and evaluation of this system at such a large scale. The results of this are indeed useful for the community to know about, and will likely stimulate further discussions about how far we can really push behavioural cloning methods. All reviewers recommended to accept the paper, and I believe that we can all learn some useful insights from this work.
Decision: Accept (Poster)