Learning Multi-Stage Tasks with One Demonstration via Self-Replay
Authors: Norman Di Palo
Edward Johns
Venue: CoRL2021 Poster (2021)
Keywords: Imitation Learning
Multi-Stage Imitation Learning
Robot Manipulation

Abstract: In this work, we introduce a novel method to learn everyday-like multi-stage tasks from a single human demonstration, without requiring any prior object knowledge. Inspired by the recent Coarse-to-Fine Imitation Learning, we model imitation learning as a learned object reaching phase followed by an open-loop replay of the operator's actions. We build upon this for multi-stage tasks where, following the human demonstration, the robot can autonomously collect image data for the entire multi-stage task, by reaching the next object in the sequence and then replaying the demonstration, repeating in a loop for all stages of the task. We evaluate with real-world experiments on a set of everyday multi-stage tasks, which we show that our method can solve from a single demonstration.
Videos and supplementary material can be found at this webpage: https://www.robot-learning.uk/self-replay.

Learning Multi-Stage Tasks with One Demonstration via Self-Replay
Norman Di Palo
Edward Jonhs The Robot Learning Lab
The Robot Learning Lab Imperial College London
Imperial College London
United Kingdom di w.di-pall200iimeriaallll ac.uk
United Kingdom
johns@imperial uk
Abstract: In this work, we introduce a novel method learn everyday-lic like multi- stage tasks from single human demonstration, without requiring any prior object knowledge. Inspired by the recent Coarse-to-Fine Imitation Learning method, we model imitation learning as a learned object reaching phase followed by an open- loop replay of the demonstrator's 's actions. We build upon this for multi-stage tasks where, following the human demonstration, the robot can autonomously collect image data for the entire multi-stage task, by reaching the next object in the se- quence and then replaying the demonstration, and then repeating in a loop for all stages of the task. We evaluate with real-world experiments on set of everyday- like multi-stage tasks, which show that method can solve from single demonstration. Videos and supplementary material can be found at this webpage.
Keywords: Multi-Stage Imitation Learning, Manipulation
Introduction
One of the principal, long-term goals of robot learning, is to enable robot to learn new skill from human demonstrations in a simple and efficient way, without significant effort from the human, and without them requiring expert knowledge of the underlying algorithm. Imitation Learning techniques have been widely adopted recent years, but these methods often require a considerable human effort. In Sec. 2, we describe the shortcomings of techniques like Behavioural Cloning, Meta Learning, and Reinforcement Learning, that have hindered their widespread adoptions outside of laboratory settings. Sev- eral manipulation tasks involve multiple stages, e.g. grasping and then using a tool (Fig.1), 1), or re-arranging set of objects. But generally, the more stages task involves, the more diffi- cult it is to learn [1. 2], further exposing the limitations of the aforementioned techniques.
Figure 1:
An example of two stage task: picking up knife, and then using it to cut a toy vegetable.
In this work, we introduce method that allows human op- erator to teach a robot to solve a new, multi-stage manip- ulation task, with single demonstration, without the need for any prior data or knowledge about the object. Our method capable of solving series of everyday-like tasks, and can be effectively used by a non-expert, given its simplicity and need for minimal human effort. The main assumption is that most everyday manipulation tasks can be divided into two phases: coarse, object reaching phase, and fine and precise, object interaction phase (Fig. 1, 2). As such, multi- task can be decomposed into several stages, each of which alternates between these two phases. This decomposition forms the basis of our method, which allows the robot to collect all the experience it requires autonomously,
Sth Conference Robot Learning (CoRL. 2021), London, UK.

---

without requiring further human supervision, such as repeated environment resetting. We call our method Self- Self-Supervised Learning with Actions Replay, or Self-Replay. Self-Replay is inspired by the Coarse-to-Fine Imitation Learning method [3], which also models tasks with coarse and fine trajectories, but which can only solve single-stage tasks. Here, we develop a more general method that can learn wide variety of multi-stage tasks, whilst still only requiring a single human demon- stration.
Self-Replay is simple but effective. We evaluated our method on a set of real-world, multi-stage tasks, inspired by everyday primitive skills such as object grasping. object placing, object pushing, shape insertion, and food cutting. Our method is able to solve multi-stage combinations of these skills, whereas in the literature such tasks generally require least tens of demonstrations, or re- peated manual environment resets, or engineered laboratory setups [4, 5, 6. 7, 8]. We also provide a series of ablation studies and comparisons with existing techniques in the literature, to further understand the contribution of each component of our method. Our method only requires an uncal- ibrated wrist-mounted camera. While the training should take place in an uncluttered environment, we demonstrate how our method can tackle visual distractors at test time.
Our contributions are the following: (I) We introduce Self-Replay, a novel self-supervised learn- ing method to solve everyday, multi-stage manipulation tasks from a single human demonstration, with no additional human effort, prior knowledge, or engineering needed. (2) As well as extending
it to multi-stage tasks, we extend the data collection method proposed in [3] with a more efficient, active variant, that we call Active Self-Replay. (3) We designed a novel test-time pipeline that al- lows the robot to tackle multi-stage tasks, whilst also recovering from external disturbances and errors, and being robust to visual distractors. (4) We benchmark our method on a varied set of real- world, everyday-like manipulation tasks, and compare it against other methods from the recent robot learning literature.
2 Related Work
The recent Imitation Learning [9, 10, 11, 12] literature has proposed a series of techniques teach a robot novel skill, starting from demonstrations, in efficient way. However, most techniques often require considerable amount of work, prior knowledge, or supervision from a human operator. Behavioural Cloning methods [13, 4, 14, 5, 7, 6. 1, 15, 16] learn policy network directly from human demonstrations. While providing demonstrations is conceptually easy, simple tasks may require tens even hundreds of them, making the process laborious. When dealing with multi-stage tasks, the amount of demonstrations often increases considerably [1. 14]. Meta Learning methods [17, 18, 19, 20, 21] require one few demonstrations to learn to solve new task. However, they need large amounts of data collected before hand on similar tasks, another laborious process, and the amount of previously collected data also increases when tackling multi-stage tasks [21]. Plus, they can often only adapt to tasks that are very similar to what was observed during meta-training [17, 18, 19, 20]. Reinforcement Learning methods [2] can use demonstrations to bootstrap the agent's policy [22, 23] or shape its reward function [24, 25). Even if they require a few demonstration, the autonomous, reinforcement learning phase needs constant human supervision, as the environment need to be reset after each episode. Additionally, reward function needs to be shaped and provided at each time-step the agent, which may require additional engineering of the setup [2]. To solve multi-stage tasks, the recent literature has also proposed several methods that combine ideas from different fields: auto-regressive sequence modelling [26], neural task programming [27], exploiting spatial symmetries [28], hierarchical decomposition [1], explicit task planning [29], etc., but each technique requires considerable human effort in terms of either data collection or engineering.
3 Method
3.1
Background on Coarse-to-Fine Imitation Learning
One of the main concepts our method builds upon is shifting the imitation learning problem from learning model that emulates the demonstration in its entirety, to learning to align the end-effector to the pose where the interaction demonstration started, so that it can repeat the exact actions per- formed by the operator. The end-effector therefore needs to reach the same relative position to the object that it had at the start of the demonstration. Following this framework, introduced in [3] as

---

Sampling poses the workspace
Demonstration Self-Replay
Ao
A1
$60
Pb1
An example of the training steps for two-stage task: picking cup and placing it plate. The human operator brings the robot to the bottleneck pose on top of the cup (I), and records the object interaction phase by grasping the cup and lifting (2). The operator then brings the robot on top of the plate (3), and records the second object interaction phase (4), by bringing down the cup and releasing The operator then resets the environment by placing back the cup. The Self-Replay phase starts: the robot gathers data to learn how to reach $t,0 exploring the workspace (5). When the data collection phase ends, the robot moves to $6,0 and replicates the first object interaction by executing the recorded actions Ao (6). It then repeats these steps for the second stage (7,8). We suggest to watch the video on our website for full example.
Figure 2:
Coarse-to-Fing Fine Imitation Learning, the human operator provides a demonstration composed of two phases (Fig. 1, 2): they first bring the robot to pose close to the object, that is defined as the bottle- neck pose. The bottleneck pose is the pose of the end-effector at which the interaction phase starts. Intuitively, we can imagine it being fixed in the object's imaginary frame of reference, as it rigidly moves with it. From there, the operator starts the interaction phase, in which they demonstrate how to precisely interact with the object, e.g. picking up tool, inserting a plug, etc. (Fig. 1). The assumption, proved effective in [3], is that if the robot can reach the bottleneck pose accurately, hence being in the same relative pose to the object as was during the demonstration, then repli- cating exactly the actions recorded during the demonstration is sufficient to correctly interact with the object. The goal of the robot hence becomes to learn reach the bottleneck pose accurately on novel configurations of the environment, i.e. novel poses of the objects composing the task, without the need to explicitly learn a policy to model the interaction phase.
The bottleneck pose is not an explicit part of the object or of its frame, but is chosen by the operator when providing the demonstration based how they need to interact with the object. How does the human operator actually choose the bottleneck pose for each stage, separating the reaching and interaction phases? The bottleneck pose is arbitrary, and several choices are equally possible. However, as described in [3], during the interaction phase the actions are replicated in an open-loop manner, and external sources of noise can accumulate an error that proportional the length of the interaction phase. Hence, it is suggested to select a bottleneck pose that is sufficiently close to the object.
In a multi-stage task, the operator provides several interaction demonstrations, one for each stage (examples in Fig. 1, 2). Hence, we need to define one bottleneck pose per stage. We denote each bottleneck pose as 8t,N, where N denotes the N-th stage of the task, represented with the spatial coordinates Ta, yb: Zb, Ob. where 0 is the rotation around the vertical, z-axis. We extract this pose from the robot's kinematics, effectively being a pose in 3D with rotation around the z axis, being it the pose where the demonstration of the interaction phase starts. More details are provided in following sections. We denote = {I, y, 2, 0} as a position and orientation of the end-effecttr. We define as the observation, an RGB image that the robot receives from its wrist- mounted camera. Using a wrist-mounted camera, the observations are function only of the relative pose between the end-effector and the objects. This is a fundamental aspect, as it allows the robot to autonomously gather data that can generalise to novel situations, as described in later sections. Each pose 8 is computed using the kinematics of the robot. At training time we also have the exact pose of the bottleneck 8s,N, extracted from the human demonstration, hence we can always reach precisely through inverse kinematics. At test time this information unknown and must be deducted from observations. Hence the robot has to learn how to map each RGB observations
3

---

Stage N
Stage N
Reaching the roact moves toward
addmmaeed bottlemeck until 2)60)) predicts that has been reached.
Interacting with object
robot then replicatio demonstration actions.
Reaching the bottleneck
The steps repented thel task completed
bottleneck
n(or) +
Aj
D
Ot
Ty(ot) og
Ty(on) roached
Figure 3: A visual description of the method solving novel configurations of a task test time. Each stage is solved by first reaching the bottleneck pose, and then replaying the interaction actions. The image is passed through all the networks described in 3.2. The video in the Supplementary Material provides additional examples of this.
Or, where / denotes the current time-step, action de that moves the end-effector closer to 86, in any novel configuration of the task it needs to solve. The actions are the end-effector spatial velocities, angular velocity around the z-axis, and a binary command to open or close the gripper, ez Uy Ug C. All the actions are given in the end-effector frame.
3.2 Tackling Mtlti-Stage Tasks: Overall Architecture
In the previous section we described how Coarse-to- Saarse-to-Fine Imitation Learning [3] proposes to solve a novel configuration of task for which it received a single demonstration: the end-effector must learn to reach the bottleneck pose, the relative pose it had with the object at the beginning of the operator demonstration, and then it can replay the actions executed during the demonstration. To solve a multi-stage task, we need to extend this method to alternate series of bottleneck reaching and actions replay phases. To do this, we introduce novel framework of neural networks and novel supe-suue data collection algorithm, Self-Replay.
To solve a multi-stage task at test time, the robot needs to understand in which stage of the task it at any time, how to reach the bottleneck of the predicted stage, and if that pose has been reached with sufficient precision, so it can start the interaction phase. All this information must be extracted from an observation, 0g the form of an RGB image at each time-step step t, as at test time the absolute pose of the bottleneck is not provided, nor the stage that the robot is currently in.
We designed framework of three different networks do so: first, stage-recognition network, n(..)) - j E [0, N). that predicts the stage of the task the robot is currently in, in the form of a discrete class. This prediction conditions the remaining pipeline, as for each stage the bottleneck to reach is different. We predict the current stage at each time-step instead of simply starting from the first and shifting disturbances. If, for example, the end-effector drops an object during stage, it needs to realise that it must go back to a previous stage to pick it up again. We then use bottleneck reaching network т (o,)) + a,, that given the current RGB image captured from the what action to take in the end-effector coordinate frame to reach the bottleneck pose, sb.v. Finally, bottleneck classification network, network that predicts if the bottleneck pose has been reached, r(or) - {0, 1}. If the network predicts that 8h,j has been reached, i.e. is closer than a threshold, the robot executes the series of object interaction actions recorded during the demonstration for the j-th stage. The general pipeline is visually described in Fig. 3. Apart for the stage-recognition network, we train different network for each task stage, and use the stage prediction to select which one to use. We could have used a single network for each type, adding j to its inputs to condition its output, but we found training separate networks to result in better performance. In the following sections, we describe how Self-Replay can autonomously collect all the data needed to train the aforementioned networks, and hence learn to solve the multi-stage task, starting from single demonstration.
the next after each interaction phase be robust to failures and external
camera, predicts
Test Time Pipeline In Fig. 3 we visually describe the pipeline at test time, starting from an RGB observation or The networks collaborate to predict (1) in which stage the robot is actually in, hence understanding what bottleneck pose should be reached (2) how the robot should move to approach

---

the bottleneck of the predicted stage (3) if the bottleneck has been reached, and hence it is time to execute the interaction phase to complete the stage. If the bottleneck considered non reached, the actions computed by (T:) are executed by the robot, and then a new observation is obtained. All these estimations are computed from the same input observation or and recomputed independently at each time-step.
3.3 Providing the Demonstration
The human operator only needs to provide a single demonstration of the task, and then reset the environment as it was before the demonstration. No additional actions are needed from the operator. For each stage of the task, the human operator first brings the robot's 's end-effector to the object's s bottleneck pose S6,N through free-space motion. In this phase, only the final bottleneck pose is recorded. They then demonstrate how to interact with the object moving the robot's end-effector. In this phase, all the actions commanded by the operator recorded as Ay a where N denotes the N-th stage. The operator then repeats these steps for all the stages composing the task. An example can be seen in Fig. 2.
The human operator then needs to reset the environment only one time as was at the beginning of the demonstration, as the object interaction phases may have changed the initial configuration of the objects. This is in strong contrast with other approaches, like reinforcement methods, that require hundreds or thousands of resets (2), or classic behavioural cloning methods, in which the human operator needs to provide tens of demonstrations [5, 8]. resetting the environment each time.
boaring-basee
3.4 Collecting Data via Self-Replay
After providing a single demonstration of the full task and resetting the environment, bringing the objects to their starting position, all the following data collection and training phases are self- supervised, hence the human operator can disengage and no more human interventions are needed.
As described in Sec. 3.1, 3.2, the agent needs to collect data to train the models that will allow to accurately reach the bottleneck pose of each stage from any environment configuration and starting pose. As described before, the use of a wrist-mounted camera makes the RGB observations a function of the relative, and not absolute, pose between end-effector and objects. Hence, moving the end-effector is generally equivalent to moving the object to observe it from different poses. To gather data in self-supervised fashion we use a method we call Self-Replay.
Random Self-Replay Our algorithm extends the method proposed in [3] by being able to au- tonomously shift between stages to autonomously gather all the data it needs to solve the multi-stage task. We also propose novel active variant in 3.4. The general approach is to sample random pose of the end-effector in the workspace, and from there linearly go back to the bottleneck $b,N, while recording the observations encountered in the path. The robot explores the area above the bottleneck, to avoid collisions with the objects, and we assume that area to be free from obstacles. More precisely, at each time-step the agent stores data in the form (ot, Se,t, Sb.N. .N. where o is the image taken from the camera in that pose, Se = Fe, ye: Ze, 8g is the pose the end-effector at the current time-step, 86,N is the bottleneck pose, and N denotes the stage of the task we're we're currently in as an integer. For each pose we can accurately compute the ground-truth relative displacement between the current end-effector pose and the bottleneck, and therefore analytically compute what end-effector movement would bring it closer to the bottleneck. This allows us to then train a model to predict the optimal movement from an RGB observation. This method of data gathering is defined Random Self-Replay, as we randomly sample poses in the workspace to explore different relative poses between end-effector and objects. While randomly sampling poses would be enough, we propose more efficient and robust active exploration strategy, called Active Self-Replay, that we describe in Sec. 3.4. When the data collection of task stage completed, the robot needs shift to the next stage. To do so, it reaches the bottleneck pose S6.N, and then replicates the recorded actions AN to interact with the object (e.g. moving an object, grabbing an object, etc.). After this phase, the algorithm goes to the next task stage N + N + 1, repeating the aforementioned steps 1. By autonomously moving between stages, the robot
(Fig. 2), with the next bottleneck pose 11 can gather all the data it needs for all the stages without any additional human intervention.
5

---

Cut
Pan
IaR PICK
Stack
Figure 4: The set of tasks we tested method Additional information can be found in the Supplementary Material and the video on our website.
Active Self-Replay We additionally propose an active, self-supervised data collection method. While both methods alternate between collecting data in a self-supervised fashion and then replay- ing the demonstration's is actions shift the next stage, the steps of data collection are different. The first stage is identical to the Random Relf-Replay phase described before. It collects data re- peating the Random Self-Replay cycle M times, gathering an initial dataset. However, differently from Random Self-Replay, we then train our models on this data also during the training phase: after having trained the networks on this initial dataset, the robot will then sample poses from its workspace that maximise the error of the networks, hence areas that have not been explored enough and require more training data. This allows the robot to actively collect more informative data, in- stead of re-sampling already explored areas. We use a gradient-free sampling based optimiser, and provide more details on the optimisation procedure we implemented in the Supplementary Material.
The data collection and network training steps are repeated until the robot is unable to find a pose in its workspace where the network's 's error is larger than a threshold herrs after maximum number of steps defined beforehand.
4 Experiments
We evaluated our proposed method on a set real-world, everyday multi-stage manipulation tasks. Based these experiments, now answer the following questions: (1) Is our method able to solve a series of common, multi-stage tasks with single demonstration? (2) the Active Self-Replay algorithm more efficient and robust than the Random variant? (3) How does
method compare to similar state-of state-of-the-ar methods from the literature? (4) Is reproducing the actions executed by the human operator during object interaction enough to solve the task, or would a learned policy perform better?
Additional experiments are presented in the Supplementary Material, including a comparison with Reinforcement Learning from Demonstrations, more detailed analysis on the time and sample efficiency of our method compared to other baselines from the literature, comparison of replaying the operator's demonstration against learning it with a policy, and an investigation of the robustness of our method to distractors. Additional details regarding neural networks and algorithms, and the code for this work, can be found in the Supplementary Material.
4.1 Experimental Setting and Tasks
Due to the governmental restrictions during the COVID-19 pandemic, we were unable to conduct experiments in our institution's laboratory. Therefore, we decided to set up a robotics laboratory at home, building smaller, 6-DOF manipulator with a parallel gripper: the TinkerKit Braccio. Despite the small size of this robot, we were able to conduct all the experiments successfully and answer the above questions, albeit with reduced workspace. Nevertheless, the results obtained here can effortlessly be replicated on a larger robot. We use wrist-mounter RGB camera, that captures 800 x 600 pixels images, which we resize to 64 x 64. We control the robot using a keyboard. Our setup is lightweight and easy to reproduce, only requiring a wrist-mounted RGB camera with no calibration needed. All the networks in this work are convolutional neural networks trained with supervised learning on a single GPU.
Manipulation Tasks We designed a set of 8 tasks, inspired by the recent imitation learning literature [3, 13, 5, 7, 8], and by everyday tasks to test our method. We included variety of common multi-stage human tasks, like picking-and-placing cutting food, shape insertion, etc. (Fig. 4). Detailed information can be found on our website and Supplementary Material. To generate a test set for the experiments, we instantiate novel configurations of each task by moving the objects in the

---

Table 1: Percentage of successes of the variants of our method, Random Self-Replay (R. S-R) and Active Self-Replay (A. S-R) and a series of baselines on 20 configurations of each tasks. Experi- ments are repeated over random seeds and show mean and 1-std.
Task
Method
Cup
Fork
Stack
Cut
Insert
Push
Pan
Ham.
FlowC.
40 441.1
24 +:2.3
13 4:9
5 #4.1
2.3
13 2.3
12 22.3
18 4:2.3
SIFT
4:2.3
0
0 =00
0 +0
0
0
0::0
3 4:2.3
LMP-1
4:2.3
+0
0 0:0
0 ++0
0
0
0
0:0
RIL-1
0::0
0
0::0
0::
0
322
2
2
LMP-20
22.3
48 16.
38 162
23
20 14.0
33 22.3
$7.700
53
RIL-20
60:44.0
52 14.7
38 14.
17 12.3
123
37 2233
162
52
DDPGID
2+
0+
0+0.
0+0.
0+
0+
0+0.
7
BC-20
6.2
2.0
12+2.3
7.7
2.2
2.0
283
42 10.
R. S-R
83
4.7
583
2.3
4.7
63
62
62 2.3
A.
2.3
83 2.3
4.7
73
4.7
4.7
73747
73 13110 4.7
workspace. Position are sampled inside the 30em x 20cm workspace, and orientation is sampled in 120 degrees range. The end-effector starts 15 em above the table. all the tasks, the robot end-effector move in 3D space and only rotate around the vertical z-axis. For each task, the success criteria is judged by the human operator at the end of the last interaction phase. We provided examples of successes and failures in the Supplementary Material.
4.2 Results
Learning Multi-Stage Tasks from a Single Demonstration We tested our method by providing single demonstration of each task. Recording the demonstration takes roughly a minute to the human operator. video detailing the experiments can be found on our website We compared Active Self-Replay and Random Self-Replay, (Sec. 3.1) (Table 1). Both variants are trained on the same amount of self-collected data, which collected autonomously by the robot, following the methods described in 3. We then used the data to train the networks described in Sec. 3.2, keeping the architecture, number of parameters and random weights initialisation fixed on both Active and Random variants. We then set up test set of 20 random initial configurations of each task by placing the objects randomly in the environment, testing both methods on this same test set and averaging the results over 3 random seeds. At test time, the robot behaves following the method we described Sec. 3.2 and Fig. 3. In our experiments we observed how, although Random Self- Replay achieves strong performance, failed some particularly challenging configurations of the harder tasks, e.g. when the difference in orientation of the object with respect to what observed during the demonstration was large. Indeed, observe improvements of Active over Random Self- Replay on the harder tasks, like Cut, that requires precise orientation of the knife, and Insert, which also requires precise orienting the object above the shape (Table 1).
Comparison with Baselines from the Literature. We also compared Self-Replay to series of techniques from the literature, to highlight the improvements of our method. Further details these experiments can be found in the Supplementary Material. We divide the baselines we developed and compared into three categories:
One Demonstration Methods: these methods are the most similar in design to our method, as they only require a single demonstration from the human operator, and additional prior or posterior work. These methods are FlowControl [30], a recent statc-of-the-ar technique that used a learned optical-flow model to compute how to align the end-effector, from its current observation, to the bottleneck pose. We also compared a non searning-based to automatically extract matching features from the current observation and the bottleneck pose observation, used to compute how to align the end-effector to the bottleneck pose.
eeypoint-detectin algorithm, SIFT [31],
methods from the re- cent literature to learn multi-stage tasks from demonstrations: Latent Motor Plans (LMP) [14] and Relay Imitation Learning (RIL) [1]. Both these methods use human demonstrations to learn goal- conditioned policy, that can also learn to decompose a longer task into sub-tasks. We also compare these to classic Behavioural Cloning (BC).
2 Multi-Stage Behavioural Cloning: we implemented two state-of-ther
7

---

3. Reinforcement Learning from Demonstrations: we developed Deep Deterministic Policy Gra- dient from Demonstrations (DDPGfD) [22, 23), that takes human demonstrations train a policy, and then uses autonomous Reinforcement Learning to increase its performance. Although the agent autonomously explores the environment during the RI. phase, human supervision is still needed to reset the environment after each episode. We use around 30 minutes of exploration, the same time we spend providing demonstrations to BC methods.
We compare our results on a series of everyday-like tasks in Table 1. We provide I and 20 demos to the Behavioural Cloning methods. We provide 1 demonstration to DDPGfD, as the human operator also has to spend a considerable amount of time supervising the autonomous exploration phase and resetting the environment. We show that all the baselines fail when receiving only a single demonstration. LMP and RIL perform better than BC, while DDPGfD is unable to solve these tasks consistently. In the Supplementary Material, we also compare the time efficiency of these methods, as motivated by [32], and their sample efficiency. Here, we show that our method is not only more time efficient for the human operator, but can also achieve better performance when using the same number of datapoints than state-of-the-e thrt---rr- baselines. Those experiments additionally show that RIL and LMP require around 40-50 demonstrations to reach the performance we obtain with a single demonstration, hence being 40x to 50x more time expensive than Self-Replay.
Comparison with Behavioural Cloning on an Increasing Number of Stages
We designed a set of experiments to compare the efficiency and performance of our method on tasks with increasing number of stages while still receiving only single demon- stration, with respect to Behavioural Cloning (BC), where instead the human operator pro- vides several demonstrations of the task. In partieular, we compare it to Relay Imitation Learning (RIL) [1]. a recent variation that extends Behavioural Cloning to multi-stage tasks. We designed stacking task (Fig. 4), where the robot has stack varying number of cubes in the correct order, where we can easily add more stages by using more objects. In our method, we only provided a single demonstration, run- ning Active Self-Replay to autonomously gather data. We compared to RIL with 10 and 30 demonstrations. For RIL, we provided end-to-end demos of the multi-stage task from 10 or 30 different initial configurations of the environ- ment, recording RGB observations and actions in the end- effector frame the form of {(ou, ao)...., (or, ar) We then trained a policy network fB with supervised learning on these demonstrations to map [Bt( + ag We used the same network architecture we use our bottleneck reaching network (Sec. 3.2).
Table 2: Percentage of successes of our method against end-to-end Behavioural Cloning method, Re- lay Imitation Learning, with 10 or 30 demonstrations, over 20 config- urations of each task.
Stages
Method
2
4
Ours
100
90
80
In Table 2, 1 stage corresponds to only grasping the first ob- ject, 2 stages also placing it on a goal x, y position on the table, 4 stages to also place the second object on top of the first. We test the two methods on the same test set of 10 con- figurations of the task, measuring the number of successes. We show how Behavioural Cloning, extended to tackle multi-stage tasks with the Relay Imitation Learning method, quickly starts to de- grade in performance when the number of stages increases, while our method always obtains large number of successes.
RIL-10
60
40
15
RIL-30
90
75
35
Conclusions
In this work, presented a novel method to learn solve multi-stage, everyday-like manipulation tasks from a single human demonstration. We build on the foundation of [3], which introduced the concept of Coarse-to-Fine Imitation Learning: we introduce both novel learning method, Self- Replay, and a framework of neural networks to tackle multi-stage tasks. We empirically demon- strated how our method can solve wide range of tasks, often used in the literature as benchmarks, while requiring minimal human work, no prior knowledge, engineering, or modelling of the objects manipulate.
8

---

Acknowledgments
This work was supported by the Royal Academy of Engineering under the Research Fellowship scheme. We wish to thank the reviewers who provided insightful comments that allowed to improve the paper significantly. We also wish to thank Vitalis Vosylius for his feedback initial draft of the paper.
References
[1] A. Gupta, Kumar, C. Lynch, Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.1195: 2019.
[2] M. Riedmiller, R. Hafner, T. Lampe, M. Neunert, J. Degrave, T. Wiele, V. Mnih, N. Heess, and J. T. Springenberg. Learning by playing solving sparse reward tasks from scratch. In International Conference on Machine Learning, pages 4344-4353 PMLR, 2018.
[3] E. Johns, Coarse-to-fine imitation learning: Robot manipulation from a single demonstration.
In IEEE International Conference on Robotics and Automation (ICRA), 2021.
[4] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality telcoperation. 2018.
[5] A. Zhan, P. Zhao, L. Pinto, P. Abbeel, and M. Laskin. A framework for efficient robotic manipulation. arXiv preprint arXiv:20/2.0797 2020..
[6] Levine, C. Finn, Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies.
The Journal of Machine Learning Research, volume 17, number 1, 1334-1373, 2016.
[7] Florence, L. Manuelli, and R. Tedrake. orrfssppppriieeddecrrroooooo in visuomotor policy learning. IEEE Robotics and Automation Letters, 5(2): 499, 2019.
[8] R. Rahmatizadeh, P. Abolghasemi, L. Bölöni, and S. Levine. Vision based multi-task ma- nipulation for inexpensive robots using end-to-end learning from demonstration. 2018 IEEE Internationaa Conference on Robotics and Automation (ICRA), 3765, IEEE, 2018.
[9] S. Ghasemipour, Zemel, and Gu. divergence minimization perspective on imita- tion learning methods. Conference Robot Learning. 1259-1277, 2019.
[10] T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, and J. Peters. An algorithmic perspective on imitation learning, arXiv preprint arXiv:1811.06711 2018.
[11] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), volume 50, number 2, 1-35, 2017.
[12] S. Ross, G. Gordon, and Bagnell. reduction of imitation learning and structured predic- tion to no-regret online learning. Proceedings of the fourteenth international conference on artificial intelligence and statistics, 627-635, 2011.
[13] N. Di Palo and E. Johns. Safari: Safe and active robot imitation learning with imagination. arXiv preprint arXi::2011.09580 2020.
[14] C. Lynch, M. Khansari, T. Xiao, V. Kumar, Tompson, S. Levine, and Sermanct. Learning 2019.
latent plans from play. arXiv preprint
[15] C. Finn, X. Y. Tan, Duan, T. Darrell, S. Levine, and Abbeel. Deep spatial autoencoders for visuomotor learning. IEEE International Conference Robotics and Automation (ICRA), 519,519, IEEE, 2016.

---

[16] M. Laskey, J. Lee, R. Fox, A. Dragan, and K. Goldberg. Dart: Noise injection for robust imitation learning. arXiv preprint arXiv / 703.09337, 2017.
[17] Y. Duan, M. Andrychowicz, B. Stadie, O. J. Ho, Schneider, Sutskever, P. Abbeel, and
W. Zaremba. One-shot imitation learning, Advances in neural information processing systems, 1087-1098, 2017.
[18] S. James, M. Bloesch, and A. Davison. Task-embedded control networks for few-shot imi- tation learning. arXiv preprint arXiv: 1810.03377 2018.
[19] T. Yu, C. Finn, A. Xie, S. Dasari, T. Zhang. P. Abbeel, and S. Levine. One-shot imitation from observing humans via addaine meta-learning. arXiv preprint urXiv: 1802.01557. 2018.
[20] C. Finn, T. Yu, T. Zhang. P. Abbeel, and S. Levine. One-shot visual imitation learning via meta-learning. arXiv preprint IrXiv:/709.044 04905, 2017.
[21] T. Yu, P. Abbeel, S. Levine, and C. Finn. One-shot hierarchical imitation learning of compound visuomotor tasks. arXiv preprint arXiv: 2018.
[22] M. Vecerik, Hester, J. Scholz, Wang. O. Pietquin, B. Piot, N. Heess, T. Rothörl, Lampe, and M. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint (rXXiv:707.088111 2017.
[23] A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 6292-6299, IEEE, 2018.
[24] J. Ho and S. Ermon. Generative adversarial imitation learning. Advances in Neural Information Processing Systems, pages
2016.
[25] K. L. J. Fu and S. Levine. Learning robust rewards with adversarial inverse reinforcement learning. International Conference on Learning Representations, 2018.
[26] S. Pirk, K. Hausman, A. Toshev, and M. Khansari. Modeling long- horizon tasks as sequential interaction landscapes. arXiv preprinl arXiv:2006.04884 2020.
[27] D. Xu, S. Nair, Zhu, J. Gao, A. Garg, L. Fei-Fei, and Savarese. Neural task programming: Learning to generalize across hierarchical tasks. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3795-3803 IEEE, 2018.
[28] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. arXiv preprint arXiv:2010. /4406, 2020.
[29] D.-A. Huang, D. Xu, Y. Zhu, A. Garg, S. Savarese, L. Fei-Fei, and J. C. Niebles. Continuous re- laxation of symbolic planner for one-shot imitation learning. arXiv preprint NtXX:::19080066600 2019.
[30] M. Argus, L. Hermann, J. Long, and T. Brox. Flowcontrol: Optical flow based visual servoing. arXiv preprint M:Xi::20007.00291,00000
[31] D. G. Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60(2):91-110, 2004.
[32] E. Johns. Back to reality for imitation learning. In Conference on Robot Learning (CoRL), Blue Sky Track, 2021.
[33] E. Valassakis, N. Di Palo, and E. Johns. Coarse-to-fine for sim-to-real: Sub-millimetre preci- sion across the workspace. arXiv preprint 1tXX::2005.1128.3 2021.
10


Reviews:
[{"review_id": "yM9inLqyEiW", "reviewer": "Reviewer_yNGA", "summary": "This paper extends previous work [1], to effectively solve multi-stage manipulation tasks from a single demonstration. The human expert provides a single kinesthetic task demonstration and decomposes it into bottleneck frames. The robot reaches bottleneck frames, replays the transformed demo at that frame, moves onto the next bottleneck, and repeats until the task is over. The real meat of this method is in the self-supervised tricks and active data collection required to  make error free bottleneck predictions from vision. This process requires minimal human effort and clearly yields huge dividends: the proposed work is far better than behavior cloning at multistage tasks despite using fewer demonstrations.\n\n\n[1] Johns, Edward. \"Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration.\" arXiv preprint arXiv:2105.06411 (2021).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* The paper is well written and convincingly explores the design decisions needed to make this method work in practice. Specifically, the improvements to random self replay and active components are sensible ways to boost performance.\n* The experiments consider a great deal of tasks and strongly support the hypothesis. The inclusion of robot results despite COVID is a big plus.\n\n\n**Weaknesses**\n* The paper could be seen as a relatively straightforward extension of prior work. I believe the added components are significant enough, but would like to see more thorough ablations done.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper proposes a non-trivial suite of self-supervised methods in order to extend Coarse-to-Fine imitation learning [1] to multi-stage tasks. By definition this work is incremental, but thorough robot experiments help make up for that. This is just enough to warrant acceptance. However, I should stress that more ablations and baselines could make the work significantly stronger. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "* I'd add more ablations to show how your method compares vs more naive implementations. For example, what would happen if you omitted the \"stage transition\" in random self-replay and just predicted where all the bottlenecks were? This is a simpler method but would likely fail due to object occlusions that arise after the first stage. Seeing these comparisons would be helpful.\n* Related to prior point - it's hard to tell if Active Self-Replay is really helping over the more naïve variant. One explanation is that the workspace (from video) is too limited to make difference seem apparent. Testing on more configurations/in wider workspace could make this difference more significant.\n* This work does not compare against many prior imitation learning baselines. After all, there are prior network architectures and methods (e.g. [2,3]) that improves on behavior cloning for more complicated multi-stage tasks. Thus, more baselines would be helpful.\n\n[2] Lynch, Corey, et al. \"Learning latent plans from play.\" Conference on Robot Learning. PMLR, 2020.\n\n[3] Yu, Tianhe, et al. \"One-shot hierarchical imitation learning of compound visuomotor tasks.\" arXiv preprint arXiv:1810.11043 (2018)."}}, {"review_id": "ej8NnvLyKh_", "reviewer": "Reviewer_ViCo", "summary": "The authors propose to learn policies from a single demonstration and multiple bottleneck states (identified by the demonstrator). The robot is then reset to random states from which it learns to reach the next bottleneck state. The robot is able to do so on its own by \"linearly interpolating from random state\" to bottleneck state. They show this strategy allows them to learn policies quickly on a real robot on a number of manipulation tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1) The proposed approach is interesting, modular, and general enough to be applied to many robotics problems.\n2) Real-world robot experiments on multiple tasks to validate their approach.\n3) Nice ablation study of how behavior cloning succeeds on tasks with less stages but as the stages increases more complicated algorithms are required.\n\nWeakness\n1) One of the new contributions of this work has been explored before in [1]. In [1] the authors suggest having checkpoints with which one could reset the environment from which the agent can explore. This is similar to the Self-Replay part of this work where the authors start exploring from pre-defined checkpoints or bottleneck states.\n\n2) The proposed idea seems difficult to scale to multiple demonstrations or handle variations in workspace without manual intervention. For example, the authors train a bottleneck classifier treating this one state as a positive and other states as negative. If at test time the configuration of objects is different it would make it difficult for the bottleneck classifier to work properly.\nThere is some recent work on unsupervised video representation learning [2] which claims that key events and phases can be discovered in a self-supervised way. The authors might consider employing something like this to get a scalable bottleneck state from multiple demonstrations.\n\n3) Evaluation missing relevant baselines:\nThe baseline considered in this work is FlowControl which is a policy learning method that uses optical flow. It is not clear how that is the best baseline for the proposed approach that tackles long-horizon tasks. Approaches like [3],[4] or [5] which solve long horizon tasks by breaking it into smaller tasks seem more suited here.\n\n4) How many extra interaction steps is required by the proposed algorithm in the Random SelfPlay stage to achieve good performance? The lower bound seems to be pure behavior cloning where no extra interaction steps is required. It will be good to see trend of that graph, so that we can estimate if better performance can be achieved by more steps in the SelfPlay stage.\n\nReferences\n\n[1] Ecoffet, Adrien, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. \"First return, then explore.\" Nature 590, no. 7847 (2021): 580-586.\n\n[2]Dwibedi, Debidatta, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. \"Temporal cycle-consistency learning.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1801-1810. 2019.\n\n[3] Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations. Ajay Mandlekar, Danfei Xu, Roberto Martín-Martín, Silvio Savarese, Li Fei-Fei\n\n[4] D. Xu, S. Nair, Y. Zhu, J. Gao, A. Garg, L. Fei-Fei, and S. Savarese. Neural task programming: Learning to generalize across hierarchical tasks. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3795–3802. IEEE, 2018.\n\n[5] Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, Karol Hausman. Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While the paper is quite well-written and has real-world experiments for multiple (which are difficult to setup), the method is not scalable, missing some baselines comparisons and has limited technical contribution. It was however interesting to see their proposed approach get good performance for manipulation tasks in the real world. Hence I am not recommending the acceptance of the paper in its current form.\n\n**UPDATE**\nI thank the authors for newer experiments and additional analysis. It definitely adds more value to the paper. \n\nIt seems one factor that makes their method generalize using few samples is the use of a camera on the wrist. It would be important to show if the method can generalize to new configurations from third-person views or the wrist-view is crucial for success. If it is, it should be reflected in title/abstract.\n\nBy scalable I meant addressing the scenario when multiple demonstrations are available? Since it is a leaning based method, there has to be configurations (single demonstrations) that make it difficult to generalize to newer (out of distribution) settings.\n\nHowever with the newer experiments, I can recommend a weak accept if the authors include limitations of their approach. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "1.Comparison with relevant baselines will help the paper. ", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "TdxRAv24sok", "reviewer": "Reviewer_tkPW", "summary": "This paper tackles multi-stage tasks given a single demonstration. In order to do so, the algorithm is composed a stage-recognition network, which classifies which stage the agent is in; a bottleneck reaching network (one for each stage), which gives a policy to reach the bottleneck state; and a bottleneck classification network (one for each stage), which predicts whether the bottleneck state has been reached. At each bottleneck state, the robot simply replays the actions from the demonstrations to complete the stage. The data to train these networks is collected in a self-supervised manner.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- This work studies imitation of a multi-stage task from a single demonstration under high-dimensional image observations. This is a challenging setting and is highly relevant to the CoRL community. Additionally, the experiments are conducted on a real robot.\n\n- The proposed approach only requires 30-60 minutes of self-supervised data collection before it learns to successfully perform the demonstrated task. Hence, the only human supervision required is a single demonstration with labels for each stage and the corresponding bottleneck state.\n\n- The trained networks are able to generalize to new configurations of the scene due to training on observations collected from a wrist camera, which does not tie the observations to a specific configuration of the objects. This is a pretty neat trick and naturally allows the robot to generalize to new configurations of the scene.\n\nWeaknesses:\n- The experimental evaluation only includes one comparison to prior work on a pair of tasks, so it’s difficult to understand how well this compares to existing systems for this setting. The main text points to the appendix for a comparison to RL from demonstrations, but the results are missing.\n\n- It’s also unclear how significant the results are from just the reported success rates. For example, the difference between Random Self-Replay and Active Self-Replay is just one more successful trial out of ten on some of the tasks. It would be helpful to test on a larger number of configurations.\n\n- This approach assumes that prior to achieving the bottleneck pose for an object, the object remains fixed in the scene. However, it seems like the robot could potentially disturb the object during the self-supervised data collection. Currently, the actions used to reach the bottleneck pose are the difference between the bottleneck and current pose, which would be fine if the direct trajectory to the bottleneck pose is obstacle-free and does not disturb the objects. Otherwise, the robot would likely move the objects and need more human intervention.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I’m recommending a weak accept. The overall results are impressive for a difficult problem, but it’s difficult to assess the performance of the approach and the difficulty of the tasks without more comparisons.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "- It’s noted that training separate policies for each stage leads to better performance, compared to training a single policy conditioned on the task id. What if the policy was conditioned on the bottleneck state? Would this joint training help?\n\n- In the self-supervised data collection phase, presumably the robot can disturb the scene. For example, the cup in the Cup Pick & Place task could be knocked over. Does this not require human intervention to correct the scene?\n\n- How sensitive is the performance to the choice of the bottleneck poses? It’d be useful to see how different bottleneck choices for the same demonstration affects performance.\n\n- It would be helpful to break up Sec. 3.4 by outlining the objectives used to train the networks.\n\n- The link to the website seems to be missing from the paper."}}, {"review_id": "NBVf75SD16", "reviewer": "Reviewer_qL1E", "summary": "* This work extends \"coarse to fine\" imitation learning to the multi-stage setting.\n* Introduces a way to scale up demonstration collection for multi-stage tasks with the following structure: 1) \"move to bottleneck 1\" phase, e.g. move in free space to pregrasp of an object, 2) \"bottleneck 1\" replayable open-loop object interaction, e.g. picking up a fork, 3) \"move to bottleneck 2\", e.g. carry an object above a plate, 4) \"bottleneck 2\" replayable open-loop object interaction, e.g. place the fork on a plate.\n* Turns one demonstration for a particular object configuration into many (for the same particular object configuration) using the following scheme: autonomously reach the first bottleneck pose from many different positions, replay the bottleneck 1 actions, autonomously approach bottleneck two from a variety of poses, replay bottleneck 2 actions.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n* Well motivated problem setting: collecting demonstrations for multistage tasks is cumbersome and time-intensive.\n* This work introduces a clever and scalable way to collect many demonstrations from only one, assuming certain (fairly strict) requirements about the tasks and environment are met. I can imagine some important commercial applications that meet the requirements of this method.\n* Clear description of the method and assumptions around task structure.\n* Compelling qualitative results on a real robot.\n\n# Weaknesses\n* **Introducing strict assumptions, but not adequately discussing them.** A significant limitation of the setup is that the set of automatically created demonstrations are only valid for one particular object configuration. The paper addresses this by introducing a different assumption: that the agent perceives the scene only through a wrist-mounted camera. The wrist-mounted camera \"makes the RGB observations a function of the relative, and not absolute, pose between end-effector and objects. Hence, moving the end-effector is generally equivalent to moving the object to observe it from different poses.\"  This is furthermore only true when there is no clutter or visual variation in the background. If there is, then the RGB observations of even the wrist camera depend on the absolute position of the scene and the particular configuration of the background. This assumption is never stated but it is nevertheless important. Requiring a wrist-mounted camera (and only a wrist-mounted camera) as well as an uncluttered scene are rigid assumptions. The issue is, without these assumptions, the autonomous data collection becomes far less useful as it is indexed on a particular object configuration---changing the object configuration even slightly requires recollecting all new data.  It's fine to restrict attention to the setting of \"wrist-mounted visual manipulation in uncluttered scenes\", but this set of assumptions should be very clearly stated early on in the introduction. L124 in Section 3 mentions how important the wrist-camera is to the method: \"[the wrist-mounted camera] is a fundamental aspect, as it allows the robot to autonomously gather data that can generalise to novel situations, as described in later sections\". The intro should be significantly reworded to accurately convey these (significant) limitations to the reader.\n\n* **Insufficient discussion of the other important sources of variation in imitation**. It is true that imitation learning requires a lot of human effort, and it is a good goal to reduce this effort. But this paper does not appear to address the important sources of effort in behavioral cloning. It provides a way to collect a very large number of demonstrations for one particular scene configuration, covering different initial poses and different trajectories between bottleneck states. Covering pose variation is important, but only *one* of the many kinds of variation that necessitate a large number of demonstrations when doing behavioral cloning. Especially in the \"everyday\" setting this work claims to operate in, multiple demonstrations are needed to cover changes in the initial positions of the objects, changes in lighting conditions, changes in background, presence of distractor objects etc. If any of these change, all the data automatically collected by this method seems just as brittle as a large set of manually collected demonstrations from a single world state. This is important missing context. Without this context, statements like \"...we introduce a novel method to learn everyday-like multistage tasks from a single human demonstration\" seem quite strong, as it does not appear to actually improve upon the most important sources of \"everyday\" variation.  \n\n\n* **No limitations discussed on just how coarse the \"coarse\" phase can be**. The paper states that it can automatically expand a demonstration set with \"without requiring any prior object knowledge\". It is hard to imagine how this is true for, say, a cup of coffee, a bowl of candy, a plate with food on it, etc, without requiring object-specific specific scripting like “the container needs to remain upright\". Clearly how you script the automatic \"move to random poses\" phase of collection depends on the object you are dealing with and the scene you are operating in. Otherwise the robot may do unsafe things like collide with objects. Language like \"without requiring any prior object knowledge\" (which appears multiple times in the paper) should be significantly toned down or the assumptions about which objects this method pertains to should be made more explicit.\n\n* **Strong conclusions from weak experiments.** Experiments are reported without standard deviation over multiple training seeds. Without knowing the standard deviation of the success of the methods, how can we tell if we are observing an actual improvement or just sampling error? \"Results therefore show that learning the interaction phase from demonstrations does not provide any *statistically significant* benefit\" (emphasis mine). The experiments do not currently carry enough information to assess statistical significance. This is also quite a broad conclusion given the major assumptions around the task setup.  \"In our method, we only provided a single demonstration, running Active Self-Replay for around 40 minutes to autonomously gather data. We compared it to BC with 10 and 30 demonstrations.\" How many minutes of data were the 10 and 30 demonstrations? Was your method simply trained on more data? Without knowing this it's difficult to assess the benefit of the method. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "UPDATE:\nIn light of the following, I'm updating my score:\n* The authors updated the paper to clearly state the strict dependencies of the method upfront (uncluttered scenes at training time, wrist mounted camera).\n* They added new experiments showing that despite these assumptions, the method handles novel configurations of the task even in the presence of unseen visual distractors.\n* They added seeded real world evaluations that help communicate the significance of the quantitative results.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "* The paper would benefit from clarifying the actual sources of task variation that this work addresses. Clearly state what sources of variation it does not address. The claims that this solves \"everyday\" tasks should be softened substantially if it does not indeed address everyday sources of task variation.\n* The paper would additionally benefit from clarifying the strict dependencies of this method early on (wrist-mounted camera, uncluttered scene). \n* Claims of statistical significance should not be made without experiments that support those statements. Without rigorous experiments, it is difficult to assess the quality of the proposed method."}}]

Author rebuttal:

Meta-review: In this paper, the authors propose a method to solve multi-stage task from one demonstration. Multiple bottleneck states are defined and recognized. It is well recognized that the system is able to collect scalable data in a self-supervised fashion. However, there are some concerns about how to handle multiple demonstrations, some practical situations, etc. Also, more comparison experiments with stronger baselines are lacking.  

In the rebuttal session, the authors have responded to the reviewers' concerns and more details and baselines are provided. And the reviewers have reached a consensus of accepting this paper.
Decision: Accept (Poster)