Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper tackles multi-stage tasks given a single demonstration. In order to do so, the algorithm is composed a stage-recognition network, which classifies which stage the agent is in; a bottleneck reaching network (one for each stage), which gives a policy to reach the bottleneck state; and a bottleneck classification network (one for each stage), which predicts whether the bottleneck state has been reached. At each bottleneck state, the robot simply replays the actions from the demonstrations to complete the stage. The data to train these networks is collected in a self-supervised manner.
strengths_and_weaknesses: Strengths: 
- This work studies imitation of a multi-stage task from a single demonstration under high-dimensional image observations. This is a challenging setting and is highly relevant to the CoRL community. Additionally, the experiments are conducted on a real robot.

- The proposed approach only requires 30-60 minutes of self-supervised data collection before it learns to successfully perform the demonstrated task. Hence, the only human supervision required is a single demonstration with labels for each stage and the corresponding bottleneck state.

- The trained networks are able to generalize to new configurations of the scene due to training on observations collected from a wrist camera, which does not tie the observations to a specific configuration of the objects. This is a pretty neat trick and naturally allows the robot to generalize to new configurations of the scene.

Weaknesses:
- The experimental evaluation only includes one comparison to prior work on a pair of tasks, so it’s difficult to understand how well this compares to existing systems for this setting. The main text points to the appendix for a comparison to RL from demonstrations, but the results are missing.

- It’s also unclear how significant the results are from just the reported success rates. For example, the difference between Random Self-Replay and Active Self-Replay is just one more successful trial out of ten on some of the tasks. It would be helpful to test on a larger number of configurations.

- This approach assumes that prior to achieving the bottleneck pose for an object, the object remains fixed in the scene. However, it seems like the robot could potentially disturb the object during the self-supervised data collection. Currently, the actions used to reach the bottleneck pose are the difference between the bottleneck and current pose, which would be fine if the direct trajectory to the bottleneck pose is obstacle-free and does not disturb the objects. Otherwise, the robot would likely move the objects and need more human intervention.
summary_of_recommendation: I’m recommending a weak accept. The overall results are impressive for a difficult problem, but it’s difficult to assess the performance of the approach and the difficulty of the tasks without more comparisons.
issues: - It’s noted that training separate policies for each stage leads to better performance, compared to training a single policy conditioned on the task id. What if the policy was conditioned on the bottleneck state? Would this joint training help?

- In the self-supervised data collection phase, presumably the robot can disturb the scene. For example, the cup in the Cup Pick & Place task could be knocked over. Does this not require human intervention to correct the scene?

- How sensitive is the performance to the choice of the bottleneck poses? It’d be useful to see how different bottleneck choices for the same demonstration affects performance.

- It would be helpful to break up Sec. 3.4 by outlining the objectives used to train the networks.

- The link to the website seems to be missing from the paper.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors propose to learn policies from a single demonstration and multiple bottleneck states (identified by the demonstrator). The robot is then reset to random states from which it learns to reach the next bottleneck state. The robot is able to do so on its own by "linearly interpolating from random state" to bottleneck state. They show this strategy allows them to learn policies quickly on a real robot on a number of manipulation tasks.
strengths_and_weaknesses: Strengths
1) The proposed approach is interesting, modular, and general enough to be applied to many robotics problems.
2) Real-world robot experiments on multiple tasks to validate their approach.
3) Nice ablation study of how behavior cloning succeeds on tasks with less stages but as the stages increases more complicated algorithms are required.

Weakness
1) One of the new contributions of this work has been explored before in [1]. In [1] the authors suggest having checkpoints with which one could reset the environment from which the agent can explore. This is similar to the Self-Replay part of this work where the authors start exploring from pre-defined checkpoints or bottleneck states.

2) The proposed idea seems difficult to scale to multiple demonstrations or handle variations in workspace without manual intervention. For example, the authors train a bottleneck classifier treating this one state as a positive and other states as negative. If at test time the configuration of objects is different it would make it difficult for the bottleneck classifier to work properly.
There is some recent work on unsupervised video representation learning [2] which claims that key events and phases can be discovered in a self-supervised way. The authors might consider employing something like this to get a scalable bottleneck state from multiple demonstrations.

3) Evaluation missing relevant baselines:
The baseline considered in this work is FlowControl which is a policy learning method that uses optical flow. It is not clear how that is the best baseline for the proposed approach that tackles long-horizon tasks. Approaches like [3],[4] or [5] which solve long horizon tasks by breaking it into smaller tasks seem more suited here.

4) How many extra interaction steps is required by the proposed algorithm in the Random SelfPlay stage to achieve good performance? The lower bound seems to be pure behavior cloning where no extra interaction steps is required. It will be good to see trend of that graph, so that we can estimate if better performance can be achieved by more steps in the SelfPlay stage.

References

[1] Ecoffet, Adrien, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. "First return, then explore." Nature 590, no. 7847 (2021): 580-586.

[2]Dwibedi, Debidatta, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. "Temporal cycle-consistency learning." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1801-1810. 2019.

[3] Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations. Ajay Mandlekar, Danfei Xu, Roberto Martín-Martín, Silvio Savarese, Li Fei-Fei

[4] D. Xu, S. Nair, Y. Zhu, J. Gao, A. Garg, L. Fei-Fei, and S. Savarese. Neural task programming: Learning to generalize across hierarchical tasks. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3795–3802. IEEE, 2018.

[5] Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, Karol Hausman. Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning
summary_of_recommendation: While the paper is quite well-written and has real-world experiments for multiple (which are difficult to setup), the method is not scalable, missing some baselines comparisons and has limited technical contribution. It was however interesting to see their proposed approach get good performance for manipulation tasks in the real world. Hence I am not recommending the acceptance of the paper in its current form.

**UPDATE**
I thank the authors for newer experiments and additional analysis. It definitely adds more value to the paper. 

It seems one factor that makes their method generalize using few samples is the use of a camera on the wrist. It would be important to show if the method can generalize to new configurations from third-person views or the wrist-view is crucial for success. If it is, it should be reflected in title/abstract.

By scalable I meant addressing the scenario when multiple demonstrations are available? Since it is a leaning based method, there has to be configurations (single demonstrations) that make it difficult to generalize to newer (out of distribution) settings.

However with the newer experiments, I can recommend a weak accept if the authors include limitations of their approach. 
issues: 1.Comparison with relevant baselines will help the paper.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: * This work extends "coarse to fine" imitation learning to the multi-stage setting.
* Introduces a way to scale up demonstration collection for multi-stage tasks with the following structure: 1) "move to bottleneck 1" phase, e.g. move in free space to pregrasp of an object, 2) "bottleneck 1" replayable open-loop object interaction, e.g. picking up a fork, 3) "move to bottleneck 2", e.g. carry an object above a plate, 4) "bottleneck 2" replayable open-loop object interaction, e.g. place the fork on a plate.
* Turns one demonstration for a particular object configuration into many (for the same particular object configuration) using the following scheme: autonomously reach the first bottleneck pose from many different positions, replay the bottleneck 1 actions, autonomously approach bottleneck two from a variety of poses, replay bottleneck 2 actions.

strengths_and_weaknesses: # Strengths
* Well motivated problem setting: collecting demonstrations for multistage tasks is cumbersome and time-intensive.
* This work introduces a clever and scalable way to collect many demonstrations from only one, assuming certain (fairly strict) requirements about the tasks and environment are met. I can imagine some important commercial applications that meet the requirements of this method.
* Clear description of the method and assumptions around task structure.
* Compelling qualitative results on a real robot.

# Weaknesses
* **Introducing strict assumptions, but not adequately discussing them.** A significant limitation of the setup is that the set of automatically created demonstrations are only valid for one particular object configuration. The paper addresses this by introducing a different assumption: that the agent perceives the scene only through a wrist-mounted camera. The wrist-mounted camera "makes the RGB observations a function of the relative, and not absolute, pose between end-effector and objects. Hence, moving the end-effector is generally equivalent to moving the object to observe it from different poses."  This is furthermore only true when there is no clutter or visual variation in the background. If there is, then the RGB observations of even the wrist camera depend on the absolute position of the scene and the particular configuration of the background. This assumption is never stated but it is nevertheless important. Requiring a wrist-mounted camera (and only a wrist-mounted camera) as well as an uncluttered scene are rigid assumptions. The issue is, without these assumptions, the autonomous data collection becomes far less useful as it is indexed on a particular object configuration---changing the object configuration even slightly requires recollecting all new data.  It's fine to restrict attention to the setting of "wrist-mounted visual manipulation in uncluttered scenes", but this set of assumptions should be very clearly stated early on in the introduction. L124 in Section 3 mentions how important the wrist-camera is to the method: "[the wrist-mounted camera] is a fundamental aspect, as it allows the robot to autonomously gather data that can generalise to novel situations, as described in later sections". The intro should be significantly reworded to accurately convey these (significant) limitations to the reader.

* **Insufficient discussion of the other important sources of variation in imitation**. It is true that imitation learning requires a lot of human effort, and it is a good goal to reduce this effort. But this paper does not appear to address the important sources of effort in behavioral cloning. It provides a way to collect a very large number of demonstrations for one particular scene configuration, covering different initial poses and different trajectories between bottleneck states. Covering pose variation is important, but only *one* of the many kinds of variation that necessitate a large number of demonstrations when doing behavioral cloning. Especially in the "everyday" setting this work claims to operate in, multiple demonstrations are needed to cover changes in the initial positions of the objects, changes in lighting conditions, changes in background, presence of distractor objects etc. If any of these change, all the data automatically collected by this method seems just as brittle as a large set of manually collected demonstrations from a single world state. This is important missing context. Without this context, statements like "...we introduce a novel method to learn everyday-like multistage tasks from a single human demonstration" seem quite strong, as it does not appear to actually improve upon the most important sources of "everyday" variation.  


* **No limitations discussed on just how coarse the "coarse" phase can be**. The paper states that it can automatically expand a demonstration set with "without requiring any prior object knowledge". It is hard to imagine how this is true for, say, a cup of coffee, a bowl of candy, a plate with food on it, etc, without requiring object-specific specific scripting like “the container needs to remain upright". Clearly how you script the automatic "move to random poses" phase of collection depends on the object you are dealing with and the scene you are operating in. Otherwise the robot may do unsafe things like collide with objects. Language like "without requiring any prior object knowledge" (which appears multiple times in the paper) should be significantly toned down or the assumptions about which objects this method pertains to should be made more explicit.

* **Strong conclusions from weak experiments.** Experiments are reported without standard deviation over multiple training seeds. Without knowing the standard deviation of the success of the methods, how can we tell if we are observing an actual improvement or just sampling error? "Results therefore show that learning the interaction phase from demonstrations does not provide any *statistically significant* benefit" (emphasis mine). The experiments do not currently carry enough information to assess statistical significance. This is also quite a broad conclusion given the major assumptions around the task setup.  "In our method, we only provided a single demonstration, running Active Self-Replay for around 40 minutes to autonomously gather data. We compared it to BC with 10 and 30 demonstrations." How many minutes of data were the 10 and 30 demonstrations? Was your method simply trained on more data? Without knowing this it's difficult to assess the benefit of the method. 
summary_of_recommendation: UPDATE:
In light of the following, I'm updating my score:
* The authors updated the paper to clearly state the strict dependencies of the method upfront (uncluttered scenes at training time, wrist mounted camera).
* They added new experiments showing that despite these assumptions, the method handles novel configurations of the task even in the presence of unseen visual distractors.
* They added seeded real world evaluations that help communicate the significance of the quantitative results.
issues: * The paper would benefit from clarifying the actual sources of task variation that this work addresses. Clearly state what sources of variation it does not address. The claims that this solves "everyday" tasks should be softened substantially if it does not indeed address everyday sources of task variation.
* The paper would additionally benefit from clarifying the strict dependencies of this method early on (wrist-mounted camera, uncluttered scene). 
* Claims of statistical significance should not be made without experiments that support those statements. Without rigorous experiments, it is difficult to assess the quality of the proposed method.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper extends previous work [1], to effectively solve multi-stage manipulation tasks from a single demonstration. The human expert provides a single kinesthetic task demonstration and decomposes it into bottleneck frames. The robot reaches bottleneck frames, replays the transformed demo at that frame, moves onto the next bottleneck, and repeats until the task is over. The real meat of this method is in the self-supervised tricks and active data collection required to  make error free bottleneck predictions from vision. This process requires minimal human effort and clearly yields huge dividends: the proposed work is far better than behavior cloning at multistage tasks despite using fewer demonstrations.


[1] Johns, Edward. "Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration." arXiv preprint arXiv:2105.06411 (2021).
strengths_and_weaknesses: **Strengths**
* The paper is well written and convincingly explores the design decisions needed to make this method work in practice. Specifically, the improvements to random self replay and active components are sensible ways to boost performance.
* The experiments consider a great deal of tasks and strongly support the hypothesis. The inclusion of robot results despite COVID is a big plus.


**Weaknesses**
* The paper could be seen as a relatively straightforward extension of prior work. I believe the added components are significant enough, but would like to see more thorough ablations done.
summary_of_recommendation: This paper proposes a non-trivial suite of self-supervised methods in order to extend Coarse-to-Fine imitation learning [1] to multi-stage tasks. By definition this work is incremental, but thorough robot experiments help make up for that. This is just enough to warrant acceptance. However, I should stress that more ablations and baselines could make the work significantly stronger. 
issues: * I'd add more ablations to show how your method compares vs more naive implementations. For example, what would happen if you omitted the "stage transition" in random self-replay and just predicted where all the bottlenecks were? This is a simpler method but would likely fail due to object occlusions that arise after the first stage. Seeing these comparisons would be helpful.
* Related to prior point - it's hard to tell if Active Self-Replay is really helping over the more naïve variant. One explanation is that the workspace (from video) is too limited to make difference seem apparent. Testing on more configurations/in wider workspace could make this difference more significant.
* This work does not compare against many prior imitation learning baselines. After all, there are prior network architectures and methods (e.g. [2,3]) that improves on behavior cloning for more complicated multi-stage tasks. Thus, more baselines would be helpful.

[2] Lynch, Corey, et al. "Learning latent plans from play." Conference on Robot Learning. PMLR, 2020.

[3] Yu, Tianhe, et al. "One-shot hierarchical imitation learning of compound visuomotor tasks." arXiv preprint arXiv:1810.11043 (2018).
review_title: null

Meta-review: In this paper, the authors propose a method to solve multi-stage task from one demonstration. Multiple bottleneck states are defined and recognized. It is well recognized that the system is able to collect scalable data in a self-supervised fashion. However, there are some concerns about how to handle multiple demonstrations, some practical situations, etc. Also, more comparison experiments with stronger baselines are lacking.  

In the rebuttal session, the authors have responded to the reviewers' concerns and more details and baselines are provided. And the reviewers have reached a consensus of accepting this paper.
Decision: Accept (Poster)