Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: In the paper, authors have worked on enabling vision-based robotic systems to carry out novel manipulation tasks. With this goal in mind, authors have collected large robot imitation data with corresponding sentence instruction and human demonstration. Authors have used this data to train imitation learning agent to perform tasks in the real world. To make it possible to either use language instruction or video demonstration as input to define the task, authors conditioned policy on embedding learned from these inputs. They have also built infrastructure to collect data in an online fashion where human experts only jumped in when the policy went in the wrong direction for executing the task. They have shown that data collected in this fashion helped in executing tasks better compared to passive data.
strengths_and_weaknesses: Strengths
* Collected a large amount of robot imitation data with vision and language instructions
* Successfully able to learn policy conditioned on either vision or sentence instruction, which will generalize to novel tasks 
* Showed that language embedding pre-trained on some other task work well  in conditioning imitation learning policy
* Showed that system able to generalize to different objects as well as tasks based on language embedding

Weakness
* Incremental algorithmic improvement: Lots of research has already been done in imitation learning based on human video demonstration or language instruction. This work mainly combine both the input modality, language & vision, for conditioning the imitation learning policy.
summary_of_recommendation: * Collected big dateset for robot imitation learning with human video and language instructions. This dataset can be helpful to other researchers in the field of imitation learning.
* Showed that pre-trained language embedding is good enough to encode task for imitation learning.
* Able to show the learned system generalize well with the unseen task and objects based on language instructions
* Able to show data collected in active manner, by involving expert human when the learned policy goes wrong, compared to passive data collection, helps in learning better policy.  

issues: * Author have showed that language embedding trained on some other task can be useful in conditioning the imitation learning policy, however for videos, authors have tried to learn embedding from collected video demonstration. It would interesting to know how the video embedding learned form other large video datasets works for conditioning imitation learning policy . Also large drop in the performance for unseen task based on video embedding, shown in table-3, shows that video embedding over-fitted to training videos.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The main contribution of the paper is a VR based interactive imitation learning system for providing demonstrations (and feedback) to a robot. This system is used to collect a very large (> 25K) set of demonstrations across a variety of manipulation tasks. The authors then use this large set of demonstrations to train an imitation learning agent to perform these tasks (and generalize to new variants of these tasks). An important part of the system is that fact that it is interactive (the HG-Dagger component) which allows for providing feedback to the policy.

Overall the paper demonstrates that with a sufficient number of demonstrations the agent is able to generalize to novel tasks with moderate success.

strengths_and_weaknesses: Overall I think that this paper can be considered as a “systems paper” where the main contribution is the interactive imitation learning system. The comments below should be considered in this context.

Strengths:
The problem considered (language-conditioned robotic manipulation in 7-DOF) is both interesting, relevant and very challenging.
Using language (and to a lesser extent video) is an interesting modality for conditioning robot tasks.
The experimental evaluation is extensive
The authors are very honest about the strengths/weaknesses of their approach.


Weaknesses:

I am not sure what I have learned and/or what conclusions I should draw having read the paper. Overall I think it is accepted in the robotics community that with enough demonstrations (and a few other tricks, such as HG-Dagger, state differences as actions, etc.) you can achieve reasonable performance with an imitation learning agent. This seems to take that premise and test it out with a very large set of demonstrations on a variety of tasks. This is an interesting endeavor, but the conclusions aren’t particularly clear to me. Overall it seems that the method presented requires a very significant number of demonstrations (25K demonstrations collected over several months) to achieve moderate levels of performance (success rates are below 50% on the held-out tasks). Should we consider this as a negative result and discontinue this line of investigation? Do the authors learning/control approach might need to be reformulated instead of just “scaling up” and collecting another 25K demonstrations?

The “place the bottle in the ceramic bowl” task is an example of the very large number of demonstrations that this system requires. The single-task model trained with 1000 demonstrations was only able to achieve a 5% success rate. I think that 1000 demonstrations is quite a lot.  so a system that isn’t able to learn with this number of demonstrations is a bit worrying. I think it is interesting that the multi-task model is able to achieve 50% on this task, but that relies on 25K demonstrations that took months to collect. Given the number of demonstrations required are we going to be able to scale this to get the robots to accomplish useful tasks in industrial or home settings?


The network architecture for the policy is simple and straightforward, which is a positive. However, coming back to the earlier point, the method is very data hungry and requires lots of demonstrations to achieve even moderate performance. The authors might consider other more data efficient approaches given the challenge of collecting robot demonstrations in the real world.

In Table 3 we see that the performance of the model conditioned on the one-hot encoding is effectively identical (40% vs. 42%) to the model conditioned on the language encoding. This makes it seem that what the language encoder is doing (at least on the train tasks) is effectively acting as a one-hot encoder. Thus is this architecture getting any value from using language? I agree with the authors that language is a flexible and general way of specifying tasks, but I would have maybe expected language to impart other benefits that would transfer across tasks in ways that one-hot encodings wouldn’t (e.g. an understanding of colors such as in the “red bowl” and “purple bowl” language instructions). I would be curious to hear the author’s thoughts on this.


summary_of_recommendation: Recommendation: weak reject

Overall I think that the paper is well written and executed but that the underlying idea is not particularly novel and the performance of the method is not particularly good. I am not sure what exactly I learned from reading the paper. As such I am recommending weak reject, although the true rating is more somewhere in between weak reject and weak accept.


**Updates post-rebuttal**

I am updating my review to a **weak accept** (as selected in the Recommendation section) with my true rating being slightly higher than weak accept but below strong accept.

The authors have done an excellent and thorough job responding to reviewer concerns and updating the paper accordingly. I think the main improvement has been in clarifying the contribution of the work, namely a large scale empirical study. Given this I think that there is something to learn from the paper and think it merits the updated rating.

That being said I think that the actual method in question still has significant drawbacks/issues/challenges that culminate in low success rates. The approach is not the one you would necessarily take to get high reliability on a specific task, and the authors acknowledge this. The point of this paper is not to use task-specific tricks (like TransporterNetworks) to accomplish specific tasks effectively but rather to do an empirical study of a general purpose system. I also agree with the authors that the tasks and approach they are taking (e.g. continuous 7-DOF control) is very challenging. I want to make it clear that I understand this and am not trying to hold it against them for taking on this challenge and being honest about the performance of the method.


Investigating the relative merits of general purpose approaches (like the one presented) and more tailored/engineered solutions (e.g. TransporterNetworks) is an interesting direction for future work that I would like to see taken up and debated by the community.

issues: When discussing demonstrations collected using HG-Dagger the authors should expand a bit on which policy was being executed when these HG-Dagger interventions were done. Did interventions from one-policy help a different policy to improve, or did different trained policies make different types of “mistakes”.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper presents BC-0: a *one-shot* imitation learning system for vision-based robot manipulation; I will elaborate on the *one-shot* label (rather than the titles zero-shot) later in the review. As this is an imitation learning method, data collection is key in order to help elevate covariate shift. The data collection has two phases: (1) expert-only, where the agent is trained on full human-provided demos (from a VR system), and (2) a "shared-autonomy" mode, which uses HG-Dagger (where the human intervenes and temporarily take full control when they believe the policy is likely to make an error). Policy learning is done similarly to TECNets [2-3] where the paired examples of a human video/text and corresponding robot demo are trained to embed close together, while also training for a behaviour cloning loss. The action mode for the arm is 7 DoF: 6 DoF for the pose, and another for the gripper. Results show that when training on a 100-task demonstration dataset, the method generalises to 29 unseen tasks when given a single user-provided language or video commands.

TLDR: It is a one-shot imitation learning system that is conditioned on either videos or natural language, where the data is collected via shared autonomy.
strengths_and_weaknesses: **Strengths**
- Heavy use of real world experiments (with a caveat, see weaknesses).
- Obvious that a lot of time and effort went into the data collection process.
- The combination of language and video is a logical and appropriate next step from [1,2,3,4].
- The shared autonomy data collection is a nice touch.

**Weaknesses**
- Although having real world results is great, it is also important for the sake of reproducibility, for the paper to supply some for of simulation results, perhaps even on a standard set of tasks, e.g. RLBench, MetaWorld, Robosuite, etc. This is very important for robotics, where comparing methods is difficult. 
- How does this method compare to others? The only baselines provided were ablations of their own method; the paper rightly point to a few works already in this area, and so comparison to these seems important. If someone wanted to reimplement this idea, then they could use your reported simulation results to ensure they had implemented your method correctly. 
- Following the president set by previous work [1-4], this is not zero-shot learning, and seems wrong to label the work as such. It is one-shot learning; as a demonstration must be given to give intent to the control agent. Of course, the lines become blurred when considering *only* text input to convey intent, but as the paper features an equal weighting of video intent, then it is misleading to identify the work as zero-shot. 
- The experimental results when comparing to other methods could be improved. The paper does not compare to other methods expect for ablations of itself. The paper rightly point to a few works already in this area, and so comparison to these seems important. How does this work compare to [3, 4] when trained on the same dataset (without the language component)?
- The contributions seem minor; the work heavily uses methods from [2, 3, 4] and HG-Dagger [29]. I.e there is not too much to be gained/learned by reading this paper. 
summary_of_recommendation: The contributions do seem minor (the work heavily uses methods from [2, 3, 4] and HG-Dagger [29]), however the work is still the first I am aware of to have the option to give either video or language intent (most use video). This, in addition to the poor reproducibility via no simulation results is the main reason for my weak accept. 

--------------

***Post rebuttal***

Overall, I am happy with the author response. After rereading, it is now much clearer what lessons the community can take away, and indeed, a study of this scale (in real world) has not been done before. I am happy to up my rating from a weak accept to a strong accept; I think many at CoRL would find this study interesting.
issues: - Some form of simulation results; this will make reproducibility of the method easier, especially as the robot used in this paper seem like a custom robot. If someone wanted to reimplement this idea, then they could use your reported simulation results to ensure they had implemented your method correctly. 

**Questions for the authors:**
- The success rate for using just video and just language were interesting. I wonder, what would be the results of supplying both at test time? I.e. receiving two embeddings z, taking the mean, and then conditioning on the control net?
review_title: null

Meta-review: Reviewers found the paper to have strengths in the significant effort put into the large-scale real-world evaluation, but weaknesses in the novelty of the proposed method. There is a consensus that the novelty is somewhat limited, and the proposed method is primarily a simple combination of existing methods brought together in a full system, rather than a significantly novel method itself. I would like to hear from the authors their opinion on how novel their proposed method is, beyond the overall system itself, or whether the authors believe the paper is primarily an empirical study, rather than a paper proposing a novel method. The large-scale real-world data collection and experiments are certainly encouraging, but Reviewer KYDU still argues that there is not a huge amount we can actually learn from this, since the proposed method still requires a very large number of demonstrations, and so results do not show any significant leap in data efficiency. Comparisons to existing alternative frameworks (rather than just ablations) on data efficiency, could be important in highlighting how this method sits with respect to alternative frameworks. So overall, the reviews are mixed. The system and data collection are impressive, but I would now like to see a discussion focussing on what the authors believe the key novelties are in the proposed method, and what the authors believe we, as a community, can learn from the method itself. I would also like to hear from the authors about whether they believe their method can truly be considered "zero-shot" when a demonstration still needs to be provided for a novel task, following the point raised by Reviewer aF8M.

------

The review process led to some lengthy and interesting discussions between the authors and reviewers. There is consensus, amongst both the authors and the reviewers, that the contribution of the paper is not in the novelty of the method (which is primarily an integration of a number of existing ideas into a system), but the implementation and evaluation of this system at such a large scale. The results of this are indeed useful for the community to know about, and will likely stimulate further discussions about how far we can really push behavioural cloning methods. All reviewers recommended to accept the paper, and I believe that we can all learn some useful insights from this work.
Decision: Accept (Poster)