Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper identifies challenges of learning from offline human demonstrations for robot manipulation and analyzes the performance of 3 BC algorithm variants (BC, BC-RNN, HBC) and 3 batch-RL algorithms (BCQ, CQL, and IRIS) across 5 simulated and 3 real-robot manipulation tasks of varying difficulty, 3 different dataset generation sources, two observation spaces, and various hyper-parameter choices, while providing various relevant insights.
strengths_and_weaknesses: Strengths:
- Identification of challenges of learning from offline human data
- Extensive evaluation across various dimensions such as model choice, observation space choice, dataset source, and hyperparameters for manipulation tasks with varying difficulty
- Useful insights on performance across these variables with suggestions for addressing the challenges

Suggestions for Improvement:

- Can the paper dive a little deep into what differences in the nature of the mixed quality machine-generated datasets when compared to mixed-quality human datasets make it difficult for SOTA RL algorithms such as BCQ to perform comparably with human datasets? 

- The effect of observation spaces in the performance for IL algorithms could be more profound than just low-dimension (object state estimation) vs. high-dimension (visual raw data). The observation space is also sensitive to frame of reference (object reference or robot reference) and multi-modality sensing. For example: how would the trends change if force information was available as an observation in addition to the visual data? Or, how would the performance change if the robot states (gripper opening, end-effector poses etc.) are expressed in the object coordinate frame vs. robot-base coordinate frame? Also, what about the transport task made it difficult to be learned using the high-dimensional observation space? And, how does the performance vary when there is uncertainty in the low-dimensional object pose?

- While the paper mentions the promise of using more data for complex tasks with high-dim observation space, collecting huge human datasets is very challenging realistically. Can the paper comment on the feasibility as well?

- The paper can become stronger with comments on how these variables affect the performance in the presence of covariate shift, which BC algorithms generally succumb to 

- Can the insights extend beyond robot manipulation problems to other robotic applications? Kindly discuss

- The recommendations given in Section 4.4 (C5 - Hyperparameter choice) are conditioned on a given dataset size. Some of the overfitting concerns with larger models may not exist with larger datasets. Can the paper discuss the recommendations in light of other variables that may affect the performance? I would also revise L6 in the Discussion section by adding caution to the statement that the results transfer to real-world settings, especially because this is conditioned on the difficulty of a task. The paper would benefit by detailing what about a task makes it difficult in this context, is it the long-horizon factor, is it the precision factor, is it multi-arm (agent) factor, or is it all of the above, and are there other aspects of a task that could make it difficult for these learning tasks? These discussions can make the paper much stronger and more useful to the community in general.

summary_of_recommendation: This is an extensive study and the insights would be valuable to the community. I strongly support accepting the manuscript.

issues: See “suggestions for improvement” section above
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents an exploratory study aimed at identifying over-arching challenges in learning manipulation skills from offline human-labelled datasets. Specifically, by varying factors such as the learning algorithm, dataset size, optimality of demonstrations, and task complexity, the study compared the performance of policy learned under various permutations of choices. Finally, the work distills six key lessons learned from the study.
strengths_and_weaknesses: **Strengths**

+ The paper presents a timely and important study that explores the challenges involved in learning manipulation tasks from offline demonstrations. 
+ The paper is well-written and very easy to read.
+ Making the data and code available would make a big impact in the research communicate and help benchmark and compare new approaches.
+ I really appreciate the careful and comprehensive design of the study that includes variations across various important axes such as task complexity, user expertise, perception modality, and learning paradigm (offline RL and imitation).
+ Almost all design choices are carefully motivated and explained.

**Weaknesses**

- The paper is missing a few key existing works that attempt to benchmark learning from demonstrations (Rana et al., ICRA, 2020; Lemme et al., J. Beh. Robo., 2015) . It is important to acknowledge these contributions and contextualize the contributions made by this work within those of existing studies. While do not believe that these work subsume all contributions of the proposed work, there are non-trivial overlaps.
- It is helpful to see the lessons learned clearly articulated and have numerical evidence to support them. However, the reported lessons are not surprising and seem intuitive. 
- The evaluation metrics seems to be limited to success rates. It would be more illuminate to analyze more metrics such as time taken to perform the task, smoothness of motion, and (subjective) user perception.

**References**

- Rana, et al. "Benchmark for Skill Learning from Demonstration: Impact of User Experience, Task Complexity, and Start Configuration on Performance." 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020.
- Lemme, Andre, et al. "Open-source benchmarking for learned reaching motion generation in robotics." Paladyn, Journal of Behavioral Robotics 6.1 (2015).
summary_of_recommendation: On the one hand, the paper presents a timely and comprehensive study of the challenges involved in learning manipulation policies from offline datasets. The study design is commendable and making the datasets and code available to the public will likely have a big impact on the research community. On the other hand, the paper does not acknowledge a couple of key prior studies with similar aspirations, and evaluates the performance of each permutation of factors solely based on task success rate.
issues: 1. Would it be possible to compute more metrics (in addition to success rate) in the future if all the data and code are made available?
2. Given that all the experiments are conducted on a single robot platform (the Franka arm) brief discussion surrounding the generalizability of these finds across different robot morphology would be very helpful.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: 
This paper conducts extensive evaluations on learning from offline demonstrations for robot manipulation tasks. Specifically, it studies six offline learning algorithms (i.e. BC, BC-RNN, HBC, BCQ, CQL, and IRIS) on five simulated and three real-world robot manipulation tasks of varying complexity with datasets of varying quality. This work presents a variety of lessons learned from the study and makes recommendations for future work accordingly. In sum, this work presents systematical analyses of existing methods focused on robot manipulation with offline demonstrations. I believe this kind of work benchmarking existing methods is one key to measure the advance of the field. While I have some concerns regarding limited robot configuration and demonstrations, I am leaning toward accepting this paper. Yet, my research interests are not directly related to this research direction, and therefore my evaluations should be taken lightly. 
strengths_and_weaknesses: 
## Paper strengths and contributions
**Motivation**
Most of the recent papers studying robot manipulation with offline datasets usually set up their own environments/tasks and collect their own demonstrations. This makes it very difficult to draw a fair comparison among the methods proposed in these papers or measure the advance in this field. Therefore, I believe systematically benchmark existing methods would be beneficial to the community.

**Interesting insights**
This work presents a variety of insights on learning robot manipulation tasks with offline demonstrations. The following findings are particularly interesting to me:
- Leveraging history is important when learning human demonstrations based on the performance gap between BC and BC-RNN.
- Offline RL methods perform well when learning from agent-generated demonstrations but not from human demonstrations
- Including end effector velocity and joint information hurts agents trained on low-dim observations
- Selecting the policy using the best loss on validation data or using the final training checkpoint yields a significant performance drop compared to the real best policies.

**Clarity**
- The overall writing is clear, including what the goals are, how the experiments are set up and conducted, etc.
- The challenges are clearly stated.
- The findings are well-organized.
- The presentation of the experimental results is clear.
- Most of the analyses of the experimental results are detailed, comprehensive, and convincing.

**Environments & tasks & demonstrations**
The effort on setting put the environments in simulation and real-world as well as collecting demonstrations is highly appreciated.

**Diverse demonstrations**
- Evaluating methods learning from demonstrations of varying quality provides informative findings. 
- Comparing models learning from agent-produced demonstrations and human-generated demonstrations presents interesting insights.
- Training models to learn from different numbers of demonstrations yields results that align with intuitions.

## Paper weaknesses and questions

**Limited robot configuration**
This work only conducts studies with a Franka robotic arm in both simulated and real-world environments. Therefore, it is not easy to judge if the majority of the findings presented in this work would apply to other robot configurations. While I do recognize the difficulty of accessing a variety of robot configurations, especially for an academic lab, I believe it is still possible to conduct studies with other type of robots at least in simulation to justify the findings. 
 
**Converged machine-generated demonstrations**
The machine-generated dataset consists of demonstrations collected from checkpoints along with the training of expert policies. It makes sense to do so to collect suboptimal demonstrations and it may seem fair since human-generated datasets probably also contain trajectories collected when human demonstrators just start learning to use the teleoperation platform. However, human demonstrators do have an understanding of the tasks from the start and have prior knowledge such as physics. Therefore, I believe the suboptimal trajectories from the machine-generated datasets could be much worse than the suboptimal ones in the human-generated datasets, which can even enlarge the performance gap between models learning from the machine and human-generated demonstrations.

**Short-horizon tasks**
This paper mainly studies robot manipulation tasks with short horizons. While tasks such as Tool Hang and Transport consisting of multiple subtasks and might take hundreds of time steps, they do not examine robots' ability to plan. Therefore, the evaluations are all about the precision of the execution but not about the ability to interpret the intentions or the high-level plans of demonstrators.

**Domain gap**
The studies presented in this work only concern the setup where the agent and demonstrators share exactly the same environment. Yet, in many cases of real-world applications, robots might need to deal with subtle differences between the environment it faces and the one from the demonstration. I believe it would be also important to study this case where this domain gap exists.
summary_of_recommendation: 
This work presents systematical analyses of existing methods focused on robot manipulation with offline demonstrations. I believe this kind of work benchmarking existing methods is one key to measure the advance of the field. While I have some concerns regarding limited robot configuration and demonstrations, I am leaning toward accepting this paper. Yet, my research interests are not directly related to this research direction, and therefore my evaluations should be taken lightly. 
issues: 
Described in the strengths and weaknesses section.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper conducts a study of evaluating several imitation learning and batch offline RL algorithms when trained on offline demonstrations in simulated and real-world environments. In particular, such algorithms as Behavioral Cloning (BC), BC with recurrent neural networks, Hierarchical BC, Batch-Constrained Q-Learning (BCQ), Conservative Q-Learning (CQL) and IRIS are evaluated. Experiments include simulated and real lifting, simulated and real pick-and-place, simulated high-precision pick-and-place, simulated bi-manual transport, simulated and real articulate tool hanging tasks. Datasets used for experiments comprise machine-generated, and human-generated data, whereas human demonstrations are categorized based on the level of experience of the demonstrator. Overall the main takeaways from experiments are that using recurrent models is important for learning from human demonstrations, batch offline RL algorithms do not work well with human-generated data, offline policy selection is important ant it is difficult in the real world, including certain observation dimensions can lead to increased overfitting and more data leads to increased performance. Paper also provides some practical ways to tune hyper-parameters, and demonstrates that using algorithms and parameters selected in simulation transfers to learning on a real robot.
strengths_and_weaknesses: Strengths:
- The problem of learning from offline demonstration data is important for initializing robot policies, improving their safety and scaling up robot learning.
- The paper is well-written and easy to understand and follow.
- The paper conducts a wide range of experiments, including real-world robot evaluations on non-trivial tasks.
- The paper contains a range of practical insights that will be useful for robot learning practitioners.

Weaknesses:
- In general, although offering a great comparison of various algorithms, the paper doesn’t introduce novel algorithmic components.
- It is not always completely clear how true rewards are used during the optimization. For example, machine-generated and can-paired data contain both, success and failures. It looks like BC-based algorithms are not using the success/failure label and batch RL algorithms are using them. It would be useful if paper stated more clearly how these ground truth labels are used.
- The paper makes a conclusion that batch RL algorithms do not work well with human-generated data. However, it is indicated that for machine-generated data, 6 checkpoints with 300 rollouts each are included in the dataset, whereas human-generated datasets only include 200 or 300 demonstrations in total, so at least 6 times less data. Another conclusion that can be made from these experiments is that batch RL algorithms work worse with a low amount of data. It would be interesting to see experiments that disambiguate these two conclusions.
summary_of_recommendation: The paper provides a great comparison of various algorithms for offline imitation learning, but lacks some novelty in theoretical contributions.
issues: Please refer to "Strengths And Weaknesses" for a detailed list of issues.
review_title: null

Meta-review: All of the reviewers provided scores for accepting the paper. The authors addressed most of the points raised by the reviewers in their rebuttal. I encourage the authors to address the remaining points raised in the reviews for the final version of the paper.
Decision: Accept (Oral)