What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
Authors: Ajay Mandlekar
Danfei Xu
Josiah Wong
Soroush Nasiriany
Chen Wang
Rohun Kulkarni
Li Fei-Fei
Silvio Savarese
Yuke Zhu
Roberto Martín-Martín
Venue: CoRL2021 Oral (2021)
Keywords: Imitation Learning
Offline Reinforcement Learning
Robot Manipulation

Abstract: Imitating human demonstrations is a promising approach to endow robots with various manipulation capabilities. While recent advances have been made in imitation learning and batch (offline) reinforcement learning, a lack of open-source human datasets and reproducible learning methods make assessing the state of the field difficult. In this paper, we conduct an extensive study of six offline learning algorithms for robot manipulation on five simulated and three real-world multi-stage manipulation tasks of varying complexity, and with datasets of varying quality. Our study analyzes the most critical challenges when learning from offline human data for manipulation. Based on the study, we derive a series of lessons including the sensitivity to different algorithmic design choices, the dependence on the quality of the demonstrations, and the variability based on the stopping criteria due to the different objectives in training and evaluation. We also highlight opportunities for learning from human datasets, such as the ability to learn proficient policies on challenging, multi-stage tasks beyond the scope of current reinforcement learning methods, and the ability to easily scale to natural, real-world manipulation scenarios where only raw sensory signals are available. We have open-sourced our datasets and all algorithm implementations to facilitate future research and fair comparisons in learning from human demonstration data at https://arise-initiative.github.io/robomimic-web/

What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
Ajay Mandlekar', Danfei Xul, Josiah Wong', Soroush Nasiriany", Chen Wang',
Rohun Kulkarni', Li Fei-Fei', Silvio Savarese', Yuke Zhu?, Roberto Martín-Martín'
Stanford University, "The University of Texas at Austin
Abstract: Imitating human demonstrations is a promising approach to endow robots with various manipulation capabilities. While recent advances have been made in imitation learning and batch (offline) reinforcement learning, a lack of open-source human datasets and reproducible learning methods make assessing the state of the field difficult. In this paper, we conduct an extensive study of six offline learning algorithms for robot manipulation on five simulated and three real-world multi-stage manipulation tasks of varying complexity, and with datasets varying quality. Our study analyzes the most critical challenges when learning from offline human data for manipulation. Based on the study, we derive a series lessons including the sensitivity to different algorithmic design choices, the dependence the quality of the demonstrations, and the variability based the stopping criteria due to the different objectives in training and evaluation. We also highlight opportunities for learning from human datasets, such the ability to learn proficient policies on challenging, multi-stage tasks beyond the scope current reinforcement learning methods, and the ability to easily scale to natural, real-world manipulation scenarios where only raw sensory signals are available. We have open-sourced our datasets and all algorithm implementations facilitate future research and fair comparisons in learning from human demonstration data at https: / / /araaareeee ieiiiiiiiiiti www.t........................... werrroooiii mi c-web/
Keywords: Imitation Learning, Offline Reinforcement Learning, Manipulation
Introduction
Human supervision has been at the heart of the most significant recent advances in several domains such computer vision [1-4] and natural language processing [5-7]. By intelligently extracting information from large-scale human-labeled datasets, autonomous machines have been able to reach near- even super-human performance decades-old problems probllemss such image recognition and question answering. Roboticists have also attempted to tackle robot manipulation through learning from human datasets, using the paradigms of Imitation Learning [8-10) and Batch (Offline) Reinforcement Learning [1 1-13], where datasets consisting of robot arm trajectories, action labels at each timestep, and possibly reward labels, used to train closed-loop policies.
As other domains, large offline datasets offer several benefits such as scale, portability, and reproducible evaluations to measure progress. Recently, there has been considerable progress in offline learning for robot manipulation from human demonstrations [9, 14, 10]. Despite these advances, the offline learning paradigm has not been nearly disruptive in robotics as in other disciplines- - there is large gap between autonomous robot manipulation capabilities and the wide range of tasks that humans can solve effortlessly using physical and cognitive intelligence. What has inhibited the use of large human-provided datasets to address this gap?
In contrast to other domains where supervised learning has been successful, robotic manipulation a time-evolving dynamical system, requiring fine-grained real-time control to guide robot arms successfully through tasks consequently, data collection can present technical challenges requiring
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

(a) Lift
(b) Can
(c) Tool Hang
(d) Square
(c) Lift (Real)
(f) Can (Real)
(g) Tool Hang (Real)
(h) Transport
Figure 1: Tasks. We collect datasets across 6 operators of varying proficiency and evaluate offiine policy learning methods 8 challenging manipulation tasks that test wide range of manipulation capabilities including pick-and-place, multi-arm coordination, and high-precision insertion assembly.
specialized systems [15], which can explain why large-scale human- provided datasets [16, 17] have not been very prevalent. Learning from such datasets can also present several challenges. Human demonstrations can differ from Sethinne-geeerater datasets (a recent trend in benchmarks for offline policy learning [18, 19]) due to a non- non-Markovian decision process, since humans may not act purely based the current observation. There also be significant variance in both data quality and solution strategy when collecting data from multiple humans [20]. Differences from classic supervised learning, such as mismatch between training and evaluation objectives (task success rate), make selecting final policy challenging [21, 22), especially in real-world settings where evaluating each policy on robot can be infeasible. Finally, offline learning is sensitive state and action space coverage (dataset size) and agent design decisions.
buman-provided
Studying these challenges in the context of robot manipulation and human-provided datasets could be a stepping stone closing the gap between robot and human manipulation capabilities. Unfortunately, lack of suitable benchmark and human datasets have made studying this setting difficult. Prior works are either limited to studying simple 2D environments [23] or using data generated from hard-coded policies [24, 25]. In this paper, address this need by presenting study of data-driven offline policy learning methods on several numan-providee
robot manipulation datasets. We collect task demonstrations from human teleoperators across a broad range of simulated and real world manipulation tasks and investigate several factors that play role in learning from such data.
From our results, we point out several lessons to guide future research in leveraging human supervision for robot manipulation effectively. We find that history-dependent models can extremely effective in learning from single and multi-human datasets while state-of-the-art batch RI algorithms struggle to learn from such datasets, and that the choice of observation space and hyperparameters play a substantial role in training proficient policies. We also find that there is substantial promise for solving more complex tasks using large-scale human datasets and that our insights directly transfer to real-world scenarios, making this important setting to explore further.
2 Challenges in Offline Learning from Human Datasets
In this section, we outline five challenges offline learning from human datasets that motivate different factors that we investigate in our study.
(C1) Data from Non-Markovian Decision Process. Human demonstrations can differ substantially from machine-generaan demonstrations because humans may not act purely based on single current observation. External factors (teleoperation device, past actions, history of episode) may all play a role. Prior work [20] has noted substantial benefits from leveraging models that are history-dependee and with temporal abstraction to learn from human demonstrations. We investigate various design choices related to such architectures in this study.

---

Dataset
BC
BC-RNN
BCQ
CQL
HBC
IRIS
(MG)
2.5
70.7434
11
6..0+22.88
47.3+4.11
96.0+114
Can (MG)
3.4
6..7
0.9
0.3
40.7
6.5
Lift (PH)
100.0::0.0
1.0
100.0:10.00
92.7
0.0
100.0:10.0
Can (PH)
95.3
100.0
8..
38.0+
+
Square (PH)
1..
84.0
50.0
2.5
82.6
7..
Transport (PH)
17.3:4:2.5
71.3 6.6
7.3
0.0
4..6
41.3
Tool Hang (PH)
29.3:1:.9
19.3
0.0:
0.0
30.0
11.32.55
Lift (MH)
100.0 100.0=0.00
100
100.0
56.7 40.3
1.0
1.0
Can (MH)
86.0++4.3
100 0 0.0
6.2
22.0
91.3+
92.7
Square (MH)
6.6
4.3
14.0
0.7
60.7
52.7
Transport (MH)
11.3+22.5
65.3+774
2.6+
0.0
14.0+
10.7++0.9
Table I: Results on Din-eiinniional Observations. We present success rates averaged over 3 seeds for each method across the low-dim Machine-Generated (MG), Proficient Human (PH), and nulti-Human (MH) datasets. The results show that methods that model temporal correlations (BC-RNN, HBC, IRIS) exhibit strong performance on human datasets. Furthermore, while Batch RI. algorithms like BCQ are proficient on meneine-generated data, they perform poorly on human datasets.
(C2) Variance in Demonstration Quality from Multiple Humans. Prior work [20, 17] has found that data collected from several humans differ substantially in both demonstration proficiency and solution strategy. Differences in supervisor proficiency can manifest in many ways, such as large variations in trajectory length and noise in robot movement mistakes (e.g. missed grasps). In our study, we evaluate offline policy learning algorithms on such datasets. While recent batch RI algorithms have shown excellent ability to learn from mixed quality machine-generated datasets [26, 27], we empirically find that they fail to learn well from mixed quality human data.
(C3) Dependence Dataset Size. Offline policy learning is sensitive the state and action space coverage in the dataset, and by extension, the size the dataset itself. In our study, we investigate how dataset sizes affect policy performance. This analysis is useful understand the value of adding more data - an important consideration since collecting human demonstrations can be costly.
(C4) Mismatch between Training and Evaluation Objectives. Unlike traditional supervised learn- ing, where model selection can be achieved by using the model with the lowest validation loss [21], offline policy learning often suffers from the fact that the training objective is only a surrogate for the true objective of interest (e.g. task success rate), and policy performance can change significantly from epoch to epoch. This makes it difficult to select the best trained model [19, 28, 29]. In our study, we evaluate each policy checkpoint online in the environment in simulation, and report the best policy success rate per training run. We use these ground-truth values to understand the effectiveness of different selection criteria, and confirm that offline policy selection is an important problem, especially in real-world scenarios where large-scale empirical evaluation is difficult.
(C5) High Sensitivity to Agent Design Decisions. Prior studies on machine generated datasets have shown that offline policy learning can be extremely sensitive to hyperparameter choices [19, 28]. In our study, we explore how agent design decisions affect policy performances, including the choice of agent architecture, agent observation space, and hyperparameter choices per algorithm. This results in several practical conclusions that should prove useful to researchers and practitioners alike. We further show that important design decisions made through our study in simulation directly translate to effective policy learning on real world tasks and datasets.
machine-generated
3
Study Design
3.1 Tasks
We conducted our study across 5 simulated and 3 real world tasks. The tasks were chosen test broad range of manipulation capabilities. See Fig and Appendix E for more details.
Lift (sim+ + real)). The robot arm must lift small cube. This is the simplest task.
Can (sim real). The robot must place coke can from large bin into smaller target bin. Slightly more challenging than Lift, since picking the can is harder than picking the cube, and the can must also be placed into the bin.
Square (sim). The robot must pick a square nut and place it on a rod. Substantially more difficult than Lift and Pick Place Can due to the precision needed to pick up the nut and insert it on the rod.
3

---

Dataset
BC
BC-RNN
BCQ
CQL
HBC
IRIS
Can- Worse
5..7422.5
92.0=116
29.3 10.9
4.0+3.3
78.7 334
77.34 1,9
Can-Okay
7..0
95.3=19
58.0
22.0=4.3
97.3=099
9..0=0.0
Can-Better
2.5
9..
62.0+5.9
20.7++7.4
96.0 0.0 96.7 0.9
Wkase-Okay
74.7++5.7
98.7
50.7
18.74
17.3+1.99
Can- Worse-Better Better
7..0=44.3
100.0 0.0
48.0
20.7
90.0= 1.6
2..
Can Okay Better
90.7-2 1.9
100.0 :1 0.0
68.7 2.5
30.7:1..77
99.3 0.9
98.0 1.6
Square-Wort
2..0++4.3
39.3+
5.3
0.0
44.-6688
38.7
Square-Okay
27.3=2344
45.3
6.7
0.0=0.0
52.0=2.8
3.30
Square-Better
58.7
66.0
32.0++..3
0.7
61.3= 1.9
60.0 1.6
28.7:22.5
55.3=009
8.7
2.7
50.7=4.1
4.5
Square-Worr Worse- Better
46.7:2..7
6.2
1..3
1.3
65.3=3.4
5..7
Square Okay-Better Better
56.7444.1
74.0+22
22.0++4.3 4.3
1.3+0.9
63.3 1.3--4. 4.1
56.7
3.8
Can-Paired
64.0=9.1
7..0
4..7+119
6.0
70.7=552
1.9
Table 2: Results Suboptimal Human Data. We present success rates averaged over 3 seeds for each method across different subsets of the Multi-Human datasets, corresponding to mixtures of demonstrations from "Better". "Adequate", and "Worse" human operators, and finally on diagnostic dataset with paired success and failure human trajectories for each starting initialization. Results indicate that BC-RNN is strong baseline, and that Batch RL methods perform poorly across all datascts, even on the simple diagnostic dataset.
Transport (sim). Two robot arms must transfer hammer from a closed container on shelf to target bin on another shelf. One robot arm must retrieve the hammer from the container, while the other arm must clear the target bin by moving a piece of trash to the nearby receptacle. Finally, one arm must hand the hammer over to the other, which must place the hammer in the target bin.
Tool Hang (sim real). A robot arm must assemble frame consisting of base piece and hook piece by inserting the hook into the base, and hang a wrench on the hook. This is the most difficult task due to the multiple stages that each require precise, and dexterous, rotation-heavy movements.
3.2 Data Collection
To study the effect of dataset source, we collected data from three different sources Machine- Generated, Proficient-Human and Multi-Human (more details in Appendix B).
Machine-Generated (MG). We collected these datasets by first training a state-of-the-art RL al- gorithm [30] on the Lift and Can task, taking agent checkpoints that are saved regularly during training, and collecting 300 rollout trajectories from each checkpoint. Consequently, these datasets are comprised of mixtures of expert and suboptimal data, and resemble datasets from common offline RL benchmarks [18. 19]. We excluded other tasks because they could not be solved by the RL algorithm even with substantial tuning. See the appendix for more details.
Aaaiienn-Humar (PH) and Multi-Human (MH). Datasets are collected by humans through Robo- Turk [15, 17]. a remote teleoperation platform. The PH datasets consist of 200 demonstrations collected by single, experienced teleoperator, while the MH datasets consist of 300 demonstrations, collected by 6 telcoperators of varying proficiency, each of which provided 50 demonstrations. The 6 teleoperators consisted a "better" group of 2 experienced operators, an "okay" group of adequate operators, and a "worse" group of 2 inexperienced operators. These data subsets in the Multi-Human data allowed us to investigate the ability of algorithms to deal with mixed quality human data.
Observation Modalities. To study the effect of observation modalities, we capture a diverse set of sensor streams when collecting the dataset, including end-effector, gripper fingers, and joints, ground- truth object poses, and images from an external camera and wrist-mounted camera per robot arm (see Appendix E). We have two observation spaces- "low-dim" and "image". Both include end-effector poses and gripper finger positions, and only differ in whether ground-truth object information is used (low-dim) or whether that information replaced by the available camera observations (image).
3.3 Training and Evaluation Protocols
There are several approaches to offline imitation learning [31-34, 9, 10, 25, 35, 36] and offline reinforcement learning [26, 27, 37-43) (see Appendix for more discussion on related work). We chose to evaluate 6 algorithms in this study Behavioral Cloning (BC), BC with an RNN policy (BC-RNN), Hierarchical Behavioral Cloning (HBC) [10], Batch-Constrained Q-Learning (BCQ) [26]. Conservative Q-Learning (CQL) [27], and IRIS [20]. BC-RNN, HBC, and IRIS have all been used

---

100
100
120
Catisee
Dataset
Dataset
Welotive Succees Kare tharge
75
Suar
Chaagge
75
75
60
60
800
Squaee
Traeper
Retetive Successs
-100
-100
larger
larger MEP
smaaker J0NN ci
-100
larger
shallowe
amaler
EEF
.0811
itand
UR
GMM
Comv
dim
Observation Space
(b) Low-Dim Hyperparameters
(c) Image Hyperparameters
Figure 2: Effect of Observation Space and Hyperparameter Choice. We show how the success rate that BC-RNN obtains can drop drastically due to changes to the observation space and hyperparameter seltings.
Hrmme
Profolessihmmar
1385
kg
Teak
TIMMO0T
Traport
(a) Low-Dim
(b) Image
Figure 3: Effect of Dataset Size. We study how the BC-RNN success rate changes when lowering the quantity data to 20% and 50%. Results show that less complex tasks (Lift, Can) be learned with a fraction of the data, while more complex tasks might benefit from even larger human datasets.
in prior work to learn offline from teleoperated human demonstrations, while BCQ and CQL are commonly-used offline RL. algorithms (see Appendix C). We use binary task completion rewards for all our experiments. Each agent is trained for N epochs, where each epoch consists of gradient steps, and evaluated every E epochs, by running 50 rollouts in the environment and reporting the success rate over maximum horizon. For each agent, we report the maximum success rate over the coarse of training, and average over seeds. For low-dim agents, N = 2000, M 100, and E 50, and for image agents, N 600, M 500, and 20 (see Appendix B.2).
Experiments
In this section, we present each factor that explored in study, and note the relevant challenges from Sec. that each pertains to.
4.1 Algorithm Comparison on Single and Multi-Human Demonstrations (C1, C2)
We trained and evaluated all algorithms the Sefficient-Humaan (PH) and Multi-Human (MH) datasets and report the average success rates across 3 seeds Table 1.
Observation history is crucial for good performance. There is substantial performance gap between BC-RNN and BC, which highlights benefits vistory-dependence. The performance gap is larger for longer-horizon tasks (e.g. 55% for Transport (PH) compared to ~ 5% for Square (PH)) and for multi hhlmin data compared to single-human data (c.g. ~ 25% for Square (MH) compared to ~ 5% for Square (PH).. Interestingly, results are lower for MH datasets compared to PH datasets, even though the MH datasets contain 100 more demos (300 demos vs. 200 demos). This most likely stems from the presence of suboptimal and multimodal data in the MH datasets.
Batch RL algorithms perform poorly on Human Datasets. Recent batch (offline) RL algorithms such as BCQ and CQL have demonstrated excellent results in learning from suboptimal and multi- modal agent-generatee
datasets. Our results confirm the capacity of such algorithms to work well BCQ in particular performs strongly on our agent-generates MG datasets that consist of diverse mixture good and poor policies. Surprisingly though, neither BCQ nor CQL performs particularly datasets. This puts the ability of such algorithms learn from more
well on these
5

---

Relative Bucooss Rato Change
Sepue (PM as)
Datanet
Dataset
BC
BC-RNN
BCQ
CQL
Square (PH ini)
Transpon (Pr L15)
Lift (PH)
10000:0.00
100.0::.00
98.0116
52.0 13.0
Thanspan (PH in
Can (PH)
97.3= 1.9
9.9
86.7
0.7
Square (PH)
62.0=4.9
8..0=0.0
41.344.1
Transport (PH)
55.3 - 6.2
4.3
0.7+0.99
Tool Hang (PH)
20.0=
6.1
3.3
Lift (MH)
100.0+0.0
100.0+0.00
93.3++0.9
11.3+
Can (MH)
0,9
1.6
77.3+688
0.0
-100
Square (MH)
46.0= 1.6
76.7
17.3+
Vallo (BC)
Vildd (BC-RNN) (BC) Last (SC-RhN)
Transport (MH)
2.5
4.
0.0
(a) Effect Policy Selection Criteria
Table 3: Results on Image Observations.
Figure 5: (left) Effect of Policy Selection Criteria. We compare how performance decreases when choosing the policy to ovaluate by using the lowest validation loss, or when using the final trained checkpoint, with respect to the best policy performance. (right) Results on Image Observations. We present success rates for cach method across the image observation human datasets. BC-RNN maintains ncarly the same performance as lcarning from ground-truth observations, providing an optimistic view for learning with real-world raw sensory observations.
natural dataset distributions into question (instead of those collected via RI. exploration or pre-trained agents). There is opportunity for future work in batch RL to resolve this gap.
4.2 Learning from Suboptimal Human Data (C2)
To further investigate how algorithms deal with suboptimal human data, we split our MH datasets into smaller subsets based on the proficiency of the human operators. The MH-Better, MH-Okay, and MH-Worse are the 100 demo subsets corresponding to the "better", 2 "okay", and 2 "worse" operators respectively, while MH Worse-Okay, MH Worse-Better, and MH Okay-Better are the 200 demo subsets corresponding to the mixture of the previous subsets. Similar data mixtures have been used for evaluations in batch RL [18]. Appendix B shows the average trajectory lengths in each data subset - lower quality datasets contain demonstrations that take more time to solve the task.
BC-RNN is strong baseline suboptimal human data, but there is room for improvement. Table 2 shows that BC exhibits a large performance gap between the Better and Worse 100-demo subsets (roughly 27% and 35% for Can and Square respectively). Interestingly, BC-RNN is able to nearly eliminate this gap in performance on the Can task, but not on the Square task. However, BC-RNN outperforms BC all datasets (7%-35% improvement). Comparing results on the 100 Better demonstrations and 100 Okay demonstrations to the 200 Worse-Better demonstrations and 200 Worse-Okay demonstrations further allows
analyze how adding 100 "worse" demonstrations impacts the performance of each algorithm. Most algorithms decline in performance while BC-RNN able to uniformly improve from the added data. Comparing the performance of BC-RNN on the 200-demo Square mixture datasets (55.3%, 3%.7%%.. 73.3%, 74.0%) to the high-quality 200-demo Square (PH) dataset (84.0%) shows that there is still room for algorithms improve on the use of this data.
Diagnostic dataset shows that Batch RL struggles in simpler settings as well. The final row of Table 2 shows additional results on a diagnostic dataset termed Can-Paired, where single operator collected 2 demonstrations for each of 100 task initializations one successful demonstration, and one where the
tossed outside of the bin (task failure), for total of 200 demonstrations. There a strong expectation for batch RL algorithms to be able to distinguish between actions leading successful placement and actions leading to task failure, but even this simple setting, most algorithms suffer. providing a pessimistic view of the state-of-the-art. The 5% improvement that IRIS provides over BC-RNN suggests that introducing depery-deeeedeece into state-of-the-a batch RL algorithms might be a promising direction for future work.
4.3 Effect of Observation Space (C5)
Learning from image observations can match low-dim agent performance. In Table 3, we present policy learning results when using image observations instead of ground-truth object locations - an important setting for real-world policy learning. BC-RNN still maintains superior performance improvements over BC on the complex Square and Transport tasks, and with the exception of Transport (MH), maintains nearly the same performance as learning from ground-truth observations. This result provides an optimistic view for learning with real-world raw sensory observations.

---

Features used for robot proprioception can matter. In Fig 2a, we study the effect of adding end ef- fector velocities to the observations (+ EEF Vel), and joint positions and velocities to the observations (+ Joint). Surprisingly, we find that including end effector velocity information, and joint information hurts agents trained on low-dim observations substantially (49%-88%) relative performance drop), while image-based agents are more tolerant to the inclusion of this extra information (2%-29% relative performance drop). We hypothesize that performance drops might be due to overfitting to the presence of this extra information not needed for solving these tasks. Thus, practitioners should take care to engineer the robot observation space and exclude possibly irrelevant information information-hidinn -hiding can be powerful paradigm for training proficient robots [44].
Image randomization and wrist observations can be erucial for manipulation tasks. In Fig 2a. we report performance drops from removing pixel shift image randomization (- Rand) and the wrist camera (- Wrist) from image-based agents to understand their importance. We see that removing randomization results in 47% and 35% relative performance drops on Square and Transport respec- tively, and removing wrist images results in 9% and 43% relative drops. Consequently, both wrist camera images and image randomization play a substantial role in producing performant policies. We confirm the importance of each for visuomotor imitation in the real world as well (see Sec 4.7). Wrist observations likely help the robot improve gripper alignment for grasping and randomization helps the policy develop invariance for portions of the image that are not important for action prediction.
4.4 Effect of Hyperparameter Choice (C5)
In this section, we take our default hyperparameters for BC-RNN and study the effect of changing a subset of them to report practical recommendations for learning from human datasets (see Appendix I for BCQ and CQL). We present our results in Fig 2b (low-dim) and Fig 2c (image).
(larger LR) Increasing the learning rate from lc-4 to 1e-3 affects the performance of image-agents substantially (drop of 35%-63%), while low-dim agents are more tolerant to the change. (no GMM) Using a deterministic policy instead of learning a GMM action distribution results in significant relative performance drops the MH datasets (especially low-dim Transport, with drop of 58%). (larger MLP) Using a larger MLP size at each RNN timestep reduces performance uniformly, suggesting that it is possible to overfit to dataset actions if network architectures are too large. (shallow Conv) Using a shallow convolutional network [45] instead of the ResNet backbone [46] for encoding image observations reduces performance significantly with relative drops of 25%-62% suggesting that large-capacity visual encoders are crucial for visuomotor imitation. (smaller RNN dim) Reducing the size of the RNN hidden dimension from 400 to 100 (low-dim) and 1000 400 (image) uniformly decreases performance (drops of 3%-58%), showing the importance of a large RNN hidden dimension. (Recommendations) We recommend tuning the LR (especially for image agents) and network structure (MLP size, size of RNN dim) carefully. Opting to use a GMM policy and ResNet encoder appears to be uniformly better.
4.5 Selecting a Policy to Evaluate (C4)
Model selection in offline policy learning can be challenging for this reason, in simulation experiments, we evaluated every policy checkpoint online and reported the best one. This is not feasible for real-world settings, making offline policy selection desirable. In Fig 4a, show that this can be non-trivial, by showing the relative performance drop when selecting the policy using the best loss validation data (common in supervised learning), and when using the final training checkpoint as well (common in offline RL [47, 18, 27]) - in both cases, the selected policy is significantly worse than the best one (10% to 100% decrease). See Appendix G for more detailed results and discussion. This motivates the need for better offline evaluation metrics.
4.6 Effect of Dataset Size (C3)
To study how dataset size impacts performance, we formed smaller 20% and 50% subsets of our human datasets by sampling trajectories. We evaluate low-dim and image BC-RNN agents across these subsets Table 27 and Table 28. There are several promising results here. We first note that less complex tasks (Lift, Can) can yield proficient policies (75%-1005) success rate) using a small fraction of the data (20%). Second, while policies trained on more complex tasks (Square, Transport) suffer substantially when using 50% or 20% of the data, the converse is also true adding more data
7

---

(e.g. moving from 20% to 50% or 50% to 100% size) can result in significant policy improvement. This confirms the value of using large human datasets as a means to obtain proficient policies for challenging and complex manipulation tasks.
4.7 Applicability to Real-World Settings
Here, we show that design decisions made in simulation can potentially transfer to real world settings. We collected 3 additional real-world datasets with a Franka robotic arm - Lift (Real), Can (Real), and Tool Hang (Real). Each consists of 200 trajectories collected by one operator. We train BC-RNN and report the final policy checkpoint success rate, over 30 rollouts, due to the time-consuming nature of real world policy evaluation. We also emphasize that no real-world hyperparameter tuning took place, so our results are a lower bound. We were able to train proficient Lift (96.7% % and Can (73.3%) %) policies, and the Tool Hang (3.3%) policy is able to generate some task successes, despite the extremely difficult nature of the task. Furthermore, as in Sec. 4.3, we validate the importance of pixel shift randomization and the wrist camera by ablating each component on the Can task, and show that including both is the difference between a proficient and non-proficient real-world policy Can (- Rand) (26.7% 7), Can (- Wrist) (43.3%).
Discussion
In this section, summarize the lessons from study and make recommendations for future work.
(L1) Models with temporal abstraction can be extremely effective learning from human datasets. In Sec 4.1 and Sec 4.2, we demonstrated that history-dependent models (BC-RNN, HBC, and IRIS) are particularly effective in learning from human datasets compared to algorithms that do not take temporal context into account.
(L2) Need to improve the ability of batch (offline) RL to learn from suboptimal human datasets. Sec 4.2 and Appendix I demonstrated that oftt-offtthe-at batch RL algorithms are excellent at learning from suboptimal machine generrted datasets but much worse learning from suboptimal human datasets. They even struggled with diagnostic dataset with paired good and bad human demonstration trajectories while IRIS was able to improve slightly on BC-RNN, suggesting that combining vistory-dependence with value learning might be good place to start for improving batch RL methods [48-51]. This also demonstrates need to start benchmarking new batch RI algorithms on human datasets instead of purely on machine-genede
datasets.
(L3) Improving offfine policy selection is important for real world settings. See 4.5 demonstrated the need for better ways to select an evaluation policy in an offline manner. We hope that our datasets can help supplement other efforts [29].
(L4) Observation space plays a large role and hyperparameters matter. Sec 4.3 demonstrates that policies trained on low-dim observations can be very sensitive to the choice of robot propriocep- tion, while pixel shift randomization and wrist camera images are critical for effective visuomotor policy learning. The choice of observation space for imitation merits careful consideration other work has also confirmed the importance of feature representations used for offline policy learn- ing [52, 53]. Sec 4.4 and Appendix made practical recommendations for choosing hyperparameters to learn from human data.
(L5) There is substantial promise for solving more complex tasks using large-scale human datasets. Sec 4.6 showed that adding more data can result in significant policy improvement on complex tasks. Table and Sec 4.7 shows that could learn proficient policies the Tool Hang task, our most complex task, without any hyperparameter tuning on the task or dataset. Together, these results show the potential of large human datasets means solve challenging and complex manipulation tasks.
(L66 Study results transfer to real-world settings. In Sec 4.7, we showed that we could directly apply hyperparameters that were tuned on simulated tasks directly to real-world datasets and tasks. This provides promise for using in simulation, while also being confident that conclusions can transfer to real- settings.
tasks, datasets, and codebase to enable reproducible evaluation
Going forward, we hope that the datasets, tasks, code, and subsequent insights of our study will serve researchers and practitioners alike.
8

---

Acknowledgments
We would like to thank Albert Tung for helping with the RoboTurk data collection system, Jim Fan for providing timely lab cluster support, and Helen Roman for helping order items for the physical robot tasks. Ajay Mandlekar acknowledges the support of the Department of Defense (DoD) through the NDSEG program. We acknowledge the support of Toyota Research Institute ("TRI"); this article solely reflects the opinions and conclusions of its authors and not TRI any other Toyota entity. We acknowledge the support the US Army Research Office (award W91 1911NF-15 1--59-)-999) and the National Science Foundation (award CNS- CNS-1955522 1955523). This work relates to Department of Navy award N00014-14-1-0671 issued by the Office of Naval Research.
References
[1] J. Deng, W. Dong, R. Socher, Li, K. Li, and Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[2] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097-1105,, 2012.
[3] J. Redmon, S. Divvala, R. B. Girshick, and A. Farhadi. You only look once: Unified, real-time object detection. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 779-788, 2016.
[4] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
[5] Rajpurkar, R. Jia, and P. Liang. Know What You Don't Know: Unanswerable Questions for SQuAD. arXiv preprint arXiv:1806.0388 2018.
[6] L. Floridi and M. Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30(4):681-694 2020.
[7] Devlin, M.-W. Chang, K. Lee, and Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
[8] D. A. Pomerleau. Alvinn: An autonomous land vehicle neural network. In Advances in neural information processing systems, pages 305-313, 1989.
[9] Zhang, McCarthy, O. Jow, D. Lee, Goldberg, and Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. arXiv preprint arXiv:1710.04615, 2017.
[10] A. Mandlekar, D. Xu, R. Martín-Martin S. Savarese, and L. Fei-Fei. Learning to generalize across long-horizon tasks from human demonstrations, arXiv preprint arX1v:2003.06085, 2020.
[11] S. Lange, T. Gabel, and M. Riedmiller. Batch reinforcement learning. In Reinforcement learning, pages 45-73. Springer, 2012.
[12] S. Levine, A. Kumar, G. Tucker, and Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.016643 2020.
[13] S. Cabi, S. Gómez Colmenarejo, A. Novikov, K. Konyushkova, S. Reed, R. Jeong, K. Zolna, Y. Aytar, D. Budden, M. Vecerik, et al. Scaling data- robotics with reward sketching and batch reinforcement learning. arXiv, pages arXiv-1909, 2019.
[14] P. Florence, L.. Manuelli, and R. Tedrake. Self-supervised correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters, 5(2):492-499, 2019.
[15] A. Mandlekar, Y. Zhu, A. Garg, J. Booher, M. Spero, A. Tung. J. Gao, J. Emmons, A. Gupta, E. Orbay, Savarese, and I.. Fei-Fei. RoboTurk: Crowdsourcing Platform for Robotic Skill Learning through Imitation. In Conference on Robot Learning, 2018.

---

[16] P. Sharma, L. Mohan, L. Pinto, and A. Gupta. Multiple interactions made easy (mime): Large scale demonstrations data for imitation. In Conference on robot learning, pages 906-915. PMLR, 2018.
[17] A. Mandlekar, J. Booher, M. Spero, A. Tung, A. Gupta, Y. Zhu, A. Garg, S. Savarese, and L. Fei-Fei. Scaling robot supervision to hundreds of hours with roboturk: Robotic manipulation dataset through human reasoning and dexterity. arXiv preprint arXiv:1911.04055 2019.
[18] Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning, arXiv preprint arXiv:2004. 02.:200240.221. 2020.
[19] C. Gulcehre, Z. Wang, A. Novikov, T. L. Paine, S. G. Colmenarejo, K. Zolna, R. Agarwal, J. Merel, Mankowitz, C. Paduraru, et RI unplugged: Benehmarks for offline reinforcement learning. arXiv preprint wrXiv:2006.138880 2020.
[20] A. Mandlekar, F. Ramos, B. Boots, S. Savarese, L. Fei-Fei, A. Garg, and D. Fox. Iris: Implicit reinforcement without interaction at scale for learning control from offline robot manipulation data. In JEEE International Conference on Robotics and Automation (ICRA), pages 4414-4420. IEEE, 2020.
[21] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635, 2011.
[22] D. Precup. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series, page 80, 2000.
[23] Toyer, R. Shah, A. Critch, and S. Russell. The magical benchmark for robust imitation. arXiv preprint arXiv:201 2020.
[24] S. James, Z. Ma, D. R. Arrojo, and A. J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019 3026,, 2020.
[25] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, al. Transporter networks: Rearranging the visual world for robotic manipulation. arXiv preprint arXiv:2010. 14406, 2020.
[26] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. In International Conference on Machine Learning, pages 2052-2062. PMLR, 2019.
[27] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning, arXiv preprint arXiv:2006. 04779, 2020.
(28) T. L. Paine, C. Paduraru, A. Michi, C. Gulcehre, K. Zolna, A. Novikov, Z. Wang, and N. de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint arXiv:2007009055, 2020.
[29] Fu, M. Norouzi, O. Nachum, G. Tucker, Z. Wang, A. Novikov, M. Yang, M. R. Zhang, Y. Chen, A. Kumar, et al. Benchmarks for deep off- off-policy evaluation. arXiv preprint 11X.::2003.1690 2021.
[30] Haarnoja, A. Zhou, P. Abbeel, and Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference Machine Learning, pages 1861-1870. PMLR, 2018.
[31] A. J. Ijspeert, J. Nakanishi, and S. Schaal. Movement imitation with nonlinear dynamical systems in humanoid robots. Proceedings 2002 IEEE International Conference Roboties and Automation, 2: 2:1398-1403 vol.2, 2002.
[32] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine. One-shot visual imitation learning via meta-learning. Conference Robot Learning, volume abs/1709.00e 0090044 2017.
[33] A. Billard, S. Calinon, R. Dillmann, and Schaal. Robot programming by demonstration. In Springer Handbook of Robotics, 2008
10

---

[34] S. Calinon, F. D' D'halluin, E. L. Sauser, D. G. Caldwell, and A. Billard. Learning and reproduction of gestures by imitation. IEEE Roboties and Automation Magazine, 17:44-54, 2010.
[35] C. Wang, R. Wang, D. Xu, A. Mandlekar, L. Fei-Fei, and S. Savarese. Generalization through hand-eye coordination: An action space for learning spatially-invariant visuomotor control. arXiv preprint arXiv:2103.00375, 2021.
[36] A. Tung, J. Wong, A. Mandlekar, R. Martin-Martin, Zhu, L. Fei-Fei, and S. Savarese. Learning multi-arm manipulation through collaborative teleoperation. arXiv preprini arXiv:20/2.06736, 2020.
[37] Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 1rXiv::911.113300 2019.
[38] Z. Wang, A. Novikov, K. Zolna, J. T. Springenberg, S. Reed, B. Shahriari, N. Siegel, J. Merel, C. Gulcehre, N. Heess, et al. Critic regularized regression. arXiv preprint arXiv:2006. 15134, 2020.
[39] N. Y. Siegel, J. T. Springenberg, F. Berkenkamp, A. Abdolmaleki, M. Neunert, T. Lampe, R. Hafner, N. Heess, and M. Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprini arXiv:2002.08399 2020.
[40] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel: Model-based offline reinforcement learning, arXiv preprint arXiv:2005.0590 2020.
[41] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005. 202200055.3339. 2020.
[42] S. K. S. Ghasemipour, D. Schuurmans, and S. S. Gu. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. arXiv preprint arXiv:2007.10091, 2020.
[43] T. Yu, A. Kumar, R. Rafailov, A. Rajeswaran, S. Levine, and C. Finn. Combo: Conservative offline model-based policy optimization. arXiv preprint arXiv:2102.08366 2021.
[44] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47), 2020.
[45] C. Finn, X. Y. Tan, Y. Duan, T. Darrell, S. Levine, and Abbeel. Deep spatial autoencoders for visuomotor learning. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 512-519. IEEE, 2016.
[46] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[47] R. Agarwal, D. Schuurmans, and M. Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pages 104-114. PMLR, 2020.
[48] A. Ajay, Kumar, Agrawal, Levine, and O. Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010.136/ 2020.
[49] K. Pertsch, Y. Lee, and J. Lim. Accelerating reinforcement learning with learned skill priors. arXiv preprint arXiv:2010.11944, 2020.
[50] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, Abbeel, A. Srinivas, and Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint arXiv:2106. 01340, 2021.
[51] M. Janner, Q. Li, and S. Levine. Reinforcement learning as one big sequence modeling problem. arXiv preprint arXiv: 102.020000033 2021.
[52] M. Yang and Nachum. Representation matters: Offline pretraining for sequential decision making. arXiv preprint arXiv:2102.0588 2021.

---

[53] O. Nachum and M. Yang. Provable representation learning for imitation with contrastive fourier features. arXiv preprint arXiv:2105. /2272, 2021.
[54] M. Hersch, F. Guenter, S. Calinon, and A. Billard. Dynamical system modulation for robot learning via kinesthetic demonstrations. IEEE Transactions on Robotics, 24(6):1463-114 1467, 2008.
[55] Kormushev, S. Calinon, and D. G. Caldwell. Imitation learning of positional and force skills demonstrated via kinesthetic teaching and haptic input. Advanced Robotics, (5(5):5116003 2011.
[56] B. Akgun, M. Cakmak, J. W. Yoo, and A. L.. Thomaz. Trajectories and keyframes for kines- thetic teaching: A human-robot interaction perspective. In Proceedings of the seventh annual ACM/IEEE international conference on Human-Robor Interaction, pages 391-398, 2012.
[57] A. Bajesy, D. P. Losey, M. K. O' O'Malley, and A. D. Dragan. Learning robot objectives from physical human interaction. Proceedings of Machine Learning Research, 78:217-226, 2017.
[58] A. Bajesy, D. P. Losey, M. K. O'Malley, and A. D. Dragan. Learning from physical human corrections, one feature at time. In Proceedings of the 2018 ACM/IEEE International Conference on Human-Robe
Interaction, pages 141-149, 2018.
[59] A. Mandlekar, D. Xu, R. Martin Martin-Marttn, Y. Zhu, L. Fei-Fei, and S. Savarese. fluman-intther the-loop imitation learning using remote telcoperation. arXiv preprint
2020.
[60] R. Zhang, F. Torabi, L. Guan, D. H. Ballard, and P. Stone. Leveraging human guidance for deep reinforcement learning tasks. arXiv preprint arXiv: / 11999999999,6.. 2019.
[61] R. Loftin, B. Peng. J. MaeGlashan, M. L. Littman, M. E. Taylor, Huang, and D. L. Roberts. Learning behaviors via human-delivered discrete feedback: modeling implicit feedback strate- gies to speed up learning. Autonomous agents and multi-agent systems, 30(1):30-59, 2016.
[62] MacGlashan, M. K. Ho, R. Loftin, B. Peng, D. Roberts, M. E. Taylor, and M. L. Littman. Interactive learning from policy-dependent human feedback. arXiv preprint arXiv: / 170100609,, 2017.
[63] Christiano, Leike, T. Brown, M. Martic. S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, pages 4299-4307, 2017.
[64] A. Singh, L. Yang, K. Hartikainen, C. Finn, and S. Levine, End-to-end robotic reinforcement learning without reward engineering. arXiv preprint 1rXiv:1904.0785% 2019.
[65] Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning, pages 1329- 1338. PMLR, 2016.
[66] T. Wang, X. Bao, I. Clavera, J. Hoang, Y. Wen, E. Langlois, S. Zhang, G. Zhang, P. Abbeel, and J. Ba. Benchmarking model-based reinforcement learning. arXiv preprint arXiv:1907.00000 1907.02077, 2019.
[67] M. Andrychowicz, A. Raichuk, P. Stańczyk, M. Orsini, S. Girgin, R. Marinier, L. Hussenot, M. Geist, O. Pietquin, M. Michalski, et al. What matters in on-policy reinforcement learning? large-scale empirical study. arXiv preprint arXiv:2006.05990 2020.
[68] P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. Deep reinforce- ment learning that matters. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
[69] R. Memmesheimer, L. Mykhalchyshyna, V. Seib, and D. Paulus. Simitate: A hybrid imitation learning benchmark. arXiv preprint arXiv:
2019.
[70] L. Hussenot, M. Andrychowizz, D. Vincent, R. Dadashi, A. Raichuk, L. Stafiniak, S. Girgin, R. Marinier, N. Momehev, S. Ramos, et al. Hyperparameter selection for imitation learning. arXiv preprint arXhv:2105.12034 2021.
12

---

[71] M. A. Rana, D. Chen, J. Williams, V. Chu, S. R. Ahmadzadeh, and S. Chernova. Benchmark for skill learning from demonstration: Impact of user experience, task complexity, and start config- uration on performance. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 7561-7567. IEFE, 2020.
[72] A. Lemme, Y. Meirovitch, M. Shansari-Zadeh T. Flash, A. Billard, and J. J. Steil. Open- source benchmarking for learned reaching motion generation in robotics. Paladyn, Journal of Behavioral Robotics, 6(1,, 2015.
[73] L. Fan, Y. Zhu, J. Zhu, Z. Liu, O. Zeng, A. Gupta, J. Creus-Costa, S. Savarese, and L. Fei-Fei. Surreal: Open-source reinforcement learning framework and robot manipulation benchmark. In Conference on Robot Learning, pages 767-782. PMLR, 2018.
[74] X. Lin, Y. Wang. J. Olkin, and D. Held. Softgym: Benchmarking deep reinforcement learning for deformable object manipulation. arXiv preprint oxXiv2201..07211 2020.
[75] Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning, pages 1094-1100. PMLR, 2020.
[76] Y. Zhu, J. Wong, A. Mandlekar, and R. Martín-Martin robosuite: A modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009. 12293, 2020.
[77] O. Ahmed, F. Träuble, A. Goyal, A. Neitz, Y. Bengio, B. Schölkopf, M. Wüthrich, and S. Bauer. Causalworld: robotic manipulation benchmark for causal structure and transfer learning. arXiv preprint arXiv:2010.04200 2020.
[78] D. P. Kingma and Ba. Adam: A method for stochastic optimization. arXiv preprint mrXiv:1412.66 2014.
[79] D. P. Kingma and M. Welling, Auto-encoding variational bayes. arXiv preprint 2013.
[80] T. P. Lillicrap. J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 150900271, 2015. 2015.
[81] I.. Biewald. Experiment tracking with weights and biases, 2020. URL / wandb. com/ Software available from wandb.com.
[82] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033, 2012.
[83] O. Khatib, A unified approach for motion and force control of robot manipulators: The operational space formulation. IEEE Journal on Robotics and Automation, 3(1):43-53, 1987.
[84] M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learning with augmented data. arXiv preprint arXiv:2004. 14990, 2020.
[85] I. Kostrikov, D. Yarats, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.13647 2020.
[86] S. Young, D. Gandhi, S. Tulsiani, A. Gupta, Abbeel, and L. Pinto. Visual imitation made easy. arXiv e-prints, pages arXiv-2008, 2020.
[87] A. Zhan, P. Zhao, L. Pinto, P. Abbeel, and M. Laskin. A framework for efficient robotic manipulation. arXiv preprint arXiv:2012.078 2020.
[88] M. Hoffman, B. Shahriari, J. Aslanides, G. Barth-Maron, Behbahani, T. Norman, A. Ab- dolmaleki, A. Cassirer, F. Yang, K. Baumli, S. Henderson, A. Novikov, S. G. Colmenarejo, S. Cabi, C. Gulcehre, T. L. Paine, A. Cowie, Z. Wang, B. Piot, and N. de Freitas. Acme: A research framework for distributed reinforcement learning. arXiv preprint 27X.9.0006.00979 2020. URL https:://// /ttp//aaxvvv////////////////aav/ org/
00979.
13


Reviews:
[{"review_id": "xfoo2mj37hh", "reviewer": "Reviewer_wVDW", "summary": "\nThis paper conducts extensive evaluations on learning from offline demonstrations for robot manipulation tasks. Specifically, it studies six offline learning algorithms (i.e. BC, BC-RNN, HBC, BCQ, CQL, and IRIS) on five simulated and three real-world robot manipulation tasks of varying complexity with datasets of varying quality. This work presents a variety of lessons learned from the study and makes recommendations for future work accordingly. In sum, this work presents systematical analyses of existing methods focused on robot manipulation with offline demonstrations. I believe this kind of work benchmarking existing methods is one key to measure the advance of the field. While I have some concerns regarding limited robot configuration and demonstrations, I am leaning toward accepting this paper. Yet, my research interests are not directly related to this research direction, and therefore my evaluations should be taken lightly. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n## Paper strengths and contributions\n**Motivation**\nMost of the recent papers studying robot manipulation with offline datasets usually set up their own environments/tasks and collect their own demonstrations. This makes it very difficult to draw a fair comparison among the methods proposed in these papers or measure the advance in this field. Therefore, I believe systematically benchmark existing methods would be beneficial to the community.\n\n**Interesting insights**\nThis work presents a variety of insights on learning robot manipulation tasks with offline demonstrations. The following findings are particularly interesting to me:\n- Leveraging history is important when learning human demonstrations based on the performance gap between BC and BC-RNN.\n- Offline RL methods perform well when learning from agent-generated demonstrations but not from human demonstrations\n- Including end effector velocity and joint information hurts agents trained on low-dim observations\n- Selecting the policy using the best loss on validation data or using the final training checkpoint yields a significant performance drop compared to the real best policies.\n\n**Clarity**\n- The overall writing is clear, including what the goals are, how the experiments are set up and conducted, etc.\n- The challenges are clearly stated.\n- The findings are well-organized.\n- The presentation of the experimental results is clear.\n- Most of the analyses of the experimental results are detailed, comprehensive, and convincing.\n\n**Environments & tasks & demonstrations**\nThe effort on setting put the environments in simulation and real-world as well as collecting demonstrations is highly appreciated.\n\n**Diverse demonstrations**\n- Evaluating methods learning from demonstrations of varying quality provides informative findings. \n- Comparing models learning from agent-produced demonstrations and human-generated demonstrations presents interesting insights.\n- Training models to learn from different numbers of demonstrations yields results that align with intuitions.\n\n## Paper weaknesses and questions\n\n**Limited robot configuration**\nThis work only conducts studies with a Franka robotic arm in both simulated and real-world environments. Therefore, it is not easy to judge if the majority of the findings presented in this work would apply to other robot configurations. While I do recognize the difficulty of accessing a variety of robot configurations, especially for an academic lab, I believe it is still possible to conduct studies with other type of robots at least in simulation to justify the findings. \n \n**Converged machine-generated demonstrations**\nThe machine-generated dataset consists of demonstrations collected from checkpoints along with the training of expert policies. It makes sense to do so to collect suboptimal demonstrations and it may seem fair since human-generated datasets probably also contain trajectories collected when human demonstrators just start learning to use the teleoperation platform. However, human demonstrators do have an understanding of the tasks from the start and have prior knowledge such as physics. Therefore, I believe the suboptimal trajectories from the machine-generated datasets could be much worse than the suboptimal ones in the human-generated datasets, which can even enlarge the performance gap between models learning from the machine and human-generated demonstrations.\n\n**Short-horizon tasks**\nThis paper mainly studies robot manipulation tasks with short horizons. While tasks such as Tool Hang and Transport consisting of multiple subtasks and might take hundreds of time steps, they do not examine robots' ability to plan. Therefore, the evaluations are all about the precision of the execution but not about the ability to interpret the intentions or the high-level plans of demonstrators.\n\n**Domain gap**\nThe studies presented in this work only concern the setup where the agent and demonstrators share exactly the same environment. Yet, in many cases of real-world applications, robots might need to deal with subtle differences between the environment it faces and the one from the demonstration. I believe it would be also important to study this case where this domain gap exists.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "\nThis work presents systematical analyses of existing methods focused on robot manipulation with offline demonstrations. I believe this kind of work benchmarking existing methods is one key to measure the advance of the field. While I have some concerns regarding limited robot configuration and demonstrations, I am leaning toward accepting this paper. Yet, my research interests are not directly related to this research direction, and therefore my evaluations should be taken lightly. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "\nDescribed in the strengths and weaknesses section.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {}}, {"review_id": "s0xL-L0nM2G", "reviewer": "Reviewer_mRsh", "summary": "The paper conducts a study of evaluating several imitation learning and batch offline RL algorithms when trained on offline demonstrations in simulated and real-world environments. In particular, such algorithms as Behavioral Cloning (BC), BC with recurrent neural networks, Hierarchical BC, Batch-Constrained Q-Learning (BCQ), Conservative Q-Learning (CQL) and IRIS are evaluated. Experiments include simulated and real lifting, simulated and real pick-and-place, simulated high-precision pick-and-place, simulated bi-manual transport, simulated and real articulate tool hanging tasks. Datasets used for experiments comprise machine-generated, and human-generated data, whereas human demonstrations are categorized based on the level of experience of the demonstrator. Overall the main takeaways from experiments are that using recurrent models is important for learning from human demonstrations, batch offline RL algorithms do not work well with human-generated data, offline policy selection is important ant it is difficult in the real world, including certain observation dimensions can lead to increased overfitting and more data leads to increased performance. Paper also provides some practical ways to tune hyper-parameters, and demonstrates that using algorithms and parameters selected in simulation transfers to learning on a real robot.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The problem of learning from offline demonstration data is important for initializing robot policies, improving their safety and scaling up robot learning.\n- The paper is well-written and easy to understand and follow.\n- The paper conducts a wide range of experiments, including real-world robot evaluations on non-trivial tasks.\n- The paper contains a range of practical insights that will be useful for robot learning practitioners.\n\nWeaknesses:\n- In general, although offering a great comparison of various algorithms, the paper doesn’t introduce novel algorithmic components.\n- It is not always completely clear how true rewards are used during the optimization. For example, machine-generated and can-paired data contain both, success and failures. It looks like BC-based algorithms are not using the success/failure label and batch RL algorithms are using them. It would be useful if paper stated more clearly how these ground truth labels are used.\n- The paper makes a conclusion that batch RL algorithms do not work well with human-generated data. However, it is indicated that for machine-generated data, 6 checkpoints with 300 rollouts each are included in the dataset, whereas human-generated datasets only include 200 or 300 demonstrations in total, so at least 6 times less data. Another conclusion that can be made from these experiments is that batch RL algorithms work worse with a low amount of data. It would be interesting to see experiments that disambiguate these two conclusions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper provides a great comparison of various algorithms for offline imitation learning, but lacks some novelty in theoretical contributions.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please refer to \"Strengths And Weaknesses\" for a detailed list of issues.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "akwaqYvNJ-i", "reviewer": "Reviewer_XeZV", "summary": "This paper identifies challenges of learning from offline human demonstrations for robot manipulation and analyzes the performance of 3 BC algorithm variants (BC, BC-RNN, HBC) and 3 batch-RL algorithms (BCQ, CQL, and IRIS) across 5 simulated and 3 real-robot manipulation tasks of varying difficulty, 3 different dataset generation sources, two observation spaces, and various hyper-parameter choices, while providing various relevant insights.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Identification of challenges of learning from offline human data\n- Extensive evaluation across various dimensions such as model choice, observation space choice, dataset source, and hyperparameters for manipulation tasks with varying difficulty\n- Useful insights on performance across these variables with suggestions for addressing the challenges\n\nSuggestions for Improvement:\n\n- Can the paper dive a little deep into what differences in the nature of the mixed quality machine-generated datasets when compared to mixed-quality human datasets make it difficult for SOTA RL algorithms such as BCQ to perform comparably with human datasets? \n\n- The effect of observation spaces in the performance for IL algorithms could be more profound than just low-dimension (object state estimation) vs. high-dimension (visual raw data). The observation space is also sensitive to frame of reference (object reference or robot reference) and multi-modality sensing. For example: how would the trends change if force information was available as an observation in addition to the visual data? Or, how would the performance change if the robot states (gripper opening, end-effector poses etc.) are expressed in the object coordinate frame vs. robot-base coordinate frame? Also, what about the transport task made it difficult to be learned using the high-dimensional observation space? And, how does the performance vary when there is uncertainty in the low-dimensional object pose?\n\n- While the paper mentions the promise of using more data for complex tasks with high-dim observation space, collecting huge human datasets is very challenging realistically. Can the paper comment on the feasibility as well?\n\n- The paper can become stronger with comments on how these variables affect the performance in the presence of covariate shift, which BC algorithms generally succumb to \n\n- Can the insights extend beyond robot manipulation problems to other robotic applications? Kindly discuss\n\n- The recommendations given in Section 4.4 (C5 - Hyperparameter choice) are conditioned on a given dataset size. Some of the overfitting concerns with larger models may not exist with larger datasets. Can the paper discuss the recommendations in light of other variables that may affect the performance? I would also revise L6 in the Discussion section by adding caution to the statement that the results transfer to real-world settings, especially because this is conditioned on the difficulty of a task. The paper would benefit by detailing what about a task makes it difficult in this context, is it the long-horizon factor, is it the precision factor, is it multi-arm (agent) factor, or is it all of the above, and are there other aspects of a task that could make it difficult for these learning tasks? These discussions can make the paper much stronger and more useful to the community in general.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This is an extensive study and the insights would be valuable to the community. I strongly support accepting the manuscript.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See “suggestions for improvement” section above\n", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "7uJZTUK62qW", "reviewer": "Reviewer_YubL", "summary": "The paper presents an exploratory study aimed at identifying over-arching challenges in learning manipulation skills from offline human-labelled datasets. Specifically, by varying factors such as the learning algorithm, dataset size, optimality of demonstrations, and task complexity, the study compared the performance of policy learned under various permutations of choices. Finally, the work distills six key lessons learned from the study.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n+ The paper presents a timely and important study that explores the challenges involved in learning manipulation tasks from offline demonstrations. \n+ The paper is well-written and very easy to read.\n+ Making the data and code available would make a big impact in the research communicate and help benchmark and compare new approaches.\n+ I really appreciate the careful and comprehensive design of the study that includes variations across various important axes such as task complexity, user expertise, perception modality, and learning paradigm (offline RL and imitation).\n+ Almost all design choices are carefully motivated and explained.\n\n**Weaknesses**\n\n- The paper is missing a few key existing works that attempt to benchmark learning from demonstrations (Rana et al., ICRA, 2020; Lemme et al., J. Beh. Robo., 2015) . It is important to acknowledge these contributions and contextualize the contributions made by this work within those of existing studies. While do not believe that these work subsume all contributions of the proposed work, there are non-trivial overlaps.\n- It is helpful to see the lessons learned clearly articulated and have numerical evidence to support them. However, the reported lessons are not surprising and seem intuitive. \n- The evaluation metrics seems to be limited to success rates. It would be more illuminate to analyze more metrics such as time taken to perform the task, smoothness of motion, and (subjective) user perception.\n\n**References**\n\n- Rana, et al. \"Benchmark for Skill Learning from Demonstration: Impact of User Experience, Task Complexity, and Start Configuration on Performance.\" 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020.\n- Lemme, Andre, et al. \"Open-source benchmarking for learned reaching motion generation in robotics.\" Paladyn, Journal of Behavioral Robotics 6.1 (2015).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "On the one hand, the paper presents a timely and comprehensive study of the challenges involved in learning manipulation policies from offline datasets. The study design is commendable and making the datasets and code available to the public will likely have a big impact on the research community. On the other hand, the paper does not acknowledge a couple of key prior studies with similar aspirations, and evaluates the performance of each permutation of factors solely based on task success rate.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "1. Would it be possible to compute more metrics (in addition to success rate) in the future if all the data and code are made available?\n2. Given that all the experiments are conducted on a single robot platform (the Franka arm) brief discussion surrounding the generalizability of these finds across different robot morphology would be very helpful."}}]

Author rebuttal:

Meta-review: All of the reviewers provided scores for accepting the paper. The authors addressed most of the points raised by the reviewers in their rebuttal. I encourage the authors to address the remaining points raised in the reviews for the final version of the paper.
Decision: Accept (Oral)