MimicPlay: Long-Horizon Imitation Learning by Watching Human Play
Authors: Chen Wang
Linxi Fan
Jiankai Sun
Ruohan Zhang
Li Fei-Fei
Danfei Xu
Yuke Zhu
Anima Anandkumar
Venue: CoRL 2023 Oral (2023)
Keywords: Imitation Learning
Learning from Human
Long-Horizon Manipulation

Abstract: Imitation learning from human demonstrations is a promising paradigm for teaching robots manipulation skills in the real world. However, learning complex long-horizon tasks often requires an unattainable amount of demonstrations. To reduce the high data requirement, we resort to human play data - video sequences of people freely interacting with the environment using their hands. Even with different morphologies, we hypothesize that human play data contain rich and salient information about physical interactions that can readily facilitate robot policy learning. Motivated by this, we introduce a hierarchical learning framework named MimicPlay that learns latent plans from human play data to guide low-level visuomotor control trained on a small number of teleoperated demonstrations. With systematic evaluations of 14 long-horizon manipulation tasks in the real world, we show that MimicPlay outperforms state-of-the-art imitation learning methods in task success rate, generalization ability, and robustness to disturbances. Code and videos are available at https://mimic-play.github.io.

MimicPlay: Long-Horizon Imitation Learning by Watching Human Play
Chen Wang', Linxi F Sun', Ruohan Zhang', Li Fei-Fei', Danfei Xu Stanford, "NVIDIA, 'Georgia Tech, 'UT Austin, CClttch, 'Equal Advising
, Yuke Zhu?4i, Anima Anandkumar
Abstract: Imitation learning from human demonstrations is promising paradigm for teaching robots manipulation skills in the real world. However, learning complex long-horizon tasks often requires an unattainable amount of demonstrations. To reduce the high data requirement, we resort human play data- people freely interacting with the environment using their hands. Even with different morphologies, hypothesize that human play data contain rich and salient information about physical interactions that can readily facilitate robot policy learning. Motivated by this, we introduce hierarchical learning framework named MIMICPLAY that learss latent plans from human play data to guide low-level visuomotor control trained on a small number of teleoperated demonstrations. With systematic evaluations of 14 long- horizon manipulation tasks in the real world, we show that MIMICPLAY outperforms satt--o-ttheart imitation learning methods task success rate, generalization ability, and robustness to disturbances. Code and videos are available at caamii-play.gihhubee
sequences of
Keywords: Imitation Learning, Learning from Human, Long-Horizon Manipulation
seconds human dala segmenk (fast)
MimicPlay
Human play data
Unlabelled in-domali videoo
High-level planner
seconds segment (slow)
Latent plan
Robot demos
Mulki-tak Ideeper
Low-leve control
Figure I: Human able to complete a long-horizon task much faster than a teleoperated robot. This observation inspires to develop MIMICPLAY, a hierarchical imitation learning algorithm that learns a high-level planner from cheap human play data and a low-level control policy from a small amount of multi-task teleoperated robot demonstrations. We show that MIMICPLAY significantly improves sample efficiency and robustness of imitation learning algorithms in long-horizon tasks.
1 Introduction
Efficiently teaching robots to perform general-purpose purpose manipulation tasks is a long-standing challenge.
Imitation Learning (IL. has recently made considerable strides towards this goal, especially through supervised training using either human teleoperated demonstrations or trajectories of expert policies [1, 2]. Despite this promise, IL methods have been confined learning short-horizon primitives, such as opening door or picking up a specific object. The time cost and labor intensity of collecting long-horizon demonstrations, especially for complex real-world tasks with wide initial and goal condition distributions, remains key barrier to the widespread adoption of IL methods.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Two connected directions have emerged in recent literature to scale up imitation learning to complex long-horizon tasks: hierarchical imitation learning and learning from play data. The former aims to increase learning sample efficiency by decoupling end-to-end deep imitation learning into the learning of high-level planners and low-level visuomotor controllers [3, 4]. The latter leverages an alternative form of robot training data, named play data [5]. Play data is typically collected through suman-teleoperated robots interacting with their environment without specific task goals guidance. Prior works show that data collected this way covers more diverse behaviors and situations compared to typical task-oriented demonstrations [5, 6). The methods for learning from such play data often seek to uncover such diverse behaviors by training a hierarchical policy [5], where the high-level planner captures the distribution of intent and the low-level policies learn goal-directed control. However, collecting such play data in the real world can be very costly. For example, C-BeT [6] requires 4.5 hours of play data to leam manipulation skills specific scene, and TACO-RL [7] needs 6 hours of play data for one 3D tabletop environment.
In this work, we argue that the data required for learning high-level plan and low-level control can come in different forms, and doing so could substantially reduce the cost of imitation learning for complex long-horizon tasks. Based on this argument, we introduce a new learning paradigm, in which robots learn high-level plans from human play data, where humans use their hands to interact with the environment freely. Human play data much faster and easier to collect than robot teleoperation data (Fig. 1). allows us to collect data at scale and cover a wide variety of situations and behaviors. We show that such scalability plays key role in strong policy generalization. The robot then leamss low-level manipulation policies from small amount of demonstration data, which is collected by humans teleoperating with the robots. Unlike human play data, demonstration data is expensive to collect but does not lead to issues due to the mismatch between human and robot embodiments.
To scale imitation learning to long- horizon manipulation tasks, we present MIMICPLAY, a new imitation leaming algorithm that leverages the complementary strengths of two data sources mentioned above: human play data and robot teleoperation data. MIMICPLAY trains a goal- coal-conditooned latent planner from human play data to predict the future 3D human hand trajectories conditioned on the goal images. Such latent plans provide rich 3D guidance (what to do and where to interact) at cach time step, tackling the challenging long-horizon manipulation problem by converting it into a guided motion generation process. Conditioned on these latent plans, the low-level controller incorporates state information essential for fined-grained manipulation to generate the final actions. We evaluate our method on 14 real-world long-horizon manip- ulation tasks in six environments. Our results demonstrate significant improvement over state-of-the-art imitation learning methods in terms of sample efficiency and generalization abilities. Moreover, MIMIC- PLAY integrates human motion and robotic skills into joint latent plan space, which enables an interface that allows using human videos directly as "prompts" for specifying goals in robot manipulation tasks.
To summarize, the main contributions of our work are as follows:
A novel paradigm for learning 3D-aware latent plans from cheap human play data.
A hierarchical framework that trains a plan-guided multi-task robot controller accomplish challenging long- long-horizon manipulation tasks sample-efficiently.
In 14 real-world long-horizon evaluation tasks, MIMICPIAYY shows state-of-the-an performance with generalization to novel tasks and robustness against disturbance, which further allows prompting robot motion with human videos.
2 Related Work
Imitation learning from demonstrations. Imitation Learning (IL) has enabled robots to successfully perform various manipulation tasks [8-15]. Traditional IL algorithms such as DMP and PrMP [16-19] enjoy high learning sample efficiency but are limited in their ability to handle high-dimensional observations and settings that require closed-loop control. In contrast, recent II. methods built upon deep neural networks can learn reactive policies from raw demonstration data [20, 2, 3, 21-24]. While these methods offer greater flexibility, they require large number of human demonstrations to leam even simple aa--and-place tasks, which remains labor- and resource- innuunceinteenvvee [25, 26). Our work instead proposes

---

to leverage human play data, which does not require robot hardware and can be collected efficiently, to reduce the need for on-robot demonstration data dramatically.
Hierarchical imitation learning. Our idea of learning hierarchical policy from demonstrations is also related to prior works [27, 3, 5, 4, 28). However, all previous methods focus on learning both planning and control with a single type of data- tell--teloperatedd robot demonstrations, which is expensive to collect. Our approach uses cheap human play data for learning high-level planning and a small number of robot demonstrations for learning low-level control, which significantly strengthens the model's planning capability while keeping a low demand on demonstration data.
Learning from human videos. A plethora of recent research has explored leveraging large-scale human video data to improve robot policy leamming [29-39]. Closely related to our work are R3M [40] and MVP [41], which use an Internet-scale human video dataset Ego4D [42] to pretrain visual representations for subsequent imitation learning. However, due to diversity in the data source and large domain gaps, transferring the pre-traines representation to a specific manipulation task might be difficult. Notably, Hansen al. [43] found simple data augmentation techniques could have similar effects these pre-trained representations. To reduce the domain gap, another thread of work [29, 44, 35, 45, 34, 46, 47) utilizes in in-domain human videos, where human directly interacts with the robot task environment with their own hands. Such type of data has a smaller gap between human and robot domains, which allows sample-efficient reward shaping for training RI. agents [35, 45-47] and imitation learning [29, 44, 34]. However, these works focus on learning either task rewards or features from human videos, which doesn't t directly help the low-level robot action generation. In this work, we extract meaningful trajectory-level task plans from human play data, which provides high-level guidance for the low-level controller for solving challenging long-horizon manipulation tasks.
Learning from play data. Our idea of leveraging human play data is heavily inspired by learning from play [5, 48, 6], an alternative imitation learning paradigm that focuses on multi-task learning from play data, form of teleoperated robot demonstration provided without a specific goal. Although play data exhibits high diversity in behavior [5], requires the laborious teleoperation process (4.5 hours [6] and 6 hours [7]). In this work, we instead learn from human play data, where humans freely interact with the scene with their hands. This method of data collection is not only time-effective, requiring mere 10 minutes, but it also provides rich trajectory-level guidance for the robot's 's motion generation. Consequently, the robot only requires minimal amount of teleoperation data, empirically less than 30 minutes, to translate the guidance into its own motor commands and successfully perform complex long-horizon mogg-ooizonn manipulation tasks.
3 MimicPlay
Training a robot for long-horizor tasks challenging, as requires high-level planning determine where and what to interact during different task stages, well low-level motor controls to handle how to achieve the goals. MIMICPLAY is based on the key insight that high-level planning can be effectively leamed from human play data that are fast to collect. Meanwhile, low-level control skills are best acquired from teleoperated demonstration data that do not have any embodiment gap. In particular, due to the difference in the embodiments of humans and robots, it is critical to find an intermediate representation that can bridge the gap between the two data sources. MIMICPLAY addresses this challenge by learning a 3D-aware latent planning space to extract diverse plans from cost-effective human play data. The overview of MIMICPLAY illustrated in Fig. 2.
3.1 Collecting human play data
Human play data. We leverage human play data, where a human operator freely interacts with the scene with one hand driven by their curiosity. For instance, in the kitchen, the operator might open the oven then pull out the tray or pick up a pan and place it on the stove. This type of data contains rich state transitions and implicit human knowledge of objects' affordance and part functionalities. More importantly, collecting human play data is cheaper and much more efficient than teleoperation, as it does not require task labeling environment resetting and takes only small fraction of time human operator can finish task that would take 90-second robot teleoperation time in just five seconds (Fig. 1). In this work.
3

---

Smalll smssent
play (mslli-iee))
mobtt
Goal
Goal BF gV
Ginal
Lasent planeer P
plan
Latent planner
Latest plan Ph
Latent planner
Latesa plan
Carcen image of
P
image of
image oz
GMM decoder
Robox policy
Roboc
Roboc policc
octoo
3T land l
traeetory
ling My Proprio.c de
Writ lin Mr Prupeius 42
Robot nution Oig
(4) Training stage
Learning latent plans
(b) Training stage 2- Plan-guided imitation learning
(c) Testing stage
Figure 2: Overview of MIMICPLAY. (a) Training Stage using cheap human play data to train goal-conditioned trajectory generation model to build latent plan space that contains high-level guidance for diverse task goals. (b) Training Stage 2: using a small amount of teleoperation data to train a low-level robot controller conditioned on the latent plans generated by the pre-trained (frozen) planner. (c) Testing: Given single long- monghoorioon task video prompt (either human motion video robot teleoperation video), MIMICPLAY generates latent plans and guides the low-level controller to accomplish the task.
collect 10 minutes of human play video as the training dataset for each task environment, which is approximately equivalent to 3-hour dataset of robot teleoperation video.
Tracking 3D human hand trajectories. When performing the play, human constructs a sequence of movements within their mind, then engages its hand to physically interact with the environment. This interaction creates hand trajectory that contains rich information regarding the individual's 's underlying intentions. Based on the hand trajectories, the robot can learn to mimic human's 's motion planning capability by reconstructing the hand trajectory conditioned on the goals. We will show how to train a latent planner from human hand trajectory data in Sec. 3.2. However, common human video datasets comprise single-view observations, providing only 2D hand trajectories. Such trajectories present ambiguities along the depth axis and suffer from occlusions. We instead use two calibrated cameras to track 3D hand trajectories from human play data. We use an off-the-shelf hand detector [49] to identify hand locations from two viewpoints, reconstructing a 3D hand trajectory based on the calibrated camera parameters. Details of the data collection process and system are introduced in the Appendix.
3.2 Learning 3D-aware latent plans from human play data
Given a long-term task represented by goal image, the policy should generate actions conditioned on this goal. We formalize the problem into a hierarchical policy learning task, where a goal-conditioned high-level planner P distills key features from the goal observation 9e and transforms them into low-dimensional latent plans Pt- These plans are then employed to guide the low-level motor controller toward the goal. However, learning such a vision-based motion planner requires large dataset since it needs to be capable of handling the multimodality inherent in the goal distribution. We address this issue by leveraging cheap and easy-to-collect data cancee--mann play data.
Learning multimodal latent plans. With the collected human play data and corresponding 3D hand trajectory 7, we formalize the latent plan learning process as goal-conditioned 3D trajectory generation task. More specifically, observation encoder F,, implemented convolutional networks, processes the visual observations of and goal image sf from the human video y into low-dimensional features, which are further processed by an MLP-based encoder network into a latent plan vector (as shown in Fig. 2(a)). Based on the latent plan Pt and the hand location 4, an MI P-baaeed decoder network generates the prediction of the 3D hand trajectory. However, simple regression of the trajectory cannot fully cover the rich multimodal distribution of human motions. Even for the same human operator, one task goal can be achieved with different strategies. To address this issue, we use an MLP-based Gaussian Mixture Model (GMM) [50] to model the trajectory distribution from the latent plan pt. For GMM Eq. (1) shows:
p(T = Σ p(T(0.z)p(210)
(1)
z

---

Kilicen
(b) Sludy
Flawor
(d) W Whitehnard
Sandwich
(1 Cloth
Figure 3: Evaluation Tasks. design six environments with long-horizon tasks for a Franka Emika robot arm, with initial (left) and goal (right) states shown in images. Tasks include: (a) Kitchen environment: 3 individual tasks including cooking food with oven. (b) Study Desk environment: individual tasks including tidying up the desk. (c) Flower: flower insertion into vase. (d) Whiteboard: crasing curve lines. (e) Sandwich: ingredient selection for cheeseburger or sandwich. (f) Cloth: folding a towel twice. See the Appendix for details.
where 0 = Ok, are the parameters of the GMM and p(r(0,z)) is Gaussian distribution ((T with z consisting of K components. A specific weight 7)x. represents the probability of the k-th component. GMMs are more expressive than simple MLPs, because they are designed to capture the multi-modality which is inherited the human play data. The final learning objective our GMM model is to minimize the negative log-likes log-likelihood the detected 3D human hand trajectory T as Eq, (2)
LGMM --FE Eyg
Σ The (c((Flu)o)))
.heee 0 S1 m =1
Handling visual gap between human and robot domains. We consider the setup where the human and the robot interact in the same environment. However, different visual appearances (for example, top vs. bottom row in Fig. 1) between human and robot domains pose a challenge in transferring the learned latent planner to the downstream robot control. We introduce new learning objective to minimize the visual representation gap between the two domains. Given human video frames d 4h and on-robot video frames o" EVV" we calculate the distribution (mean and variance) of the feature embeddings outputted by the visual encoder F of the human domain Q = Fi(d") and the robot domain Q" Fl(o") in each training data batch. We then minimize the distance between Qh and Q" with Kullback-Leibler (KL) divergence loss: LKI. DKL (Q']l Note that, our approach does not require paired human-robot video data. yhhh and V can be different behavior and solving different tasks. Only the image frames from the video are used to minimize the representation gap between the two domains. The final loss function for training the latent planner is:
C CaMM CxI, where À is a hyperparameter that controls the weights between the two losses.
3.3 Plan-guided multi-task imitation learning
MIMICPLAY focuses multi-task imitation learning settings, where single policy is trained to perform multiple tasks conditioned on different goals. Prior works often learn multi-task visuomotor policies end-to-end from scratch [6, 48, 5]. However, given the large goal space, these methods require a large amount of teleoperation data for policy training (4.5 hours [6] and 6 hours [7]). In this work, we leverage the latent planner P, pretrained with cost-effective human play data (10 minutes), to condense high-dimensionall inputs into low-dimensional latent plan vectors Pt- Since these latent plans pt can offer rich 3D guidance for formulating low-level robot actions ae the low-level policy the conversion between the owwdiimensional plans Pt and actions a, task it can learn efficiently due
focus on learning
5

---

Subgool (fins subgool)
Long barizon (2 3 subguals)
demos
demoxs Task-2
Task-1
Task-2
Task-3
ALL
Task-1
3 ALL
Task-3
Task-1
Task-2
Task-3
ALL
Task-1
Task-2
Task-3 ALL
GC (BC-RNN) [20]
0.1
0.0
0.1
0.07
0.1
0.2
0.2
0.17
0.0
0.0
0.0
0.00
0.0
0.0
0.1
0.03
GC-BC (BC-trans) [52]
0.2
0.0
0.0
0.07
0.3
0.7
0.6
0.53
0.0
0.0
0.0
0.00
0.0
0.0
0.1
0.03
C.BeT [6]
0.37
0.00
1MP [5]
0.3
0.1
0.2
0.20
0.6
0.3
0.2
0.37
0.1
0.0
0.1
0.07
0.3
0.1
0.0
0.13
R3M-BC [40]
0.9
0.0
0.0
0.30
0.5
0.4
0.0
0.30
0.0
0.0
0.0
0.00
0.5
0.0
0.0
0.17
Ours (0%) human)
1,0
0.5
0.3
0.60
1,0
0.5
0.5
0.67
0.5
0.1
0.3
0.23
0.4
0.3
0.5
0.40
Ours
1.0
0.8
0.7
0.83
1.0
0.9
0.8
0.90
0.7
0.3
0.4
0.47
0.7
0.6
0.8
0.70
Table 1: Quantitative evaluation results in the Kitchen environment.
to the decreased dimensionality. In the following, we introduce how to generate the latent plan Pt and the details of training the plan-guided low-level controller x with a small amount of data.
Video prompting for latent plan generation. Instructing robot to perform visuomotor long-horizon tasks is challenging due to the complexity of goal specifications. Our latent planner P, learned from human play videos, is capable of interpolating 3D-aware task-level plans directly from human motion videos, which can serve as an interface for promoting long-horizon robot manipulation. More specifically, we use a one-shot video V (either human video y or robot video V") as a goal specification prompt sent to the pre-trained latent planner to generate robot-executable latent plans Pt. The one-shot video first converted into sequence of image frames. At each time step, the high-level planner P takes one image from the sequence goal-image input Ge and generates a latent plan pt guide the generation of low-level robot action ag. After executing at, the next image frame in the sequence is used as a new goal image. During the training (Fig. 2(a)(b)), the goal image of (o) E V") is specified as the frame H steps after the current time step in the demonstration. H is a uniformly sampled integer number within the range of [200,600] (10-30 seconds), which performs as a data augmentation process.
Transformer-basee plan-guided imitation. Decoupling planning from control allows the lowelevel policy π to focus on learning how to control the robot by following the guidance pt- The plan-guided imitation learning process is illustrated in Fig. 2(b). However, to execute fine-grained behaviors like grasping an oven handle, merely having high-level guidance is insufficient. It is equally important to consider low-level specifics of the robot end-effector during the action-gent generation process. Therefore, we convert the robot's wrist camera observation we and proprioception data e, into oin-dinnsiional feature vectors, both with shape of Raxdd We then combine these features with the generated latent plan pt to create a one-step token embedding s; = [u. The sequence of these embeddings over T time steps, (Ta. is processed through a transformer architecture(51] fuan- The transformer-bas based policy, known for its efficacy in managing long-horizon action generation, produces an embedding of action prediction Iy in an autoregressive manner. The final robot control commands a are computed by processing the action feature dt through a two-layer fully connected network. To address the multimodal distribution of robot actions, we utilize an MLP-based Gaussian Mixture Model (GMM) [50] for action generation. Details regarding the model architecture are outlined in the Appendix.
Multi-task prompting. Learning from human play data enables the planner to handle diverse task goals. We demonstrate this empirically by designing all of our evaluation environments to be multi-task and share the same planner P and the policy и models across all tasks in the same environment. For each training sample, the prompting video is uniformly sampled from the training videos of the same task category.
4 Experiment Setups
Environments and Tasks, We create six environments with 14 tasks (Fig. 3), featuring tasks such contact- rich tool use, articulated-object handling, and deformable object manipulation. Three tasks are designed for the Kitchen environment and four for the Study Desk environment, focusing on long-horizon tasks with different goals. We assess methods using Subgoal and Long horizon task categories. The Study Desk environment examines compositional generalization with three unseen tasks: Easy, Medium, and Hard. The Easy task combines two trained tasks, while the Medium and Hard tasks require novel motions for unseen subgoal compositions, i.e., the transition from subgoal A to subgoal B is new. The horizon of all tasks is between 2000 to 4000 action steps, which equals to 100-200 seconds of robot execution (20Hz control frequency). For more details about the environments and simulation results, please refer to the Appendix.

---

Triined takks
Unseen tooks
Spatial gomerlication
Extreme long botizon
Defommalee
Task-I
Task-2
Task
Task 4 ALL
Easy
Medium Hard
ALL
GC-BC (BC-trans (ranstss) [52]]
0.0
0.00
0.0
0.0
Flower Whiieboudd
Sundich
Cluth
|LLL
LMP [S]
0.0
0.0
0.0
0.00
0.0
0.0
0.0
0.00
LMP singl
0.1
0.0
0.3
Q13
Ours (9%% human)
0.2
0.3
0.1
0.2
0.20
0.2
0.1
0.0
0.10
LMP
0.0
Q05
Ours (50% human)
0.1
0.4
0.30
0.3
0.27
R3M-iingle
022
025
Ours (wo KL)
0.7
0.3
0.2
0.38
0.4
0.2
0.0
0.20
R3M (40)
0.1
0.1
0.2
0.2
015
Ours (wos GMMM)
0.4
0.2
0.2
0.3
0.28
0.2
0.0
0.0
0,07
Our single
0.5
0.5
0.6
0.7
0.58
Ours
0.7
0,4
0.5
0.55
0.7
0,5
0.2 0,47
Ours
0.4
0.2
08
0.8
0.55
Table 2: Ablation evaluation results in the Study Desk environment (20 demos).
Table 3: Quantitative evaluation results of
multi-task learning.
Baselines. We evaluate five methods: 1) GC-BC (BC-RNN) and 2) GC-BC (BC-trans), goal-conditioned BC variants of [20] using RNN and GPT-based transformer architectures, respectively; 3) C-BeT [6], an algorithm using Behavior Transformer [53] to learn from teleoperated robot play data; 4) LMP [5], a method that learns to generate plans and actions in an end-to-end fashion from robot play data; and 5) R3M-BC, a goal-conditioned BC variant of [20] using R3M pre-trained visual representation [40]. All methods, including ours, train on the same robot telcoperation demos (20 or 40 demos per task). Besides this common dataset, baselines add an extra 10-minute robot demos, while MIMICPLAY uses 10-minute human play data. The total data collection time is consistent across methods. Task success rate (%) is the primary metric.
5 Results
Learning latent plans from human play data significantly improves performance. Our method outperforms Ours (0% human) by more than 23% in long-horizon task settings over all trained tasks, as shown in both Tab. (ALL) and 2 (Trained tasks ALL). This result showcases that learning a latent plan space does not need to rely fully on teleoperated robot demonstration data. 10-minute of cheap and unlabelled human play data brings large improvements in the task success rate and sample efficiency.
Hierarchical policy is important for learning long-horizon tasks. Ours (0% humnn)) trained with our two-stage framework outperform prior end-to-end learning methods in the long-horizon task settings by more than 15%, as is shown in Tab. (ALL) and 2 (Trained tasks ALL). This result shows that end-to-end learning for planning and control less effective than learning to act based on pre-trained latent plans for long-horizon tasks. We also find the same conclusion in simulation results as is introduced in the Appendix.
Latent plan pre-training benefits multi-task learning, In Tab. 3, we study how each method performs when training each task with a separate model. For end-to-end learning approaches (e.g.. LMP and R3M- BC), training task-specific models will lead better performance (LMP-single vs. LMP; R3M-single vs. R3M). These results showcase the difficulty of learning multiple tasks with single model. However, our approach has the smallest performance drop in multi-task training (Ours-single vs. Ours). These findings highlight the advantage of learning plan-guided low-level robot motions based on the pre-trained latent plan space. However, we do observe an uneven performance drop with our method (the success rate of the whiteboard task drops from 0.5 to 0.2). We hypothesize this is due to the reason that the length of the demon- stration for the whiteboard task is shorter than the other tasks, which leads to an imbalanced training dataset.
GMM is crucial for learning latent plans from human play data. In Tab. our full with GMM model largely outperforms Ours (wo GMM). Although being trained with full human play data, Ours (wlo GMM) even fails to match the performance of Ours (0% human) in the generalization task settings. We visualize the trajectories generated by all of the model variants in the Appendix, where we found Ours (w/o GMM) has the worst quality of trajectory generation. This result highlights
0.8
(human prompt) Ours (robot prompt)
Success (%6
0.6
0.4
the importance of using the GMM model handle the multimodal distribution of the hand trajectory when learning the latent plan space from human play data.
0.2
0.0
Task-2
Task-3
Medium
ALL
KL loss helps minimize the visual gap between human and robot data. In Tab. 2, although Ours (w/o KL) baseline outperforms most baselines in trained tasks, its success rate is 17% lower than Ours. In the generalization setting, Ours (w/o
Figure 4: Evaluation of multi-task policy prompted with robot/human videos in the Study Desk environment.
7

---

KL) fails to match the performance of Ours (50% human). These results showcase that the visual gap between the human play data and robot data exists, and KI. loss helps close the gap when training the latent planner. More analysis of the distribution shifts between human play data and robot data can be found in the Appendix.
The scale of the human play data matters. In Tab. 2, we compared the model variants with 50% human play data (Ours (50% human)) and found it fails to match the performance of Ours, which has access to 100% human play data. Most critically, in the unseen task settings, using more human play data to cover unseen cases in the training set significantly benefits generaliza- tion (Ours vs Ours (50% human)).
Observation latent plan before disturbance
Human disturhance
Real-time replanning
Recovery
Human play data improves generalization to new subgoal compositions. In Tab. 2 unseen tasks, Ours surpasses all baselines by more than 35%. This result highlights that our approach extracts novel latent plans from human play data and guides the robot's low-level policy to gener- alize to compositions of subgoals.
Figure 5: Qualitative visualization of the latent plans before the disturbance and re-planning. Column 1: third- person view. Column 2: visualization the latent plan before disturbance. Column 3: human disturbance; the red arrow indicts the direction of disturbance. Column visualization of real-time re-plamning capabilities, which show robustness against disturbance. Column 5: robot recovers with the updated task plan.
An intuitive interface for prompting robot motion with human videos. Fig. 4 shows that our policy model, when prompted with hu- man videos, retains competitive performances as prompted with robot oracle videos across three Study Desk tasks. The reason is that MIMIC- PLAY integrates human motion and robot skills into a joint latent plan space, which enables an intuitive interface for directly specifying robot manipulation goals with human videos. More results can be found the supplementary video.
Real-time planning capability is robust against disturbance. In Fig. 5, we showcase how our model reacts to unexpected human disturbance. None of these disturbances appears in the teleoperated robot demonstration data. For instance, in the cloth folding task (Fig. third row). the human unfolds the towel after the robot has folded it, and the robot replans and folds the towel again. Since our whole system (including the vision-based latent planner, low-level guided policy, and robot control) is running a speed of 17Hz, our model is able to achieve real-time re- p-lpannnnng capability against disturbance and manipulation errors. For more details, please refer to the supplementary video.
6 Conclusion and Limitations
Existing limitations of the MIMICPLAY include: 1) The current high-level latent plan is learned from scene-specific human play data. The scalability of MIMICPLAY can greatly benefit from training on Internet-scale data. 2) The current tasks are limited to table-top settings. However, humans are mobile and their navigation behaviors contain rich high-level planning information. The current work can be extended to more challenging mobile manipulation tasks, and 3) There is plenty of room to improve on the cross-embodimest representation learning. Potential future directions include temporal contrastive learning (54) and cycle consistency learning [55] from videos.
We introduce MIMICPLAY, a scalable imitation learning algorithm that exploits the complementary strengths of two data sources: cost-effective human play data and small-scale teleoperated robot demonstra- tion. Using human play data, the high-level controller leamss goal coul-conditioned latent plans by predicting future 3D human hand trajectories given the goal image. Using robot demonstration data, the low-level controller then generates the robot actions from the latent plans. With this hierarchical design, MIMICPLAY outperforms prior arts by over 50% in 14 challenging long- horizon manipulation tasks. MIMICPLAY paves the path for future research to scale up robot imitation learning with affordable human costs.
8

---

Acknowledgments
This work is partially supported by ONR MURI N00014-21-1-2011. L. F-F is partially supported by the Stanford HAI Hoffman- Yee Research Grant. This work done during Chen Wang's internship at NVIDIA. Stanford provides the computing resources and robot hardware for this project. We are extremely grateful Yifeng Zhu, Ajay Mandlekar for their efforts developing the robot control library Deoxys[22] and RoboTurk([6).. We would also like thank Yucheng Jiang for assisting with multi-seed evaluation in simulation.
References
[1] D. A. Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in neural information processing systems. 1, 1988.
[2] T. Zhang. Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 5628-5635. IEEE, 2018.
[3] A. Mandlekar, D. Xu, R. Martín-Martúnn S. Savarese, and Fei-Fei. Learning to generalize across long-horizon tasks from human demonstrations. arXiv preprint arXiv:2003.06083 2020.
[4] K. Shiarlis, M. Wulfmeeer, S. Salter, S. Whiteson, and I. Posner. Taco: Learning task decomposition via temporal alignment for control. In International Conference on Machine Learning, pages 4654-4663, PMLR, 2018.
[5] C. Lynch, M. Khansari, Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. In Conference on robot learning. pages 1113-1132, PMLR, 2020.
[6] Z. J. Cui, Y. Wang, N. Muhammad, L. Pinto, al. From play to policy: Conditional behavior generation from uncurated robot data. arXiv preprint vrXiv:2210. 10047, 2022.
[7] E. Rosete-Beas, O. Mees, G. Kalweit, J. Boedecker, and W. Burgard. Latent plans for task-agnostic offline reinforcement learning. arXiv preprint arXiv:2209.08959 2022.
[8] S. Calinon, D'halluin, L. Sauser, D. G. Caldwell, and A. G. Billard. Learning and reproduction of gestures by imitation. IEEE Robotics & Automation Magazine, 17(2)44 54, 2010.
[9] A. Ijspeert, J. Nakanishi, and S. Schaal. Movement imitation with nonlinear dynamical systems in humanoid robots. In Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH33292), volume 2, pages 1398-1403 vol.2, 2002. 001100110//R0B012000..00114
[10] S. Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences, 3(6): 233 233-242, 1999.
[11] J. Kober and J. Peters. Imitation and reinforcement learning. IEEE Robotics & Automation Magazine, 17(2)555-62, 2010.
[12] P. Englert and M. Toussaint. Learning manipulation skills from a single demonstration. The International Journal of Robotics Research, 37(1):137-154, 2018.
[13] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine. One-shot visual imitation learning via meta- learning In Conference on robot learning, pages 357-368. PMLR, 2017.
[14] A. Billard, S. Calinon, R. Dillmann, and S. Schaal. Robot programming by demonstration. In Springer handbook of robotics, pages 1371-1394. Springer, 2008.
[15] B. D. Argall, S. Chemova, M. Veloso, and B. Browning. A survey of robot learning from demonstra- tion. Robotics and autonomous systems, 57(5):469-455 2009.

---

[16] S. Schaal. Dynamic movement primitives-a framework for motor control in humans and humanoid robotics. In Adaptive motion of animals and machines, pages 261-280. Springer, 2006.
[17] J. Kober and Peters. Learning motor primitives for robotics. In 2009 JEEE International Conference on Robotics and Automation, pages 2112-2118. IEEE, 2009.
[18] A. Paraschos, C. Daniel, J. R. Peters, and G. Neumann. Probabilistic movement prim- itives. In C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Asso- ciates, Inc., 2013. URL https://proceeddin http:://prroceddagg............. www.neeuuurcceeeee htt.r/pp..ctopppppr///////11.... Paper.
[19] A. Paraschos, C. Daniel, J. Peters, and G. Neumann. Using probabilistic movement primitives in robotics. Autonomous Robots, 42(3):52--551, 2018. 2018.
[20] A. Mandlekar, D. J. and R. Martin-Martnn. What matters in learning from offline human demonstrations for robot manipulation. In Sth Annual Conference on Robot Learning, 2021. URL httph:tttt//ttteett : openreviev. net/ for/ff ?
J. Wong, S. Nasiriany, C. Wang, R. Kulkarmi, L. Fei-Fei, S. Savarese, Y. Zhu,
[21] P. Florence, L. Manuelli, and R. Tedrake. Self-supervised correspondence in visuomotor policy learning IEEE Robotics and Automation Letters, 5(2):492-499, 2019.
[22] Y. Zhu, A. Joshi, P. Stone, and Y. Zhu. VIOLA: Object-centric imitation learning for vision- based robot manipulation. In 6th Annual Conference on Robot Learning, 2022. URL https: / /apenzeview net forrm//ormmmmmmmm wit.iddd--hhhPPPiIee
[23] C. Wang, R. Wang, A. Mandlekar, L. Fei-Fei, S. Savarese, and D. Xu. Generalization through hand-eye coordination: An action space for learning spatially-invariant visuomotor control. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8913-8920.
IEEE, 2021.
[24] P. Florence, C. Lynch, A. Zeng, O. Ramirez, Wahid, L. Downs, A. Wong, Lee, I. Mordatch, and
J. Tompson. Implicit behavioral cloning. Conference on Robot Learning (CoRL., 2021.
[25] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, Herzog, Hsu, Ibarz, B. Ichter, A. Irpan, T. Jackson, Jesmonth, N. Joshi, R. Julian, Kalashnikov, Kuang, Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manjunath, I. Mordatch, O. Nachum, C. Parada, Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q Vuong, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. Rt-1: Robotics transformer for real-world control at scale. In arXiv preprint arXiv:2212.068177 2022.
[26] brian ichter, A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, D. Kalashnikov, S. Levine, Y. Lu, C. Parada, K. Rao, P. Sermanet, A. T. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, M. Yan, N. Brown, M. Ahn, O. Cortes, N. Sievers, C. Tan, S. Xu, D. Reyes, J. Rettinghouse, J. Quiambao, P. Pastor, L. Luu, K.-H. Lee, Y. Kuang, S. Jesmonth, K. Jeffrey, R. Ruano, Hsu, K. Gopalakrishnan, B. David, A. Zeng, and C. K. Fu. Do as can, not say: Grounding language in robotic affordances. In 6th Annual Conference on Robot Learning. 2022.
[27] A. Mandlekar, Ramos, B. Boots, S. Savarese, L. Fei-Fei, A. Garg, and Fox. Iris: Implicit reinforcement without interaction scale for learning control from offline robot manipulation data. In 2020 IEEE International Conference Robotics and Automation (ICRA), pages 4414 4420. IEEE, 2020.
[28] D. Xu, S. Nair, Zhu, Gao, A. Garg. Fei-Fei, and S. Savarese. Neural task programming: Leaming to generalize across hierarchical tasks. In 2018 IEEE International Conference Robotics and Automation (ICRA), pages 3795-3802, IEEE, 2018.
10

---

[29] H. Xiong, Q. Li, Y.-. Chen, H. Bharadhwaj, S. Sinha, and A. Garg. Learning by watching: Physical imitation of manipulation skills from human videos. In 202/ IEEE/SS International Conference on Intelligent Robots and Systems (IROS), pages 7827-7834. IEEE, 2021.
[30] N. Das, S. Bechtle, T. Davchev, D. Jayaraman, A. Rai, and F. Meier. Model-based inverse reinforce- ment learning from visual demonstrations. In Conference on Robot Learning, pages 1930-1942. PMLR, 2021.
[31] K. Zakka, A. Zeng, P. Florence, J. Tompson, J. Bohg, and D. Dwibedi. Xirl: Cross-embodiment inverse reinforcement learning. In Conference on Robot Leamnin, pages 537-546. PMLR, 2022.
[32] L. Shao, T. Migimatsu, Q. Zhang, K. Yang, and J. Bohg. Concept2robot: Learning manipulation concepts from instructions and human demonstrations. The International Journal of Robotics Research, 40(12-14):141--4444 2021.
[33] A. S. Chen, S. Nair, and C. Finn. Learnigg generalizable robotic reward functions from" in-the-wild" human videos. Robotics: Science and Systems (RSS), 2021.
[34] P. Sharma, D. Pathak, and A. Gupta. Third-person visual imitation learning via decoupled hierarchical controller. Advances in Neural Information Processing Systems, 32, 2019.
[35] L. Smith, N. Dhawan, M. Zhang, P. Abbeel, and S. Levine. Avid: Learning multi-stage tasks via pixel-level translation of human videos. arXiv preprint arXiv: 2019.
[36] K. Schmeckpeper, A. Xie, O. Rybkin, S. Tian, K. Daniliidis, S. Levine, and C. Finn. Learning predictive models from observation and interaction. In European Conference on Computer Vision, pages 708-725. Springer, 2020.
[37] A. D. Edwards and C. L.. Isbell. Perceptual values from observation. arXiv preprint 195.09190..07861 2019.
[38] Schmeckpeper, O. Rybkin, K. Daniilidis, S. Levine, and C. Finn. Reinforcement learning with videos: Combining offline observations with interaction. In J. Kober, F. Ramos, and C. Tomlin, editors, Proceedings of the 2020 Conference on Robot Learning, volume 155 of Proceedings of Machine Learning Research, pages 339-354. PMLR, 16-18 Nov 2021. URL https: peattp::/prroceee http///oocceddeeoooo ngs. mlr.press / v / htt.:///sccceecpppeele.....m html.
[39] K. Shaw, S. Bahl, and D. Pathak. Videodex: Learning dexterity from internet videos. CoRI, 2022.
[40] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual representation for robot manipulation. 6th Annual Conference on Robot Learning, 2022. URL https:// openreview net forumrruuiioo (itmmmddddtGgggGgoo00
[41] T. Xiao, I. Radosavovic, T. Darrell, and Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2003.06/73, 2022.
[42] K. Grauman, A. Westbury, E. Bymee Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang. M. Liu, X. Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995-19012, 2022.
[43] N. Hansen, Z. Yuan, Y. 7e, T. Mu, A. Rajeswaran, H. Su, H. Xu, and X. Wang. On pre-training for visuo-motor control: Revisiting a learning- from-ss-com baseline. arXiv preprint arXiv:2212.05749, 2022.
[44] Y. Liu, A. Gupta, P. Abbeel, and S. Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1 1118-1125. IEEE, 2018.
[45] S. Kumar, J. Zamora, N. Hansen, R. Jangir, and X. Wang. Graph inverse reinforcement learning from diverse videos. Conference on Robot Learning (CoRL), 2022.
11

---

[46] M. Sieb, Z. Xian, A. Huang. O. Kroemer, and K. Fragkiadaki. Graph-structured visual imitation. In Conference on Robot Learning, pages 979-989. PMLR, 2020.
[47] S. Bahl, A. Gupta, and D. Pathak. Human-to-robot imitation in the wild. arXiv preprint arXiv:2007.09450, 2022.
[48] E. Rosete-Beas, O. Mees, G. Kalweit, J. Boedecker, and W. Burgard. Latent plans for task agnostic offline reinforcement learning. In Proceedings of the 6th Conference on Robot Learning (CoRL), 2022.
[49] D. Shan, J. Geng, M. Shu, and D. Fouhey. Understanding human hands in contact at internet scale. CVPR, 2020.
[50] C. M. Bishop. Mixture density networks. 1994.
[51]
Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[52] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam,
G. Sastry, A. Askell, et al. Language models are few-shot learners, Advances in neural information processing systems, 33:1877-1901, 2020.
[53] N. M. M. Shafiullah, Z. J. Cui, A. Altanzyaa, and L. Pinto. Behavior transformers: Cloning k modes with one stone. arXiv preprint 2tti::2006.1155511 2022.
[54] I. Dave, R. Gupta, M. N. Rizve, and M. Shah. Telr: Temporal contrastive learning for video representation. Computer Vision and Image Understanding, 219:103400 2022.
[55] D. Dwibedi, Y. Aytar, Tompson, P. Sermanet, and A. Zisserman. Temporal cycle-consistency learning. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition. pages 1801-1810, 2019.
[56] A. Mandlekar, Y. Zhu, A. Garg. J. Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta, E. Orbay, et al. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In Conference on Robot Learning, pages 879-893 PMLR, 2018.
[57] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[58] O. Khatib. A unified approach for motion and force control of robot manipulators: The operational space formulation. IEEE Journal on Robotics and Automation, 3(1):43-53, 1987. doi: 109//RA. 1987.80
[59] A. Graves and A. Graves. Long short-term memory. Supervised sequence labelling with recurrent neural networks, pages 37-45, 2012.
[60] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning, 2023.
[61] Y. Zhu, J. Wong, A. Mandlekar, R. Martín-Martín, A. Joshi, S. Nasiriany, and Y. Zhu. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint 2rX 2020.
[62] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSI International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012. 10. (O0//ROS.2002.63322
[63] C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Martin-Martin, C. Wang, G. Levine, M. Lingelbach. J. Sun, al. Behavior- 1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference on Robot Learning, pages 80 80-93, PMLR, 2023,
12

---

[64] L. van der Maaten and G. Hinton. Learning Research, 9:2579-2605, 2008. URL http://www. iml www.o//ppr////////....../
Visualizing data using t-SNE. Journal of Machine
vandermaateenOa.hen html.
13

---

A Implementation details
Here we lay down the details of the data collection, training, and testing process.
Collecting human play data and training details. The human play data is collected by letting a human operator directly interact with the scene with single hand for 10 minutes for each scene. The entire trajectory T is recorded at the speed of 60 frames per second and is used without cutting or labeling. The 3D hand trajectory is detected with off- the-sheeff multi-view human hand tracker [49]. The total number of video frames within 10 minutes of human play video is around 36k. We train one latent planner for each environment with the collected human play data. For the multi- cnlliieevvroomentt setup (for the experiments in Tab. 3), we merge the human play data from each scene to train a single latent planner. The latent planner contains two ResNet-18 [57] networks for image processing and MLP-based encoder-decoder networks together with a GMM model, which has K 5 distribution components. We train 100k iterations for the latent planner which takes single GPU machine for 12 hours.
Collecting robot demonstrations and training details. The robot teleoperation data is collected with an IMU-based phone telcoperation system RoboTurk [56]. The control frequency of the robot arm is 17-20Hz and the gripper is controlled at 2Hz. For each task, we collect 20 demonstrations. In the experiments, we also have 40 demonstration dataset for testing the sample efficiency of different approaches. The robot policy model a Gyylst transformer [52]. which consists of four multi-head layers with four heads. We train 100k iterations for the policy with single GPU machine in 12 hours. For a fair comparison with method, the baseline approaches trained without human play data have five more demonstrations during training the latent planner P and the low-level policy
Video prompting. In this work, we use a one-shot video V (either human video y or robot video V") to prompt the pre-trained latent planner to generate corresponding plans pt = (Oo .44 (.fe E V. During training (Fig. 2(b)), we specify the goal image of (o E VV)) as the frame H steps after the input observation of in the robot demonstration. H is uniformly sampled integer number within the range of [200,600], which equals 10-30 seconds in wall-clock time. here is the 3D location of the robot's end-effecttr. During inference (Fig. 2(c)), we assume access to a task video (either human or robot video) which is used as source of goal images. The goal image will start at the 200 frame of the task video and move to the next i frame after each step. We use i= in all our experiments. Based on the inputs, the latent planner generates a latent plan feature embedding Pt of shape Rixd, which used as guidance for the low-level robot policy.
Data visualization. We visualize the collected human play data and robot demonstration data in Tab. 9. For the human play data, we use an off-the-shet hand detector [49] to localize the hand's 2D location on the left and right image frame. which are visualized as red bounding boxes in Tab 9. For the robot demonstration data, we directly project the 3D location of the robot end-effector to the left and right image frames, which are visualized as blue bounding boxes in Tab 9.
Testing. We perform real-time inference on Franka Emika robot arm with control frequency of 1 17Hz- directly from raw image inputs to 6-DoF robot end-effector and gripper control commands with our trained models. The robot is controlled with the Operational Space Control (OSC) [58].
B Experiment setups
Environments. We design six environments with a total of 14 tasks for a Franka Emika robot arm, as illustrated in Fig. 3. These environments feature several manipulation challenges, such as contact-rich tool manipulation (cleaning the whiteboard), articulated-object manipulation (opening the oven and the box on the study desk), nigh-precision tasks (inserting flowers and turning the lamp by pressing the button), and deformable object manipulation (folding cloth).
Tasks, We design three tasks in the Kitchen environment and four tasks in the Study desk environment. All these tasks have different goals. In this work, we focus on long-horizon tasks that require the robot to complete several subgoals. To better analyze the performance of each method, we define the Subgoal task category that only counts whether the first subgoal of the task has been achieved and the Long horizon task category which is the full task. In the Study desk environment, we design three tasks for testing the
14

---

Subgoal (first subgoal)
Long Horizon (2 subgoals)
Task-1
Task-2
Task-3
Task
Task-5
Task-1
Task-2
Task-3
Task-4
T
GC-BC (BC-RNNN) 0,96
0.00
+.00
+.00
0.00
0.00
0.00 0.00 +.00 0.00
GC-BC (BC-trans) 0,05
0.01 #o00
0.00
0.01 #0.00
0.00
4.00
0.00
0.00 a.0 a.00
Ours (0% human)
1.02 0.70
0.74 0.0
0,78 00.4
0.58 2.00
1.29 0.62 0u5 0.67 1:009
Table Quantitative evaluation results in simulation (success rates % averaged over 5 seeds)
compositional generalization ability of the models to novel task goal sequences, which are not included in the training dataset. These three tasks are classified as Easy, Medium, and Hard depending on their difference compared to the training tasks. The Easy task is a simple concatenation of two trained tasks and their subgoals. The Medium task contains an unseen composition of pair of subgoals that is not covered by any trained tasks, i.e., the transition from subgoal A subgoal B is new. The model needs to generate novel motions to reach these subgoals. The Hard task contains two such unseen transitions. For the rest four environment, each scene has one task goal and features different types of challenges in manipulation, e.g., generalization to new spatial configuration, extremely long horizon, and deformable object manipulation.
Baselines. We compare with five prior approaches: (1). GC-BC (BC-RNN) [20]: Goal-conditioned behavior cloning algorithm [5] implemented with recurrent neural networks (RNN) [59]. (2). GC-BC (BC- trans) [52]: Another goal-conditioned behavior cloning algorithm implemented with GPT-like transformer architecture. (3). C-BeT [6]: Goal-conditioned learning from teleoperated robot play data algorithm implemented with Behavior Transformer (BeT) [53]. (4), LMP [5]: A learning from teleoperated robot play data algorithm designed to handle variability in the play data by learning an embedding space. LMP (single) a variant by training each task with a separate model. (5). R3M-BC [40]: goal-conditioned imitation learning framework that leverages R3M visual representation pre-trained with internet-scal human video dataset Ego4D [42]. R3M-BC (single) is variant by training each task with separate model.
Ablations. We compare four variants of our model to showcase the effectivenesss of our architecture design: (I). Ours: MIMICPLAY with full collection (10 min) of human play data. Ours (single) is a variant by training each task with a separate model. (2). Ours (0% human): variant of our model without using human play data. The pre-trained latent plan space trained only with the teleoperated robot demonstrations. (3). Ours (50% human): variant of our model where the latent planner is trained with 50% of human play data (5 min). (4). Ours (w/ GMM); variant without using the GMM model for learning the latent plan space from human play data. (5). Ours (wo KL): Our approach without using KL loss for addressing the visual gap between human and robot data when pre-training the latent planner.
C Supplementary Experiment Results
Results in simulation. To extensively evaluate the methods with more testing trials and training seeds, we conduct an experiment in simulation LIBERO [60], which is multitask robot manipulation benchmark based on robosuite [61] and MuloCo [62]. We choose LIBERO due to its utilization of the BDDL language [63] for goal specification, which facilitates the multitask evaluation for learning from play data. Note that, in our main paper, we leverage human play data. However, in simulation, there is no way to get such dataset, which will always end up being robot teleoperation. Therefore, in this experiment, we use the same teleoperated robot play dataset to train both high-level planner and low-level controller, and report the results of Ours (0% human) and baselines in Table 4. For each method, we train with 5 random seeds and report the average success rate over 100 testing trials. The results showcase the advantage of MIMICPL. PLAY's 's hierarchical policy learning framework over the baselines, which is consistent with the real-world results (Tab. 1, 2). The implementation code is available at https:://eee /ttpt/ggttttttt////// com/ wtwp://www//to WMwp://iiiiiiii.iimmmee lay.
Visualization of the trajectory prediction results. We visualize the 3D trajectory decoded from the latent plan by projecting it onto the 2D image in Fig. 6. In the last two rows, we showcase the results of
15

---

Ours(P-huannl
Plewer
craning
Sandwich
Genes.
Easy
Whiteboard deaning
Geses.
Medium
arrunging
(a) Trajectory prediction results decoded from the latent plans
(b) t-SNE visualization of the latent plans
Figure Qualitative visualization of the learned latent plan. (a) Visualization of the trajectory prediction results decoded from the latent plans learned by different methods. The fading color of the trajectory from blue green indicates the time step from to 10. (b) t-SNE visualization of latent plans, the latent plans of the same task tend to cluster in the latent space.
100
Robei dots
Reivet date
50
Dist. ovedlap
overlap
0np2
com-22
Distribution overlap: 35%
-105
Distribution overlap: 58%
-150
-150
300
-355
150
c
(a) Distribaatinn of Ours (wlo KL.)
(b) Distribution ovs Ours
Figure 7: t-SNE visualization of the generated feature embeddings by taking human data and robot data as inputs. The slashes refer to the overlap region two data distributions. (a) Feature visualization results of method without using KL divergence loss. (b) Feature visualization results of our method with KL divergence loss. Our approach covers 23% more area than the baseline.
two unseen subgoal transitions. The trajectory generated by model is most similar to the ground truth trajectory, while Ours (0% human) overfitted to the subgoal transitions in the training set and generates the wrong latent plan. For instance, in the training data, the robot only leams to open the box after turning off the lamp, meanwhile in the Easy setting of generalization tasks, the robot is prompted to pick up the pen after turning off the lamp. Ours (0% human) variant still outputs latent plan to open the box, which causes the task fail since the box is already open.
Transformer architecture helps multi-task learning. In Tab. 1, GC-BC (BC-trans) with the GPT transforeer architecture outperforms GC-BC (BC-RNN) by more than 30% in 40-demos Subgoal setting. However, the performance of GC-BC (BC-trans) quickly drops the same level GC-BC (BC-RNN) in
16

---

20-demos settings. The result showcases that training vision-based transformer policy end-to-end requires more data.
Visualization of the learned latent plans. We use t-SNE [64] to visualize the generated latent plans conditioned on different tasks, as shown in Fig. 6(b). We find that the latent plans of the same task tend to cluster in the latent space, which shows the effectiveness of our approach in distinguishing different tasks.
Analysis of the visual gap between human and robot data. As introduced in the method Sec. 3.2, to minimize the visual gap between human play data and robot demonstra- tion data, we use KL divergence loss over the feature embeddings outputted by the visual en- coders. In Fig. 7, we use t-SNE to process and visualize the learned feature embeddings gener- ated by Ours and the model variant Ours (w/o KI.) the 2D distribution plots. To better vi- sualize the distribution overlap, we use slashes to highlight the overlap area in both plots. We observe that our approach with KI. loss has 23% larger overlap between the human data and the robot data compared to Ours (wo KL). This result showcases the effectiivneess of our KL di- vergence loss and supports the result in Tab. (Ours (w/o KL) is inferior to Ours in task success rate).
Right camera
Left camera
Human play
(a) Human play data collection
Right camera
Left camera
D Details of system setups
Wrist camera
We illustrate the system designs for the data collection in Fig. 8. The human play data is collected by having a human operator directly interact with the environment with one of its hands (Fig. 8(a)). The left and right cameras record the video the speed of 100 frames per second. During the collection process of human play data, specific task goal is given and the human operator freely interacts with the scene for interesting behaviors based on its curiosity. For each scene in our experiments, we collect 10 minutes of human play data.
Phone teleoperation (RoboTurk
(b) Robot demonstration data collection
System setups for the data collection. (a) Human play data collection. human operator directly interacts with the scene with one of its hand and perform interesting behaviors based on its curiosity without a spe- cific task goal. (b) Robot demonstration data collection. human demonstrator uses phone teleoperation sys- tem to control the 6 DoF robot end-effector. The gripper of the robot is controlled by pressing a button on the phone interface.
Figure 8:
The robot teleportation demonstration col- lected with phone teleoperation system Robo- Turk [56] (Fig. 8(b)). The left, right, and end- effector wrist cameras record the video at the speed of 20 frames per second, which is aligned with the control speed of the robot arm (20Hz).
Each sequence of robot demonstration has pre-defined task goal. During the data collection, the human demonstrator completes the assigned sub-goals one by one and finally solves the whole task. For each training task in experiments, collect 20 demonstrations, In the Kitchen environment, we collect 40 demonstrations for each task to figure out which approach is more sample inefficiency.
17

---

E
Details of the task designs
The definition of our long-horizon tasks is listed below. For each task, the initial state and subgoals are pre-defined. The whole task completed if and only if all subgoals are completed in the correct order.
E.1 Kitchen
Task-1
Initial state: A drawer is placed on the left side the table. The drawer is not fully open and contains pumpkin and lettuce. A closed microwave oven is placed on the right side of the desktop. A bowl and inside the bowl. A pan is placed top of the stove.
placed the lower edge of the tabletop. There is camot
Subgoals: a) Open the microwave oven door. b) Pull out the microwave oven tray. c) Pick up the bowl. d) Place the bowl on the microwave tray.
Task-2
Initial state: same as Kitchen Task-1.
Subgoals: a) Open the drawer. b) Pick up the carrot. c) Put the carrots in the drawer.
Task-3
Initial state: same as Kitchen Task-1.
Subgoals: a) Pick up the pan. b) Place the pan on the table. c) Pick up the bowl. d) Place the bowl on the stove.
E.2
Study desk
Task-1
Initial state: The book is the rack. The lamp is on. The box is opened and closed in random state. The pen is located either in the center of the table or in the box.
Subgoals: a) Turn off the lamps. b) Pick up the book, c) Place the book on the shelf position.
Task-2
Initial state: The location of the book is either on the shelf or on the rack. The lamp is off. The box is closed. The pen is in the center of the table.
Subgoals: Turn the lamps. Open the box. e) Pick up the pen. d) Put in the box.
Task-3
Initial state: The book is on the rack. The state of the lamp is random. The box is closed. The pen in the center the table.
Subgoal a) Open the box. b) Pick up the pen. c) Place the pen in the box. d) Pick up the book.
e) Place the book on the shelf.
T
Initial state: The location of the book either on the shelf or on the rack. The lamp is on. The box is closed. The pen located either in the center of the table or in the box.
Subgoals: a) Open the box. b) Turn off the lamp.
Easy
Initial state: The location of the book is either on the shelf or the rack. The lamp off. The box is closed. The pen is located either in the center of the table or in the box.
Subgoals: a) Turn on the lamp. b) Open the box. c) Turn off the lamp.
Medium
Initial state: The location of the book is either the shelf the rack. The lamp is on. The box is closed. The pen is in the center of the table.
18

---

Subgoals: a) Open the box. b) Turn off the lamp. c) Pick up the pen. d) Place the pen in the box.
Hard
Initial state: The book is on the shelf. The lamp is on. The box is closed. The pen is located
either in the center of the table or in the box.
Subgoals: a) Turn off the lamp. b) Open the box. c) Pick up the book. d) Place the book the shelf.
E.3 Flower
Initial state: Two flowers and vase are placed on the table. The vase will randomly be placed on the top left top right comner of the table.
Subgoals: a) Picking up a flower. b) Insert the flower into the vase. c) Pick up the other flower. d) Insert flower into the vase.
E,4 Whiteboard
Initial state: A whiteboard and board eraser are placed on the table. The board eraser is placed on the left side of the whiteboard.
Subgoals: a) Pick up the board eraser. b) Moves over the curve line. c) Erase the curve line. d) Return the eraser to the original location.
E,5 Sandwich
Initial state: A circular ingredient selector is placed in the upper right corner of the table. Half of the circle holds ingredients for sandwich (bread, lettuce, sliced tomato) and half holds ingredients for cheeseburger (bread, cheese, burger patty). A white plate is placed in the lower left corner of the table.
Subgoals for sandwich: Rotate the ingredient selector to the right position. Pick up a piece of bread from it and place it the plate. b) Rotate the ingredient selector the correct position. Pick up the lettuce and place top of the bread. c) Rotate the ingredient selector the right position. Pick up the sliced tomato and place it on top of the lettuce. d) Rotate the ingredient selector to the right position. Pick up another piece of bread and place it on top of the tomato.
E.6 Cloth
Initial state: An unfolded brown cloth is randomly placed on the table.
Subgoals: a) The robot folds the cloth in half once to become 1/2 of original size. b) The robot folds the cloth once more to become 1/4 of its original size.
F Training hyperparameters
We list the hyperparameters for training the models Tab. 5 for the latent planner P and Tab. the robot policy क. The hyperparameters that are named starting with GMM related the MLP-based GMM model. The hyperparameters that named starting with GPT related the transformer architecture. We also list the hyperparameters for the baseline GC-BC (BC-trans) in Tab. 7.
19

---

Hyperpnmmeerr
Defauh
Batch Size
16
Heperpaameer
Detuit
Leuming rue (LR)
lc-4
Hyperpaameter
Batch Size
Default
16
I Learing (L.)
Num
Beach Size
Lengengh
le-4
16
Trin See Length
Num Epoch
Epoch
Factor
1300, 600)
1000
10
Leurning Rate (LR)
le-4
Decay
MLP Dins
[400, 400]
Num Epoch
1000
Decay Epech
600)
Image Encoder- Wrist View
ResNe: 15
LR Decay
None
Iragee Enerder Wrid View
MLP Dime
Rc-Ner-18
4400 4001
Image Encoder Right View
Image Encoder- LencnderLLett View
ResNes 18
ResNet 18
KL Weights À
1000
Image Featuee Dam
64
Imuge
MLP Dims
[400, 400]
GMMM Num Modes
5
GMM Mades
Image Encoder Left View
ResNer-18
GMM
0.01
GMM Min Sta
0.01
Image Encoder Right View
ResNet-18
GMM Sid
Softlas
GMM Sid Activation
GPT Block Sizr
Sotiplus
10
Image Feature Dim
64
Head
GMM Num Modes
5
GPT Num Layer
Layer
GMM Min Std
0.0001
GPT Embeed Size
6d6
GPT Embed Size
656
GMM Activation
Softplus
GPT Deogout Kate
GPT MI MIP Dins.
0.1 (696, 128]
GPT Drspout Rate
CIPT MI Dins
[666, 128]
0.1
Table 5: Hyperparameters- Ours (La-
Table 6: Hyperparameters
Table 7: Hyperparameters
tent Planner P)
Ours (Robot Policy T)
GC-BC (BC-trans)
G Network Architecture
Transformer-based policy network. The embedding sequence I time steps is represented 8[a++2]
-00-::mm----------PPP-PP-00- (etTT-t--T)) which passes through a transformer architecture [51]. The transformer model frass processes the input embeddings using its N layers of self-attention and feed-forward neural networks. Given an embedding sequence of time steps, fuans generates the embedding of trajectory prediction an autoregressive way - -T frams (Ww.T.1..11... 11:T11P1:77 1). where ø7 is the predicted action embedding at time step T. The transformer architecture gather context and dependencies from the entire history trajectory each step. The final robot control commands a computed by processing the action feature 2; with two-layer fully-connected network. To handle the multimodal distribution of robot actions, we also use MLP-basee GMM model [50] for the action generation.
the multi-head self-attention mechanism
20

---

View
Left view
Right view
0.1s
/ 0.2s
t= 0.3s
t=0. 0.4s
t= 0.5s
L= 0.1s
t= 0.2s
0.3s
t= 0.4s
t= 0.5s
0.6s
t= 0.7s
0.8s
t= 0.9s
t=00s
t= 0.7s
0.88
0-0.s
1.0s
Kitchen (human play data
1=1.1
t= 1.2s
+= 1.3s
+= 1.4s
f= 1.5s
t= 1=1.1
1=1.2s
f= 1.3s
t= 1=144
f= 1.5s
t=
==
t=
t=
t=
t=
f=
t=
f
t= 254.0k
246.6s 6s
248.4s
250.2s
252.0s
254.0s
246.66
248.4s
250.2
252.0s
f 255.8s
f= 259.4s
Kitchen (Robot
257.6s
261.2s
263.0s
255.8s
257.66 66s
259.4s
261.2 .21.2s
263.0K Os
multi- task demos)
l= 264.8s
L= 266.6s
[
t=
t= 272.0s
L=
(=
1=
L= 272.0s
268.4s
270.2s
264.8s
266.6s
268.4s
270.2s
2.6s
2-7.7s
2.8s
2-2.
3.0s
t= 2.7s
t-8s8
2-2.5
t=1.1s
f=2.ss
f= 3.3s
f=4.4s
3.5s
f f= 3.1s
3.22s
f= 3.3s
1.4.s
t=3 3.5s
Study desk (human play data)
3.6s
3.78
3.8s
3.9%
4.0s
3.6s
3.7s
3.8s
1.3%%
4.0s
f 950.6s
f= 952.4s
t= 956.0s
948.8s
950.6s
952.4s
954.2s
956.0s
948.8s
954.2s
1= 957.4s
-= 959.2s
1 961.0s
t= 963.0s
-= 964.6s
t= 957.4s
[=
-=
t= 964.66
Study
959.2s
961.0s
963.0s
desk
(Robot
multi-
task
demos)
f 964.6s
= 966.6is
f= 968.4s 4s
f 972.0s
970.2s
964.6s
966.6s
968.4s As
970.2s
972.0s
Figure Dataset visualization.
21


Reviews:
[{"review_id": "IhgVKyDG_0Y", "reviewer": "Reviewer_5FXC", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "* Allows robots to leverage less burdensome unlabeled in-domain human play data to learn high level plans, while learning low-level plan execution from labeled multitask robot demonstrations.\n* Same method allows for conditioning directly on human video to guide task execution.\n", "summary_of_recommendation": "This work presents compelling real world evidence that large amounts of in-domain human play data can improve the performance of a multitask imitation learning robot doing difficult long-horizon manipulation tasks. It feels important to broadcast this conclusion to the community, as it may be an important and scalable future resource for robot learning, and may open the door to follow-on work such as learning from out of domain human play data.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "The main question I have is: are any of the real world results leveraging human data statistically significantly better than baseline methods, but it is understandable if these answers are difficult to come by in real world evals.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths\n* Well motivated novel approach. Unlabeled in-domain human play data indeed seems like a promising direction for future scaling, and this work proposes a clever way of incorporating this type of data into a hierarchical imitation framework. In particular, the use of 3d tracked hand trajectories as high-level pretraining supervision seems well motivated.\n* Evidence that the proposed method can indeed incorporate human play data to obtain higher task performance in real world manipulation scenarios, and that more human play data leads to better final policies.\n* Solid, well designed experiments and ablations, e.g. studying the effect of human data scale or the impact of the KL term in solving for correspondence mismatch.\n* Evidence that the proposed approach can be conditioned with either robot goal images or human goal images, possibly allowing for more flexible test time conditioning scenarios.\n* Clear description of methodology and setting.\n* Compelling real world results.\n\n## Weaknesses\n* Strong claims of significance without evidence of statistical significance. The paper makes multiple claims of “significant improvement” (e.g. L54, L227) without presenting confidence intervals run over multiple trials. While it is perhaps understandable that multiple trials were not run for each real world experiment since it is expensive to do so, claims of significance should be toned down. For example, it might be helpful is the variance in success of the same model run twice in the real world on the same suite of benchmarks? 1%? 10%? 20%? \n\n## Nit\n* L121: “we collect 10 minutes of human play video as the training dataset for each task environment, which is approximately equivalent to a 3-hour dataset of robot teleoperation video.” What is the conversion rate here? It would be helpful to describe how 10 minutes of human play is equivalent to 3 hours of robot teleoperation in more detail.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "Cw8_G2uDFd", "reviewer": "Reviewer_4uup", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper combines human play data—video sequences of people interacting with the environment in an unstructured way–and teleoperation data to teach a robot long-horizon skills. Using human play data, one can first learn a latent planner that takes as input raw video of a human (or robot) and infers a representation that enables high-quality reconstruction of the agent’s end-effector trajectory. The robot’s low-level motor control policies are trained from teleoperation data which perform the same task, but are conditioned on the output of the pre-trained latent planner. These two modules are then combined during testing where a long-horizon video prompt is given to the robot and it generates latent plans and low-level controls to accomplish the task. The paper showcases favorable quantitative and qualitative performance on many real-world robotics tasks and in hardware on a Franka robot. ", "summary_of_recommendation": "Although I have several questions I'd like to have addressed during the rebuttal about the underlying assumptions of this method, I believe this is a strong and relevant contribution to CoRL. Thus, my recommendation is accept. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "* In the video prompting, how sensitive is the method to the H parameter when selecting the goal image? It seems like choosing a good time horizon and the right “image waypoint” would matter a lot for generating a useful plan. If the goal image is too far in the future, the planning problem is too hard, but if the goal image is too similar to the current observation then there isn’t much signal for how to progress. \n* In stage 1, learning the latent planner requires a multi-view camera setup to collect human play data. Is this assumption carried through to stage 2 and testing? If so, it seems like a strong assumption to have a calibrated multi-view camera setup all the time at deployment / testing time. How does this method perform with only a single-view camera setup? \n* In the “transformer-based plan-guided imitation” section it seems like the 3D end-effector representation is not used for the robot (only in phase 1 for the human). Why get rid of this intermediate representation? This would make the cross-embodiment problem easier (e.g., as done in Bahl et al, “Human-to-Robot Imitation in the Wild” RSS 2022). \n* It seems like many of the core ideas from the prior work [34] “Third-person visual imitation learning via decoupled hierarchical controller” and the proposed work are quite similar (e.g., learning from human video demos a hierarchical model that decouples goal prediction from low-level control). One of the core differences is the human play data, and then several design differences. Why was this method not benchmarked against and how do you think it would compare?  \n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n* The paper is well written, clearly positioning itself in the landscape of prior work, and clearly explaining the methodology. \n* The multimodal high-level latent planner that was decoupled from the low-level control was an intuitive way to model the problem\n* Thorough analysis on the many components involved in this work, including robustness against unexpected disturbances and many robotic tasks\n* The hardware results and videos are neat! \n\nWeaknesses:\n* One of the main weaknesses is that assumptions underlying this approach could be clearer. For example, it is unclear if the multi-view camera assumption in stage 1 is consistent throughout the rest of the stage 1 and testing. It is also unclear if the robot’s wrist camera is used only during stage 2, or in testing. Similarly, it is tough to understand how the “small batch of on-robot data” that is paired with the human play data during stage 1 training is collected. I found these assumptions to be hidden or omitted from the text, but they are central to understanding the impact of this framework. \n* Nitpick – there is not an explicit limitations section (it is just part of the conclusion), and the submission instructions requested “All Submissions should include a Limitations section.”\n\nOverall comments:\n* “...10 minutes of human play video as the training dataset for each task environment, which is\napproximately equivalent to a 3-hour dataset of robot teleoperation video.” How did you arrive at this conversion? Did you empirically observe this?\n* “...to address the multimodal distribution of robot actions, we utilize an MLP-based Gaussian Mixture Model (GMM) [50] for action generation” at deployment time, do you sample from this GMM to get the action to execute, or take the MLE? It would be helpful to state how you use the properties of the GMM at deployment time. \n* “...H is a uniformly sampled integer number within the range of [200,600], which equals 10-30 seconds in wall-clock time.” How long was the typical task horizon? It would be helpful to know what fraction of the task these subgoals constituted. \n* One section that was unclear was how the latent planner – which according to figure 2 is only trained from human play data – is expected to generalize to visual inputs of the robot doing the task. The text seems to imply that “given human video frames and frames from a small batch of on-robot data '' that there is some amount of robot teleoperation data that is used in stage 1 training. However, this seems tough to get in practice. For example, if the human is just executing their natural play data, then to get on-robot data means that another human teleoperator needs to reproduce all the play motions on the robot? This does not seem very scalable or easy to reproduce; at he same time, it seems like a key step to making the planner useful in stage 2, so more details would be appreciated. \n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "5kveuAxSj7D", "reviewer": "Reviewer_17ab", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a system for developing a stronger method for long Horizon manipulation planning. This has been an area of difficulty for the community for many years, especially when applying learning-based methods that struggle to be able to deal with the exploration challenges of solving long Horizon tasks. The method proposes a two-level process where in the first level, a large amount of human interaction data is collected, and a latent model is learned to be able to generate trajectories from this human experience. Then from this human experience, data and additional model is trained to be able to prompt the model to generate good plans to solve individual tasks. The method does show some promising performance on real hardware, but there are details missing from the paper that makes it difficult to understand the method.", "summary_of_recommendation": "The paper proposes a potentially solid method to enable the community to train robotic controllers to perform better at long-horizon planning problems. However, the writing lacks details, making it difficult to fully understand the proposed method. If these details are addressed it will improve the paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Comments:\n- It's not easy to understand the details about how the cross-modal representation is learned for a model that works both for human play data and for robotics data. There only seem to be a few sentences on this topic online 162. this seems to be an important feature or hurdle that the modal needs to overcome in order to be successful. Without these details, it's difficult to understand how the model should work properly.\n- In addition, the paper is missing information about the one-shot video prediction method that is used to help generate trajectories. this is described online 184; there should at least be citations towards the models that are used to do this contextual video generation. How is this video prompting being performed? How are the prompts being designed such that it will result in the video generation model producing a better plan?\n- One question that comes to mind while reading the paper is whether or not there is enough data to train the transformer model? Transformers often need much larger amounts of data to be very successful so it's unclear if the use of the transformer in this paper is truly helpful compared to the amount of data collected for the experiments.\n- The concept that training a latent model helps increase the success of planning over longer Horizons is not a novel contribution. this has been studied in many prior papers and has been shown to be increasingly important, especially for longer Horizon tasks.\n- The results shown in the tables are missing important information and should include more significant figures. To all readers to better process the results of the experiments, more details are the quantities should be reported as well as confidences/variances over the computed quantities.\n- For showing the importance of the GMM method starting in line 246, the authors note that the method is very important for providing better trajectory generation. The authors need to provide more information on how they determine which trajectory generation is better. what type of metric is used to define this type of better trajectory generation? This should be in the main paper.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- The method does grow on an interesting pair of times in the community about how to take better advantage of data that is much cheaper to collect, in particular data of humans interacting with their own hands inside of the workspace that the robot arm will end up operating in.\n- The method also proposes using a GMM model to help capture the multimodality of human demonstrations better.\n- This appears to result in some strong performance for the overall method.\n\nWeaknesses:\n- There are a number of papers that use a similar paradigm of trying to train robotic manipulation models from play data. The paper would benefit from situating itself amongst this literature better.\n- There are many components of the method section that are not explained clearly enough to understand how the method is designed.\n- More information on how the methods are compared and how performance is defined will shed more light on understanding the performance differences between these method environments.\n\n\n------- \nI have updated my score after three discussions.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "0ZC6Iern7g", "reviewer": "Reviewer_mxMa", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "(see below for summary)", "summary_of_recommendation": "This work is well-motivated and performant, with strong potential impact in an order of magnitude reduction with on-robot data required. There are extensive empirical investigations on multiple domains with baselines from different domains. The presentation is fairly good, but minor improvements and clarifications can be made. I also have a few minor points about the future scalability of the method, but it does not detract much from my holistic rating. Overall, I believe the work is impactful and significant, and makes a strong contribution to hierarchical robot learning under real world data considerations. \n\n[Post-Rebuttal Update]\nThe reviewers have addressed my questions and I maintain my positive rating of Strong Accept. I believe this is a valuable contribution and strongly argue for acceptance.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "My main questions would be to clarify my points raised in the weakness section above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**:\n\nThis work introduces MimicPlay, a hierarchical learning from play data framework that decouples high-level latent plan learning from low-level control learning. The motivation is that in the standard robot play setting, both high level plans as well as low-level control must be learned from the same data source (teleoperated robot play data) despite these two problems having very different problem requirements. Thus, MimicPlay uses human play data in the same domain to learn high level latent plans, which are then used by a low-level policy that learns from a much smaller amount of teleoperated robot data (<30 demos) compared to prior works. The high level planner learn to predict future human hand 3D poses given the initial state and a goal state (10-30s in future) with a GMM to handle the inherent multimodality of play data, coupled with a KL divergence loss to enforce learned visual encoders to minimize the distribution shift between robot and human domain data. The low-level policy is a standard latent goal conditioned multitask imitation learning policy. There are quite extensive experiments on 14 tasks across 6 different environments, including comparisons with baselines such as GC-BC, LMP, C-BeT, and R3M; in these experiments, MimicPlay is the most performant, especially in the lower data regime. In addition, an ablation study shows that the multimodal GMM is crucial for performance, as well as the human data.\n\n**Strengths**:\n- This work is strongly motivated, since the expense of collecting real data is non-trivial and any work that reduces the data requirement (such as by offloading high-level plan learning to much cheaper human data) will be impactful if successful.\n- The empirical results are extensive, comparing to baseline across other play methods (ie. LMP and C-BeT) as well as representation learning methods (R3M). \n- The technical soundness is strong, with the ablations showing the importance of the GMM as well as the human data, both of which are core claims of the method.\n- The method is elegant and performant, with the cross-morphology results with prompting from either human or robot videos very compelling.\n\n**Weaknesses**:\n- The hand orientations are 3D only without orientation information. Since the 3D pose prediction (opposed to 2D from prior works) is noted as a minor contribution, it would be helpful to #1 study the impact of expanding 2D to 3D plans and #2 clarify why orientation is not included in the plans.\n- The KL term for bridging the robot visual distribution to the human visual distribution may not scale when both of these distributions become more complex. For example, robot data may become more challenging with in a more complex setting (ie. mobile manipulation) or with more realistic hardware (ie. un-calibrated camera) or with more difficult tasks (ie. home setting). And in parallel, human data may become more challenging with more diverse environments (ie. Ego4D / EpicKitchens) or realistic hardware (ie. no 3D ground-truth pose if there are no calibrated RGBD cameras). Therefore, I am worried that the KL term alone may not scale well to more realistic settings.\n- Related to the previous point, not only will the visual distribution shift cause challenges with scale, also the plans themselves may only be useful if the human play data is in the exact same domain as the robot demos.\n- Some implementation details were not clear in the main work. For example, data requirements for the human play as well as robot demos (how many?) were not clear. Or, it's unclear what is meant for \"baselines get 10x more demos\" -- is this 10x robot demos that are used for both plan learning and low-level control learning? For the robot data, did they receive demos or play data (my understanding is MimicPlay does not use any robot play data but only demos; on the other hand, the baseline play methods do not use any demos but only robot play data)", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method which uses a combination of human play data and robot teleoperation data, in order to train manipulation policies that address a variety of tasks, including some of reasonably long horizon. This paper addresses several important themes in robot learning, including: (i) it benefits from using third-person data of humans, (ii) it benefits from using data which is not necessarily task-specific ("play" data), (iii) it shows relevance to "long horizon" manipulation tasks, (iv) the human demos can be thought of as a type of multimodal prompting for addressing manipulation tasks.

In addition to the relevance of its general topics, the authors work was generally well received by the mix of reviewers.  Multiple reviewers commented on the strength of the motivation and the value of the empirical results.  3 reviewers started with Strong Accepts, and after a robust rebuttal discussion with the 4th reviewer, they increased to Weak Accept.  One of the key differentiators that came up in the rebuttal with the 4th reviewer, is that this work uses not just (a) "human" data i.e. of task-specific human demonstrations, or (b) "play" data i.e. of robots collecting non-task-specific data, but it specifically uses (a+b) "human play" data, in which humans are collecting non-task-specific data.

There are many other details of this paper that provide valuable technical ideas and analysis, including the details of the high-level/low-level policy interaction, the 3D human hand tracking, and reasonably comprehensive ablations.

Despite overall complementary reviews, the authors still have several things to do to best incorporate all feedback from reviewers, including adding more seeded evaluations (which was mentioned by multiple reviewers) and describing the additional missing details discussed.  Also the authors mention in the rebuttal they plan to open-source their code, and I would highly recommend to do so it they would like to maximize the impact of their work.
Decision: Accept (Oral)