MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations
Authors: Ajay Mandlekar
Soroush Nasiriany
Bowen Wen
Iretiayo Akinola
Yashraj Narang
Linxi Fan
Yuke Zhu
Dieter Fox
Venue: CoRL 2023 Poster (2023)
Keywords: Imitation Learning
Manipulation

Abstract: Imitation learning from a large set of human demonstrations has proved to be an effective paradigm for building capable robot agents. However, the demonstrations can be extremely costly and time-consuming to collect. We introduce MimicGen, a system for automatically synthesizing large-scale, rich datasets from only a small number of human demonstrations by adapting them to new contexts. We use MimicGen to generate over 50K demonstrations across 18 tasks with diverse scene configurations, object instances, and robot arms from just ~200 human demonstrations. We show that robot agents can be effectively trained on this generated dataset by imitation learning to achieve strong performance in long-horizon and high-precision tasks, such as multi-part assembly and coffee preparation, across broad initial state distributions. We further demonstrate that the effectiveness and utility of MimicGen data compare favorably to collecting additional human demonstrations, making it a powerful and economical approach towards scaling up robot learning. Datasets, simulation environments, videos, and more at https://mimicgen.github.io .

MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations
Ajay Mandlekar", Soroush Nasiriany 2* Bowen Wen Iretiayo Akinola',
Yashraj Narang', Linxxi Fan', Yuke Zhu Dieter Fox!
equal contribution, 'NVIDIA, 'The University of Texas at Austin
Abstract: Imitation learning from a large set of human demonstrations has proved to be an effective paradigm for building capable robot agents. However, the demonstrations can be extremely costly and time-consuming to collect. We intro- duce MimicGen, a system for automatically synthesizing large-scale, rich datasets from only a small number of human demonstrations by adapting them to new con- texts. We use MimicGen to generate over 50K demonstrations across 18 tasks with diverse scene configurations, object instances, and robot arms from just 2000 hu- man demonstrations. We show that robot agents can be effectively trained on this generated dataset by imitation learning to achieve strong performance in long- horizon and high-precision tasks, such as multi-part assembly and coffee prepa- ration, across broad initial state distributions. We further demonstrate that the effectiveness and utility of MimicGen data compare favorably collecting ad- ditional human demonstrations, making it a powerful and economical approach towards scaling up robot learning. Datasets, simulation environments, videos, and more at ht tstpps: / /tm//mmimmeee/ee//ee gi github....... io.
Keywords: Imitation Learning, Manipulation
Introduction
Imitation learning from human demonstrations has become effective paradigm for training robots to perform a wide variety of manipulation behaviors. One popular approach is to have human op- erators telcoperate robot arms through different control interfaces [1.2], resulting in several demon- strations of robots performing various manipulation tasks, and consequently to use the data to train the robots perform these tasks their own. Recent attempts have aimed to scale this paradigm by collecting more data with a larger group of human operators over a broader range of tasks [3-6]. These works have shown that imitation learning on large diverse datasets can produce impressive performance, allowing robots to generalize toward new objects and unseen tasks. This suggests that critical step toward building generally capable robots is collecting large and rich datasets.
However, this success does not come without costly and time-consuming human labor. Consider case study from robomimic [7], in which the agent is tasked with moving a coke can from one bin into another. This is a simple task involving a single scene, single object, and single robot; however, relatively-large dataset of 200 demonstrations
required to achieve a modest success rate 73.3%. Recent efforts at expanding to settings with diverse scenes and objects have required orders of magnitude larger datasets spanning tens of thousands of demonstrations. For example, [3] showed that a dataset over 20,000 trajectories enables generalization to tasks with modest changes in objects and goals. The nearly 1.5-year data collection effort from RT-1 [5] spans several human operators, months, kitchens, and robot arms to produce policies that can rearrange, cleanup, and retrieve objects with 97% success across handful of kitchens. Yet it remains unclear how many years of data collection would be needed to deploy such a system to kitchens in the wild.
We raise the question how much of this data actually contains unique manipulation be- haviors? Large portions of these datasets may contain similar manipulation skills applied in different contexts or situations. For example, human operators may demonstrate very sim- ilar robot trajectories to grasp a mug, regardless of its location on one countertop or an-
7th Conference on Robot Learning (CoRI. 2023), Atlanta, USA.

---

other. Re-purposing these trajectories in new contexts can be way to generate diverse data without much human eftort. In fact, several recent works build on this intuition and propose imitation learning methods that replay previous human demonstrations [8-11].
While promising, these methods make as- sumptions about specific tasks and algorithms that limit their applicability. Instead, we seek to develop general-purpose system that can be integrated seamlessly into existing imita- tion learning pipelines and improve the perfor- mance of a wide spectrum of tasks.
Small human demonetrations
Large, broad dataset generated automatically with MimicGen
Damo
Diverss configuratione
Duro
objject
In this paper, we introduce a novel data col- lection system that uses small set of hu- man demonstrations to automatically gener- ate large datasets across diverse scenes. Our system, MimicGen, takes small number of human demonstrations and divides them into object-centric segments. Then, given a new scene with different object poses, it se- lects one of the human demonstrations, spa- tially transforms each of its object-centric seg- ments, stitches them together, and has the robot follow this new trajectory to collect a new demonstration. While simple, found that this method is extremely effective at generating large datasets across diverse scenes and that the datasets can be used train capable agents through imitation learning.
Dero
Divarae roioct hardware
MimicGen Overview. We introduce data generation system that can produce large diverse datasets from a small number of human demonstrations by re-purposing the demonstrations to make them appli- cable in new settings. We apply MimicGen to generate configurations, objects, and
Figure 1:
data across diverse robot hardware.
We make the following contributions:
We introduce MimicGen, a system for generating large diverse datasets from small number of human demonstrations by adapting the human demonstrations to novel settings.
We demonstrate that MimicGen is able to generate high-quality data train proficient agents via imitation learning across diverse scene configurations, object instances, and robot arms, all which unseen in the original demos (see Fig. 1). MimicGen is broadly applicable to wide range of long-horizon and high-precision tasks that require different manipulation skills, such as pick-and- place, insertion, and interacting with articulated objects. We generated 50K+ new demonstrations for 18 tasks across 2 simulators and a physical robot arm using only ~200 source human demos.
Our approach compares favorably to the alternative of collecting more human demonstrations using MimicGen to generate an equal amount of synthetic data (e.g. 200 demos generated from 10 human vs. 200 human demos) results in comparable agent performance this raises important questions about when it is actually necessary to request additional data from a human.
2 Related Work
Some robot data collection efforts have employed trial-and-crror [12-17] and pre-programmed demonstrators in simulation [18-22]. but it can be difficult scale these approaches to more com- plex tasks. One popular data source is human demonstrators that teleoperate robot arms [2-6,23-27], but collecting large datasets require extensive human time, effort, and cost. Instead, MimicGen tries to make effective use of a small set of human samples to generate large datasets. We train policies from our generated data using imitation learning, which has been used extensively in prior work [1, 19, 25, 28-34J. Some works have used offline data augmentation to increase the dataset size for learning policies [7,35-45] in this work we generate new datasets online. Our data generation method employs a similar mechanism to replay-based imitation approaches [8-11, 1, 46-48J, which solve tasks by having the robot replay prior demonstrations. More discussion in Appendix E.
Problem Setup
Imitation Learning. We consider each robot manipulation task as Markov Decision Process (MDP), and aim to learn a robot manipulation policy that maps the state space S to the action space A. The imitation dataset consists of N demonstrations D {()) ags s ai.,,, ..., ..., ,,,,s,,,) )).Y where each = ~ D(-) is sampled from the initial state distribution D. In this work, we use Behavioral Cloning [28] to train the policy with the objective arg ming [(sa))>D log (als)))

---

Parse source demonstrations into segments
Pipeline for generating new trajectories
Ohhbiin regiwer
Damo
Chowrralan
Demo
Ceracade regirnet
Thamslor segreri
Figure 2: MimicGen System Pipeline. (left) MimicGen first parses the demos from the source dataset into segments, where each segment corresponds an object-centric subtask (Sec. 4.1). (right) Then, generate new demonstrations for new scene, MimicGen generates and follows sequence of end-effector target poses for each subtask by (1) choosing a segment from a source demonstration (chosen segments shown with blue border in figure above), (2) transforming it for the new scene, and (3) executing it (Sec. 4.2).
Problem Statement and Assumptions. Our goal is use source dataset Dare that consists of a small set of human demonstrations collected on a task M and use it to generate a large dataset D either the same task or task variants (where the initial state distribution D, the objects, or the robot arm can change). To generate a new demo: (1) start state is sampled from the task we want to generate data for, (2) a demonstration T Pare chosen and adapted to produce new robot trajectory r', (3) the robot executes the trajectory r' on the current scene, and if the task is completed successfully, the sequence of states and actions is added the generated dataset 'D (see Sec. 4 for details of each step). We next outline some assumptions that our system leverages.
Assumption 1: delta end effector pose action space. The action space A consists of delta-post commands for an end-effector controller and a gripper open/close command. This is a common action space used in prior work [3-7.33]. This gives us an equivalence between delta-pose actions and controller target poses, and allows us to treat the actions demonstration as a sequence of target poses for the end effector controller (Appendix N).
Assumption 2: tasks consist of a known sequence of object-centric subtasks. Let {or...., OK} be the set of objects in a task M. As in Di Palo et al. [11], assume that tasks Sm (os,,)) )), where the ma-
consist of a sequence of object-centric subtasks (S1 (os,) ). S2(00,)) nipulation in each subtask (,s,s) is relative to a single object's coordinate frame (os, E O). We assume this sequence is known (it is typically easy for a human to specify - see Appendix K).
Assumption 3: object poses can be observed at the start of each subtask during data collection. We assume that we can observe the pose of the relevant object os, at the start of each subtask (os,) during data collection (not, however, during policy deployment).
Method
We describe how MimicGen generates new demonstrations using small source dataset of human demonstrations (see Fig. for an overview). MimieGen first parses the source dataset into segments one for each object-centric subtask in task (Sec. 4.1). Then, to generate demonstration for new scene, MimicGen generates and executes a trajectory (sequence of end-effector control poses) for each subtask. by choosing reference segment from the source demonstrations, transforming according to the pose of the object in the new scene, and then executing the sequence of target poses using the end effector controller (Sec. 4.2).
4.1 Parsing the Source Dataset into Object-Centric Segments
Each task consists of a sequence of object-centric subtasks (Assumption 2, Sec. 3) we would like to parse every trajectory т in the source dataset into segments [T where each segment T; corresponds to a subtask S,(os,). for each subtask, we assume access to metrics that allow the end of each subtask to be detected automatically (see Appendix K for full details). After this step, every trajectory T E Dare has been split into contiguous sequence of segments (T1, T2, .... TM one per subtask.
this work, to parse source demonstrations into segments

---

4.2 Transforming Source Data Segments for a New Scene
To generate a task demonstration for a new scene, MimicGen generates and executes a segment for each object-centric subtask in the task. As shown in Fig. 2 (right), this consists of three key steps for each subtask: (1) choosing a reference subtask segment in the source dataset, (2) transforming the subtask segment for the new context, and (3) executing the segment in the scene.
Choosing a reference segment: Recall that MimicGen parses the source dataset into segments that correspond to each subtask Dare = { ..., (.......)))))) Thm) where .V = |Parel- At the start of each subtask (os,,) ), MimicGen chooses corresponding segment from the set {T() can be chosen at random or by using the relevant object poses (more details in Appendix N).
N 1==1- These segments
Transforming the source subtask segment:
We can consider the chosen source subtask segment Ty for subtask (os.)) ) as a sequence of target poses for the end effector controller (Assumption I, Sec. 3). Let T be the homogeneous 4x4 matrix that represents the pose of frame A with respect to frame B. Then we can write T; (Tw", W ....; TWx) where C. is the controller target pose frame at timestep 1, W is the world frame, and K is the length of the segment. Since this motion is assumed to be relative to the pose of the object os, (frame Oo with pose TW) at the start of the segment, we will transform 72 according to the new pose of the corresponding object in the current scene (frame O with pose TW)) so that the relative poses between the target pose frame and the object frame are preserved at each timestep (T = =To') resulting in the transformed sequence where Tw = To (T (derivation in Appendix M). As an
T: = (T Tw" ..... T example, see how the source segment and transformed segment in the right side of Fig. 2 approach the mug in consistent ways. However, the first target pose of the new segment T might be far from the current end-effector pose of the robot the new scene T. (where E is the end-effector frame). Consequently, MimicGen adds an interpolation segment at the start of T! to interpolate
linearly from the current end-effector pose (TW)) to the start of the transformed segment T
Executing the new segment: Finally, MimicGen executes the new segment T/ by taking the target pose at each timestep, transforming into a delta pose action (Assumption 1, Sec. 3), pairing it with the appropriate gripper open/close action from the source segment, and executing the new action.
The steps above repeat for each subtask until the final segment has been executed. However, this process can be imperfect - small trajectory deviations due to control and arm kinematics issues can result in task failure. Thus, MimicGen checks for task success after executing all segments, and only keeps successful demonstrations. We refer to the ratio between the number of successfully generated trajectories and the total number of attempts as the data generation rate (reported in Appendix P).
This pipeline only depends on object frames and robot controller frames - this enables data gener- ation to take place across tasks with different initial state distributions. objects (assuming they have canonical frames defined), and robot arms (assuming they share a convention for the end effector control frame). In our experiments, we designed task variants for each robot manipulation task where we vary either the initial state distribution (D), an object in the task (O), or the robot arm (R), and showed that MimicGen enables data collection and imitation learning across these variants.
5 Experiment Setup
We applicd MimicGen to a broad range of tasks (see Fig. 3) and task variants, in order to showcase how it can generate useful data for imitation learning across diverse set of manipulation behaviors, including pick-and-plaa aand-paac,, contact-rich interactions, and articulation.
Tasks and Task Variants. Each task has default reset distribution (Do) (all source datasets were collected on this task variant), a broader reset distribution (D1), and some have another (D2), meant to pose even higher difficulty for data generation and policy learning. Consider the Threading task shown in Fig. 5 5- in the Do variant, the tripod is always initialized in the same location, while in the Dj variant, both the tripod and needle can move, and in the D2 variant, the tripod and needle are randomized in novel regions of the workspace. In some experiments, we also applied MimicGen to task variants with different robot arm (R) or different object instances (O) within category.

---

(a) Stack Three
(h) Square
Threading
(d) Pc. Assembly
(e) Pick Place
(1) Kitchen
(g) Coflee Prep
(h) Mobile Kitchen
(i) Gear Assembly
(i) Frame Assembly
Figure 3: Tasks. We use MimicGen to generate demonstrations for several tasks - these are subset. They span a wide variety of behaviors including pick-and-place insertion, interacting with articulated objects, and mobile manipulation, and include long-horizon tasks requiring chaining several behaviors together.
We group the tasks into categories and summarize them below (full tasks and variants in Ap- pendix L). Some tasks are implemented with the robosuite framework [49] (MuJoCo backend [50]) and others are implemented in Factory [51] (Isaac Gym [52] backend). Basic Tasks (Stack, Stack Three): of box stacking tasks. Contact-Rich Tasks (Square, Threading, Coffee, Three Piece Assembly, Hammer Cleanup, Mug Cleanup): a set of tasks that involve contact-rich behaviors such insertion drawer articulation. Long-Horizon Tasks (Kitchen, Nut Assembly, Pick Place, Cof- fee Preparation): require chaining multiple behaviors together. Mobile Manipulation Tasks (Mo- bile Kitchen): requires base and arm motion. Factory Tasks Mout-Bolt-Assemble Gear Assembly, Frame Assembly): a set of high-precision assembly tasks in Factory [51].
Data Generation and Imitation Learning Methodology. For each task, one human operator col- lected a source dataset of 10 demonstrations on the default variant (Du) using a teleoperation sys- tem [2. (with the exception of Mobile Kitchen, where used 25 demos due to the large number of object variants, and Square, where we used 10 demos from the robomimic Square PH dataset [7])- MimicGen used to generate 1000 demonstrations for each task variant, using each task's source dataset (full details in Appendix N). Since data generation is imperfect, each data generation at- tempt is not guaranteed to result in task success. Attempts that did not achieve task success were discarded, and data collection kept proceeding for each task variant until 1000 task successes were collected. Each generated dataset was then used train policies using Behavioral Cloning with an RNN policy [7]. We also adopt the convention from Mandlekar et al. [7] for reporting policy performance the maximum success rate across all policy evaluations, across different seeds (full training details in Appendix O). All policy learning results are shown on image-based agents trained with RGB observations (see Appendix Q for low-dim agent results).
Experiments
We present experiments that (1) highlight the diverse array of situations that MimicGen can generate data for, (2) show that MimicGen compares favorably to collecting additional human demonstra- tions, both in terms effort and downstream policy performance on the data, (3) offer insights into different aspects of the system, and (4) show that MimicGen work real-world robot arms,
6.1 Applications of MimicGen
We outline a number of applications that showcase useful properties of MimicGen.
MimicGen data vastly improves agent performance on the source task. A straightforward ap- plication of MimicGen is to collect small dataset on some task of interest and then generate more data for that task. Comparing the performance of agents trained on the small source datasets those trained on Do datasets generated by MimicGen, we see that there is substantial improvement across all our tasks (see Fig. 4). Some particularly compelling examples include Square (11.3% to 90.7%), Threading (19.3% to 98.0%), and Three Piece Assembly (1.3% to 82.0%).
MimicGen data can produce performant agents across broad initial state distributions. As shown in Fig. 4), agents trained using datasets generated on broad initial state distributions (D,, D2) are performant (42% to 99% on D1), showing that MimicGen generates valuable datasets on new initial state distributions. several cases, certain objects in the source demonstrations never moved (the peg in Square, the tripod in Threading, the base in Three Piece Assembly, etc), but
5

---

Task
Source
D₂
26.0
100.0+0.00 92.7 +
99.3
120
Sounce Sias Comparson
Stack Three
0.7 + 0.9
3.4
Square
0.9
90.7 + 1.9
3.4
40.3 = 2.5
Threading
3.4
98.0 1.6
60.7 2.5
38.0 3.3
Cofiee
T4.0 4.3
1.0
17.3
Three Pc. Assembly
1.3.
82.0 1.6
0..71.25
13.3 3.8
Squin (10)
Sqaare (T1)
Stgare (72)
TRNNTIO
TRANTY
Hommer Cleunup
50.3 5.7
100.0-1:000 0.0
02.7 4:7
Th
Mug Cleanup
12.7 2.5
80.0 4.9
8..
deneo
daraas
damas
Kitchen
Assembly
8.4
1.0
76.0 4.3
Policy Trairing Comparison
Pick Place
0.0 0.0
50.7 1: 6.6
Coffee Preguration
12.7
07.3 + 0.9
42.0++ 0.0
Mobile Kitchen
2.0 0.0
46.7 18.4
Nut-and- Bolt Assembly
8.7
92.7 à: 2.5
81.3
72.7
4.
Gear Assembly
10.7 5.2
740
56.7
(29)
Frame Assemibyy
10.7 1, 0.8
82.0 4.3
08.7 3.4 36.7
2.5
Figure 4: (left) Agent Performance on Source and Generated Datasets. Success rates (3 seeds) of image- based agents trained with BC on the 10 source demos and cach 1 1000 demo MimicGen dataset. There is large improvement across all tasks on the default distribution (Do) and agents are performant on the broader distribu- tions (D1, D2). (top-right) MimicGen with more source human demonstrations. We found that using larger source datasets generate MimicGen data did not result in significant agent improvement. (bottom-right) Pol- icy Training Datasct Comparison. Image-based agent performance comparable on MimicGen demos and 200 human demos, despite MimicGen only using 10 source human demos. MimicGen can produce im- proved agents by generating larger datasets (200, 1000, 5000 demos), but there are diminishing returns.
data was generated (and policies consequently were trained) on regimes where the objects move in substantial regions of the robot workspace.
MimicGen can generate data for different objects. The source dataset in the Mug Cleanup task contains just one mug, but we generate demonstrations with MimicGen for an unseen mug (O1) and for set of 12 mugs (O2). Policies trained on these datasets have substantial task success rates (90.7% and 75.3% respectively) (full results in Appendix G).
MimicGen can generate data for diverse robot hardware. We apply MimicGen to the Square and Threading source datasets (which use the Panda arm) and generate datasets for the Sawyer, IIWA, and UR5e across the Do and Dj reset distribution variants. Interestingly, although the data generation rates differ greatly per arm (range 38%-74% for Square Do), trained policy performance is remarkably similar across the 4 robot arms (80%-919) ,, full results in Appendix F). This shows the potential for using human demonstrations across robot hardware using MimicGen, an exciting prospect, as teleoperated demonstrations are typically constrained to a single robot.
Applying MimicGen mobile manipulation. In the Mobile Kitchen task MimicGen yields gain from 2.0% to 46.7% (image, Fig. 4) and 2.7% to 76.7% success rate (low-dim, Table Q.1 in Appendix), highlighting that our method can be applied to tasks beyond static tabletop manipulation.
MimicGen is simulator-agnostic.. We show that MimicGen is not limited to just one simulation framework by applying it to high-precision tasks (requiring millimeter precision) in Factory [51], a simulation framework built on top of Isaac Gym [52] to accurately simulate high-precision ma- nipulation. We generate data for and train performant policies the Nut-and-Bole Assembly, Gear Assembly, and Frame Assembly tasks. Policies achieve excellent results on the nominal tasks (Dp) (82%-99%), a significant improvement over policies trained on the source datasets (9%-155 15%), and are also able to achieve substantial performance on wider reset distributions (D,, D2) (37% 37%%8199 %).
demonstrations from inexperienced human operators and different tele- operation devices. Surprisingly, policies trained on these MimicGen datasets have comparable performance to those in Fig. 4. See Appendix I for the full set of results.
MimicGen
6.2 Comparing MimicGen to using more human data
In this section, we contextualize the performance of agents trained on MimicGen data.
Comparing task performance to prior works. Zhu et al. [53] introduced the Hammer Cleanup and Kitchen tasks and reported agent performance on 100 human demonstrations for their method called BUDS. On Hammer Cleanup, BUDS achieved 68.69 (Do), while BC-RNN achieves 59.3% on our 10 source demos, 100.0% on our generated 1000 Do demos, and 62.7% on the D2 variant

---

where both the hammer and drawer move substantially. On Kitchen, BUDS achieved 72.0% (Do), while BC-RNN achieves 54.7% on our 10 source demos, 100.0% on our generated Do data, and 76.0% on the Dj variant, where all objects move in wider regions. This shows that using MimicGen to make effective use of a small number of human demonstrations can improve the complexity of tasks that can be learned with imitation learning. As another example, Mandlekar et al. [2] collected over 1000 human demos across 10 human operators on both the Nut Assembly and Pick Place tasks, but only managed to train proficient policies for easier, single-stage versions of these tasks using combination of reinforcement learning and demonstrations. By contrast, in this work we are able to make effective use of just 10 human demonstrations to generate set of 1000 demonstrations and learn proficient agents from them (76.0% and 58.7% low-dim, 53.3% and 50.7% image).
Agent performance on data generated by MimicGen can be comparable to performance on an equal amount of human demonstrations. We collect 200 human demonstrations on several tasks and compare agent performance on those demonstrations to agent performance on 200 demonstra- tions generated by MimicGen (see Fig. 4). In most cases, agent performance is similar, despite the 200 MimicGen demos being generated from just 10 human demos- a small number of human demos can be as effective (or even more) than a large number of them when used with MimicGen. MimicGen can also easily generate more demonstrations to improve performance (see Sec. 6.3), unlike the time-consumingg nature of collecting more human data. This result also raises important questions whether soliciting more human demonstrations can be redundant and not worth the labeling cost, and where to collect human demonstrations given a finite labeling bandwidth.
6.3 MimicGen Analysis
We analyze some practical aspects of the system, including (I) whether the number of source demon- strations used impacts agent performance, (2) whether the choice of source demonstrations matters, (3) whether agent performance can keep improving by generating more demonstrations, and (4) whether the data generation success rate and trained agent performance are correlated.
Can dataset quality and agent performance be improved by using more souree human demon- strations? We used 10, 50, and 200 source human demonstrations on the Square and Three Piece Assembly tasks, and report the policy success rates in Fig. 4. We see that performance differences are modest (ranging from 2% to 21%). We also tried using just I human demo - in some cases performance was much worse (e.g. Square), while in others, there was no significant performance change (c.g. Three Piece Assembly). is possible that performance could improve with more source human demos if they are curated in an intelligent manner, but this is left for future work.
Does the choice of source human demonstrations matter? For each generated dataset, we logged which episode came from which source human demonstration - in certain cases, this distribution can be very non-uniform. As an example, the generated Factory Gear Assembly task (D,) had over 850 of the 1000 episodes come from just 3 source demonstrations. In the generated Threading task (Do), one source demo had over 170 episodes while another had less than 10 episodes. In both cases, the number of attempted episodes per source demonstration was roughly uniform (since picked them at random - details in Appendix N), but some were more likely to generate successful demonstrations than others. Furthermore, we found the source demonstration segment selection technique (Sec. 4.2) to matter for certain tasks (Appendix N). This indicates that both the initial set of source demos provided to MimicGen (D.c), and how segments from these demos chosen during each generation attempt (T; for each subtask, see Sec. 4.1) can matter.
Can agent performance keep improving by generating more demonstrations? In Fig. 4, we train agents on 200, 1000, and 5000 demos generated by MimicGen across several tasks. There is a large jump in performance from 200 to 1000, but not much from 1000 to 5000, showing that there can be diminishing returns on generating more data.
Are the data generation success rate and trained agent performance correlated? It is tempting to think that data generation success rate and trained agent performance are correlated, but we found that this is not necessarily true there are datasets that had low dataset generation success rates (and consequently took long time to generate 1000 successes) but had high agent performance after training on the data (Appendix P). A few examples are Object Cleanup (Da) (29.5% generation rate, 82.0% agent rate), Three Piece Assembly (Do) (35.6% generation rate, 74.7% agent rate), Coffee (D2) (27. 77% generation rate, 76.7% agent rate), and Factory Gear Assembly (D₂) (8.2% generation rate, 76.0% agent rate). These results showease the value of using replay-based mechanisms for data collection instead of directly using them to deploy as policy as in prior works [8, 11].
7

---

(a) Do
(b) D1
(c) D2
Figure 5: (left) Reset Distributions. Each task has a default reset distribution for the objects (Do), a broader (D1), and some had more challenging and needle in the Threading task. The tripod is fixed location in Do, and D2 swaps the relative locations of tripod and needle. We generate data across diverse scene configurations by taking source demos from Do and generating data for all variants. (right) Real Robot Tasks. We apply MimicGen to two real robot tasks Stack (top row) and Coffee (bottom row). In the first column, the blue and orange regions show the source (Do) and generated (D:) reset distributions for each task. We use 10 source demos per task, and generate 100 successful demos- MimicGen has data generation success rate of 82.3% for Stack and 52.1% for Coffee.
(D2). The figure shows the sampling regions for the tripod
6.4 Real Robot Evaluation
We validate that MimicGen can be applied to real-world robot arms and tasks. We collect 10 source demonstrations for each task narrow regions of the workspace (Do) and then generate demon- strations (200 for Stack, 100 for Coffee) for large regions of the workspace (D1) (see Fig. 5). The generation success rate was 82.3% for Stack (243 attempts) and 52.1% for Coffee (192 attempts), showing that MimicGen works in the real world with reasonably high success rate. We then trained visuomotor agents using front-facing RealSense D415 camera and a woist-mounted Re- alSense D435 camera (120x 160 resolution). Over 50 evaluations, our Stack agent had 36% success rate and Coffee had 14% success rate (pod grasp success rate of 60% and pod insertion success rate of 20%). The lower numbers than from simulation might be due to the larger number of interpola- tion steps we used in the real world for hardware safety (50 total instead of 5) - these motions are difficult for the agent to imitate since there is little association between the intermediate motion and observations (see Appendix H for more experiments and discussion).
We also compared to agents trained on the source datasets (10 demos) in the narrow regions (orange regions in Fig. 5) where the source data came from the Stack source agent had 0% success rate and the Coffee source agent had 0% success rate (with an insertion rate of 0% and pod grasp rate of 94%). The Coffee (Do) task in particular has barely any variation (the pod can move vertically in a 5cm region) compared to the D, task, which is substantially harder (pod placed anywhere in the right half of the workspace). Agents trained with MimicGen data compare favorably to these agents, as they achieve non-zero success rates on broader task reset distributions.
7 Limitations
See Appendix for full set of limitations and discussion. MimicGen assumes knowledge of the object-centric subtasks in task and requires object pose estimates at the start of each subtask during data generation (Assumption 3, Sec. 3). MimicGin only filters data generation attempts based on task success, so generated datasets can be biased (Appendix R). MimicGen uses linear interpolation between human segments (Appendix N.2), which does not guarantee collision- -ree motion, and can potentially hurt agent performance (Appendix H). MimicGen was demonstrated quasi-static tasks with rigid objects, and novel objects were assumed to come from the same category.
8 Conclusion
We introduced MimicGen, data generation system that strations to generate large datasets across diverse scenes, object instances, and robots, and applied generate over 50K demos across 18 tasks from less than 200 human demos, including tasks involv- ing long-horizon and tigh-precision manipulation. We showed that agents learning from this data can achieve strong performance. We further found that agent performance MimicGen data be comparable to performance on an equal number of human demos - this surprising result motivates further investigation into when solicit additional human demonstrations instead of making more effective use of a small number, and whether human operator time would be better spent collecting data in new regions of the workspace. We hope that MimicGen motivates and enables exploring a more data-centric perspective on imitation learning in future work.
small amounts of human demon-
8

---

Acknowledgments
This work was made possible due to the help and support of Sandeep Desai (robot hardware), Ravin- der Singh (IT), Alperen Degirmenci (compute cluster), Anima Anandkumar (access robot hard- ware), Yifeng Zhu (tasks from [53] and robot control software [54]), Cheng Chi (diffusion policy experiments), Shuo Cheng (drawer design used in Coffee Preparation task), Balakumar Sundar- alingam (code release), and Stan Birchfield (dataset release).
References
[1] T. Zhang, Z. McCarthy, O. Jow, D. Lee, K. Goldberg, and P. Abbeel, "Deep imitation learning for complex manipulation tasks from virtual reality teleoperation," arXiv preprint arXiv:/710.04615, 2017.
[2] A. Mandlekar, Y. Zhu, A. Garg, Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta,
E. Orbay, S. Savarese, and L. Fei-Fei, "RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation," in Conference on Robot Learning, 2018.
[3] E. Jang, A. Irpan, M. Khansari, D. Kappler, F, Ebert, C. Lynch, S. Levine, and C. Finn, "Be-
z: Zero-shot task generalization with robotic imitation learning," in Conference on Robot Learning. PMLR, 2022, pp. 991-1002.
[4] M. Ahm, A. Brohan, N. Brown, Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog al., "Do as can, not as say: Grounding language in robotic affordances," arXiv preprint arXiv:2204.00 01661, 2022.
[5] A. Brohan, N. Brown, J. Carbajal,
Chebotar. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu et al., "Rt-1: Robotics transformer for real-world control at scale," " arXiv preprint arXiv:2212.068// 22.71177 2022.
[6] F. Ebert, Y. Yang, K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and S. Levine, "Bridge data: Boosting generalization of robotic skills with cross-domain datasets," arXiv preprint arXiv:2109.1333
2021.
[7] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang. R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martin-Martin, "What matters learning from offline human demonstrations for robot manipulation," in Conference Robot Learning (CoRI.), 2021.
[8] B. Wen, W. Lian, K. Bekris, and S. Schaal, "You only demonstrate once: Category-level manipulation from single visual demonstration," " in Robotics: Science and Systems (RSS), 2022.
[9] E. Johns, "Coarse-to-fine imitation learning: Robot manipulation from single demonstra- tion," in 2021 IEEE international conference on robotics and automation (ICRA). IEEE, 2021, pp. 4613-4619.
[10] E. Valassakis, G. Papagiannis, N. Di Palo, and E. Johns, "Demonstrate once, imitate imme- diately (dome): Learning visual servoing for one-shot imitation learning," " in 2022 JEEE/RS. International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022, pp. 8614- 8621.
[11] N. Di Palo and E. Johns, "Learning multi state tasks with one demonstration via self-replay," in Conference on Robot Learning. PMLR, 2022, pp. 1180-1189.
[12] S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen, "Learning hand-eye coordination for robotic grasping with large-scale data collection," in ISER, 2016, pp. 173-184.
[13] L. Pinto and A. Gupta, "Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours," in Robotics and Automation (ICRA), 2016 IEEE Int') I Conference on. IEEE, 2016.
[14] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke et al., "Qt-opt: Scalable deep reinforcement learning for rision-based robotic manipulation," arXiv preprint 1rXiv:1806.10280 2018.
[15] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman, "Mt-opt: Continuous multi-task robotic reinforcement learning at scale," arXiv preprint arXiv:2 104.0822,, 2021.

---

[16] K.-T. Yu, M. Bauza, N. Fazeli, and A. Rodriguez, "More than million ways to be pushed. high-fidelit experimental dataset of planar pushing," in Int'l Conference on Intelligent Robots and Systems, 2016.
[17] S. Dasari, F. Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, S. Singh, S. Levine, and C. Finn, "Robonet: Large-scale multi-robot learning." arXiv preprint arXiv: 199901102.21 1215, 2019.
[18] S. James, Z. Ma, D. R. Arrojo, and J. Davison, "RIbench: The robot learning benchmark & learning environment," IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 3019-3026, 2020.
[19] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani et al., "Transporter networks: Rearranging the visual world for robotic manipulation." arXiv preprint ar arXiv:2010. 1t..:2000..4406.. 2020.
[20] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan, "Vima: General robot manipulation with multimodal prompts," arXiv preprint
0rXiv:2210.030% 2022.
[21] J. Gu, F. Xiang, X. Li, Z. Ling, X. Liu, T. Mu, Y. Tang, S. Tao, X. Wei, Y. Yao et al., "Maniskill2: unified benchmark for generalizable manipulation skills," arXiv preprint 2023.
arXiv:2302.0466
[22] M. Dalal. A. Mandlekar, C. Garrett, A. Handa, R. Salakhutdinov, and D. Fox, "Imitating task and motion planning with visuomotor transformers," arXiv preprint arXiv:2305. /6309, 2023.
[23] A. Mandlekar, J. Booher, M. Spero, A. Tung, A. Gupta, Y. Zhu, A. Garg, S. Savarese, and L. Fei-Fei, "Scaling robot supervision to hundreds of hours with roboturk: Robotic manip- ulation dataset through human reasoning and dexterity," arXiv preprint arXiv: 191110402, 2019.
[24] A. Mandlekar, D. Xu, R. Martin-Marttn, Y. Zhu, L. Fei-Fei, and S. Savarese, -Human-in-the- loop imitation learning using remote teleoperation," arXiv preprint arXiv:2012.06733 2020.
[25] A. Tung, J. Wong, A. Mandlekar, R. Martin-Martín, Zhu, L. Fei-Fei, and S. Savarese, "Learning multi-arm manipulation through collaborative teleoperation," arXiv preprint arXiv:2012.06738, 2020.
[26] J. Wong, A. Tung, A. Kurenkov, A. Mandlekar, L. Fei-Fei, S. Savarese, and R. Martin-Martin, "Error-aware imitation learning from teleoperation data for mobile manipulation." in Confer- ence on Robot Learning. PMLR, 2022, pp. 1367-1378.
[27] C. Lynch, A. Wahid, J. Tompson, T. Ding, J. Betker, R. Baruch, T. Armstrong, and P. Florence, "Interactive language: Talking to robots real time," arXiv preprint arXiv:2210.06407 2022.
[28] D. A. Pomerleau, "Alvinn: An autonomous land vehicle in a neural network," " in Advances in neural information processing systems, 1989, pp. 305-313.
[29] A. J. Ijspeert, J. Nakanishi, and S. Schaal, "Movement imitation with nonlinear dynamical systems in humanoid robots," Proceedings 2002 IEEE International Conference on Robotics and Automation, vol. 2, pp. 1398-1403 vol.2, 2002.
[30] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine, "One-shot visual imitation learning via meta-learning." " in Conference on robot learning. PMLR, 2017, pp. 357-368.
[31] Billard, S. Calinon, R. Dillmann, and S. Schaal, "Robot programming by demonstration," in Springer Handbook of Robotics, 2008.
[32] S. Calinon, F. D' D'halluin, E. L. Sauser, D. G. Caldwell, and A. Billard, "Learning and re- production of gestures by imitation," " JEEE Robotics and Automation Magazine, vol. 17, pp- 44 54, 2010.
[33] A. Mandlekar, D. Xu, R.
S. Savarese, and L. Fei-Fei, "Learning to generalize across long-horizon tasks from human demonstrations." arXiv preprint urXiv:2003.06088 2020.
[34] C. Wang, R. Wang, D. Xu, A. Mandlekar, L. Fei-Fei, and S. Savarese, "Generalization through hand-eye coordination: An action space for learning ipatiall visuomotor control," arXiv preprint arXiv:2103.00377 2021.
10

---

[35] P. Mitrano and D. Berenson, "Data augmentation for manipulation." arXiv preprint arXiv:2205.02886, 2022.
[36] M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas, "Reinforcement learning with augmented data," arXiv preprint arXiv:2004. 11220004414999 14990, 2020.
[37] I. Kostrikov, D. Yarats, and R. Fergus, "Image augmentation is all you need: Regularizing deep reinforcement learning from pixels," " arXiv preprint arXiv:2004.J 110004116644 13649, 2020.
[38] S. Young, D. Gandhi, S. Tulsiani, A. Gupta, P. Abbeel, and L. Pinto, "Visual imitation made easy," arXiv e-prints, pp. arXiv-2008, 2020.
[39] A. Zhan, P. Zhao, L. Pinto, P. Abbeel, and M. Laskin, "A framework for efficient robotic manipulation." arXiv preprint arXiv:2012.07973 2020.
[40] S. Sinha, A. Mandlekar, and A. Garg, "S4rl: Surprisingly simple self-supervision for offline reinforcement learning in robotics," in Conference Robot Learning. PMLR, 2022, pp- 907-917.
[41] S. Pitis, E. Creager, and A. Garg, "Counterfactual data augmentation using locally factored dynamics," Advances in Neural Information Processing Systems, vol. 33, pp. 3976-3990, 2020.
[42] S. Pitis, E. Creager, A. Mandlekar, and A. Garg, "Mocoda: Model-based counterfactual data augmentation." arXiv preprint arXiv:2210. 1187,, 2022.
[43] Z. Mandi, H. Bharadhwaj, V. Moens, S. Song, A. Rajeswaran, and V. Kumar, "Cacti: A framework for scalable multi-task multi-scene visual imitation learning." arXiv preprint 1.Xiv:2212.05711, 2022.
[44] T. Yu, T. Xiao, A. Stone, J. Tompson, A. Brohan, S. Wang, J. Singh, C. Tan, J. Per- alta, B. lchter et al., "Scaling robot learning with semantically imagined experience," arXiv preprint arXiv:2302. 11550, 2023.
[45] Z. Chen, S. Kiami, A. Gupta, and V. Kumar, "Genaug: Retargeting behaviors to unseen situations via generative augmentation," arXiv preprint 2xXiv:2302.06671, 2023.
[46] V. Vosylius and E. Johns, "Where to start? transferring simple skills to complex environ- ments," " arXiv preprint arXiv:2212.0611 22211106111111 2022.
[47] A. Chenu, O. Serris, O. Sigaud, and N. Perrin-Gilbert, "Leveraging sequentiality in reinforce- ment learning from single demonstration," arXiv preprint arXiv:2211.04786, 2022.
[48] J. Liang. Wen, K. Bekris, and A. Boularias, "Learning sensorimotor primitives of sequen- tial manipulation tasks from visual demonstrations." " in 2022 International Conference on Robotics and Automation (ICRA). IEEE, 2022, pp. 8591-8597.
[49] Y. Zhu, J. Wong, A. Mandlekar, and R. Martín Martin, "robosuite: A modular simulation framework and benchmark for robot learning," in arXiv preprint arXiv: 12293, 2020.
[50] E. Todorov, T. Erez, and Y. Tassa, "Mujoco: A physics engine for model-based control," in JEEE/RSJ International Conference on Intelligent Robots and Systems, 2012, pp. 0026-5033.
[51] Y. Narang. K. Storey, I. Akinola, M. Macklin, P. Reist, L. Wawrzyniak, Y. Guo, A. Mora- vanszky, G. State, M. Lu et al., "Factory: Fast contact for robotic assembly," arXiv preprint arXiv:2205.0355 2022.
[52] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa et al., "Isaac gym: High performance gpu-based physics simulation for robot learning," " arXiv preprint a1Xiv:2108. 10470, 2021.
[53] Y. Zhu, P. Stone, and Y. Zhu, "Bottom-up skill discovery from unsegmented demonstrations for long-horizon robot manipulation," IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 4126 4133, 2022.
[54] Y. Zhu, A. Joshi, P. Stone, and Y. Zhu, "Viola: Imitation learning for baseon manipula- tion with object proposal priors." " 6th Annual Conference on Robot Learning, 2022.
[55] S. Nasiriany, T. Gao, A. Mandlekar, and Y. Zhu, "Learning and retrieval from prior data for skill-based imitation learning." in Conference on Robot Learning (CoRI.), 2022.
11

---

[56] O. Mees, L. Hermann, E. Rosete-Besse and W. Burgard, "Calvin: A benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks," IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 7327-7334, 2022.
[57] S. LLvvine A. Kumar, G. Tucker, and J. Fu, "Offline reinforcement learning: Tutorial, review, and perspectives open problems," arXiv preprint urXiv:2005.0100 2020.
[58] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, "Sim-tt-real transfer of robotic control with dynamies randomization," in 2018 IEEE international conference on robotics and automation (ICRA). IEEE, 2018, pp. 3803-3810.
[59] M. Kaspar, J. D. M. Osorio, and J. Bock, "Sim2real transfer for reinforcement learning without dynamics randomization." in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020, pp. 4383-4388.
[60] A. Allshire, M. Mittal, V. Lodaya, V. Makoviychuk, D. Makoviichuk, F. Widmaier, M. Wüthrich, S. Bauer, A. Handa, and A. Garg, "Transferring dexterous manipulation from gpu simulation
remote real-world trifinger," in 2022 IEEF/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022, pp. 11 802-11 809.
[61] M. Khansari, D. Ho, Y. Du, A. Fuentes, M. Bennice, N. Sievers, S. Kirmani, Y. Bai, and E. Jang, "Practical imitation learning in the real world via task consistency loss," arXiv preprint arXiv:2202.0I
2022.
[62] A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makoviichuk, K. Van Wyk, A. Zhurkevich, B. Sundaralingam et al., "Dextreme: Transfer of agile in-hand manipulation from simulation reality," arXiv preprint arXiv:2210. 13702, 2022.
[63] O. Khatib, "A unified approach for motion and force control of robot manipulators: The operational space formulation," IEEE Journal on Robotics and Automation, vol. 3, no. 1, pp. 43-53, 1987.
[64] S. Dasari, J. Wang, J. Hong. S. Bahl, Y. Lin, A. Wang, A. Thankaraj. K. Chahal, B. Calli, S. Gupta al. "Rb2: Robotic manipulation benchmarking with a twist," arXiv preprint arXiv:2203.0800 2022.
[65] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine, "Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning,"" in Conference on robot learning. PMLR, 2020, pp. 1094-1100.
[66] T. Mu, Z. Ling, F. Xiang, D. Yang, X. Li, S. Tao, Z. Huang, Z. Jia, and H. Su, "Maniskill: Generalizable manipulation skill benehmark with large-scale demonstrations, arXiv preprint arXiv:2107. 11..11077.44.,... 2021.
[67] J. Zhang and K. Cho, "Query-effiiient imitation learning for end-to-end autonomous driving." arXiv preprint arXiv: 1605.60055000 2016
[68] R. Hoque, A. Balakrishna, C. Putterman, M. Luo, D. S. Brown, D. Seita, B. Thananjeyan, E. Novoseller, and K. Goldberg, 'Lazydagger: Reducing context switching in interactive imitation learning," in 202/ IEEE 17th International Conference on Automation Science and Engineering (CASE). IEFF, 2021, pp. 502-509.
[69] R. Hoque, A. Balakrishna, E. Novoseller, A. Wilcox, D. S. Brown, and K. Goldberg, "Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning," arXiv preprint arXiv:2109.08277 2021.
[70] S. Dass. K. Pertsch, H. Zhang, Y. Lee, J. Lim, and S. Nikolaidis, "Pato: Policy assisted teleoperation for scalable robot data collection," " arXiv preprint arXiv:2212.04708, 2022.
[71] Y. Zhang, H. Ling. J. Gao, K. Yin, J.-F. Lafleche, A. Barriuso, A. Torralba, and S. Fidler, "Datasetgan: Efficient labeled data factory with minimal human effort," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 10 145- 10 155.
[72] D. Li, H. Ling, S. W. Kim, K. Kreis, S. Fidler, and A. Torralba, "Bigdatasetgan: Synthesiz- ing imagenet with pixel-wise annotations." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. pp. 21 330-21 340.
[73] A. Kar, A. Prakash, M.-Y. Liu, E. Cameracci, J. Yuan, M. Rusiniak, D. Acuna, A. Torralba, and S. Fidler, "Meta-sim: Learning to generate synthetic datasets," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 4551 1551-4
12

---

[74] J. Devaranjan, A. Kar, and S. Fidler, "Meta-sim2: Unsupervised learning of scene structure for synthetic data generation," in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVII 16. Springer, 2020, pp. 715-733.
[75] S. W. Kim, J. Philion, A. Torralba, and S. Fidler, "Drivegan: Towards a controllable high- quality neural simulation," in Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2021, pp. 5820-5829.
[76] D. Paschalidou, A. Kar, M. Shugrina, K. Kreis, A. Geiger, and S. Fidler, "Atiss: Autoregres- sive transformers for indoor scene synthesis," Advances in Neural Information Processing Systems, vol. 34, pp. 12013-12026, 2021.
[77] S. Tan, K. Wong. S. Wang. S. Manivasagam, M. Ren, and R. Urtasun, "Scenegen: Learning to generate realistic traffic scenes," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 892-901.
[78] C. Chamzas, C. Quintero-Penaa Z. Kingston, A. Orthey, D. Rakita, M. Gleicher, M. Toussaint, and L. E. Kavraki, "Motionbenchmaker: A tool to generate and benchmark motion planning datasets," IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 882-889, 2021.
[79] C. Lynch,,, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet, "Learning latent plans from play." in Conference on Robot Learning, 2019.
[80] K. Pertsch, Y. Lee, Y. Wu, and Lim,
soaionnsttratooguuddedd reinforcement learning with learned skills," in Conference on Robot Learning, 2021.
[81] A. Ajay, A. Kumar, P. Agrawal, S. Levine, and O. Nachum, "Opal: Offline primitive discov- ery for accelerating offline reinforcement learning," " in International Conference on Learning Representations, 2021.
[82] K. Hakhamaneshi, R. Zhao, A. Zhan, P. Abbeel, and M. Laskin, "Hicrarchical few-shot imita- tion with skill transition models," in International Conference Learning Representations, 2021.
[83] Kumar, A. Singh, F. Ebert, Y. Yang, C. Finn, and S. Levine, "Pre-training for robots: Of- fline rl enables learning new tasks from handful of trials," arXiv preprint arxiv:2210.05178, 2022.
[84] M. A. Fischler and R. C. Bolles, "Random sample consensus: a paradigm for model fit- ting with applications to image analysis and automated cartography," Communications of the ACM, vol. 24, no. 6, pp. 381-395, 1981.
[85] M. Ester, H.-P. Kriegel, J. Sander, X. Xu et al., "A density-based algorithm for discovering clusters in large spatial databases with noise." in kdd, vol. 96, no. 34, 1996, pp. 226-231.
[86] He, G. Gkioxari, Dollár, and R. Girshick, "Mask r-cnn," in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2961-2969.
[87] M. Danielczuk, M. Matl, S. Gupta, A. Li, A. Lee, J. Mahler, and K. Goldberg, "Segmenting unknown 3d objects from real depth images using mask r-cnn trained on synthetic data," in 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019, pp. 7283-7290.
[88] B. Wen, C. Mitash, S. Soorian, A. Kimmel, A. Sintov, and K. E. Bekris, "Robust, occlusion- aware pose estimation for objects grasped by adaptive hands," in 2020 IEEE International Conference Robotics and Automation (ICRA). IEEE, 2020, pp. 6210-6217.
[89] Z. Zhang, "Iterative point matching for registration of free-form curves and surfaces," Inter- national journal of computer vision, vol. 13, no. 2, pp. 119-152, 1994.
[90] B. Wen and K. Bekris, "Bundletrack: 6d pose tracking for novel objects without instance or category-level 3d models," in IROS, 2021.
[91] T. Lee, J. Tremblay, V. Blukis, B. Wen, B.-U. Lee, I. Shin, S. Birchfield, I. S. Kweon, and K.-J. Yoon, "Tta-cope: Test-time adaptation for category- object pose estimation," in CVPR, 2023.
[92] Y. Liu, Y. Wen, S. Peng, C. Lin, X. Long, T. Komura, and W. Wang, "Gen6d: Generalizable model-free 6-dof object pose estimation from rgb images," in Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXXII. Springer, 2022, pp. 298-315.
13

---

[93] J. Sun, Z. Wang, S. Zhang, X. He. H. Zhao, G. Zhang, and X. Zhou, "Onepose: One-shot object pose estimation without cad models," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 2925-6834.
[94] B. Wen, Tremblay, V. Blukis, S. Tyree, T. Muller, A. Evans, D. Fox, J. Kautz, and S. Birch- field, "Bundlesdf: Neural 6-dof tracking and 3d reconstruction of unknown objects," CVPR, 2023.
[95] E. Valassakis, N. Di Palo, and E. Johns, "Coarse-to- for sim-to-real: Sub-millimen millimetre pre- cision across wide task spaces," in 2021 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS). IEFE, 2021, pp. 5989-5996.
[96] P.-L.. Guhur, S. Chen, R. G. Pinel, M. Tapaswi, I. Laptev, and C. Schmid, "Instruction-driven history-aware policies for robotic manipulations." in Conference on Robot Learning. PMLR, 2023, pp. 175-187.
[97] H. Ha, P. Florence, and Song, "Scaling up and distilling down: Language-gund robot skill acquisition," arXiv preprint arXiv:2307 2023.
[98] A. Agarwal, A. Kumar, J. Malik, and D. Pathak, "Legged locomotion in challenging terrains using egocentric vision," in Conference on Robot Learning. PMLR, 2023, pp. 403-415
[99] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, "Domain randomization for transferring deep neural networks from simulation to the real world," in 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, 2017, pp. 23-30.
[100] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. "Diffusion policy: Visuomotor policy learning via action diffusion," arXiv preprint arXiv:2303.041.37 2023.
[101] Y. Zhu, Z. Wang, J. Merel, A. Rusu, T. Erez, S. Cabi, S. Tunyasuvunakool, J. Kramár, R. Had- sell, N. de Freitas et al., "Reinforcement and imitation learning for diverse visuomotor skills," arXiv preprint arXiv:/802.000 09564, 2018.
[102] A. Mandlekar, F. Ramos, B. Boots, S. Savarese, L. Fei-Fei, A. Garg, and D. Fox, "Iris: Im- plicit reinforcement without interaction at scale for learning control from offline robot manip- ulation data," in IEEE International Conference Robotics and Automation (ICRA). IEEE, 2020, pp. 4414-4420.
[103] L. Chen, R. Paleja, and M. Gombolay, "Learning from suboptimal demonstration via self- supervised reward regression," in Conference robot learning. PMLR, 2021, pp. 1262- 1277.
[104] D. Brown, W. Goo, P. Nagarajan, and S. Niekum, "Extrapolating beyond suboptimal demon- strations via inverse reinforcement learning from observations," in International conference on machine learning. PMLR, 2019, pp. 783-792.
[105] R. Jeong, J. T. Springenberg, J. Kay, D. Zheng, Y. Zhou, A. Galashov, N. Heess, and F. Nori, "Learning dexterous manipulation from suboptimal experts," arXiv preprint 2020.
arXiv:2010.085.0
[106] H. Xu, X. Zhan, H. Yin, and H. Qin, "Diseriminator-weigteee offline imitation learning from suboptimal demonstrations," in International Conference on Machine Learning. PMLR, 2022. pp. 24725-24744 742.
[107] M. Yang. S. Levine. and O. Nachum, "Trail: Near- dearmoptimal imitation learning with suboptimal data," arXiv preprint arXiv:2 14770, 2021.
[108) A. S. Morgan, B. Wen, J. Liang, A. Boularias, A. M. Dollar, and K. Bekris, "Vision-driven compliant manipulation for reliable, high-precision assembly tasks," RSS, 2021.
14

---

Appendix
A Overview
The Appendix contains the following content.
Author Contributions (Appendix B): list of each author's contributions the paper
FAQ (Appendix C): answers to some common questions
Limitations (Appendix D): more thorough list and discussion of MimicGen limitations
Full Related Work (Appendix E): more thorough discussion on related work
Robot Transfer (Appendix F): full set of results for generating data across robot arms
Object Transfer (Appendix G): full set of results for generating data across objects
Real Robot Results (Appendix H): additional details and discussion the real robot periments, including an explanation for the lower training results the real world
Different Demonstrators (Appendix I): results that show MimicGen works just as well when using source demos from suboptimal demonstrators and from different teleoperation devices
Motivation for MimicGen over Alternative Methods (Appendix J): motivation for Mim- icGen over offline data augmentation and replay-based imitation
Additional Details on Object-Centie Subtasks (Appendix K): more details and intuition
subtasks, including examples
Tasks and Task Variants (Appendix L): detailed descriptions all tasks and task variants
Derivation of Subtask Segment Transform (Appendix M): derivation of how MimicGen transforms subtask segments from the source data
Data Generation Details (Appendix N): in-depth details on how MimicGen generates data
Policy Training Details (Appendix O): details of how policies were trained from Mimic- Gen datasets via imitation learning
Data Generation Success Rates (Appendix P): data generation success rates for each of our generated datasets
Low-Dim Policy Training Results (Appendix Q): full results for agents trained on low- dim observation spaces (image agents presented in main text)
Bias and Artifacts in Generated Data (Appendix R): discussion some undesirable properties of MimicGen data
Using More Varied Source Demonstrations (Appendix S): investigation on whether hav- ing source demonstrations collected on more varied set of task initializations. is helpful
Data Generation with Multiple Seeds (Appendix T): results that show there is very little variance in empirical results across different data generation seeds
Tolerance to Pose Estimation Error (Appendix U): investigation of MimicGen's toler- ance to pose error
15

---

B Author Contributions
Ajay Mandlekar led the overall project, implemented the MimicGen code, ran most of the experi- ments in the paper, and wrote the paper.
Soroush Nasiriany implemented the Mobile Manipulation code, ran the Mobile Manipulation periments, and also helped with simulation environment development, including the mug assets. He also wrote parts of the paper.
Bowen Wen developed the perception pipeline used in real-world experiments and wrote parts of the paper.
Iretiayo Akinola designed the Factory [51] tasks and assets and developed infrastructure to ensure compatibility with human teleoperation, imitation learning, and MimicGen.
Yashraj Narang helped design the Factory [51] tasks and assets, and advised on the project.
Linxi Fan advised on the project.
Yuke Zhu advised on the project and wrote parts of the paper.
Dieter Fox advised on the project, and provided experiment ideas.
16

---

C
FAQ
1. How can reproduce experiment results?
We have released datasets, simulation environments, and instructions on reproducing the policy learning results at https: / /mimi cgen, .eeenggtthii tut...u..... io. We also hope that the availability of our datasets helps the community develop hetter policy learning methods.
2. What are some limitations of MimicGen?
See Appendix D for discussion.
3. Why are policy learning results worse in the real world than in simulation?
See Appendix H for discussion and an additional experiment.
4. Since data generation relies on open-loop replay of source human data, it seems like MimicGen only works for low-precision pick-and-place tasks.
We demonstrated that MimicGen can work for a large variety of manipulation tasks and be- haviors beyond standard pick-and-place tasks. This includes tasks with non-trivial contact- rich manipulation (Gear Assembly has Imm insertion tolerance, and Picture Frame As- sembly needs alignment of holes with 4mm tolerance each), long- oonghhorizon manipulation (up to 8 subtasks), and behaviors beyond pick-and-pll such as insertion, pushing, and ar- ticulation see Appendix I. for full details. The tasks also have pose variation well beyond typical prior works using BC from human demos
5. Is MimicGen robust to noisy object pose estimates during data generation?
In the real world, we use the initial RGBD image to estimate object poses (see Appendix H). Thus, MimicGen compatible with pose estimation methods and has some tolerance to pose error. We further investigated tolerance to pose estimate errors in simulation (see Ap- pendix U) and found that while data generation rates can decrease (so data collection will take longer), policies trained the generated data maintained the same level of perfor- mance.
6. Several recent works apply offline data augmentation to existing datasets create more data. What are the advantages of generating new data online like MimicGen does?
Offline data augmentation can be effective for generating larger dataset for robot manip- ulation 35-45:55] however, it be difficult generate plausible interactions without prior knowledge of physics [35] or causal dependencies [41,42], especially for new scenes, objects, or robots. In contrast, by generating new datasets through environment interaction, MimicGen data is guaranteed to be physically-consisten. Additionally, in contrast to many offline data augmentation methods, MimicGen is easy to implement and apply in practice, since only a small number of assumptions are needed (see Sec. 3). See more discussion in Appendix J.2.
7. What is the advantage of using replay-based imitation for data generation and then training policy with BC (like MimicGen does) over using it as the final agent? Replay-based imitation learning methods are promising for learning manipulation tasks us- ing a handful of demonstrations to MimicGen, which uses similar mechanisms during data generation, but trains end- to-end closed-loop agent from the generated data. First, replay-based agents generally conform to a specific policy architecture, while MimicGen datasets allow full compatibil- ity with a wide spectrum of offline policy learning algorithms [57]. Second, replay-based methods are typically open-loop, since they consist of replaying demonstration blindly, while agents trained on MimicGen datasets can have closed-loop, reactive behavior, since the agent can respond to changes in observations. Finally, as pendix P), in many cases, the data generation success rate (a proxy for the performance of replay-basch methods) can be significantly lower than the performance of trained agents. See more discussion in Appendix J.I.
but they have some limitations compared
in Sec. 6 (and Ap-
8. Why might a data generation attempt result in a f'ailure?
One reason is that the interpolation segments are unaware of the geometry in the scene and consist of naive linear interpolation (see Appendix N.2), so these segments might result in unintended collisions. Another is that the way source segments are transformed do not
17

---

consider arm kinematics, so the end effector poses where segments start might be difficult reach. A third reason that certain source dataset motions might be easier for the controller to track than others.
9. When can MimicGen be applied to generate data for new objects?
We demonstrated results geometrically similar rigid-body objects from the same cate- gory (e.g. mugs, carrots, pans) with similar scales. We also assumed aligned canonical coordinate frames for all objects in category. and that the objects are well-described by their poses (e.g. rigid bodies, not soft objects). Extending the system for soft objects or more geometrically diverse objects is left for future work.
10. Can MimicGen data contain undesirable characteristies?
See Appendix R for a discussion.
11. Give a breakdown of how MimicGen was used to generate 50K demos from 200 hu- man demos.
Here breakdown. It should be noted that this breakdown does not include our real robot demonstrations (200 demos generated from 20 source demos) or any extra datasets generated for additional experiments and analysis presented in the appendix.
175 source demos: 10 source demos for each of 16 simulated tasks in Fig. 4 (except Mobile Kitchen, which has 25)
36K generated demos: 1000 demos for each of the 36 task variants in Fig. 4
12K generated demos: robot transfer experiment (Appendix F) had tasks, each of which had 2 variants (Do, D1) and 3 new robot arms for 12 x 1000 demos.
2K generated demos: object transfer experiment (Appendix G) had 1000 demos for the O, (new mug) and O2 (12 mugs) variants.
18

---

D Limitations
In this section, we discuss limitations of MimicGen that can motivate and inform future work.
1. Known sequence of object-centric subtasks. MimicGen assumes knowledge of the object-centric subtasks in a task (which object is involved at each subtask) and also as- sumes that this sequence of subtasks does not change (Assumption 2, Sec. 3).
2. Known object poses at start of each subtask during data generation. During data gener- ation, at the start of each object-centric subtask, MimicGen requires an object pose estimate of the reference object for that subtask (Assumption 3, Sec 3). However, we demonstrated that we can run MimicGen in the real world, using pose estimation methods (Sec. 6,4 and Appendix H), and has some tolerance to errors in pose estimates (Appendix U). Another av- enue for real world deployment is to generate data and train policies in simulation (where object poses readily available) and then deploy simulation-trainn agents in the real world [58-62]- - this is left for future work.
3. One reference object per subtask. MimicGen assumes each task is composed of a se- quence of subtasks that are each relative to exactly one object (Assumption 2, Sec. 3). Being able support subtasks where the motion depends on more than one object (for example, placing an object relative to two objects, or on a cluttered shelf) is left for future work.
4. Naive filtering for generated data. MimicGen has naive way to filter data generation attempts (just task success rates). However, this does not prevent the generated datasets from being biased, or having artifacts (see discussion in Appendix R). Developing better filtering mechanisms is left for future work.
5. Naive interpolation scheme and no guarantee on collision-free motion. MimicGen uses naive linear interpolation scheme to connect transformed human segments together (Ap- pendix N.2). However, this method is not aware of scene geometry. and consequently can result in unintended collisions if objects happen to be in the way of the straight line path. We opted for this simple approach to avoid the complexity of integrating a planner and ensuring it uses the same action space (Operational Space Control [63]). We also saw that longer interpolation segments could be harmful to policy learning from generated data (Ap- pendix H). Similarly, ensuring that motion plans are not harmful to policy learning could be non-trivial. Developing better-quality interpolation segments (e.g. potentially with motion planning) that are both amenable downstream policy learning and safer for real-world operation is left for future work.
6. Object transfer limitations. While MimicGen can generate data for manipulating dif- ferent objects (Appendix G), we only demonstrated results geometrically similar rigid- body objects from the same category (e.g. mugs, carrots, pans) with similar scales. We also assumed aligned canonical coordinate frames for all objects in category, and that the objects are well-deseribed by their poses (e.g. rigid bodies, not soft objects). Extending the system for soft objects more geometrically diverse objects is left for future work.
7. Task limitations. MimicGen was demonstrated on quasi-static tasks it is unlikely to work on dynamic, non quasi-static tasks in its current form. However, a large number of robot learning works and benchmarks use quasi-static tasks [1,3-7, 14, 18, 19,22,30,33.5 55, 64-66), making the system broadly applicable. We also did not apply MimicGen to tasks where objects had different dynamics from the source demonstrations (e.g. new friction values). However, there is potential for MimicGen to work, depending on the task. Recall that each data generation attempt, MimicGen tracks a target end effector pose path (Sec. 4.2) - this allows data generation for robot arms with different dynamics (Appendix F), and could potentially allow it to work for different object dynamics (e.g. potentially pushing a cube across different table frictions).
8. Mobile manipulation limitations. In Sec. 6.1, we presented results for MimicGen on the Mobile Kitchen task, which requires mobile manipulation (base and arm motion). Our current implementation has some limitations. First, it assumes that the robot does not move the mobile base and arm simultaneously. Second, we simply copy the mobile base actions from the reference segment rather than transforming it like we do for end effector actions. We found this simple approach sufficient for the Mobile Kitchen task (more details
19

---

in Appendix N.5). Future work could integrate more sophisticated logic for generating base motion (e-g. defining and using a reference frame for each base motion segment, like the object-centric subtasks used for arm actions, and/or integrating a motion planner for the base).
9. No support for multi-arm tasks. MimicGen only works for single tasks extending it to generate datasets for multi-manual manipulation [25] is left for future work.
20

---

E Full Related Work
This section presents a more thorough discussion of related work than the summary presented in the main text.
Data Collection for Robot Learning. There have been several data collection efforts to try and address the need for large-scale data in robotics. Some efforts have focused on self-supervised data collection where robots gather data on tasks such as grasping through trial-and-error [12-17]. RoboTTur [2, 23-26] is a system for crowdsourcing task demonstrations from human operators us- teleoperation and video streams provided in web browsers. Several related efforts [3-6,2] 27] also collect large datasets (e.g. 1000s of demonstrations) by using large number of human operators over extended periods of time. In contrast, MimicGen tries to make effective use of a small number of human demonstrations (e.g. 10) to generate large datasets. Some works have collected large datasets using pre- pregrgrrrmmm demonstrators in simulation [18-22]; however, it can be difficult to scale these approaches up to more complex tasks, while we show that Mimic- Gen can be applied to a broad range of tasks. Prior work has also attempted to develop systems that can selectively query humans for demonstrations when they are needed, in order to reduce human operator time and burden [67-70]. In contrast, MimicGen only needs an operator to collect a few minutes of demonstrations at the start of the process. Generating large synthetic datasets has been a problem of great interest other domains as well [71-77], and has also been used as a tool for benchmarking motion planning [78].
ing smartphone-base
Imitation Learning for Robot Manipulation. Imitation Learning (IL) seeks train policies from set of demonstrations. Behavioral Cloning (BC) [28] is standard method for learning policies offline, by training the policy to mimic the actions in the demonstrations. It has been used extensively in prior work for robot manipulation [1, 29-3499-4- in this work, we use BC to train single-task policies from datasets generated by MimicGen. However, MimicGen can also be used to generate datasets for a wide range of existing offline learning a algorithms that learn from diverse multi-task 55, 79-83]. Some works have used offline data augmentation to increase the dataset size for learning policies [7,35 45] in this work we collect new datasets.
datasets 55,
Replay-Based Imitation Learning. While BC simple and effective, it typically requires several demonstrations to learn task [7]. To alleviate this, many recent imitation learning methods try to leam policies from only handful of demonstrations by replaying demonstrations in new scenes [8- 11,46-48) Some methods [9-11] use trained networks that help the robot end effector approach poses from which demonstration can be replayed successfully. In particular, Di Palo al. [11] proposes an approach to replay parts of a single demonstration to solve multi-stage tasks - this is similar to the way MimicGen generates new datasets. However they make a number assumptions that we do not (4D position and yaw action space vs. our 6-DoF action space, single wrist camera view to enable spatial generalization). Furthermore, this work and others use demonstration replay as component of the final trained agent- - in contrast, use Consequently, these prior approaches are complementary to our data generation system, and in principle, could be used as part of alternative schemes for data generation. In this work, we focus on the general framework of using such demonstration replay mechanisms to generate data that can be seamlessly integrated into existing imitation learning pipelines, and opt for approach that emphasizes simplicity (more discussion in Appendix J). Our experiments also show that there can be a large benefit from collecting large datasets and training agents from them, instead of directly deploying a replay-based agent.
data generation mechanism.
21

---

Robot Transfer
In Sec. 6, we summarized results that show MimicGen can generate data for diverse robot hardware. Recall that we took the source datasets from the Square and Threading tasks (which use the Panda arm) and generated datasets for the Sawyer, IIWA, and URSe robots across the Dp and D₃ reset distribution variants (see Fig. F.1). Here, we present the complete set of results.
have large spread across robots (range 20%-74% for Da. see Table F.I), the policy success rates are significantly higher and remarkably similar across robots (for example, 80%-911 %% Square Do and 89%-98% Threading Do based agent results in Table F.2 and low-dim agent results in Table F.3). This shows the potential for using human demonstrations across robot hardware using MimicGen, an exciting prospect, as telcoperated demonstrations are typically constrained to a single robot.
Notice that although the data generation
the full image-
Panda
Sawyer
IIWA
UR5e
Figure F.1: Robots used in Robot Transfer Experiment. The figure shows the rohot used for data
generation. Source datasets were collected on the Panda arm (blue horder) and used to generate data for Sawyer, IIAA and URSe arms (orange border).
Task Variant
Panda
Sawyer
IIWA
URSe
Square (Dg)
73.7
55.8
37.7
64.7
Square (Di)
48.9
38.8
34.1
Threading (Dp)
51.0
28.8
20.4
21.4
Threading (D2)
39.2
23.7
11.5
18.5
Table F.1: Data Generation Rates on Different Robot Hardware. The success rates of data generation are diffcrent across different robot arms (yet agents trained on these datascts achieve similar task success rates).
Task Variant
Panda
Sawyer
IIWA
URSe
Square (D₂)
=
86.0
80.0
84.7 0.9
Square (D,)
73.3 + 3.4
60.7 2.5
48.0 1..0+:3.. 3.3
56.0 + 4.3
Threading (Do) 98.0 :: 1.6
88.7 7.5
94.0 3.3
91.3 -:: 0.9
Threading (D,)
60.7 1 2.5
50.7 1 3.8
49.3
1.1.1 4.1
60.7 2.5
Table F.2: Agent Performance on Different Robot Hardware. We use MimicGen to produce datasets across
different robot arms using the same set of 10 source demos (collected on the Panda arm) and train image-hased agents on each dataset seeds). The success rates comparable across the different robot arms, indicating that MimicGen can generate high-quality data across robot hardware.
Task Variant
Panda
Sawyer
ПIWA
URSe
Square (Do)
98.0 + 1.6
87.3 1.9
79.3
82,0 4: 1.6 67.3 3.4
Square (Di)
80.7 =
69.3 2.5
1.9
Threading (Do)
97.3 + 0.9
96.7
93.3 033++0.. 0.9
96.0 1.6
Threading (D.)
72.0
67.3 4.7
80.0
Table F3: Low-Dim Agent Performance
Different Robot Hardware. We use MimicGen to produce
datasets across different robot arms using the same of 10 source demos (collected the Panda arm) and train agents on each dataset seeds). The success rates are comparable across different robot arms, indicating that MimicGen can generate high-quality data across robot hardware.
22

---

G
Object Transfer
In Sec. 6, we summarized results that show MimicGen can generate data for different objects. Recall that we took the source dataset from the Mug Cleanup task and generated data with MimicGen for an unseen mug (O1) and for set of 12 mugs (O2). Here, we present the complete set of results (Table G.1) and also visualize the mugs used for this experiment (Fig. G.1).
The Mobile Kitchen task that generated data for also had different object variants show the 3 pans and 3 carrots in Fig. G.2. Results for this task are in Fig. 4 (image-based agents) and in Table Q.I (low-dim agents).
While these results are promising, we only demonstrated results on geometrically similar rigid- objects from the same category (e.g. mugs, carrots, pans) with similar scales. We also assumed aligned canonical coordinate frames for all objects in a category, and that the objects are well- described by their poses (e.g. rigid bodies, not soft objects). Extending the system for soft objects or more geometrically diverse objects is left for future work.
Task
Do
O1
02
Mug Cleanup (DGR)
29.5
31.0
24.5
Mug Cleanup (SR, image)
80.0
4.9
90.7
+
75.3 5.2
Mug Cleanup (SR, low-dim)
82.0
2.8
88.7 1 4.1
66.7 2.5
Table G.I: Object Transfer Results. We present data generation rates (DGR) and success rates (SR) of trained agents on the O, and O₂ variants of the Mug Cleanup task, which have unseen mug, and set of 12 mugs new mug per episode) respectively.
Figure G.I: Objects used Object Transfer Experiment. The figure shows the mug used in the Mug Cleanup Do task (blue border), the unseen one the O₃ task (orange border), and the complete set mugs in the O2 task.
Figure G.2: Objects used in Mobile Kitchen task. The figure shows the pans and carrots used in the Mobile Kitchen task. each episode random pan and carrot are selected and initialized in the scene.
23

---

H
Real Robot Results
Interpolation Comparison
100
80
Succees Rabe
Num Steps
50
0
Stock (D1)
Square (D1)
TPX (D1)
Coffee (D1)
Threading (D1) PickPlae (DO)
Task
Figure H.1: Effect of Increasing Interpolation Steps. Comparing the effort interpolation steps on trained amage-based agents. Using an increased amount of interpolation can cause agent performance to decrease significantly. This could explain the gap between real-world and simulation agent performance.
In this section, we first provide further details how we applied MimicGen to the real world tasks in Fig. 5, then we provide additional experiment results that help to explain the gap in trained policy performance between simulation and real.
Real Robot Data Collection Details. Recall that during data generation, MimicGen requires pose estimates at the start of each object-centric subtask (Assumption Sec. 3). To do this, use front-view Intel RealSense D415 camera which has been calibrated (e.g. known extrinsics). We first convert the RGBD image point cloud and remove the table plane via RANSAC [84]. We then apply DBSCAN [85] clustering to identify object segments of interest, though alternative seg- mentation methods such [86, 87] are also applicable. In the Stack task, the cube instances distinguished by their color. In the Coffee task, the coffee machine and the pod are distinguished based on the segment dimensions. Finally for each identified object segment, we leverage [88] for global pose initialization, followed by ICP [89] refinement. Note that while the current pose esti- mation pipeline works reasonably well, our framework is not specific to certain types of perception methods. Recent [90-94] and future advance in state estimation could be used to apply MimicGen in real-workes settings with less assumptions about the specific objects.
Gap in Policy Performance between Sim and Real. While we saw a significantly high data col- lection success rate 8 for Stack, 52. % for Coffee), we saw much lower policy success rate on these tasks than in simulation (36% vs. 100% for Stack, and 14% vs. ~90% for Coffee), as described in Sec. 6). While there considerably less data in the real world due to the time-consuming nature of real-world data collection (100 demos instead of 1000 demos), there were also other factors that could explain this gap.
As a safety consideration, our real-world tasks used much larger interpolation segments Tiinterp 25, Rixeed 25 instead of the simulation default (Mreterp 5, Mlsxed 0) (see Appendix N.2 and Appendix N.6). We hypothesized that the increased duration of the interpolation segments made them difficult imitate, since there little association between the motion and what the agent sees in the observations (the motions are slow, and do not generally move towards regions of interest). To further investigate this, we ran an experiment in simulation where used the same settings for interpolation for a subset of our tasks. The results are presented in Fig. H.1
We that for certain tasks, the larger interpolation segments cause agent performance to decrease significantly - for example image-based agents on Stack D decrease from 99.3% success to 68. 68.7% success, and image based agents on Pick Place decrease from 50.7% to 11.3%. These results confirm that the larger segments (together with the smaller dataset size) may have been responsible for lower real world performance. Developing better-quality interpolation segments that are both safe for real-world operation and amenable to downstream policy learning is left for future work.
Combining MimicGen with sim-to-real policy deployment methods (58-62,95-98] is another ex- citing avenue for future work simulation does not suffer from the same bottlenecks as real-worid data collection (slow and time-consuming, requiring multiple arms and human supervisors to reset

---

the task), making simulation an ideal setting for MimicGen to generate large-scale diverse datasets. Recent sim2real efforts have been very promising several works [61. 95-98] have been able to transfer policies trained via imitation learning from sim to real. Furthermore, MimicGien is entirely complementary domain randomization techniques [99], which could also be applied to assist in transferring policies to the real world.
Improved Performance with More Flexible Policy Models. One promising avenue improve real-world learning results is to develop and/or apply imitation learning algorithms that can better deal with multimodal and heterogeneous trajectories. We trained Diffusion Policy [100], recent state-of-the ofrttr-art imitation learning model, on our real-world Stack dataset. The new agent achieved success rate of 76% across 50 evaluations a significant improvement over the 36% success rate achieved by BC-RNN, This result provides an optimistic outlook on producing capable agents from real-world MimicGen data.
25

---

I Different Demonstrators
Task
Do
D1
D2
Stack Three (Op. A, image)
92.7 1:
86.7 3.4
Stack Three (Op. image)
86.0 +
69.3 5.0
Threading (Op. A, image)
98.0 1.6
00.7
38.0 3.3
Threading (Op. B, image)
98.0 1.6
58.0 + 4.3
38.0 8.6
Three Pc. Assembly (Op. A, image)
82.0 1.6
62.7 + 2.5
13.3 + 3.8
Three Pc. Assembly (Op. B, image)
76.0 = 1.6
54.7 # 6.8
5.3 = 1.9
Stack Three (Op. A, low-dim)
88.0 + 1.6
90.7 0.9
Stack Three (Op. B, low-dim)
82.7 + 0.9
84.0 + 3.3
Threading (Op. low-dim)
97.3
72.0 1: 1.6
60.7 1 6.2
Threading (Op. B, low-dim)
9..3
76.0 + 4.3
70.0 1.6
Three Pc. Assembly (Op. A, low-dim)
74.7
61.3 + 1.9
38.7
Three Pc. Assembly (Op. B, low-dim)
77.3
65.3
46.0
Table L1: MimicGen with Different Demonstrators. We show that policies trained on MimicGen data can achieve similar performance even when the source demonstrations come from different demonstrators. Oper- ator used different teleoperation device than Operator A, policy training results on generated datasets are comparable for both maage-based and low-dim agents.
Task
Do
D1
D2
Square (Better, image)
90.7 1.9
73.3 3,4
49.3 + 2.5
Square (Okay, image)
90.0 = 1.6
50.0 7.1
Square (Worse, image)
90.7 0.9
2.5
45.3 4.1
Square (Better, low-dim)
98.0
8..7
1.9
Square (Okay, low-dim)
95.3 0.9
8..0
60.7 + 1.9
Square (Worse, low-dim)
95.3 0.9
76.7 5.0
52.7 + 1.9
Table 1.2: MimieGen with Lower Quality Demonstrators. We show that policies trained on MimicGen data can achieve similar performance even when the source demonstrations come from lower quality demonstrators. We compare across source datasets from the "Better", "Okay". and "Worse" subsets of the robomimic Square- MH dataset [7], which was collected by operators of different proficiency. Policy training results on generated datasets are comparable for both image-based and low-dim agents.
While most of our experiments datasets from one particular operator, show that Mimic- Gen can easily use demonstrations from different operators of mixed quality. We first collected 10 source demonstrations from a different operator on the Stack Three, Threading, and Three Piece Assembly tasks - this operator also used a different teleoperation device (3D mouse [49, 101J). We also used 10 demonstrations from one of the "Okay" operators and of the "Worse" opera- tors in the robomimic Square-MH dataset [7] to see if MimicGen could use lower-quality datasets. These source datasets were then provided MimicGen to generate 1000 demonstrations for all task variants, and subsequently train policies - the results are summarized in Table I.I (different demonstrator with different teleoperation device) and Table 1.2 (lower quality demonstrators).
Interestingly, the operator using a different teleoperation interface produced policies that were ex- tremely similar in performance to original results (deviations of 0% to 17%). Furthermore, the policies produced from the datasets generated with the "Worse" and "Okay" operator data are also extremely similar in performance (deviations of 0% to 14%). This is quite surprising. as the robomimic study [7] found that there can be significant difficulty in learning from datasets produced by less experienced operators. Our results suggest that in the large data regime, the harmful ef- fects of low-quality data might be mitigated. This is an interesting finding that can inform future work into learning from suboptimal human demonstrations [102-107].
26

---

J Motivation for MimicGen over Alternative Methods
In this section, we expand on the motivation for using data generation with MimicGen over two alternatives reepla--basaee based imitation learning and offline data augmentation.
J.1 Replay-Based Imitation Learning
Several recent works learn policies using only a handful of demonstrations by replaying the demon- strations in new scenes 18-11, 46-4).. While these methods are promising, there are some limita- tions. One limitation is that their learned policy usually uses demonstration replay as part of their agent. This means that the policy is often composed of hybrid stages (such as self-supervised net- work that learns to move the arm to configurations from which replay will be successful and a replay stage). By contrast, MimicGen uses a similar mechanism to generate datasets this allows full compatibility with a wide spectrum of offline policy learning algorithms [57]. These datasets also allow for evaluating different design decisions (such as different observation spaces and learning methods), including the potential for multi-task benchmarks consisting of high-quality human data. Furthermore, by easily allowing datasets to be created and curated, MimicGen can facilitate future work to investigate how dataset composition can influence learned policy proficiency.
Another limitation is that replay-based imitation methods are typically open-loop, since they consist of replaying demonstration blindly (the trajectory executed by the robot cannot adapt to small errors). By contrast, agents trained on MimicGen datasets can have closed-loop, reactive behavior, since the agent can respond to changes in observations.
Finally, as we saw in Sec. (and Appendix P), in many cases, the data generation success rate (a proxy for the performance of replay-bused methods) can be significantly lower than the performance of trained agents (one reason for this might be because of only training the policy on the successful data generation attempts, and another might be due agent generalization).
J.2 Offline Data Augmentation
Several works have used offline data augmentation to increase the dataset size for learning poli- cies [7,35-45]. Since this process is offline. it can greatly increase the size of the dataset. In fact, this can be complementary to MimicGen- we leverage pixel shift randomization [7, [7,36-39] when training image-based agents on MimicGen data.
However, because data augmentation offline, can be difficult to generate plausible interactions without prior knowledge of physics [35] causal dependencies [41,42] especially for new scenes, objects, or robots. Instead, MimicGen opts for generating new datasets through environment in- teraction by re-purposing existing human demonstrations consistent data, since generation online, In contrast to many offline data augmentation methods, MimicGen is easy implement and apply in practice, since only small number of assumptions are needed (see Sec. 3).
this automatically leads to physically-
Similar to MimicGen, some recent works [43-45] have also shown an ability to create datasets with new objects, but these works typically change distractor objects that are not involved in manipu- lation this leads to encouraging behavioral invariances (e.g. tell the policy to apply the same actions, even if the background and irrelevant objects are changed). By contrast, MimicGen gener- ates datasets with new objects that are a critical part of the manipulation task it seeks to generate data by adapting behavior to new contexts.
27

---

K Additional Details on Object-Centric Subtasks
Mug Grasp (reference: mug)
Mug Place (reference: machine)
Pod Grasp (reference: pod)
Pod Insert (reference: machine)
Start
Figure K.I: Illustrative Example of Object-Centric Subtasks. In this example, the robot must prepare a cup of coffee by placing the mug the machine, and the coffee pod into the machine. This task is easily broken down into sequence of object-centric subtasks this figure shows the end of each subtask, and the relevant object for each subtask. There is mug grasping suhtask (motion relative to mug), mug placement subtask (motion relative machine), a pod grasping subtask (motion relative to pod), (motion relative to machine). The robot can solve this task by sequencing motions relative to each object frame (one per subtask).
pod insertion subtask
Object-centric subtasks (Assumption 2 in Sec. 3) are key part of how MimicGen generates new demonstrations. In this section, we provide more details on how they are defined, and how sub- task segments are parsed from the source demonstrations. We also show some examples to build intuition.
K.1 How Tasks can be broken up into Object-Centric Subtasks
assume that tasks consist of known sequence of object-centric subtasks. Let O = foi,... Ok) } be the set of objects in a task M. As in Di Palo al. [11], assume that tasks consist of sequence of object-centric subtasks (Si (os, S2(s,)) Sas (05m)), where the manipulation in each subtask S;(os,) is relative to single object's (os, O) coordinate frame. We assume this sequence is known.
We first restate Assumption
Specifying the sequence of object-centric subtasks is generally easy and intuitive for a human to do. As first example, consider the coffee preparation task shown in Fig. K.1 (and Fig. 2). A robot must prepare cup of coffee by grasping a mug, placing it on the coffee machine, grasping coffee pod, inserting the pod into the machine, and closing the machine lid. This task can be broken down into a sequence of object-centric subtasks: mug- grasping subtask (motion is relative to mug). a mug- placement subtask (motion relative to machine), pod-grasping subtask (motion relative to pod), and a final pod-insertion and lid-closing subtask (motion relative to machine). Consequently, the robot solve this task by sequencing several object-centric motions together. This is the key idea behind how MimicGen data generation works - it takes a set of source human demos, breaks them up into segments (where each segment solves subtask), and then applies each subtask segment in a new scene. The subtasks are visualized in Fig. K.I.
We also emphasize that wide variety of tasks can be broken down into object-centric subtasks (e.g. Assumption 2 applies to a wide variety of tasks, especially those that are commonly considered in the robot learning community). Fig. K.2 illustrates subtasks for some of our tasks (more discussion in Appendix K.3 below).
K.2 Parsing the Source Dataset into Object- Centric Subtask Segments
We now provide more details on the parsing procedure described in Sec, 4.1. Recall that would like to parse every trajectory J in the source dataset into segments { ).M. where each segment т₃ corresponds to a subtask (,s,,) ). We assume access metrics that allow the end of each subtask to be detected automatically. In our running example from Fig. 2, this would correspond to metrics that use the state of the robot and objects to detect when the mug grasp, mug placement, pod grasp, and machine lid close occurs. This information is usually readily available in simulation, as it often required for checking task success. With these metries, we can easily run through the set of demonstrations, detect the end of each subtask sequentially, and use those as the subtask
28

---

Square
Threading
Start
Siaap tnd:: netdl))
Thread tul trpod)
Gear Assembly
Start
Urapp (nt gear)
Inar Chank traft baset
Stack Three
red cuba)
tiue
trati red outet
Three Piece Assembly
GraRD
Snepp
Figure K.2: Object-Centric Subtasks for Selected Tasks This figure shows the end of cach object-centric subtask (and the reference object) for subset of the tasks in the main text. MimicGen assumes that this subtask structure is known for each task; however, specifying this subtask structure is generally easy and intuitive for hurman.
boundaries, to end up with every trajectory E Dac split into a contiguous sequence of segments (T1, F2, TMr). one per subtask.
However, another alternative that requires no privileged information (and hence suitable for real world settings) is to have a human manually annotate the end of each subtask. As the number of source demonstrations usually small, this easy for a human operator do, either while collecting each demonstration or annotating them afterwards. In this work, we opted for the former method (automated subtask end metries) because they were readily available for our tasks or easy to craft.
K.3 Specific Examples
We provide some examples this section of how some tasks are broken up into object-centric subtasks. The examples are provided in Fig. K.2. For each task below, outline the object- centric subtasks, and the subtask end detection metrics used for parsing the source human demos into segments that correspond to each subtask. Note that these metries are only used for parsing the source human demos and are not assumed to be available during policy execution.
Square. There are 2 subtasks grasping the nut (motion relative to nut) and inserting the nut onto the peg (motion relative to peg). To detect the end of the grasp subtask, we check for contact between the robot fingers and the nut. For the insertion subtask, we just the task success check.
Threading. There are subtasks - grasping the needle (motion relative to needle) and threading the needle into the tripod (motion relative to tripod). To detect the end of the grasp subtask, we check for contact between the robot fingers and the needle. For the threading subtask, we just use the task success check.
29

---

Gear Assembly. There are 2 subtasks - grasping the gear (motion relative to gear) and inserting the gear into the base and turning the crank (motion relative base). To detect the end of the grasp subtask, we check if the gear has been lifted by a threshold. For the insertion subtask, we just use the task success check.
There are 4 subtasks - grasping the red block (motion relative to red block), placing the red block onto the green block (motion relative green block), grasping the blue block (motion relative to blue block), and placing the blue block onto the red block (motion relative to red block). To detect the end of each grasp subtask we check for contact between the robot fingers and the relevant block. For each place subtask, we check that the relevant block has been lifted and is in contact with the block that should be underneath it.
Stack Three
Three Piece Assembly. There are subtasks - grasping the first piece (motion relative to first piece), inserting the first piece into the base (motion relative to base), grasping the second piece (motion relative to second piece), and inserting the second piece onto the first piece (motion relative first piece). To detect the end of each grasp subtask, check for contact between the robot fingers and the relevant piece. For each insertion subtask, we re-use the insertion check from the task success check.
30

---

L Tasks and Task Variants
(+)
(c) Square
(e) Threading
(n) Three Pc. Assembly
(g) Hammer Cleanup
(hi Mug Cleanup
(i) Pick Place
Nut Assembly
(k) Kitchen
(l) Coffee Prepuration
Mobile Kitchen
Nut-Bolt Assembly
Gear Assembly
(p) Frame Assembly
Figure I..: Tasks (all). We show all of the simulation tasks in the figure above. They span wide variety of hehaviors including pick-and-place, precise insertion articulation, and mobile manipulation, and include long-horizon tasks requiring chaining several hchaviors together.
In this section, provide more detailed descriptions of each of tasks and task variants. The tasks (Fig. L.1) and task variants (especially their reset distributions) are best appreciated on the in Sec.
website (https: : /////mm //mimii egen. gi tuub. io). We group the tasks into categories and describe the goal, the variants, and the object-centric subtasks in each task. As mentioned in Sec. 3 and Appendix. N.1, the tasks have delta-pose action space (implemented with an Operational Space Controller [63]). Control happens at 20 hz.
Basic, A basic set of box stacking tasks.
Stack [49] Stack x 0.16)) region (Do) and large (0.4m x 0.4m) region (D1) with random top-down rotation. There are subtasks (grasp red block, place onto green). We also develop version of this task in the real-world (Fig. 5), where the Do region is a 0.21m x 0.30m box and D region is a 0.44m x 0.85m box.
block on green one. Blocks are initialized in a small (0. 16m16m
Stack Three Same Stack, but additionally stack blue block the red one. Blocks initialized in a small (0.20m x 0.20m) region (Do) and large (0.4m x 0.4m) region (Dj) with random top-down rotation. There are 4 subtasks (grasp red block, place onto green, grasp blue block, place onto red).
Contact-Rich, A set of tasks that involve contact-rich behaviors such insertion drawer articu- lation. In each Do variant, at least one object never moves.
Square [7]. Pick square nut and place peg. (Dn) Peg moves, nut is placed in small (0.005m x 0. 15m) region with random top-down rotation. (D₂) Peg and nut move in large regions. but peg rotation fixed. Peg is initialized in 0.4m x 0.4m box and nut is initialized in 0.23m x 0.5 Im box. (D2) Peg and nut move in larger regions (0.5m x 0.5m box of initialization for both) and peg rotation also varies. There are subtasks (grasp nut, place onto peg).
Threading [24]. Pick needle and thread through a hole on a tripod. (Do) Tripod fixed, needle moves in modest region (0.15m 0. 1m box with 60 degrees of top-down rotation variation). (D₂) Tripod and needle move large regions on the left and right portions of the table respectively. The needle is initialized in 0.25m 0. box with 240 degrees
31

---

of top-down rotation variation and the tripod is initialized in a 0.25m x 0.1m box with 120 degrees of top-down rotation variation. (D2) Tripod and needle are initialized on the right and left respectively (reversed from D1). The size of the regions is the same as Dj. There are subtasks (grasp needle, thread into tripod).
Coffee [24]. Pick a coffee pod, insert into coffee machine, and close the machine hinge. (D₂) Machine never moves, pod moves in small (0.06m x 0.06m) box. (D2) Machine and pod move in large regions on the left and right portions of the table respectively. The machine is initialized in a O. lm x 0.1m box with 90 degrees of top-down rotation variation and the pod initialized in a 0.25m x 0.13m box. (D2) Machine and pod are initialized on the right and left respectively (reversed from D1). The size of the regions is the same as Dj. We also develop a version of this task in the real-world (Fig. 5) - in Do. the pod is initialized in a 0.05m vertical strip and in Di, the pod is initialized in a 0.44m 0.35m box. There are subtasks (grasp pod, insert-into and close machine).
Three Piece Assembly, Pick one piece, insert into the base, then pick the second piece, and insert into the first piece to assemble a structure. (Dn) base never moves, both pieces move around base with fixed rotation a 0.44m 0.44m region. (D1) All three pieces move in workspace (0,44m x 0.44m region) with fixed rotation. (D2) All three pieces can rotate (the base has 90 degrees of top-down rotation variation, and the two pieces have 180 degrees of top-down rotation variation). There are 4 subtasks (grasp piece 1, place into base, grasp piece 2, place into piece 2).
Hammer Cleanup [53]. Open drawer, pick hammer, and place into drawer, and close drawer. (Do) Drawer is fixed, and hammer initialized in a small 0.08m x 0.07m box with 11 degrees of top-down rotation variation. (D1) Drawer and hammer both move in large regions. The drawer is initialized in 0.2m 0.1m box with 60 degrees of top-down rotation variation and the hammer is initialized in a 0.4m x 0.12m box with a random top- down rotation. There are 3 subtasks (open drawer, grasp hammer, place into drawer and close).
Mug Cleanup. Similar to Hammer Cleanup but with a mug and with additional variants. (Do) The drawer does not move and the mug moves in a 0.3m 0.15m box with a random top-down rotation. (D1) The mug moves in a 0.2m x 0.1m box with 60 degrees of top- down rotation variation and the mug is initialized in 0,4m 0.15m box with a random top-down rotation. (O₂) A different mug is used. (O2) On each task reset, one of 12 mugs is sampled. There are 3 subtasks as in Hammer Cleanup.
Long-Horizzo. A set of tasks that require chaining multiple behaviors together.
Kitchen [53]. Switch stove on, place pot onto stove, place bread into pot, place pot in front of serving region and push it there, and turn off the stove. (Do) The bread is initialized in a 0.03m x 0.06m region with fixed rotation and the pot is initialized in a 0.005m x 0.02m region with 11 degrees of top-down rotation variation. The other items do not move. (Dı) Bread, pot, stove, button, and serving region all move wider regions. Bread: 0.2m x 0.2m box with 180 degree top-down rotation variation, pot: 0.1m x 0.15m box with 60 degrees top-down rotation variation, stove: 0.17m x 0.1505m box with fixed rotation, button: 0.26m x 0.15m box with fixed rotation, serving region: 0.15m horizontal strip. There are 7 subtasks (turn stove on, grasp pot, place pot on stove, grasp bread, place bread in pot, serve pot onto serving region, and turn stove off).
Nut Assembly [49]. Similar to Square, but place both a square nut and round nut onto two different pegs. (Dp) Each nut is initialized in a small box (0.005m x 0. 0.115m region with a random top-down rotation). There are 4 subtasks (grasp each nut and place onto each peg).
Pick Place [49]. Place four objects into four different bins. (Dn) Objects are initialized anywhere within the large box (0.29m x 0.39m). We use a slightly simpler version of this task where the objects are initialized with top-down rotations between 0 and 90 degrees (instead of any top-down rotation). There are subtasks (grasp each obejct and place into each bin).
Coffee Preparation. A full version of Coffee load mug onto machine, open machine, retrieve coffee pod from drawer and insert into machine. (Tx) The mug moves in modest (0. x O. 15m)) region with fixed top-down rotation and the pod inside the drawer moves
32

---

in 0,06m x 0.08m region while the machine and drawer are fixed. (Ti) The mug is initialized in a larger region (0.35m x 0.2m box with uniform top-down rotation) and the machine also moves in modest region (0.1m x 0.05m box with 60 degrees of top-down rotation variation). There are subtasks (grasp mug, place onto machine and open lid, open drawer, grasp pod, insert into machine and close lid).
Mobile Manipulation. Tasks involving mobile manipulation.
Mobile Kitchen. Set up frying pan, by retrieving pan from counter and placing stove, followed by retrieving a carrot from sink and placing onto pan. (Dp) The pan starts in 0.2m 0.4m region in the center of the countertop (with 120 degrees of top-down rotation variation) and the carrot starts in a 0. x 0.1m region inside the sink (with 60 degrees of rotation variation). There three possible pans and three possible carrots sampled randomly for each episode. There are 4 subtasks (grasp gap, place pan, grasp carrot, place carrot). The latter three stages involve operating the mobile base.
Factory. A set of high-precision tasks in Factory [51].
Nut-and-Bon Assembly. Pick nut and align onto bolt. (Da) Nut and bolt are initialized in modest regions of size 0.2m x 0.2m with no rotation variation. (D1) Nut and bolt initialized anywhere in workspace (0.35m x 0.8m box) with fixed rotation. (D2) Nut and bolt can rotate (180 degrees of top-down rotation variation). There are 2 subtasks (pick nut and place onto bolt)
Gear Assembly. Pick gear, insert it onto a shaft containing other gears, and turn the gear crank to move the other gears. (Do) Base is fixed, and gear moves in modest region (0.1m 0.1m with no rotation variation). (D1) Base and gear move larger regions (of size 0.3m x 0.3m) with fixed rotation. (D2) Both move with rotations (180 degrees of top- down variation for the gear and degrees of top-down variation for the base). There are subtasks (grasp gear, insert into base and crank).
Frame Assembly. Pick picture frame border with 4 holes and insert onto a base with bolts rigidly attached. (D₂) Frame border and base move in small regions of size 0. 0.1m with fixed rotation. (D₃) Frame border and base move in much larger regions of size 0.3m x 0.3m with fixed rotation.
(D2) Both move with rotations (60 degrees of top-down variation for both). There are 2 subtasks (grasp frame border and insert into base).
33

---

M Derivation of Subtask Segment Transform
In this section, we provide complete derivation of the source subtask segment transformation presented in Sec. 4.2. Recall that T denotes homogenous 4x4 matrix that represents the pose of frame with respect frame B. We have chosen source subtask segment consisting of target poses for the end effector controller (Assumption Sec. 3) T3 (I TW .... TW ) where C is the controller target pose frame at timestep I, W is the world frame, and K' is the length of the segment.
We would like to transform т; according to the new pose of the corresponding object in the current scene (frame O with pose 7 so that the relative poses between the target pose frame and the object frame preserved at each timestep oogg T We can write T (Tw)) w and -
TC (IW))- Setting them equal, have
( '1 (TWO)-
Rearranging for Tw by left-multiplying by T Th we obtain
W = Tw 00 ) -TT
which is the equation we use transform the source segment.
34

---

Data Generation Details
In this section, we provide additional details on how MimicGen generates data. We first pro- vide additional details about components of MimicGen that were not discussed in the main text. This includes further discussion on how MimicGen converts between delta-pose -pse actions and con- troller target poses (Appendix N.I), more details on how interpolation segments are generated (Ap- pendix N.2), an overview of different ways the reference segment can be selected (Appendíx N.3), details on how transformed trajectories are executed with action noise (Appendix N.4), additional details on our pipeline for mobile manipulation tasks (Appendix N.5), and finally, a list of the data generation hyperparameters for each task (Appendix N.6).
N.1 Equivalence between delta-pose actions and controller target poses
We assume that the action space A consists of delta-pose commands for an end effector controller (Assumption 1, Sec. 3). As in [7], we assume that actions are 7-dimensional, where the first components the desired translation from the current end effector position, the next 3 components represent the desired delta rotation from the current end effector rotation, and the final component is the gripper open/close action. The delta rotation is represented in axis-angle form, where the magnitude of the 3-vector gives the angle, and the unit vector gives the axis. The robot controller converts the delta-pose action into absolute pose target TW by adding the delta translation the current end effector position, and applying the delta rotation to the current end effector rotation.
Consequently, at each timestep in demonstration {st, du Y it is possible to convert each action a, to controller pose target TW by using the end effector pose at each timestep. MimicGen uses this to represent each segment the source demonstration as a sequence of controller poses. MimicGen also uses this conversion execute
transformed segment during data generation it converts the sequence of controller poses in the segment to a delta-pose action at each timestep during execution, using the current end effector position.
N.2 Details on Interpolation Segments
As mentioned in Sec. 4.2, MimicGen adds an interpolation segment at the start of each transformed segment during data generation interpolate from the current end effector pose Tw and the start of the transformed segment T There are two relevant hyperparameters for the interpolation segment in each subtask segment - Rinterp and Rlxxed- We first use simple linear interpolation between the two poses (linear in position, and spherical linear interpolation for rotation) add Thinter interme-
diate controller poses between T. and T and then hold TW fixed for Rexxed steps. These intermediate poses are all added to the start of the transformed segment, and given to MimicGen to execute by one.
N.3 Reference Segment Selection
Recall that MimicGen parses the source dataset into segments that correspond to each subtask Dire = - b (Sec. 4.1). During data generation, at the start of each subtask St(os,) ) MimicGen must choose corresponding segment from the set {T of N subtask segments in Dare. suffices to choose only one source demonstration j E [1, N} since this uniquely iden- tifies the subtask segment for the current subtask. We discuss some variants of how this selection occurs.
Selection Frequency. As presented in the main text (Fig. 2), MimicGen can select a source demon- stration j (and corresponding segment) at the start of each subtask. However, in many cases, this can be undesirable, since different demonstrations might have used different strategies that are in- compatible with each other. As an example, two demonstrations might have different object grasps for the mug in Fig. 2 each grasp might require a different placement strategy. Consequently, we introduce a hyperparameter, per-subtask, which can toggle this behavior MimicGen chooses single source demonstration j at the start of data generation episode and holds it fixed (so all source subtask segments are from the same demonstration, (Ti -2. -2...., 7yy)).
it is set False,
35

---

Task
normal
no noise
replay w/ noise
Square (Da) (DGR)
73.7
80.5
88.1
Square (D1) (DGR)
48.9
50.7
Square (Da) (DGR)
31.8
33.4
Threading (Dp) (DGR)
51.0
84.5
53.8
Threading (D1) (DGR)
39.2
50.8
Threading (D2) (DGR)
21.6
27.3
Square (12) (SR, image)
90.7 + 1.9
72.0
+ 3.3
42.0 + 1.6
Square (D;) (SR, image)
+.4
56.7 0.9
Square (Dz) (SR, image)
49.3 2.5
6.6
Threading (Da) (SR, image)
98.0 1.6
59.3 6.8
74.0
Threading (D;) (SR, image)
60.7 :: 2.5
43.3
Threading (Da) (SR, image)
38.0 # 3.3
22.7 0.9
Square (D₂) (SR, low-dim)
98.0
82.0
60.7 3.4
Square (D;) (SR, low-dim)
70.0 8.4
Square (D2) (SR, low-dim)
58.7
55.3 1.9
Threading (12) (SR, low-dim)
97.3 +79
69.3 0.9
34.7
Threading (D2) (SR, low-dim)
72.0 1.6
56.7 5.0
Threading (D2) (SR, low-dim)
60.7 1 6.2
46.0 7.5
Table N.1: Effect of Action Noise. MimicGen adds Gaussian noise to actions when executing transformed segments during data generation. These results show that removing the noise can increase the data generation rate (as expected), but can cause agent performance to decrease significantly. They also show that just replaying the same task instances from the source human data with action noise is not sufficient (although does improve results over just using the source human data).
The per-subtask hyperparameter determines how frequently source demonstration selection occurs we next discuss strategies for actually selecting the source demonstration.
Selection Strategy. We now turn to how the source demonstration is selected. We found random selection to be a simple and effective strategy many cases here, we simply select the source demonstration j uniformly at random from {1, 2.... N). We used this strategy for most of our tasks. However, we found some tasks benefit from a nearest-neighbor selection strategy. Consider
selecting source demonstration segment for subtask (,s.)). We compare the pose TW of object os, in the current scene with the initial object pose TW at the start of each source demonstration segment -/ and sort the demonstrations (ascending) according to the pose distance (to evaluate the pose distance for each demonstration segment, value of the delta rotation (in axis-angle form) between the two object rotations). We then select a demonstration uniformly random from the first nn, members of the sorted list.
the L2 position distance with the angle
N.4 Action Noise
When MimicGen executes a transformed segment during data generation, it converts the sequence of target poses into delta- posteappseeeee actions a, at each timestep. We found it beneficial to apply additive noise to these actions - we apply Gaussian noise (0, 1) with magnitude a in each dimension (excluding gripper actuation). To showcase the value of including the noise ran ablation experiment (presented in Table N.1) that shows how much data generation success rate and agent performance changes when the datasets are not generated with action noise during execution (compared to our default value of a = 0.05).
As expected, the data generation success rate increases when using noise, noise cause the end effector motion to deviate from the expected subtask segment that is being followed (the most significant example is an increase of 33% on Threading Do). However, agent performance suffers, with performance drops as large as 30% on agents trained on low-dim observations, and up to 40% on agents trained on image observations.
Another natural question is whether the benefits of MimicGen come purely from action noise in- jection. To investigate this, we also ran comparison ("replay w/ noise" in Table N.1) where we took the 10 source demos, and replayed them with the same level of action noise (0.05) used in experiments until we collected 1000 successful demonstrations. We selected random source
36

---

Task
normal
no NN
no per-subtask
no NN + no per-subtask
Square (Da) (DGR)
73.7
36.7
Square (D1) (DGR)
48.9
30.6
Square (f2) (DGRR
31.8
22.4
Nut Assembly (Dp) (DGR)
50.0
27.1
Stack (Dn) (DGR)
94.3
85.1
71.6
Stack (D₂) (DGR)
90.0
76.3
63.3
Stack Three (Do) (DGR)
71.3
37.8
26.7
Stack Three (D1) (DGR)
68.9
36.0
27.5
Pick Place (Da) (DGR)
32.7
30.8
29.7
Square (D₂) (SR, low-dim)
98.0 1.6
94.7 2.5
Square (D1) (SR, low-dim)
3.4
79.3
Square (D2) (SR, low-dim)
1.
5.9
Nut Assembly (Da) (SR, low-dim)
1.6
6.77
Stack (D2) (SR, low-dim)
100.0
9
99.3 09.3
Stack (D₂) (SR, low-dim)
100.0
100.0
99.3 1.93:099
Stack Three (D₂) (SR, low-dim)
88.0 1.6
1.6
81.3 2.5
Stack Three (D:) (SR, low-dim)
90.7 0.9
78.7 + 2.5
83.3 1 0.9
Pick Place (Dp) (SR, low-dim)
58.7 + 7.5
52.0 1 3.3
56.0 + 5.9
Table N.2: Effect of Removing Selection Strategy. Some of our tasks used nearest-ncighbor selection strategy and a per-sublask selection strategy for source demonstration segments. These results show the effect of removing these selection strategies (e-g. using the default, random selection strategy). Interestingly, while the data generation rates decrease significantly, agent performance does not decrease significantly for most tasks.
demonstration at the start of each trial and reset the simulator state to its initial state before collec- tion.
This comparison shows the value of using MimicGen transform and interpolate source human segments to collect data on new configurations, instead of purely using replay with noise on the same configurations from the source data. Comparing the "replay w/ noise" column Table N. to Fig. 4, that there is appreciable increase in the success rate Do compared to just using 10 source demos (Square increases from 11.3 to 42.0, and Threading increases from 19.3 to 74.0), but training on the MimicGen dataset still achieves better performance on Du (Square: 90.7, Threading: 98).
N.5 Data Generation for Mobile Manipulation Tasks
The process of transforming source segments differs slightly for mobile manipulation tasks. A source segment may or may not contain mobile base actions. If the segment does not contain mobile base actions generate segments in the same manner our method for manipulator-only environ- ments. If a segment does contain mobile base actions we assume that the segment can be split into three contiguous sub-segments: (1) a sub-segment involving manipulator actions, (2) subsequent sub-segment involving mobile base actions, and (3) final sub-segment involving manipulator ac- tions. We generate corresponding sub-segments for each of these phases. We generate sub-segments for (1) and (3) in the same manner as our algorithm for manipulator-only environments, and we gen- erate sub-segment (2) by simply copying the mobile base actions from the reference sub-segment. We found this scheme to work sufficiently well for the mobile manipulation task in this work, but future work improve the generation sub-segment (2) (the robot base movement) to account for different environment layouts in scene, by defining and using reference frame for each base motion segment, like the object-centric subtasks used for arm actions, and/or integrating motion planner for the base. We highlight the limitations of our approach in Appendix D.
N.6 MimieGen Hyperparameters
In this section, we summarize the data generation hyperparameters (defined above) used for each task. As several tasks had the same settings, we group tasks together wherever possible.
Default. Most of our tasks used noise scale of a = 0.05, interpolation steps of Vinterp = 5. Fxxxed O, and selection strategy of random with per-subtask set to False. These tasks include
37

---

Threading, Coffee, Three Piece Assembly, Hammer Cleanup, Mug Cleanup, Kitchen, Coffee Prepa- ration, Mobile Kitchen, Nut-and-Boll Assembly, Gear Assembly, and Frame Assembly.
Nearest-Neighhor and Per-Subtask. Some of our tasks used the default values above, with the ex- ception of using nearest-neighhor selection strategy. The following tasks used nearest-neighbor (nny = 3) with per-subtask set to False: Square and Nut Assembly. Some tasks used nearest- neighbor (nny 3) with per-subtask set True: Stack, Stack Three, Pick Place. In general, found per-subtask selection to help for pick-and-place tasks. To showcase the value of using these specific selection strategies, we ran an ablation experiment (presented in Table N.2) that shows how much data generation success rate and agent performance changes when turning these strategies off during data generation. Interestingly. while the data generation rates decrease significantly, agent performance does not decrease significantly for most tasks.
Real. Our real robot tasks used different settings for safety considerations, and ensure that data could be collected in timely manner (maintain high data generation rate). All tasks used a reduced noise scale of 0.02, and higher interpolation steps of Piinterp 25, 7ifixed 25. The Stack task used a selection strategy of nearest-neighhor (nTg 3) with per-subtask set to True, and the Coffee task used a selection strategy of random with per-subtask set to False, just like their simulation counterparts.
38

---

Policy Training Details
We describe details of how policies were trained via imitation learning. Several design choices are the same as the robomimic study [7].
Observation Spaces. As in robomimic [7], train policies two observation spaces "low- dim" and "image". While both include end effector poses and gripper finger positions, "low-dim" includes ground-truth object poses, while "image" includes camera observations from front-view camera and a wrist-view camera. All tasks use images with 84x84 resolution with the exception of the real world tasks (Stack, Coffee), which an increased resolution of 120x 160. For "image" agents, we apply pixel shift randomization 363399 and shift image pixels by up to 10% of each dimension each time observations are provided to the agent.
Training Hyperparameters. We use BC-RNN from robomimic [7] with the default hyperparam- eters reported in their study, with the exception of increased learning rate (Ie-3 instead of le-4) for policies trained on low-dim observations, as we found to speed up policy convergence on large datasets.
[7], on simulation tasks, we evaluate policies using 50 rollouts per agent checkpoint during training, and report the maximum success rate achieved by each agent across seeds. On the real world tasks, due to the time-consuming nature policy evaluation, we take the last policy checkpoint produced during training, and evaluate it over 50 episodes.
Policy Evaluation.
Hardware. Each data generation run and training run used a machine (on compute cluster) with an NVIDIA Volta V100 GPU, 8 CPUs, 32GB of memory, and 128GB of disk space. In certain cases, we batched multiple data generation runs and training runs the same machine (usually to 4 runs). Real robot experiments were carried out on machine with an NVIDIA GeForce RTX 3090 GPU, 36 CPUs, 32GB of memory, and TB of storage.
39

---

P Data Generation Success Rates
In this section, we present data generation success rates for each our generated datasets. Com- paring the results in Table P.1 with our core image-based agent results (Fig. 4) and low-dim agent results (Table Q.1), we see that in many cases the agent performance is much higher than the data generation success rate. An extreme example is the Gear Assembly task which has data generation rates of 46.9% (Dn), 8.2% (Di), and 7.1% (D2) but policy success rates of 92.7% 92.7% (Do), 76.0% (D1), and 64.0% (D2). We also saw much higher agent performance than the data generation rate our robot transfer experiment (see Appendix F).
Task
Do
D1
D2
Stack
94.3
90.0
Stack Three
71.3
68.9
Square
73.7
48.9
31.8
Threading
51.0
39.2
21.6
Coffee
78.2
63.5
27.7
Three Pe. Assembly
35.6
35.5
31.3
Hammer Cleanup
47.6
20.4
Mug Cleanup
29.5
17.0
Kitchen
100.0
42.7
Nut Assembly
50.0
Pick Place
32.7
Coffee Preparation
53.2
36.1
Mobile Kitchen
20.7
Nut-and-Bolt Assembly
66.0
59.4
47.6
Gear Assembly
46.9
8.2
7.1
Frame Assembly
45.3
32.7
28.9
we generated data for, we report the data generation rate (DGR) - which is the success rate of the data generation process (recall that not all data generation attempts are successful, and MimicGen only keeps the attempts that result in task success). Comparing with Table Q.1 and Fig. 4, that several tasks have significantly higher policy learning performance than data generation rates.
Table P.1: Data Generation Rates. each
40

---

Low-Dim Policy Training Results
In the main text we focused on image observation spaces. this section we present full results for agents trained on low-dim observation spaces and show that these agents are equally perfor- mant. Results on our main generated datasets are shown in Table Q.1 (and can be compared the image-based agent results in Fig. 4), and the souree dataset size comparison and policy training data comparisons are shown Fig. Q.1 (and can compared Fig. 4).
Task
Source
Du
D2
D2
100.0
Stack Three
2.7 0.9
88.0 1 1.6
90.7
Square
80.7
1.9
Threading
9.3
97.3
12
60.7
6.2
Coffee
12.7
76.7
0.9
Three Pc. Assembly
2.7
74.7
613111
18.7 4.1
Hammer Cleanup
61.7
10000
Mug Cleanup
3.0 1.6
82.0 + 2.8
54.7 5.0
Kitchen
Nut Assembly
0.0
76.0 = 1.6
Pick Place
0.0++00
68.7 + 7.5
Coffee Preparation
2.0-4.0.0
76.0 4: 5.7
59.3 3.4
Mobile Kitchen
6.7.
76.7 10.5
Nut and Bolt Assembly
2.0
98.0 1 1.6
96.0 1 1.6
81.3
3.8
Gear Assembly
12.0
Frame Assembly
9.3
87.3 = 2.5
10.7:11
58.0=5.7
Table Q.1: Low-Dim Agent Performance
Source and Generated Datasets. For each task, present source demos and each MimicGen
the success rates seeds) of low-dim agents trained with BC on dataset (1000 demos for each reset distribution). There is large improvement across all tasks on the default distribution (Do) and agents are performant on the broader distributions (D1, D2)-
Companson (low dim)
Poliy Triinnng Daes Comparison (iw dim)
Orgaet
Ma Thrae
Figure Q.I: (left) MimicGen with more source human demonstrations. We found that using larger source datasets to generate MimicGen data did not result in significant low-dim agent improvement. (right) Policy Training Dataset Comparison. We compare agents trained on 200 MimicGen demos to 200 human demos remarkably, the performance similar, despite MimieGen only using 10 source human demos. MimicGen also produce improved low-dim agents by generating datasets we show comparison between 200, 1000, and 5000 above. However, there can be diminishing returns.
41

---

R Bias and Artifacts in Generated Data
In this section, we discuss some undesirable properties of the generated data.
Are datasets generated by MimicGen biased towards certain scene configurations? This is natural question to ask, since MimicGen keeps trying to re-use the same small set of human demonstrations on new scenes and only retains the successful traces. Indeed, there might be a limited set of scene configurations where data generation works successfully, and some scene configurations that are never included in the generated data. We conduct an initial investigation into whether such bias exists by analyzing the set of initial states in subset of our generated datasets. Specifically, we take inspiration from [79], and discretize the set of possible object placements for each object in each task into bins. Then, we simply maintain bin counts by taking the initial object placements for each episode in a generated dataset, computing the bin it belongs to, and updating the bin count. Finally, we estimate the support coverage of the reset distribution by counting the number of non-zero bins and dividing by the total number of bins.
As concrete example, consider the Threading D, variant, where the needle and tripod are both sampled from a region with bounds in x, and 0, where is a top-down rotation angle (see Fig. 5). If each dimension is discretized into n independent bins, there are total of n. bins (all combinations of the dimensions). Due to this exponential scaling, we use a small number of bins (n = 3). Note that when conducting this analysis, we had be careful ensure that the overall bin count was not too small or too large. If it was too small, each bin would correspond to a large section of the object configuration space, and the results would not be meaningful. Similarly, if was too large, there is no way for 1000 generated demonstrations to cover a meaningful portion of the support (since there can only be 1000 bins covered best).
We now present our results. For several environments, we found there to be a good amount of sup- port coverage for example, Coffee D, (98.8%), Coffee D2 (89.3%), and Square Di (92.6%). However, we also found datasets that likely have significant amounts of bias - for example, Square D2 (66.4%), Threading D1 (71 1)), Threading D2 (61 ))), Three Piece Assembly Do (67.9%), Three Piece Assembly D2 (43,5 (4).5%) and Mug Cleanup D1 (64%). This analysis is certainly im- perfect, as some datasets could still be biased towards containing certain object configurations than others (e.g. having non-uniform bin counts across the support), and there could also be different kinds of bias (such as repetitive motions). However, this analysis does confirm that there is certainly bias in some of the generated datasets. A deeper investigation into the properties of the generated data is left for future work.
Are there artifacts and other undesirable behavior characteristics in MimicGen datasets? Ar- tifacts and other undesirable behavior characteristics likely, for reasons. One reason is that MimicGen bridges transformed segments from the source dataset with interpolation segments. These interpolation segments could result in long paths and unnatural motions that are difficult to imitation. In fact, we found some evidence of this fact (see Appendix H). Another reason is that MimicGen only checks for successful task completion when deciding whether accept a gen- crated trajectory. This means that there might be undesirable behaviors such as collisions between the robot and certain parts of the world (including objects that are not task relevrele..nt.. As we move towards deploying robots trained through imitation learning, data curation efforts are of the utmost importance this is left for future work.
42

---

Using More Varied Source Demonstrations
Task
Source
Do
D1
D2
Square (src Da) (DGR)
73.7
48.9
31.8
Square D2) (DGR)
54.4
51.7
52.3
Three Piece Assembly (src Du) (DGR)
35.6
35.5
31.3
Three Piece Assembly (src D2) (DGR)
26.9
29.1
23.9
Square (src Do) (SR, low-dim)
18.7
98.0 = 1.6
80.7 4: 3.4
58.7 + 1.9
Square (src D2) (SR, low-dim)
2.0 0.0
98.0 = 1.6
84.7 17:111 1.9
60.7 + 2.5
Three Piece Assembly (sre 12) (SR, low-dim)
2.7
74.7 3.8
61.3
38.7 +
Three Piece Assembly (sre D2) (SR, low-dim)
0.0 0.0
62.0 +
57.3 4.1
32.0 +
Table S.1: Using More Varied Source Demonstrations. We present comparison of data generation success rates and policy success rates seeds) across two choices source datasets the source human demon- strations collected on Do (default used main experiments) and source human demonstrations collected on the significantly more diverse D2 reset distribution. Interestingly, while the data generation success rates differ, the policy success rates comparable, suggesting that downstream agent performance can be invariant how much the task initializations of the source demonstrations vary.
Most of our experiments used 10 source human demonstrations collected narrow reset distri- bution (Do) and generated demonstrations with MimicGen across significantly more varied reset distributions (Do. D1. D2). In this section, investigate whether having source demonstrations collected on a more varied set of task initializations is helpful. We do this by collecting 10 source human demonstrations
D2 and using it to generate data for all reset distributions (Do- D1. D2)- The results are presented in Table S.1. Interestingly, while the data generation success rates dif- fer, the policy success rates comparable, suggesting that downstream agent performance can be invariant to how much the task initializations the source demonstrations vary.
43

---

T Data Generation with Multiple Seeds
MimicGen's data generation process has several sources randommess, including the initial state of objects for each data generation attempt (which is sampled from the reset distribution D), selecting the source dataset segment that will be transformed (Appendix N.3), and the noise added to actions during execution (Appendix N.4). In all of our experiments, we only used single seed to generate datasets (our policy learning results are reported across seeds though). In this section, we justify this decision, by showing that there is very little variance in empirical results across different data generation seeds.
We generated 3 datasets (3 different seeds) for Stack Three (Do, D1) and Square (Do, D1, D2), and train low-dim policies (3 seeds per generated results, so seeds total per task variant) and summarize the results in Table T.I. The data generation success rates have very tight variance (less than 1%) and do not deviate from our reported data generation rates (Appendix P) by more than 0.6%. Furthermore, the mean policy success rates are extremely close to our reported results for low-dim agents in Table Q.1 (less than 2% deviation).
Task
Do
D1
D2
Stack Three (DGR)
71.7 0.3
69.3 + 0.4
Square (DGR)
74.4 0.5
48.5 + 0.7
32.0 = 0.9
Stack Three (SR)
89.6 2.1
92.4 + 1.6
Square (SR)
96.7 2.1
81.6 4.5 58.0 = 3.5
Table T.1: Data Generation with Multiple Seeds. We present data generation rates (DGR) and success rates
(SR) across 3 seeds of data generation, and low-dim policy training seeds per dataset (9 seeds) total. The results very close to reported results (less than 0.6% deviation in DGR, less than 2% deviation in SR) despite our results only generating datasets one seed.
44

---

U Tolerance to Pose Estimation Error
In the main text, we demonstrated that MimicGen is fully functional in real-world settings and can operate with minimal assumptions (e.g. no special tags or pose trackers) by using pose estimation methods (see Appendix for details). Consequently, the data generation process has some tolerance to pose error and can operate without having access to perfect pose estimates. In this section, we further investigate this tolerance simulation by adding 2 levels uniform noise to object poses -L is 5 mm position and 5 deg rotation noise and L2 is 10 mm position and 10 deg rotation noise [108]. As shown Table U.1, the data generation rate decreases (e-g. Square DO decreases from 73.7% to 60.9% for 1.1 and 30.5% for 1.2 and Square D2 decreases from 31.8% to 25. % for LI and 14.5% for L2), but visuomotor policy learning results are relatively robust (Square DO decreases from 90.7% to 89.3% for LI and 84.7% for L2, and Square D2 decreases from 49.3% to 47.3% for LI and 39.3% for L2).
Task
None
Level (5 mm 5 deg)
Level 2 (10 mm 10 deg)
Stack Three (D1) (DGR)
68.9
62.3
38.7
Stack Three (Dt) (SR)
86.7 + 3.4
84.0 + 2.8
80.7 + 3.4
Square (Do) (DGR)
73.7
60.9
30.5
Square (D1) (DGR)
48.9
40,2
20.2
Square (D2) (DGR)
31.8
25.1
14.5
Square (Dp) (SR)
90.7 1.9
89.3 + 2.5
84.7 + 2.5
Square (D₃) (SR)
73.3 3.4
64.0 + 1.6
62.0 + 1.6
Square (D2) (SR)
49.3 + 2.5
47.3 + 6.8
39.3 + 4.7
Coffee (Dn) (DGR)
78.2
28.9
5.6
Coffee (D₂) (DGR)
63.5
22.6
4.3
Coffee (D₂) (SR)
100.0 + 0.0
95.3 + 2.5
79.3 + 0.9
Coffee (D,) (SR)
90.7 + 2.5
83.3 + 2.5
77.3 + 4.1
Threading (Do) (DGR)
51.0
17.6
5.2
Threading (Dp) (SR)
98.0 + 1.6
94.7 + 0.9
86.7 + 1.9
Table U.I: Tolerance to Noisy Pose Estimates. We investigate how the data generation success rates (DGR) visuomotor policy success rates (SR) change when adding uniform pose noise to object poses source demonstrations and the new scene during data generation, Although the data generation rates decrease, policy success rates are robust. This shows that MimicGen can be tolcrant to noisy object pose estimation, and suitable for real-world data collection.
45


Reviews:
[{"review_id": "pARQkA8a_M9", "reviewer": "Reviewer_f5G9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a method for generating a diverse set of manipulation trajectories from a few demonstrations which could be of the same task or some variant. Their algorithm, MimicGen, takes in a few human demonstrations and parses them into object-centric subtasks. These source segments are then transformed to generate novel trajectories while preserving end-effector poses with respect to the object for the subtask.  As part of the evaluation, a large dataset is produced on various tasks from a small number of human demonstrations, both in simulation and hardware. Authors validate their method by showing that similar policy performance was achieved using a MimicGen generated dataset from a small number of human demonstrations versus the full amount from only human demonstrations. Additional experiments show that policies trained with the generated datasets achieve higher success than those using only the human demonstrations, from which the dataset was generated from, alone.", "summary_of_recommendation": "The proposed data generation method is well motivated; however, the novelty is rather weak. The main strength of the paper is the experiments which demonstrate the wide applicability of the method. There are exciting avenues for future work and utilising MimicGen in other problem settings such as sim-to-real transfer. The paper itself could be more self contained with more thorough description of methods and less reliance on appendices.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "* Given that the main contribution of the work is on trajectory generation, why was there no comparison with other trajectory generation methods in the experiments?\n* Regarding experimental setup, can the authors provide details on the reward function and computational resources used?\n* Both in abstract and introduction it is mentioned that 50k demonstrations were generated from 200 source demonstrations. However, in the experimental setup it is stated that 1000 demonstrations were generated across 18 tasks. Can you please address this discrepancy?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Quality/ Clarity\n--------------------\nThe paper is well written; however, the authors refer to the appendix in multiple instances which hampers the readability. The paper should be more self-contained. The related works covered are moved entirely onto the appendix, the parts that remain in the paper are vague. The authors should at least, move some of the most relevant works in the paper. Not clear how 50k demonstrations were produced from the ~200 human demonstrations.\n\nOriginality\n--------------\nThough trajectory generation algorithms exist, the proposed method is meant to be more general purpose and removes assumptions about the tasks and algorithms that utilise the data. Furthermore, MimicGen generates a more diverse dataset, specifically for manipulation tasks, by varying various parameters in the scenes and objects associated. However, the method itself is quite straight forward and lacks any significant novelty.\n\nSignificance\n-----------------\nThe proposed MimicGen method generates a diverse dataset from a small number of human demonstrations which reduces the amount of expensive data collection required. Removing assumptions on the task and algorithm using the data also makes their method more widely applicable to various problem settings, such as in sim-to-real frameworks.\n\nStrengths\n--------------\n* The proposed approach addresses limitations in the current literature well.\n* The authors perform a thorough experimentation with different manipulators across different tasks.\n* The results are presented well and summarised effectively.\n* Method shown to work well on real world experiments.\n\nWeaknesses\n------------------\n* No comparison with other trajectory generation methods.\n* Lacks key information regarding experimental setup such as computational resources used and formulation details of the MDP.\n* The naive interpolation scheme may heavily reduce the practicality of the proposed method, particularly in cluttered environments.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "FmFyauN_k1z", "reviewer": "Reviewer_pQft", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a data generation system MimicGen, which can synthesize successful demonstrations for randomized environment initializations. The system can handle randomizations of a number of aspects in the env, including object position and arm type. The authors then used the generated data to train a BC policy to demonstrate the efficacy of this method. Experiments are conducted in sim and real. Key ideas implemented in MimicGen include\n- Collect a small set of human demos for each task\n- Randomize the scene configuration, segment the sequence by key points on the time axis\n- Interpolate end effector movement based on object position/orientation\n- Execute the modified trajectory, and keep as a demo if successful]\n\n~50k demos were generated from ~200 human demos, a scale-up factor of 250x ", "summary_of_recommendation": "This paper provides a simple recipe for scaling up data collection, which if applied successfully can have a large impact on accelerating the acquisition of visuomotor skills using BC. The robustness of the recipe is demonstrated well in a nice array of evals in sim. The weakness of the results mainly comes from the real robot results, where the significant drop of performance compared to sim results for the select 2 evals raises questions for the applicability of the method in real. Considering the more readily available selection of data generation methods exclusive to sim, as well as the significant value from removing the need of sim2real transfer, the impact of this paper will be greatly increased if stronger real world results can be demonstrated.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "155: how is success of a subtask detected (sim and real)?\n\nFigure 4, table: please highlight the best number in each row with bold font\n\nAre there additional evidences you can provide to show the effectiveness of MimicGen in the real world?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n- This is a well presented paper. I appreciated the concise descriptions of the method, its strength and its weaknesses.\n- The proposed method was tested on and shown to be effective on a representative set of diverse tasks in sim. The experiment results and analysis are clearly presented.\n\nWeaknesses\n- Limitations to the applicability of the proposed method in the real world: the interpolation step may cause safety concern when applied in the real world. The authors implemented mitigations which resulted in performance degradation. This calls for further investigations on how to address this so the performance of the proposed method is more in line with sim results. Can MimicGen still match the performance of an equal amount of human-collected data?\n- It was unclear how success of each subtask in a trajectory is detected. How is it done in sim and real respectively? ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "Aw1f4NGb_Ju", "reviewer": "Reviewer_GHn9", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors present MimicGen - al algorithm for generating diverse robot demonstrations from a small set of initial human teleoperated demonstrations. Then they show how such a synthetically-generated dataset can be useful in a behavioural cloning scenarion where a robot learns from the dataset but only uses images as inputs to its policy.", "summary_of_recommendation": "I recommend weak reject but would be happy to change my mind depending on the rebuttal discussion.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "* It's worth elaborating, when evaluating the learned policies, how many of the test objects and object poses are new. D0, D1 and D2 are explained but it is not clear quantitatively how do they differ.\n* Can you elaborate how were the object's states inferred in the real-robot experiments?\n* It is said that when new object states are sampled, some new trajectories fail. Is this because the object positions go beyond the robots operational space? It is not clear. It is also argued that the method allows for demonstrations to be gathered and synthesised on one robot and transferred to another. Does that account for the fact that different robots might have different operations spaces and maybe not all end-effector positions of robot A are reachable by robot B?\n* Do the authors plan to release the dataset?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall the paper is well-written and presented and outlines relevant issues for the field of robot learning. Experiments on a real robot are appreciated and so is the supplementary video. The breadth of different tasks is impressive. However, I have a major issue with the methodology overall and it's actual usability - I think the 3 assumptions listed in section 2 are rather strong - i.e. we assume we have very good scene and object observability and we stay in the delta end effector pose action space, which neatly allows us to depend heavily on good motion planning. But that means we can alleviate human demonstrators and generate useful datasets *only* for tasks that are already easy to segment into parts that are each solvable by just knowing where the manipulated objects are at the beginning, where they have to be at the end and motion planning between those two. What's the value of learning a visual policy here? What about tasks where this part segmentation is not that obvious because object state is not observable or maybe is even hard to represent - e.g. cloth folding tasks and manipulating deformable objects in general. \n\nOne option would have been to demonstrate that even though we have perfect observability and state information in the simulator, we can generate visual synthetic datasets and train a policy that generalised well to the real world where such perfect information in not available. This would have made MimicGen a valuable contribution to the sim2real community but as the authors note, such transfer is not trivial and does not achieve very good success scores.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "8xC2gNg5zF", "reviewer": "Reviewer_Tm7E", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a methodology for generating a large number of diverse demonstrations from a small initial set for use in imitation learning, under the assumption that the initial demonstrations are time-consuming or difficult to collect, e.g., human teleoperation. The method is based on online data generation, in which initial trajectories are transformed to new scenes and replayed in order to collect new demonstrations. A comprehensive empirical analysis is performed, in which task success for a learned behavior cloning policy is evaluated given a) datasets with different initial state distributions, b) datasets with different target hardware, c) datasets with different simulators, and d) datasets consisting of real-world demonstrations.", "summary_of_recommendation": "In general, I find the novelty of the methodology a little questionable and am unsure if this approach yields actual improvements as compared to standard domain randomization and augmentation techniques. However, the paper is very thorough I think many of the findings could be useful to the community going forward, so I am leaning towards accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Questions:\n1) I am slightly confused by the reference segment selection process (Appendix M.3). Do the reported results in the main paper use the same source demonstration for all sub-tasks? What is considered \"normal\" in Table M.2? And what does \"no NN\" mean? Does this imply NN is used by default then? This section is not clear to me.\n\n2) If demonstrations are not segmented, would this approach still work? I.e., treat the entire demonstration as a single \"subtask\".\n\n3) I am not entirely sure why the interpolation step is necessary. Is this due to sub-tasks being chained that belong to different source demonstrations? Is it possible to transform the source demonstration in such a way that linear interpolation to the start position is not necessary? In the case of Fig. 2, couldn't the initial end effector pose just be set to the beginning of the transformed trajectory?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n* This paper is well-written and it is easy to follow the methodology; intuitively the method is straightforward and seems reasonable. The related work, limitations, and empirical analysis are very comprehensive (albeit, much of it is located in the appendix due to space limitations).\n\n* The experiments test for many interesting conditions, including changes in number of source/target demonstrations, demonstration distributions, demonstrator quality, and target hardware, in addition to data generation in real-world settings. Tested conditions also include different forms of manipulation, e.g., mobile vs tabletop. In general, the results seem to indicate strong performance gains when the generated trajectories are utilized in training.\n\n* The limitations and assumptions are thorough and clearly laid out.\n\nWeaknesses:\n* The actual novelty of the individual components is questionable. Generalization to different objects and scenes is essentially just domain randomization, and replaying prior trajectories is also well-explored (coarse-to-fine, etc). While the proposed method applies a simple transformation to the source demonstration to map it to the target scene, this is fairly simple in nature and only seems to enable variations in starting/ending locations, with transformation failures being captured through a rejection sampling scheme. I do appreciate that the limitations are thoroughly and explicitly stated, however.\n\n* While it is argued that online data generation of trajectories as proposed in this work is advantageous to offline data augmentation due to physics-consistency (Appendix B5), it is unclear if this true in practice since augmentation methods are not evaluated. It has previously been shown that training over a distribution of physics models results in more robust policies [1,2]. I wonder whether similar performance gains could not be realized through simple trajectory replay with domain randomization methods and/or perturbations, which seem like they would be significantly easier to implement in practice since trajectory transformations are not required. This is also hinted at given applying action noise to the generated trajectories yields a non-significant increase in performance (Table M.1).", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you for your submission to CoRL 2023. The reviewers have spent considerable time working with this paper and the current version is recommended for acceptance.

Please follow up with promised improvements, experiments and additional suggestions from the discussion phase. These include in particular clarity about the assumptions, added comparisons and further requested details.
Decision: Accept (Poster)