What Matters in Learning from Large-Scale Datasets for Robot Manipulation
Authors: Vaibhav Saxena
Matthew Bronars
Nadun Ranawaka Arachchige
Kuancheng Wang
Woo Chul Shin
Soroush Nasiriany
Ajay Mandlekar
Danfei Xu
Venue: ICLR 2025 Poster (2025)
Keywords: imitation learning
robotics
dataset composition
Primary area: applications to robotics, autonomy, planning

Abstract: Imitation learning from large multi-task demonstration datasets has emerged as a promising path for building generally-capable robots. As a result, 1000s of hours have been spent on building such large-scale datasets around the globe. Despite the continuous growth of such efforts, we still lack a systematic understanding of what data should be collected to improve the utility of a robotics dataset and facilitate downstream policy learning. In this work, we conduct a large-scale dataset composition study to answer this question. We develop a data generation framework to procedurally emulate common sources of diversity in existing datasets (such as sensor placements and object types and arrangements), and use it to generate large-scale robot datasets with controlled compositions, enabling a suite of dataset composition studies that would be prohibitively expensive in the real world. We focus on two practical settings: (1) what types of diversity should be emphasized when future researchers collect large-scale datasets for robotics, and (2) how should current practitioners retrieve relevant demonstrations from existing datasets to maximize downstream policy performance on tasks of interest. Our study yields several critical insights -- for example, we find that camera poses and spatial arrangements are crucial dimensions for both diversity in collection and alignment in retrieval. In real-world robot learning settings, we find that not only do our insights from simulation carry over, but our retrieval strategies on existing datasets such as DROID allow us to consistently outperform existing training strategies by up to 70\%.

Published as a conference paper at ICLR 2025
WHAT MATTERS IN LEARNING FROM LARGE-SCALE DATASETS FOR ROBOT MANIPULATION
Vaibhav Saxena', Matthew Bronars's 1 Nadun Ranawaka Arachchige Kuancheng Wang',
Kuackcheg
Woo Chul Shin', Soroush Nasiriany", Ajay Mandlekar"i Danfei Xu 1.3j
Georgia Institute of Technology
"hhe University of Texas at Austin
8 NVIDIA
ABSTRACT
Imitation learning from large multi-task demonstration datasets has emerged as a promising path for building generally-capable robots. As result, 1000s of hours have been spent on building such large-scale datasets around the globe. Despite the continuous growth of such efforts, we still lack systematic understanding of what data should be collected to improve the utility of a robotics dataset and facilitate downstream policy learning. In this work, we conduct a large-scale dataset composition study to answer this question. We develop a data generation framework to procedurally emulate common sources of diversity in existing datasets (such sensor placements and object types and arrangements), and use it to generate large-scale robot datasets with controlled compositions, enabling a suite of dataset composition studies that would be prohibitively expensive in the real world. We focus on two practical settings: (1) what types of diversity should be emphasized when future researchers collect large-scale datasets for robotics, and (2) how should current practitioners retrieve relevant demonstrations from existing datasets maximize downstream policy performance on tasks of interest. Our study yields several critical insights - for example, we find that camera poses and spatial arrangements are crucial dimensions for both diversity in collection and alignment in retrieval. In real-world robot learning settings, we find that not only do our insights from simulation carry over, but our retrieval strategies on existing datasets such as DROID allow us to consistently outperform existing training strategies by up to 70%. More results at htthtts:// misoooo--mmiii labs. gigggittuhb thub. 1o/
INTRODUCTION
Imitation learning from offline datasets has emerged as a promising method to teach robots complex real-world manipulation tasks. Importantly, prior works have found that robot performance scales favorably with the dataset size and quality (Mandlekar et al. [202]] Brohan et al. 2022). Consequently, much efforts have been invested in building large-scale robot datasets that cover diverse tasks and environments. Recent large-scale efforts such as DROID (Khazatsky et al.) (024) and the Open X Embodiment datasets (Collaboration al. (023) have amassed millions of trajectories for table-top manipulation tasks. These datasets, while still orders of magnitude smaller than their vision and language counterparts, can allow robots trained on this data to generalize different scenarios.
Despite the continuous growth and promising results of these data collection efforts, we still lack a systematic understanding of what data should be collected improve the utility of a robotics dataset. However, gaining this understanding poses sigmificant challenges. Data collection is extremely costly and time-consuming, often requiring teams of human operators, fleets of robots, and months of manual effort (Brohan et al. 2022), and the cost is compounded by the time and effort it takes to evaluate different robot models trained on these datasets. This makes testing the effectiveness of
"Equal contribution, lequal advising Correspondence vsaxena33@gateehh.o

---

Published as a conference paper at ICLR 2025
Synthelic Dalasel with Controlled Compositin
Largo-Scall Dataset
Learning from Exisling Large Scale Dataset
Compositin Study
Compostion Ditribation
kne Tatt Compons
What ouit should osfoctioon tomen the CU GOuAy?
LL
LL
Datsens
U
Whw. dela deoudd retrinve
hage Teak
buen dasnnd Sock parts
Pollity Do Zo-Ttainiing with Retrioued Dutesct
Figure I: MimicLabs overview. Our framework encompasses: (1) A procedural dataset generator that creates diverse datasets with controlled composition. (2) A large-scale dataset composition study to analyze the impact of dataset diversity and alignment inspired by practical settings. (3) Extensive real-world experiments using existing large-scale robot datasets, based on study insights.
different dataset compositions intractable. The challenge exacerbated by the sheer number of potential variations in dataset composition (e.g., object types, camera angles, lighting conditions), making it impractical to ask counterfactual questions about dataset composition ("what if the sorting task is collected with mug instead of plush toy"). As result, current data collection efforts often rely on intuition rather than systematic analysis, potentially leading to inefficient use of resources.
To this end, we propose to use simulation and synthetic data generation questions about the collection and use of large scale datasets We develop a synthetic data generator that can procedurally emulate common sources of diversity found in existing datasets, allowing to create large-scale robot datasets with precisely controlled composition. Leveraging this capability, conduct a suite of dataset composition studies from multiple practical perspectives, aiming to provide insights into the efficient construction and utilization of large-scale robot datasets in the real world. Our work makes the following major contributions, as illustrated in Fig [
testbed to answer critical
1. Synthetic data generator for controllable dataset composition. We develop a data generator to procedurally emulate common sources of diversity found in existing datasets, including sensor placements, object types and textures, and spatial arrangements. Our framework can synthesize diverse tasks and corresponding demonstration data by using a small set of human demonstrations, making it possible to generate large-scale datasets with controlled composition and enabling to conduct a suite of dataset composition studies that would be prohibitively expensive the real world.
2. Practitioner-inspired collector and retriever settings. We conduct our analysis from two practical perspectives. The collector perspective explores which dimensions of variation (DVs) should be prioritized when building large-scale datasets, focusing the utility of increasing dataset diversity on broad skill transfer. The retriever perspective examines how to best utilize existing datasets for specific target task, addressing questions such as whether to retrieve only the most similar demonstrations or train on the dataset in its full diversity. By connecting these two perspectives, we provide actionable insights on efficient dataset construction and utilization.
3. Dataset composition studies with MimicLabss Leveraging our synthetic data generator, we create the MimicLabs dataset, a large-scale dataset comprising nearly IM trajectories across over 3K task instances in 8 visually distinct simulation environments. This dataset reflects realistic scenario where multiple robotics labs collaborate to collect diverse datasets. Our experiments with MimicLabs yield several key insights. For example, we find that camera poses and spatial arrangements are crucial dimensions for both diversity collection and alignment in retrieval. High diversity in these dimensions enables better generalization, while alignment significantly boosts performance on target tasks. Conversely, we discover that object textures have minimal impact on downstream performance, suggesting that this dimension need not be prioritized in data collection retrieval strategics.
Insights that transfer to real-world settings, Our study resulted in actionable insights readily applicable to real-world settings. We validate our findings on 7 real-world manipulation tasks, where both collector and retriever experiments corroborate our simulation based insights. Particularly noteworthy are our retriever insights, which applied to DROID (Khazatsky et al.] (024), an existing large-scale dataset. Our retrieval strategy, informed by the simulation studies, outperforms the existing approach of learning from the entire DROID dataset by up 70% on certain tasks.
2

---

Published as a conference paper at ICLR 2025
@
Diversity
(
Collection
Large Dalasel
Low Diversty
Targeled Diversity
Taget Task
limiiing Distribttion
Lame Dotreet
Retrieval
Aligned Detaset
Disoriution
Target
Figure 2: Characterizing Co- Training Settings: (a) We consider four co-training settings character- ized by the diversity of their co-training datasets and their aligmment with the target dataset. This is illustrated with the object spatial DV of the coffee pod. (b) The collectors aim select and increase the diversity of selected DV(s) to improve the utility of a dataset.
(c) The retrievers aim to extract a subset from an existing large dataset by aligning specific DVs with their target task order to improve the target task performance.
2 RELATED WORK
Large-scale Robot Manipulation Datasets. There have been number of recent efforts collecting
large-scale robot manipulation datasets. These include real-world datasets covering a broad set of
manipulation tasks (Jang et al. [022]] [Brohan et al. 2022 Mandlekar et al. 2018 Lynch et al. 2022
Walke et al. 2023 Bharadhwaj et al. 2023; Fang et al. 2023] Khazatsky et al. 2024)). Similarly,
there have been efforts in collecting (Liu al. (023a) and generating (Mandlekar et al. 2023 Dalal
ct al. 2023 Nasiriany et al. 2024) data in simulation. this work, we seek to improve future data
collection and curation efforts by studying how dataset composition and different sources of diversity impact downstream policy learning.
Study of Dataset Construction and Composition. Recent works examine the role that data quality and curation play in policy generalization. Using data curation strategies to selectively weight data based on actions (Belkhale et al. 2023), interventions (Liu et 2023b), and datasets (Hejna et al.) 2024) can yield benefits without collecting additional robot data. Other works explicitly focus on the dimensions of variation present in robot manipulation and study the role that these dimensions play in policy performance. Notably, Xie et al. (2023) and Pumacay et al. (2024) study policy performance under distributions shifts across different under fixed data distributions. Gao et al. (2024) compares policy performance under different data collection strategies with combinatorial variations. While these works separately study data curation and policy evaluation across different dimensions of variation, we bring unified perspective of studying both questions and apply the resulting insights to improve existing practices in learning from large-scale dataset (Khazatsky et al. 2024
MIMICLABS STUDY DESIGN
common use for large-scale robot datasets is for a practitioner to use them to boost the performance of specific task that they would like their robot to perform. The goal of the MimicLabs study is to understand how dataset composition affects this downstream task performance. We first formalize the problem of co-training with large-scale multi-task datasets in Sec. 3.1 We then introduce a formalism for dataset composition Sec. 33.2 and describe how this allows us to compare multiple datasets. Finally, Sec. 3.3 discusses how experiments and analysis help both future dataset collectors and current robotics practitioners for making the most out of large-scale robotics datasets.
3.1 PROBLEM FORMULATION
We study practical scenario where robotics practitioner trains robot perform specific task of interest by leveraging pre-existing, large-scale robot dataset (collected across diverse scenarios) and small set task-specific demonstrations. We refer the specific task of interest as the target task. Formally, we denote the target dataset, which is the set of target task trajectories collected by the practitioner Dy
{g(0) Nr
i==1 which samples from demonstration distribution
3

---

Published as a conference paper at ICLR 2025
Dr. Here, E represents demonstration trajectory comprised of observation-action pairs. Each trajectory exhibits a certain behavior that carries out the target task in a well-defined environment setup. Similarly, we denote the co-training dataset, which the large collection of available demonstrations as De = ( y No. which are samples from a distribution Dc. The practitioner then trains a visuomotor policy कe that learns to act by optimizing for 0 as l = arg min, L (Dy J'Dc; 0) where is a behavior cloning (BC) objective. Following prior works (Fu et al. 2024| Khazatsky al. 2024). we refer to this strategy of learning from the combined dataset as "co-training.
3.2 DIMENSIONS OF VARIATION (DVs) AND CO-TRAINING SETTINGS
To characterize dataset composition, we need to formalize a demonstration distribution D. We introduce the concept of Dimensions of Variations (DVs), which are independent factors that characterize the variability in demonstration data. We denote the full set of distributions that create variation in D as {2(1), 2(K), r). where 2(k) is the distribution of variation along a single DV in the environment, and т denotes the goal of the task. We consider DD to be parameterized as 7). An important criterion for DV selection is that each DV can be measured in each demonstration of a large heterogeneous dataset such as DROID (Khazatsky et al. 2024). This allows us study different dataset compositions on existing robotics datasets by retrieving subsets of the dataset that might be measurably diverse or aligned with the target distribution. For example, we study if retrieving demonstrations from DROID that are aligned along the camera pose DV helps boost downstream performance, where we use the camera extrinsics the DV measurement. Finally, we note that our analysis framework and tools are not tied to the specific set of DVs and can be utilized for any additional DVs of interest.
Given parameterized dataset with DV distributions, we can characterize the relationships between target (Dr) and co-training (Dc) demonstration distributions. In particular, we aim to understand how the diversity of De and the alignment between De and DDr impact co-training performances. We define S-) as the support of any distribution and (S()) as measure of its size. We say 'D is diverse along DV & if (S(2())) is large. Additionally, we say that Dj and De are aligned along 2A) if ((2 (k) C S(Z()), This creates four cases when comparing Dr and De along a DV, as illustrated in Fig. with more details in Appendix [ In our experiments, considering these cases help us understand which DVs need to be diverse in the co-training dataset, and how important alignment for target task performance.
3.3 PRACTICAL SETTINGS: THE COLLECTOR AND THE RETRIEVER
We seek to understand the effects of dataset composition through two different lenses inspired by practical scenarios: a data collector and a data retriever.
Data Collector. We consider collectors as practitioners who are contributing demonstrations to large robotics dataset. Since data collection is Lime-consuming we seek to understand which DVs should be prioritized for diversity or alignment during data collection facilitate downstream task learning. Instead, in our experiments, we set Dr and De to be fully aligned along all DVs except one. This allows us study the impact of single DV (say k) in Dc. by constructing different co-training datasets (()) where each dataset corresponds to a different choice of distribution for DV k. 2(hh For example, to study the impact of different camera poses, would construct De to be aligned with DDr along spatial arrangements, textures, background, etc. and just create datasets with different camera pose distributions (2()).
Data Retriever. A data retriever is a practitioner that seeks to use a pre-existing large-scale co- training dataset De with high variation in several DVs (such as DROID) for their specific task of interest. In this setting, we aim to retrieve a subset De- C De to maximize downstream task performance, by aligning some DVs from the retrived co-training dataset z(4r with those from the target dataset 2 For example, we study whether retrieving demonstrations from DROID that have aligned camera poses with that of the downstream setup will improve learning performance. In certain
cases it can be difficult or impossible to achieve such alignment consequently, our experiment also
studies the effect of retaining diversity in DVs (e.g. a DV (k) with large support).
4

---

Published as a conference paper at ICLR 2025
MimicLabbs
Dataset
Scenes
3K Task Instunces
IM- Trajectories
50+ Objects
25+ Receptacles
S+ Camera Poses
Multiple Layouts
Figure 3: MimicLabs Dataset. We use our data generator to create large-scale, multi-task dataset to emulate a realistic scenario where multiple labs collaborate to create a dataset with large variations in diverse DVs. We leverage this dataset to conduct our dataset composition study.
4 PROCEDURAL TASK AND DEMONSTRATION GENERATION
To understand how different data composition choices influence downstream policy learning, we develop framework for procedural task and demonstration generation. This framework enables us to generate large multi-task. datasets with controlled composition, culminating in the creation of our MimicLabs dataset (Fig. 3) Our framework consists of two main components: procedural task generation and procedural demonstration generation. It takes set of dataset composition factors as input (DVs, Sec. 3.2p and uses them to generate a diverse set of task instances, followed by demonstration generation for each instance.
Procedural task specification. To enable controllable data generation for large multi-task datasets, require way to specify tasks that allows for diverse procedural generation. These task instances should vary along specific Dimensions of Variation (DVs, Sec. 3.2) which allow us to parameterize the composition of these datasets. To this end, we leverage the Behavior Domain Definition Language (BDDL) (Ghallab et al. 1998 Srivastava et al.) 2021). The BDDL grammar allows for specifying scenes, objects, their spatial arrangements, and predicates for task initialization and success. We build upon the parsing framework of LIBERO (Liu et al. (023a) and generate task specifications with controllable variation in spatial arrangements of objects and receptacles, camera poses, and textures. These are combinatorially varied, resulting in large set of task instances for collecting and generating demonstrations for our study. Details about the task specification are in Appendix D
Procedural demonstration generation. To automate demonstration generation on our procedurally generated task instances, we build upon MimicGen (Mandlekar et al., 2023). Unlike prior studies that use "scripted" experts (Xie et al. 2023) rely entirely on human teleoperation (Mandlekar al. 202T Liu al. 2023a), our approach allows for scalable data generation while preserving the fine-grained manipulation strategies used in human demonstrations. MimicGen automates data generation by decomposing source human demonstrations into object-centric manipulation segments, which are then transformed and stitched together create trajectories for novel scenes. Our key innovation lies in leveraging the BDDL task specification and the state predicates provided by the simulator largely automate the process of decomposing tasks into these segments. This automation significantly reduces the human effort required when scaling to thousands task instances. By using MimicGen in conjunction with our BDDL-based task specifications, we can scale from small set of human demonstrations to a large, diverse dataset. This approach allows us to generate demonstrations across wide variations in DVs from limited set of source demonstrations, and easily generate demonstrations for new tasks by swapping out the BDDL task specification. For example, given source demonstrations of robot picking up a bowl with fixed texture and putting a basket from a shoulder camera view, our pipeline can create datasets with varying camera poses, bowl textures and geometries, table textures, and spatial arrangements - all specified in BDDI.
The MimicLabs Dataset. We can now use our controllable multi-task dataset generation framework to collect large-scale multi-task datasets by conditioning on different dataset composition choices (DVs, Sec. b.2h. To fully leverage the power of our framework, we curate large set of DV ranges that can be combinatorially varied, including camera poses, objects, receptacles, and their textures and spatial arrangements, and instantiate over 3000 task instances with language instructions in 8 different scenes. We then collect 500 source human demonstrations for a subset of these task instances and synthesize over 1M demonstrations using our procedural data generation framework. We refer to
5

---

Published as a conference paper at ICLR 2025
Table 1: Analyzing the effects of misaligned and diversities in DVs from a collector's perspective. Details about baseline variation are in Appendix F.2 Target variations are perturbations to one DV distribution while others stay fixed. Success rates with co-training variations show how increasing variation in one DV may help skill transfer from co-training in that DV as well as other DVs.
DV w! corrat- misalignment
Target only
Co-training with different DV distributions
TbyTex
tableTex
objSput
comPoeee
16.57
43,33
90
43.33
43.33
obfTce
30
93.33
96.67
90
90
93,33
wesk-Tex
43.33
66.67
80
63.33
83.33
70
cbtSpat
46.67
33.33
This dataset is meant to reflect a realistic scenario where multiple robotics labs collaborate to collect datasets with large amounts of variation in diverse DVs, as prior work has done in real-world settings (Khazatsky et al. 2024). We illustrate the different scenes containing objects and receptacles in Fig|3] Our dataset consists of both long- and short- tasks varied across multiple DV' that can be used to either learn specific skills, stitched together to train complex behavior, or used to evaluate new robot policies for their generalization capabilities. See Appendix [[ more details of DV distributions for generating this dataset.
dataset as the MimicLabs dataset.
CORE RESULTS
We leverage our MimicLabs pipeline for demonstration generation and collect datasets of varying diversities and relative alignments. We use these datasets to gather takeaways from data collector and retriever's s standpoint, and finally validate our findings in both settings through experiments on real robot. Training details for all experiments can be found in Appendix D
5.1 EFFECT OF INDIVIDUAL DVs: COLLECTOR'S S PERSPECTIVE
Data collectors are practically limited in the number of trajectories they can collect, hence it is important to understand where time is best spent when adding demonstration diversity. We investigate which DV should be diverse the co-training dataset to maximize downstream task success. Our experiments are designed to determine when alignment is necessary for a certain DV, and when increasing diversity helps alleviate misalignment in composed datasets. To do this, we construct lorget-cotraiiin distribution pairs that are aligned along all but one DV, including a large baseline dataset with low variation and misalignment along that DV. By combining the target dataset with different co-training datasets (for each DV), we identify DVs where it is crucial to have diversity during data collection to mitigate misalignment along that and other DVs. This comprehensive approach helps collectors understand where focus their efforts best when adding diversity their datasets for optimal downstream performance.
Experimental Setup. We examine a clear table task where the robot must open cabinet's top drawer and place a bowl inside, amidst four distractor objects. This task requires two steps: openTopDrawer and pickF laceTopDrawer. We identify and independently vary five DVs: camera pose (camPose), object texture (objTex), table texture (tableTex), object spatial arrangement (objSpat), and motion primitive, Our experiments test how co-training diversity in these DVs affects transfer learning when one DV is misaligned with the target. Additionally, we investigate motion diversity and alignment effects on two task variants: (1) pick? Dtepeeaaeeettaaaeetpuu...... where the robot places the bowl in an open drawer and closes it, and (2) pul pupppppppppp
where the robot opens closed drawer before placing the bowl inside. For these variants, we maintain alignment across all other DVs. Results are in Tables 1 and I0. with our findings discussed below. Additional results on a make coffee task are in Table 9
Less diverse and misaligned camera poses prevent skill transfer. We found that co-training with dataset that has a misaligned camera pose significantly hurts the boost in performance the co- co-training dataset could have otherwise offered (in Table I] for misalignment along camPose. notice low Baseline performance). This points to the fact that camera poses are necessary to be aligned between target and co-training distributions for maximal performance boost. In this setup, find that increasing

---

Published as a conference paper at ICLR 2025
Table 2: Retriever's Perspective. Task success when co-training using various retrieval strategies,
including counterfactual cases (i.e., misaligned retrieval).
Target task
Target only
Full variation
Camera pose
Object testure
Table testure
microware mug
6.67
60
70
46.67
53.55
66.67
75.53
70
70
40
43.33
clear table
13.33
30
36,67
30
20
33.33
26.67
30
36.67
23.33
40
Object spatial
Receptudle sputial
microwane mung clear table
70
56.67
46.67
16.67
20
40
26 67
30
23.33
30
the diversity of camera poses in the co-training dataset improves the transfer, even when the target camera pose distribution mis-aligned with the co-training distribution. For the clear-table task, we see over 40% performance boost (in Table same row, see Baseline vs camPose).
Co-training with misaligned object textures, with minimal variation, was sufficient for down- stream success. We find that simply training with large set of co-training demos, without any heed to aligned or even varied object textures, led to successful task completion. Specifically, we see over 90% success in the target task despite completely misaligned object textures with co-training, Note that object geometries were consistent between target and co-training, variation which we analyze in later section. Also note that we did not vary lighting conditions or material properties in the target tasks, leading to the rendering of the object primarily dependent on its texture.
High diversity in camera poses assists transfer learning with misaligned textures. We find that co-training datasets with high diversity in camera pose can transfer to target tasks with misaligned table textures and object textures. Changing camera poses allows the robot to see larger background variations, potentially leading to this visual robustness. This observation leads us to believe that adding camera pose diversity is generally useful during data collection.
Spatial coverage in co-training is critical to downstream performance. High diversity in spatial arrangement in the target task necessitates diverse action coverage for the robot, which is not covered by any visual DV. We find that adding diversity in the spatial arrangement of objects was the only way we could significantly boost transfer learning in the presence of misalignment along this DV (in Table I] for misalignment in objSpat, see Baseline vs objSpat) While increasing diversity in camera poses adds some performance boost, the best perfffrrrn co- co-training is still the one with largest diversity in spatial arrangements. This leads us to believe that spatial arrangement is both necessary to be aligned and generally useful for diverse data collection.
Co-training with aligned motion primitives and high diversity enables better downstream performance. We find that when co-training was done with just single motion primitive, it is significantly more important to make sure that the motion was aligned with that of the target, or else performance sometimes degraded (by 20% on the easy variant of clear table, see Table 10). However, this was of less importance as diversity in motion was increased as we saw a steady increase in success rate. For both variations of the clear table task, the co-training dataset with maximum coverage of the target motion provided the maximum boost to performance in the target task, leading us to helieve that robot motion is necessary to be aligned.
5.2 EFFECT OF INDIVIDUAL. DVS: RETRIEVER'S 'S PERSPECTIVE
Now that we understand which DVs are crucial for transfer learning from co-training dataset, we try to understand if these takeaways can help us build strategies for retrieving demonstrations from large roboties datasets for maximal downstream success (as described in Sec. (3.3) In this section, we design experiments to analyze each DV that can be independently retrieved from large robotics dataset, and try to understand where it matters to have maximal alignment between target and co-training distributions. Additionally, we answer counterfactual questions covering scenarios where good retrieval is not possible, and if heavy variation in all DVs in co-training is the answer to boosting downstream performance. Overall, answer the following two questions for retrieval: (1) when does it matter to have maximal alignment between target and co- training, and (2) when alignment is not possible, does having high diversity help?
7

---

Published as a conference paper at ICLR 2025
Table MimicLabs Retrieval. Task success in the MimicLabs benchmark when co-training using various retrieval strategies. Results in bold are best co-training within the sub-experiment, but only if performance boost > 5% than target only. We provide the number of demonstrations retrieved for each experiment in Table 8
Target tack
Ademes
Turget only
Retrieving relevant objectiskill
Missing relevant objectskki
7 cbVSAM
cantFuse
oliSpar
receplpur
alf
conuPose
olySpur
recepSper
+ off
56.67
bin howl
33.33
50
70
53.35
63.35
13.M
36.67
46,67
clear table
10
23.53
20
20
36
20
23.33
20
23.33
20
20
16.67
20
36.67
43.35
46.67
43.33
43.33
40
33,33
33.33
23.33
43.33
30
microwave Ieopet
10
23.33
20
23.33
16.67
13.33
16.67
10
10
6.87
13:33
20
20
30
33.35
50
33.33
36.67
53.33
20
36.67
26.67
53.33
23.33
make coffre
23.33
6.67
333
19.33
6,67
6,67
6.67
35.33
36.67
36.67
33:33
30
axp- fabt
33.33
30
53.33
36.67
40
Experimental Setup. We analyze two tasks: microwave mug and clear table. These tasks are challenging enough to benefit from co-training, evidenced by low success rates with just 10 expert demos. For each DV, we create three co-training datasets: fully aligned with the target distribution, misaligned and low diversity, and misaligned with high diversity. We compare these against baseline dataset with full combinatorial variation in all DVs, including the target distribution. All experiments use 10 target demos and 1000 co-training demos. We include highlight results in Table 2 results in Fig. 13
with full
Camera pose aligmment and diversity significantly impact performance. Aligning camera poses in the retrieved dataset substantially boosts transfer learning, as it provides the robot with relevant examples for the target viewpoint. Our results show a large performance gap when camera poses were aligned with the target (shoulder-lefttright compared to when they were completely misaligned (agent-front, When perfect alignment is impossible, high diversity in camera poses can still improve performance by preventing overfitting specific hand-eye coordinations. This suggests that data collectors should prioritize camera pose variation to enable effective retrieval for downstream tasks
Object texture alignments have limited impact in retrieval. We find that in our setup object texture did not matter at all for either alignment or diversity. That is to say that the target demonstrations were enough for the model understand what texture it needs attend to and hence is something of minimal importance to a data collector or a retriever.
crucial for performance. Aligning spatial arrangements between co-training and target datasets significantly boosts downstream performance for both objects and receptacles. This is because spatial variations directly impact the robot's action space coverage. When perfect alignment is not possible, high variation in spatial arrangements can still improve performance, especially when target arrangements aren't fully covered. Conversely, low variation in misaligned spatial arrangements can lead to overfitting and poor skill transfer.
Spatial arrangement alignment and diversity
5.3 RETRIEVER STUDY WITH MIMICLABS
Having built some understanding of which DVs matter for downstream success when collecting data and retrieving it in pedagogical setup, we now use those takeaways build strategies for retrieving data from large simulated robotics datasets. As described in Sec. 4 we created dataset containing combinatorially varied object and spatial arrangements, camera poses, and tasks, in 8 different scenes, simulating real-world setup where mutliple labs contribute to form a large composite dataset that each lab uses to boost downstream performance for target task they care about. We consider target tasks in the Mimicl.tt labs benchmark for this experiment (increasing order of hardness): bin carrot, bin howl, clear table, microwave teapot, and make coffee. Details are in Appendix F.T
Retrieval Strategies. We perform 2 kinds of retrievals on the MimicLabs dataset for each target task: (1) retrieving demos that contain the target object or skill (such as grasping a bowl, pulling a drawer), and (2) counterfactual retrieval that ignores any demos that might contain them. The latter simulates a situation where a retriever may not find any skill that is useful for their target task but still might try to use this dataset in the hopes of some performance boost. In either case, we do subsequent retrieval on camera poses and spatial arrangements to show performance boost that the retriever might gain by aligning along these DVs. We summarize these results in Table
8

---

Published as a conference paper at ICLR 2025
store screwdrriver
bin cun
baking
serve snack
pour
put marker in cup
wipe board
Figure Real World Tasks. We leverage our insights from the simulation study conduct experiments with 7 manipulation tasks in the real world.
Retrieving skills for co-training, even in the presence of overall heterogeneity, significantly boosts downstream performance. For every target task we considered, we find that retrieving and aligning skills with co-training had a significant impact on task performance. This difference was significantly large for the easier tasks with a single atomic subtask (40% boost for bin carrot) which shows that the model was able to significantly make use of the required skill when there was little heterogeneity in target motion. Moreover, this also tells us that diversity in object geometries during data collection, which allows for eventual skill retrieval, should be a useful DV for a data collector.
Aligning camera poses enables better transfer of skills. For almost all co-training experiments where skills were aligned between target and co-training. the dataset with aligned camera poses gave further boost to co-training. It was also the best-performing co-training for make coffee with 20% boost over target-only performance compared to just partial skill alignment (picking and placing coffee pod) which showed no performance boost.
Quality often matters over quantity in co-training data. Retrieving skills in our dataset usually resulted in 10 th the amount of data than when skills could not be retrieved (O(100k) demos to O(10k) demos). However, it almost always resulted in better downstream performance. Even more, a full retrieval along camera poses and object/receptacle spatial arrangements resulted in ((1k) demos, and still sometimes outperformed other strategies. This proves that visuomotor policies are inherently prone to unstable training in the presence of heterogeneous data, highlighting the importance of diversity in data collection and subsequent retrieval.
5.4 REAL WORLD EXPERIMENTS
Finally, we seek to demonstrate that the findings from our simulation study transfer over to real- world settings. We replicate the collector and retriever experiments in the real world. Notably, we perform retriever experiments on an existing large-scale dataset (DROID) and compare full-dataset co-training as adopted in prior works with different co-training strategies inspired by our simulation study.
Experimental Setup. We design our hardware setup to match that of DROID (Khazatsky et al. 2024) a Franka robotic arm with a Robotiq gripper, mounted on a mobile platform, with several externalyy mounted cameras (details in Appendix H.. All real-world policies are trained with Diffusion Policy (Chi et al. 2023) and evaluated across 20 rollouts (details in Appendix E.. We evaluated seven different tasks across the collector and retriever perspectives of analyses. One of these tasks store serewdriver was only used for collector experiments. Two tasks bin can and baking were used for both collector and retriever while four other tasks serve snack, pour, wipe board, and put marker in cup were used only for retriever. The tasks were chosen to represent a variety of manipulation capabilities including generalization. We designed our retriever tasks around objects with sufficient demos in DROID (e.g. DROID includes 4,500 task instances that involve markers). Full details are in Appendix F3
manipulation, non- prehenshee manipulation, and large spatial
5.4.1 COLLECTOR'S ' PERSPECTIVE (REAL WORLD)
Similar to our collector experiments in simulation, we evaluate models that are co-trained on various distributions of single misaligned DV, while the rest of the DVs are fully aligned. To validate collector experiments in simulation, we focus on three DVs camPose objSpat and objTex, and evaluate whether these DVs have similar impact on real-world policy learning results. Results presented in Table 13 in Appendix K across three different tasks. See Appendix foo task details.
Main findings. Our experiments reveal several key insights about so-training effects across different DVs. Camera pose alignment proves crucial, with performance improvements of 25-50% when the co-training dataset matches the target camera distribution. Interestingly, co-training with misaligned

---

Published as a conference paper at ICLR 2025
textures can still enhance performance by 40-65%. This aligns with our simulation findings and suggests that models learn relevant motions and skills without necessarily focusing on exact object colors. Additionally, increasing the diversity of spatial distributions in both target and co-training datasets significantly boosts performance, as exemplified by the hin can task where performance improved from 30% with small spatial distribution to 60% with large one. This corroborates simulation results, demonstrating that broader spatial coverage expands the action space, leading to improved task execution.
5.4.2 RETRIEVER'S 'S PERSPECTIVE (REAL WORLD) USING DROID
To validate whether study insights broadly useful, we apply insights from the retrieval setting to the DROID dataset (Khazatsky et al., 2024), large-scale dataset collected across several robotics labs, and use it for specific manipulation tasks our robot setup.
Retrieval from DROID. To facili- tate structured retrieval from DROID, we pre-processed the dataset by fil- tering demos with language instruc- tions, labeling manipulated objects, annotating object colors using VLM, and marking object positions based on gripper actions (details in Appendix [ We evaluated retrieval strategies align- ing specific DVs, comparing them against models trained target de- mos only and co-trained with the en-
Figure 5: Retriever results on a real robot. We compare the performance models trained on retrieved datasets with those co-traie co-iriined on the entirety of DROID.
Turget tuok
Aleane
Target only
DROID
elvidal
+csefwv
eolgTes
+obiSwar
Bis can
20
60
0
15
x5
pour
70
50
0
38
78
60
0
Baking
PAY map
tire DROID dataset. Results are pre-
sented in Fig. 5
Aligned retrieval can significantly improve performance. For all of evaluated tasks, that co-training with retrieval outperforms models trained on just target demos, ranging from a 5% to 80% boost in performance. The highest increases in performance always came from retrievals that aligned camera poses or spatial locations of the target objects. For instance, for the pour task, we see 25% increase in performance when co-training with aligned camera poses, with the success rate decreasing by 15% when retrieving only the relevant object without aligning camera poses or spatial locations. Qualitatively, we found that models co-trained with aligned retrieval had more robust retrying behavior and better precision.
Random co-training with large dataset can hurt performance. All of the models we co-trained with all of DROID failed to learn the tasks, showing that too much diversity in co-training datasets can potentially lead to unstable training and diminished performance. Qualitatively, we find that models co-trained on all of DROID had erratic movement and were not precise enough to complete the target tasks. We believe this happens as co-training on a large dataset might cause a model to learn motions and features that are unnecessary for completing the target task.
Retrieval is a promising direction and standardization of dataset metadata will be beneficial. We had to design and create own metadata in addition to what was found in DROID, which is non-trivial effort. When collecting large numbers of robot demonstrations, it is probably beneficial include detailed task descriptions and metadata, and way to query this, facilitate easy and accurate retrieval for future robot learning researchers.
6 CONCLUSION
Our study offers valuable insights for both collectors and users of large-scale robotics datasets. For data collectors, we highlight the importance of prioritizing diversity in camera poses and spatial arrangements, while suggesting that extensive variation in object textures may be less critical. For practitioners, we demonstrate the benefits of strategic data retrieval, showing that aligning critical dimensions between co-training and target task distributions can significantly boost performance, often outperforming training on entire diverse datasets. These findings can inform more effective data collection and utilization strategies in the community.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENT
This work was supported in part by the National Science Foundation Award 2409016 and by a gift from Autodesk. The opinions, findings, and conclusions recommendations expressed those of the authors and do not necessarily reflect the views of the National Science Foundation or Autodesk.
REFERENCES
Suneel Belkhale, Yuchen Cui, and Dorsa Sadigh. Data quality in imitation learning, 2023. URL https://arxx org/ aww/23033555 02437
Homanga Bharadhwaj, Jay Vakil, Mohit Sharma, Abhinav Gupta, Shubham Tulsiani, and Vikash Ku- mar. Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking, 2023.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.068177 2022.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shu- Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
Open X-Embodiment Collaboration, Abby O' O'Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew Wang, Andrey Kolobov, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess- Limerick, Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang. Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Felipe Vieira Frujeri, Freek Stulp. Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guangwen Yang, Guanzhi Wang, Hao Su, Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik I Christensen, Hiroki Furuta, Homanga Bharadhwaj. Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou Chakra, Jachyung Kim, Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jay Vakil, Jeannette Bohg, Jeffrey Bingham, Jeffire Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang. Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang, Jordi Salvador, Joseph J. Lim, Junhyek Han, Kaiyuan Wang, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen, Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi "Jim" Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip, Mingtong Zhang, Mingyu Ding. Minho Heo, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Ning Liu, Norman Di Palo, Nur Muhammad Mahi Shafullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R Sanketi, Patrick "Tree" Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Mart'in-Mart in-Mart in, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Lin, Sherry Moore. Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shubham Tulsiani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany,
Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Mart"
11

---

Published as a conference paper at ICLR 2025
Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar. Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vikash Kumar, Vincent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiangyu Chen, Xiaolong Wang. Xinghao Zhu, Xinyang Geng, Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yansong Pang, Yao Lu, Yecheng Jason Ma. Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang. Yonatan Bisk, Yongqiang Dou, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yue Cao, Yueh-Hua Wu, Yujin Tang. Yuke Zhu, Yunchu Zhang, Yunfan Jiang, Yunshuang Li, Yunzhu Li, Yusuke wwaswwa, Yutaka Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen Zhang, Zipeng Fu, and Zipeng Lin. Open X-Embodiment Robotic learning datasets and RT-X models. /tttss//arrxx
org/ abs / 08864 2023.
Murtaza Dalal, Ajay Mandlekar, Caelan Garrett, Ankur Handa, Ruslan Salakhutdinov, and Di- eter Fox. Imitating task and motion planning with visuomotor transformers. arXiv preprint arXiv:2305. 16309, 2023.
Maximilian Du, Suraj Nair, Dorsa Sadigh, and Chelsea Finn. aetriiir Few-shot imitation learning by querying unlabeled datasets, 2023. URL /tttps://rxxv..orgg/////// 2304
08742
Hao- Shu Fang, Hongjie Fang. Zhenyu Tang, Jirong Liu, Junbo Wang. Haoyi Zhu, and Cewu Lu. Rh20t: A robotic dataset for learning diverse skills in one-shot. In RSS 2023 Workshop on Learning for Task and Motion Planning, 2023.
Zipeng Fu, Tony Zhao, and Chelsea Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401.02111 2024.
Jensen Gao, Annie Xie, Ted Xiao, Chelsea Finn, and Dorsa Sadigh. Efficient data collection for robotic manipulation via compositional generalization. arXiv preprini arXiv:2403.05/ 10, 2024.
Malik Ghallab, Craig Knoblock, David Wilkins, Anthony Barrett, Dave Christianson, Mare Friedman, Chung Kwok, Keith Golden, Scott Penberthy, David Smith, Ying Sun, and Daniel Weld. Pddl - the planning domain definition language. 08 1998.
Joey Hejna, Chethan Bhateja, Yichen Jian, Karl Pertsch, and Dorsa Sadigh. Re-mix: Optimizing data mixtures for large scale imitation learning. 2024. URL https:://axxiv.orggg//o 2408
14037
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pp. 99121002. PMLR, 2022.
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.129. 2024.
Diederik Kingma and Jimmy Ba. Adam:
method for stochastic optimization.
CoRR, abs/1412.6980, 2014. URL htttt::t///att semanticsch cscho ar waa..org/CorpuuTTT 6628106
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1-40, 2016.
Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, qiang liu, Yuke Zhu, and Peter Stone. LIBERO: Benchmarking knowledge transfer for lifelong robot learning. In Thirt--eventth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023a. URL https
//openreview net forum www.mmdddddxd-ddxde1..
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved basclines with visual instruction tuning, 2024. Rttt.httts::/rrttte www.:iiwwwww.rreeeeooomoe abs 7231 03744
12

---

Published as a conference paper at ICLR 2025
Huihan Liu, Soroush Nasiriany, Lance Zhang, Zhiyao Bao, and Yuke Zhu. Robot learning on the job: Human-in coeen-in-thh-att -onna--n-thhe-oop autonomy and learning during deployment. In Robotics: Science and Systems (RSS), 2023b.
Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots in real time. arXiv preprint arXiv:2210.06407, 2022.
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung. Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, Silvio Savarese, and Li Fei-Fei. RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation. Conference on Robot Learning, 2018.
Ajay Mandlekar, Danfei Xu, Josiah Wong. Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning (CoRL), 2021.
Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations, 2023. URL
/2tpr///abbb23110.o 17596
Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. Robocasa: Large-scale simulation of everyday tasks for generalist robots. In Robotics: Science and Systems (RSS), 2024.
Wilbert Pumacay, Ishika Singh, Jiafei Duan, Ranjay Krishna, Jesse Thomason, and Dieter Fox. The colosseum: benchmark for evaluating generalization for robotic manipulation. arXiv preprint arXiv:2402.08191 2024.
Vaibhav Saxena, Yotto Koga, and Danfei Xu. C3DM: Constrained-Connex Conditional Diffusion Models for Imitation Learning. Transactions on Machine Learning Research, 2024. ISSN 2835- 8856. URL
net wtt.://oooo..dadaiieeeeeeooooce
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation, 2022.
05451
Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Martin-Martin, Fei Xia, Kent Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, C. Karen Liu, Silvio Savarese, Hyowon Gweon, Jiajun Wu, and Li Fei-Fei. BEHAVIOR: benchmark for everyday household activities in virtual, interactive, and ecological environments. CoRR, abs/2108.03333 2021. URL. httpt::////aaa
borg/bb//111 2188 03332
Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Fstruch, Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning scale. In Conference on Robot Learning (CoRL), 2023.
Annie Xie, Lisa Lee, Ted Xiao, and Chelsea Finn. Decomposing the generalization gap in imitation learning for visual robotic manipulation. arXiv preprint arXiv:2307.03365, 2023.
Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. arXiv preprint arXiv:2010.14400, 2020.
13

---

Published as a conference paper at ICLR 2025
A OVERVIEW
The Appendix contains the following content.
Defining the four cases of dataset distributions (Appendix B different cases of target and co-training distributions considered in this study.
this section describes the
Description of each dimension of variation (DV) in MimicLabs (Appendix C this
section describes the different dimensions of variation we consider in our combinatorial data generation pipeline as well for study.
Task Specification via BDDL (Appendix D): this section specifies details about the param- eterization for different procedurally generated DVs in MimicLabs.
Training and Evaluation Details (Appendix E): this section lists out the policy hyperpa- rameters for BC-RNN and Diffusion Policy, as well as other co-training details.
Task Descriptions (Appendix this section provides descriptions of different target tasks we experimented on in MimicLabs and on a real robot.
Details about Retriever Experiments (Appendix G this section specifies the number of demos used for each retriever experiment in MimicLabs.
Real Experiment Setup (Appendix H): this section describes our hardware and setup for the real-world experiments.
Real Collector Datasets (Appendix m] this section elaborates on the data collection for the
real-world collector experiments.
DROID Retrieval and Metadata Processing (Appendix) D describes how retrieval was
performed on DROID for the different DVs.
Additional Results (Appendix K): this section provides additional results.
14

---

Published as a conference paper at ICLR 2025
B DEFINING THE FOUR CASES OF DATASET DISTRIBUTIONS
Given target and co-training demonstration distributions Dr and Dc. S(.) representing the support of distribution and [S(-)| the measure of its size, define four cases comparing the supports of these distributions and sizes:
1. not-diverse and misaligned:
and ((((()))))
(k)
)
=
2. diverse and misaligned: | (9 ) > ) and S
= =
3. diverse and aligned: S(2C) ( > |S(2))) | and S(2,)) c S(
4. not-diverse and aligned (perfect alignment): |S(2g)) = | and S(2 c ((
We illustrate these cases comparing Dr and De along DV in Fig. 1-4 starting from top-left going in clockwise order. As shown in our experiments, cases and (diverse not, with misalignment) are important cases to study from a collector's perspective where the goal is to find out which DV the collector should add diversity in given the worst-possible scenario i.e. misalignment, Cases 3 and are important from retriever's perspective given the assumption that large-scale robotics datasets contain high diversity along all DVs. Subsequently, study seeks to understand if retrieving datasets with perfect alignment between target and co-training datasets essential for boosting performance in the target domain.
15

---

Published as a conference paper at ICLR 2025
C DESCRIPTION OF EACH DIMENSION OF VARIATION (DV) IN MIMICLABS
Camera Pose (camPose): in the MimicLabs dataset we vary the camera pose anchored at the center of the table and also such that always points towards the center of the table while its position is varied in spherical coordinates (physics convention). Camera positions are grouped into 5 possible bins: leo and shoulder-right w.r.t the robot, and agent-left, agent-right, agent-front w.r.. an agent. Each bin in the MimicLabs dataset is comprised of a variation of 15 degrees in the polar angle and degrees in the azimuthal angle.
Figure 6: Illustrating camera distributions in MimicLabs discretized into 5 bins.
Object Texture (objTex): Each task has a target object and we vary the object color texture procedu- rally using generated fractal noise.
Table Texture (tableTex): We vary the color/texture of the table surface procedurally using fractal noise.
Targer Object Spatial Arrangement (objSpatial): The placement of the target object is randomly initialized on the table and we vary the size of this reset range.
Receptacle Spatial Arrangement (recepSpatial): For tasks that involve placement of an object into receptacle, we randomly initialize the location of the receptacle. We vary the size of this reset range.
Background Scene (scene): Simulation tasks take place in of 8 visually distinct lab environments. Target tasks for real world experiments take place in one lab is co-trained on DROID (Khazatsky ct al. 2024) data from 52 buildings.
Motion Primitives (motion): Different tasks are composed of different motions we segment these as pick, place, push and pull 1. in the data composition experiments from the collector's perspective.
16

---

Published as a conference paper at ICLR 2025
D TASK SPECIFICATION VIA BDDL
Our task specification in the MimicLabs dataset follows the following parameterization for sampling multiple DVs initialization:
spatial arrangements: specified using union of multiple bounding boxes table-top.
These bounding boxes are represented as 4-tuple specifying the start and end (x.y) locations in the robot's base frame.
camera poses: specified using union of ranges in spherical coordinates (physics convention) in a coordinate frame with the origin at the center of the table. Each range represented as 6-tuple specifying the (r, 0, () (i.e. radial, polar, azimuthal) ranges for the camera placement in a frame anchored at the center of the table.
textures: specified HSV ranges to either jitter a base texture retrieved from file into the specified range (used for table), or fully generated as fractal noise within the specified HSV range (used for objects).
demonsfration:
specified predicates that determine what motion primitives, and in what order, are useful for task completion. These predicates also split the task in an object-centric fashion that allows for large-scale reuse of human demonstrations across variety of tasks that follow same primitives (using MimicGen (Mandlekar et al., 2023)
D.1 TEXTURES IN THE MIMICLABS DATASET
sigma=5
sigma=10
sigma=20
(a) Controllable texture generation in MimicLabs (red huc).
(b) Procedurallly-genrraeed fractal textures used in Collector's 's and Retriever's 's analysis.
(c) Sample irregular textures available to use in MimicLabs.
Figure 7: Procedural texture generation in MimicLabs. Fractal textures were used in the Collector's and Retriever's analysis when controllably generating target and co-training datasets with varying object textures. We also provide various other irregular textures that can used for fabrie, cookware, appliances, etc. for creating diverse scenes and tasks within MimicLabs.
17

---

Published as a conference paper at ICLR 2025
E TRAINING AND EVALUATION DETAILS
Simulation Policy Training. While there are various approaches to imitation learning in robotics, one of them being waypoint prediction in the robot's base frame and relying motion planner to fill actions (Zeng et al., 2020) Shridhar et al. 2022 Saxena et al. 2024), we choose an end-to-end approach policy execution predicting sequences of actions (at 20Hz) and executing them in a closed-loop fashion. We use BC-RNN (Mandlekar et al.) 2021 and Diffusion Policy (Chi et al., 2023) to train imitation learning policies our experiments, and use ResNet18 as the visual backbone. We also condition our policies on language instructions using FiLM layers in the vision backbone. We train all models for simulation for 500 epochs (250k gradient steps) using the Adam optimizer (Kingma & Ba 2014) using learning rate l 4. All our co-training experiments used 10 target demonstrations and 1000 co-training demonstrations (unless otherwise specified). We evaluate checkpoints every 50 epochs and report the peak success rate. Each evaluation is obtained by rolling out the policy 30 times.
off-the-shelf
Table 4: BC-RNN Policy Hyperparameterss for Simulation Experiments
batch size
32
sequence length
10
optimizer
Adam
learning rate
le-4
image encoder
ResNet18
image resolution
(128, 128)
Table 5: BC-Transformer Policy Hyperparameters for Simulation Experiments
batch size
32
context length
10
embed dim
512
num layers
6
num heads
8
embedding dropout
0.1
attention dropout
0.1
optimizer
Adam
learning rate image encoder image resolution
le-4
ResNet18
(128, 128)
Real-World World Policy Training and Evaluation Details. We use Diffusion Policy (Chi et al. 2023) for all our real-world experiments. Our low-dimensional observations for all models were the end-effector position and orientation (represented quaternion). For collector experiments, camera (out of the four external options) for the image observations and for the retriever experiments, used both of the shoulderview cameras. We random cropping on image observations before passing them into a ResNet- 18 visual encoder and apply a Spatial Softmax (Levine et al. (0066) on the encoder outputs. These features are concatenated with the low-dim observations, processed by an additional observation processing MLP and passed into a U-Net diffusion head which predicts actions. The hyperparameters for our model are listed in Table
used single
For evaluation, we trained our models for 600 epochs for the collector experiments and for 300 epochs for the retriever experiments. Empirically, we found no difference in performance between models trained for 300 epochs and those trained for 600. We perform 20 rollouts with each model and report the success rates.
E. DATASET RE-BALANCING
Throughout our study, assume No > Nr, ie.. the number of co-training demonstrations is much larger than that collected by the practitioner for the target environment and task. Training BC policy using naive combination of these datasets could cause the policy to ignore the
18

---

Published as a conference paper at ICLR 2025
Table 6: Diffusion Policy Hyperparameters for Real Robot Experiments
batch size
128
observation horizon (To)
2
action horizon (Ta)
8
prediction horizon (Tp)
16
diffusion method
DDIM
optimizer
Adam
learning rate
le-4
image encoder image resolution
ResNet18
(128, 128)
target dataset altogether while still minimizing its training objective (Hejna et al. 2024), resulting in unstable training and bad downstream performance (Du et al., 2023). Therefore, we instead create a dataset that samples from the distribution Dr,c(wu - wDr + (1 WDD where w E [o, 1]. We implement this using sampler that creates training batches by retrieving demonstration trajectories with probability from Dy and (1 w) from De- We represent this weighted combined dataset as Drrc(w), For all our experiments, we keep w 0.5 and try ablations with w 0.3 and 0.7 on the retriever experiment (see Fig. 13
19

---

Published as a conference paper at ICLR 2025
F TASK DESCRIPTIONS
F.1 MIMICLABS TASKS
The Mimiclabss dataset contains over 3000 task instances encompassing different motion primitives for solving different tasks, varying camera poses, objects, table textures, spatial arrangements, and background scenes. We summarize the tasks included in the benchmark below:
1. pick and place it in the bin (7 instances per lab)
2. open (2 instances per lab)
3. close (2 instances per lab)
4. open X, pick Y and place it in X (14 instances per lab)
5. pick x, place
and close Y (14 instances per lab)
6. turn stove
7. turn off stove
8. make coffee
where X can be replaced by one of 7 distinct objects available in each lab for data collection and policy evaluation, can be a drawer or microwave, distinct instances of which are available each lab. In total, there are ~29 unique task instances in each lab, with skill-level overlap designed to test positive retrieval strategies. Additionally, for each task instance, we created multiple variations in camera poses (5), object and spatial arrangements (~90 combinations), which create 450 task instances in each lab, totaling to over 3000 instances across labs.
The following tasks from MimicLabs were used for experiments in Section 5.3 in increasing order of hardness:
bin carrot: easy binning task with non-precise object.
bin bowl: binning task with harder and multi-modal grasping strategy in expert demos.
clear table: the robot should pull open the top drawer the cabinct and the bowl in it.
microwave teapot: the robot should pick-place teapot into the microwave and close the microwave door.
make coffee: the robot should p ck up the coffee pod and 1 ace it in the coffee machine and close its lid; only one lab has a coffee machine and so we cannot retrieve any skills pertaining to placing the pod in the coffee machine or closing its lid.
Table 7: Task instances evaluated in the Mimicl benchmark results with corresponding camera pose and spatial arrangement configs.
Task
camPose
objSpal
recepSpat
bin carrot
shoulder-left
left 20cm x0em
right
bin bowl shoulder-right
right 20cm 20cm
left
clear table agent-front
left 20cm x 20cm
right
microwave teapot
shoulder-right
right 20cm x 20cm
left
make coffee shoulder-right
center 20cm
left
F.2 TASK DETAILS FOR COLLECTOR' EXPERIMENT IN SIMULATION
When analyzing dataset composition from a collector's 's perspective, we construct multiple variations of the clear table task. The baseline variation in the task consisted of a fixed agent- front camera pose, single hue (red) object textures, single hue (wooden/beige) table texture, and small (10cmx 10cm) spatial arrangement for the object around the center of the table.
20

---

Published as a conference paper at ICLR 2025
F.3
REAL ROBOT TASKS
bin can
baking
serve snack
pour
put marker in cup
wipe
The green and red bounding boxes define the reset distributions for the object and the receptacle in each retriever task.
Figure 8:
For real-world experiments, evaluate variety of challenging and diverse table-top manip- ulation tasks that comprise multiple motion primitives and task horizons. Our goal was to ensure that our takeaways were broadly applicable different end-to-end robot learning tasks. The reset distributions for the different objects in our retriever experiments are shown in Figure 8) During data collection, we uniformly distributed the object and the receptacle in their respective reset distributions. In evaluation, we uniformly sampled 20 poses for the objects and fixed these poses for testing all variations of each task. Below, we provide the descriptions and success criteria for each real-world task:
store screwdriver: this medium-horizon task consists picking up screwdriver, placing drawer and closing the drawer. The location of the screwdriver varies but the drawer is fixed. Success defined as the screwdriver in the drawer and the drawer closed.
bin can: this task involves one motion primitive where the robot must pick up can from the table bin. The location of the can varies, and the location the bin fixed. Success is defined as the can sitting upright in the bin.
and place
baking: this a challenging long-hone horizon task where the robot has to pick up a bowl and place inside toaster oven, push the oven tray, and close the oven. The location of the bowl varies but the oven fixed. Success is defined as the bowl on the tray and the oven completely closed.
serve snack: the robot must pick up red snack packet and put the snack and the plate vary. Success is defined as the snack completely on the plate.
plate. The locations of both
pour: the robot has pick up bowl containing beans, pour them onto plate, and return the bowl to the workspace. Success defined as all beans in the plate and the empty bowl being returned to the workspace.
put marker in cup: for this task, the robot should pick up marker and place it in cup. The locations of both the marker and the cup vary. Success is defined the marker in the cup.
wipe board: the robot should pick up a cloth towel and use it to wipe off a 5cm mark on whiteboard. The location of the towel varies, but the whitehoard and the mark fixed. Success is defined as having at least half of the mark erased.
21

---

Published as a conference paper at ICLR 2025
G NUMBER OF CO-TRAINING DEMOS FOR RETRIEVAL EXPERIMENTS ON MIMICLABS
Table 8: Number of demonstrations retrieved for each retrieval experiment the MimicLabs dataset.
Targel fask
Retriesing relevant objectsskill
Missing relevant objectskkill
abjiákil
opy
recepSpuut
nu-obysshill
cam/usr
objSppur
recepSpol
bin carrot
24000
4800
6000
12000
1200
153800
30400
38600
77200
7600
77000
clear table
46000
9200
22000
23000
3000
161800
32400
40600
81200
8000
microwave teapol
42200
8000
7400
14800
800
166600
32800
41400
82800
8200
22

---

Published as a conference paper at ICLR 2025
H REAL EXPERIMENT SETUP
We design our hardware setup to match that of DROID (Khazatsky et al. 2024 a Franka robotic arm with Robotiq gripper, mounted on mobile platform, with several externally mounted cameras. Notably, our specific robot setup did not participate in the DROID dataset collection, minimizing the chance of data pollution. We use a Franka Research 3 7-DoF robotic arm with a Robotiq 2F-85 gripper. The robot is mounted on mobile, height-adjustable platform. As illustrated in Figure 9 attached to the platform are two Zed 2 stereo cameras, and we also attach Zed Mini camera the wrist of the robot. Additionally, for our collector experiments, we attached two RealSense D435 cameras the robot workspace for a total of four external cameras and one eye-in-hand camera. During data collection, all five camera streams are recorded and synchronized to the robot's actions. We additionally record robot proprioceptive information which includes joint positions, gripper positions, and end effector poses. The action space of the robot consists of an end-effector position (3- d--imensionnl), rotation encoded in axis angles (3-dimensional), and gripper action (open/close). For teleoperating the robot and providing demonstrations, we use Meta Quest 2 headset and controller.
Figure 9: Hardware setup. The red circles show the camera set-up in the real experiment.
23

---

Published as a conference paper at ICLR 2025
1 REAL COLLECTOR DATASETS
Co-training and target demos: for the bin can and baking tasks, we used 20 target demos for every model while for store screwdriver we used 10 target demos. All models were co-trained with 100 demos.
Baseline dataset: the baseline co-training dataset had minimal variation in each DV, namely, consisted of single camera view, a single texture of the target object, and small reset distribution (25x25cm).
Co-traine Co-training dataset construction: the co-training datasets were created by sampling evenly from the ranges of partieular DV. For example,
camPose experiment where the co-training dataset was comprised of two cameras that are not target camera, the co- to-training dataset would be constructed by selecting 50 demos from one camera and 50 from the other, for total of 100.
The per-DV variation for our collector experiments is:
1. camPose: : we select one of the 4 external cameras as the target camera during evaluation and co-train models using various combinations of the cameras.
2. objTex: pick color of the target object for evaluation, and co-train models with demos of varying target object textures. Note that we do not ablate on material properties of these objects. We also keep our lighting conditions consistent within each dataset to remove any variations in object visuals due to lighting-matenn material interactions.
3. objSpat we pick spatial distribution for the target object during evaluation and co-train models with demos where the target object has varying spatial distributions. The different co-training spatial distributions are concentric boxes that increase in size until they cover the target spatial distribution completely.
DV data collection: to build the datasets for each DV, we had to vary the setup during demonstration collection. Specifically, for:
1. camPose collected 100 demos of the task, with small reset distribution and single texture. Since all four external cameras were streaming simultaneously. we could build co-training datasets with combinations of the different cameras through post-processing.
2. objTex used the data collected in 1. for baseline dataset. We then collected an additional 50 demos with a second color to build our second co- co-training dataset (50 from each) and on with third color. All demos were collected with small spatial distribution.
3. objSpal again, we used the data collected in 1. for our baseline dataset. We then collected enough demos to cover a medium reset distribution of 38x38cm and even more demos to cover large reset distribution of 50x50em. Each co-training dataset (small, medium, large) consisted demos sampled evenly from each distribution. The color of the target object remained fixed and trained and evaluated single camera pose.
24

---

Published as a conference paper at ICLR 2025
J DROID RETRIEVAL AND METADATA PROCESSING
In order to perform retrieval on DROID, we had to create additional metadata that would allow querying along specific DVs. Here we explain the creation of this metadata:
DROID has up three different language instructions describing the task executed in each demonstration. To identify the target object from the language instructions, we first merged the multiple instructions into a single coherent command for each demo. After combining the instructions, we applied syntactic parsing, focusing specifically on the direct and indirect objects of the action verbs in the combined instruction. Once the direct and indirect objects were extracted, we performed agglomerative clustering, using cosine similarity between the word embeddings of these objects. The object with the highest similarity within the primary cluster designated the target object. If multiple objects were mentioned, priority was given to the first occurrence in the instruction.
Target Object:
to get the location the manipulated object, we used the gripper state of the robot heuristic. We determined when the gripper closed and used the position of the end-effector at this timestep as the position of the manipulated object. If the gripper closed and opened multiple times during the demo, used the first gripper close for the position the object. The gripper state was averaged over a window of 15 timesteps to filter out accidental gripper closes/opens.
Object Spatial:
Object Colors: to classify the color of the target object in the demos, we first saved the initial image frame from each demo. We then used the LLaVA v1.5 7b model (Liu et al. 2024) to detect the color of the target object from this image. The prompt provided to the model was: "USER: What's ' the color of the <target object> in the image? Answer in just adjective word. ASSISTANT.".
The different types of retrieval we performed on DROID are: obj/skill, obj/skill camPose obj/skill + objTex, and obj/skill + objSpat. The process of retrieving these DVs is as follows:
obj/skill: using the language instructions in DROID, we filtered out demos where our target object was being manipulated.
obj/skill camPose from demos containing the target object, we further filter the demos where the camera positions are within 20cm of the target camera in the X- and Y-axes, and 10cm in the Z-axis. This is done using the per-demo camera extrinsic metadata provided in DROID.
obj/skill + objiex: : after filtering out demos with the target object, we further filter by selecting only demos where the color is the same as our testing setup.
obj/skill objSpat: from demos containing the target object, sclect the demos where the spatial distribution of the object is a 60x60x30cm cuboid centered at the middle of our testing distribution. The maximum size of our testing distribution was 50x I lem the retrieved spatial distributions would always cover the testing distributions.
Examples of demos after retrieval are shown in Figures TO
and
25

---

Published as a conference paper at ICLR 2025
Target
Retrieved
Figure 10: Retrieved cotraining demos for the wipe board task with the camera pose aligned. We can see that the spatial locations and textures of the objects in the demos are not necessarily aligned with target object. However, the camera pose is the same.
Target
Retrieved
Figure 11: Retrieved cotraining demos for the serve snack task with the object spatial location aligned. The camera poses and textures seen in the demos are not the same as our target task, but the location of the object (in front of the robot and level with its base) is similar.
Target
Retrieved
Figure 12: Retrieved cotraining demos for the pour task with the object texture aligned. The camera poses and spatial locations of the target object are not the same, but the color is aligned.
26

---

Published as a conference paper at ICLR 2025
K ADDITIONAL RESULTS
Table 9: Analyzing the effects misaligned and diversities in DVs from a collector's perspective. Success rates the make coffee task with different variations in co-training distributions. Target variations are perturbations to one DV while others stay fixed and perfectly-aligned with the baseline distribution. Success rates highlighted when boost in performance is 10% compared to baseline.
DV w/ turget-cotrain misalignment
Target only
Co-training with different DV distributions
Baseline
camPose
ohj
table Tex
objSpat
camPose objTex tableTex
16.67
43.33
3.33
3.33
16.67
6.67
6.67
36.67
36.67
43.33
10
16.67
10
26.67
30
33.33
objSpal
16.67
10
20
Table 10: Motion diversity in co-training. We tested two variations of the clear table task: easy (put the bowl in the drawer and close it) and hard (open the drawer and put the bowl in it). The easy variant requires learning the push and variant requires pull and pacckecccToop Co-training datasets (left to right) are ordered in increasing order of motion diversity they bring for co-training (500 demos per primitive), with the most diverse dataset covering all the motion primitives needed to solve the target task but with increased heterogeneity in robot motion.
motion primitives, while the hard
Target variation
Target only
Co-training with different motion primitives
pull
pushh
pull
pushe pioxPiaceBisker
pall rgust ckP aaceRsket pickf Tope Tower
BickDlaceBasket
36.67
63.33
83.33
tgush waw.kkk Dao
pickPlace BackPaattB
plexPlaceBasker
pull+ biekeleeellopiroeer
16.67
27

---

Published as a conference paper at ICLR 2025
Table 11: MimicLabs Retrieval. Task success in the MimicLabs benchmark when co-tra co-training using various retrieval strategies, using BC-Transformer policy. Success rates averaged over 30 rollouts, and max across 500 epochs of training. Results in bold are best co-training within the sub-experiment. We provide the number of demonstrations retrieved for each experiment in Table 8
Target lask
Ademns
Turget only
Retrieving celerant objectiskill
Missing relevant objectham
recqutpor
obj5put
bin carret
53.39
90
91.33
93.33
50
41.33
40
56.67
table
43.33
13.33
33.33
microe troped
20
13335
13.33
16.67
30
16.67
23,33
6.57
3,33
10
make coffee
50
6.67
10
20
10
16.67
13.33
10
20
3.33
3,33
13.33
Table 12: MimicLabs retrieval in pretrain- finetunfeeeeeee setup. Task success in the MimicLabs bench- mark when pre-training using various retrieval strategies and fine-tuning on the target Success rates averaged over 30 rollouts, and max across 500 epochs of training. Results in bold are best co-training within the sub-experiment. We provide the number of demonstrations retrieved for each experiment in Table 8
Target tack
Ademos
Turget only
Retrieving relevant objeetkkll
Missing relevant objectikiil
odyishii
comPose
abilpur
recopSppr
oil
abyynkilll
camPorr
ofjSpar
recepSpor
bin carrut
10
76.67
83.33
86.67
83.33
73.33
63.33
76.67
bowl
73.33
66.67
66.67
clear
46.67
46.67
Table 13: Evaluating the importance of alignment and diversity when co-training for target task in the collector setup (real-world). The grey circles above each column represent the entire possible distribution for that DV. The green circle the target distribution and the red circles are the distributions found in the cotraining datasets. For the camPose setup, this involves co-training with or cameras that are not the target camera and then co-training with all the cameras as well as only the target camera. For objTex experiments, we cotrain with 1/2/3 colors that are all different from the target color. For the objSpat experiments, we increase the reset distribution the object in the co- -o-iariinng datasets (small medium < large) until the co-training dataset covers all of the target reset distribution.
obyTex
abjSput
store screwdriier
10
5
5
15
45
30
baking
40
60
95
100
95
45
75
80
85
28

---

Published as a conference paper at ICLR 2025
Stroess ratee on the miceonne--in (reing RC-NNN Polleg) taok wring differene retrinsed detesets for uxtraiiing Cueaann Powe
Clipst Boctage
Talk Todtege
Okject Spatie)
Bonattals Spetal
Sunse Rat
Succiso miles the micronnttiiinngg (using Dilicaian Pallcy) Lack using Jilleern aetiionn dalantts far costesining
Groen Pree
Chijei Toster
Toibe Sestero
Objest Spatial
Breedaal Speiil
Sancso Bato
Smeersss the elear-abble (using BCARNN Polley" task using differen retriesed dntrocts costriining Cunoea Poee
Testune
Olsjst
Beceptac Spalial
Soroe Dator (5)
Success mntow on cour-table (reding Dilfiscon Palicy' took ladngg different retrioned dalasata rostraining
Canera
Chipsr
Textum
Obopct Spatei
Bevetac Spatiel
Stroee Coto 150
1s0
Arant's
Figure 13: Results showing success rates on two tasks using BC-RNN and Diffusion Policy for different retrieved datasets along all considered DVs. The DVs (left to right) are camera pose, object texture, table texture, object spatial, and receptacle spatial arrangements. The three bars (left right) for each experiment use = 0.7, = 0.5, and w = 0.3 respectively (see Appendix E. dataset re-balancing details). Portions of bars in shades of green shows performance improvement over baseline co-training, while hollow red portions depict performance degradation.
29


Reviews:
[{"review_id": "LEG4TLiZr0", "reviewer": "Reviewer_Gs6e", "summary": "This paper conducts a comprehensive empirical study on key factors in scaling robot manipulation datasets, offering valuable insights for both collectors and users.", "questions": "1.\tHow many tasks are included in the MimicLabs dataset? Different sections use various tasks from the dataset, but I am unsure how many tasks are there in total. According to the appendix, there seem to be five, but this is not shown in the main text or images. \n\n2.\tWhat exactly are the four co-training settings in Figure 2? There seems to be no exact definition of these settings, and different experiments actually use different numbers/types of co-training, making it harder to understand. It would be helpful to have a better description of all co-training settings. \n\n3.\tWhy are all the co-training datasets in section 5.1 misaligned with the target? Can’t it contain the target distribution (as I saw in some experiments in other sections)? \n\n4.\tIn section 5.3, the authors have an important finding that full retrieval might be less effective than other strategies using less data. However, a more common strategy used in other areas is to first pretrain on the full dataset and then fine-tune on the target dataset. I am wondering have the authors tried this setting?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. This paper collects a large-scale robot manipulation dataset in simulation with diverse dimensions of variations to enhance understanding of each factor’s impact.\n\n2. This paper provides practical suggestions on how to collect and leverage the data, and conducts real-world experiments on the DROID dataset to verify its effectiveness.", "weaknesses": "1.\tIn section 5.1, from the collector’s perspective, there’s only one task, “clear table”, which seems insufficient to draw conclusions (especially given that more tasks are actually used in the following sections).\n\n2.\tAs illustrated in the paper, object geometry is vital to robot data. However, it is a pity that it is not included in the DVs of the experiments.\n\n3.\tAlthough the MimicLabs dataset generates different textures for the experiments, the textures are still primarily pure colors according to the appendix. More complex textures, such as plaid patterns or irregular textures, aren’t included.\n\n4. There are a large number of experiments and various settings in this paper, but some of them are not explained clearly, which might be confusing (see the questions for details).\n\nOverall, this paper conducts a nice trial to investigate the key factors in large-scale robot learning and provides many useful results and practical suggestions. However, more thorough experiments and more accurate explanations of experiments would make the conclusions more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pD5HEahsmJ", "reviewer": "Reviewer_J7tN", "summary": "This paper presents a systematic study of what factors matter most in large-scale robotics datasets for manipulation tasks. The authors introduce MimicLabs, a framework for generating controlled datasets with various dimensions of variation (DVs), and analyze dataset composition from both collector and retriever perspectives. They identify camera poses and spatial arrangements as crucial factors while finding object textures less important. The study's insights are validated on real robots and existing datasets like DROID.", "questions": "1. How do you handle cases where a demonstration might be valuable according to one DV but less useful according to another? Is there a principled way to make such trade-offs?\n2. How do your findings generalize across different imitation learning approaches? Would VLA or other transformer-based policies show similar sensitivity to DVs?\n3. How does your conclusion about texture alignment being less important translate to real-world scenarios with complex lighting conditions?\n4. Given that similar objects often share manipulation strategies, why not consider 'picking a cup' and 'picking a bowl' as the same skill? Would grouping such similar skills during retrieval improve policy performance or potentially harm it?\n5. Since DROID contains diverse task variations across 52 buildings while your simulation covers 8 scenes, could the limited scene diversity affect your DV importance rankings? Specifically, would more diverse scenes reveal additional important DVs?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper addresses the crucial question of dataset composition in large-scale robotic manipulation through a systematic analysis. The two-perspective approach (collector and retriever) offers practical insights for both dataset collection and utilization. The authors conduct comprehensive experiments and clear validation in both simulation and real-world settings. The definitions and methodology based on \"DV\" are novel.", "weaknesses": "1. Results heavily rely on MimicGen's simplified environments with basic textures and lighting. The conclusion that \"texture alignment is less important\" may not hold in real-world scenarios like those in DROID with complex lighting, shadows, and material properties (as in computer vision, lighting effects are often considered as a special kind of texture). Limited scene diversity compared to real datasets like DROID (8 simulated scenes vs. 52 real buildings)\n\n2. Limited Policy Evaluation: Experiments primarily use Diffusion Policy, with only BC-RNN as comparison in MimicLabs. Performance gains might be attributed to Diffusion Policy's characteristics rather than dataset composition. Missing evaluation with modern approaches like VLA or transformer-based policies.\n\n3. No clear metric for measuring skill similarity (e.g., is \"picking a bowl\" similar to \"picking a cup\"?) Lack of systematic handling of compound skills. Binary treatment of skill alignment without considering partial similarity.\n\n4. Overlooked DV Interactions: No consideration of cases where a tejectory has conflicting values across different DVs (e.g., good camera pose but poor spatial arrangement). Missing analysis of trade-offs between different DVs when retrieving demonstrations. No principled approach for weighing the relative importance of different DVs when they conflict.\n\nThe limitations affect the generalizability and practical applicability of the paper's conclusions, particularly when scaling to more complex real-world scenarios or different learning algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eDbHejpxST", "reviewer": "Reviewer_D5QM", "summary": "This work conducts a thorough investigation on creating and utilizing robotics dataset for co-training robot policy with imitation learning. It demonstrates the most useful types of diversity for dataset collection and proposes insightful yet effective retrieval methods from existing datasets to improve task performance.", "questions": "1. In Table 1, it can be confusing why training and testing on only target distribution will yield such poor results, since overfitting to a rather small range of variation seems not to be that hard in imitation learning.\n2. In Table 1, I am confused why co-training with varied camera poses can improve policy performance to such a great extent, because there are only several (~5) poses and they are very different. Maybe some videos and extra explanations can help clarification.\n3. In Table 2 \"Object spatial\" part, why larger co-training distribution produces lower performance?\n4. In Figure 5, I am curious whether the improved performance is attributed to the co-training setting or alignment between target and training task. For example, how is the performance if the policy is trained with retrieved Droid data without target data?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This work tackles a popular yet significant problem concerning usage of large-scale robotics dataset.\n2. The authors conduct extensive experiments on collector and retriever perspective with various DVs.\n3. The paper is well-written and easy to follow.", "weaknesses": "1. The experiments in Section 5.1 only consider one task *clear table*, making the conclusions less convincing. Results on more simulation tasks or clarification on the representativeness of this task should be added.\n2. The co-train setting in this paper largely focuses on using generated/retrieved data from the same task as the test task (except for some results in Table 3). However, aligning the task setting between the target task and the tasks in a public large-scale dataset is difficult in most cases, especially for real-world tasks.  More experiments and analysis on co-training with different tasks can add to the practicality of this work.\n3. Some of the results may be confusing. Please refer to the Question section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nb48UhjCly", "reviewer": "Reviewer_CfcJ", "summary": "This paper explores the key factors that influence the success of robot manipulation policies trained on large-scale datasets. It introduces a synthetic data generator, MimicLabs, to emulate common variations in real-world datasets such as camera poses and object arrangements. The study focuses on identifying the types of diversity that should be emphasized when collecting new datasets or retrieving relevant demonstrations from existing ones to maximize downstream task performance. Using simulation and real-world tasks, the authors demonstrate that factors like camera pose and spatial arrangement are critical to both dataset diversity and alignment. These insights are validated through experiments on the DROID dataset, leading to improvements in policy learning by up to 70%.", "questions": "1. Have you tried using only wrist cameras? One question that could have been explored more thoroughly is whether using dynamic wrist cameras, which vary poses constantly, would act as a strong form of data augmentation and mitigate the problem of overfitting to fixed camera poses. If this technique were used, it could potentially reduce sensitivity to specific camera poses and improve generalization.\n2. Generalization beyond single variation dimensions: The current analysis focuses on individual DVs (like camera pose or object arrangement), but in the real world, robots often face scenarios where multiple variations occur simultaneously. Have you considered experiments that explore the interaction effects between multiple DVs, and if so, how do the insights change?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper tackles an important problem—optimizing large-scale datasets for robotic manipulation—by systematically analyzing how various dimensions of variation (DVs) in the data affect learning. It provides a novel approach through its MimicLabs synthetic data generator, which offers a controlled environment for testing different dataset compositions, addressing a key gap in robot learning research.\n2. This work has the potential to significantly impact the future of large-scale robot dataset collection and utilization. By offering actionable insights on how to prioritize data collection and improve policy learning through targeted dataset retrieval, it lays the foundation for more efficient robot learning pipelines, reducing the cost and complexity of data collection efforts.", "weaknesses": "1. Some conclusions are well-known: The paper emphasizes that “camera poses and spatial arrangements are crucial dimensions for both diversity in collection and alignment in retrieval,” which is somewhat redundant with existing research. Camera pose, in particular, has long been recognized as a challenging factor to generalize. The generalization issues of spatial arrangements are also quite intuitive since they are linked to training distribution. If the training distribution is narrow, out-of-distribution (OOD) failure at test time is expected.\n2. Dataset Dependency: The experiments and conclusions are closely tied to the DROID dataset, raising concerns about their generalizability to other robotic platforms or datasets. More varied datasets should be included in future experiments to ensure the robustness of the insights across different domains.\n3. Difficulty in real-world implementation of “Dimensions of Variation” (DVs): While the paper proposes the concept of DVs, it might be challenging to apply this approach in real-world scenarios where it is impractical to enumerate all possible variations. For example, factors such as object material or lighting conditions are not sufficiently addressed. Furthermore, real-world robot performance is often contingent on handling the complex interplay of multiple variations simultaneously rather than generalizing to each factor individually.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)