What Matters in Learning from Large-Scale Datasets for Robot Manipulation

Authors: Vaibhav Saxena, Matthew Bronars, Nadun Ranawaka Arachchige, Kuancheng Wang, Woo Chul Shin, Soroush Nasiriany, Ajay Mandlekar, Danfei Xu
Primary area: applications to robotics, autonomy, planning
Keywords: imitation learning, robotics, dataset composition

Abstract:
Imitation learning from large multi-task demonstration datasets has emerged as a promising path for building generally-capable robots. As a result, 1000s of hours have been spent on building such large-scale datasets around the globe. Despite the continuous growth of such efforts, we still lack a systematic understanding of what data should be collected to improve the utility of a robotics dataset and facilitate downstream policy learning. In this work, we conduct a large-scale dataset composition study to answer this question. We develop a data generation framework to procedurally emulate common sources of diversity in existing datasets (such as sensor placements and object types and arrangements), and use it to generate large-scale robot datasets with controlled compositions, enabling a suite of dataset composition studies that would be prohibitively expensive in the real world. We focus on two practical settings: (1) what types of diversity should be emphasized when future researchers collect large-scale datasets for robotics, and (2) how should current practitioners retrieve relevant demonstrations from existing datasets to maximize downstream policy performance on tasks of interest. Our study yields several critical insights -- for example, we find that camera poses and spatial arrangements are crucial dimensions for both diversity in collection and alignment in retrieval. In real-world robot learning settings, we find that not only do our insights from simulation carry over, but our retrieval strategies on existing datasets such as DROID allow us to consistently outperform existing training strategies by up to 70\%.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729394362024

thread_type:
official_review

<thread_object>
creator: reviewer_cfcj

<content>
summary:
This paper explores the key factors that influence the success of robot manipulation policies trained on large-scale datasets. It introduces a synthetic data generator, MimicLabs, to emulate common variations in real-world datasets such as camera poses and object arrangements. The study focuses on identifying the types of diversity that should be emphasized when collecting new datasets or retrieving relevant demonstrations from existing ones to maximize downstream task performance. Using simulation and real-world tasks, the authors demonstrate that factors like camera pose and spatial arrangement are critical to both dataset diversity and alignment. These insights are validated through experiments on the DROID dataset, leading to improvements in policy learning by up to 70%.

strengths:
1. The paper tackles an important problem—optimizing large-scale datasets for robotic manipulation—by systematically analyzing how various dimensions of variation (DVs) in the data affect learning. It provides a novel approach through its MimicLabs synthetic data generator, which offers a controlled environment for testing different dataset compositions, addressing a key gap in robot learning research.
2. This work has the potential to significantly impact the future of large-scale robot dataset collection and utilization. By offering actionable insights on how to prioritize data collection and improve policy learning through targeted dataset retrieval, it lays the foundation for more efficient robot learning pipelines, reducing the cost and complexity of data collection efforts.

weaknesses:
1. Some conclusions are well-known: The paper emphasizes that “camera poses and spatial arrangements are crucial dimensions for both diversity in collection and alignment in retrieval,” which is somewhat redundant with existing research. Camera pose, in particular, has long been recognized as a challenging factor to generalize. The generalization issues of spatial arrangements are also quite intuitive since they are linked to training distribution. If the training distribution is narrow, out-of-distribution (OOD) failure at test time is expected.
2. Dataset Dependency: The experiments and conclusions are closely tied to the DROID dataset, raising concerns about their generalizability to other robotic platforms or datasets. More varied datasets should be included in future experiments to ensure the robustness of the insights across different domains.
3. Difficulty in real-world implementation of “Dimensions of Variation” (DVs): While the paper proposes the concept of DVs, it might be challenging to apply this approach in real-world scenarios where it is impractical to enumerate all possible variations. For example, factors such as object material or lighting conditions are not sufficiently addressed. Furthermore, real-world robot performance is often contingent on handling the complex interplay of multiple variations simultaneously rather than generalizing to each factor individually.

questions:
1. Have you tried using only wrist cameras? One question that could have been explored more thoroughly is whether using dynamic wrist cameras, which vary poses constantly, would act as a strong form of data augmentation and mitigate the problem of overfitting to fixed camera poses. If this technique were used, it could potentially reduce sensitivity to specific camera poses and improve generalization.
2. Generalization beyond single variation dimensions: The current analysis focuses on individual DVs (like camera pose or object arrangement), but in the real world, robots often face scenarios where multiple variations occur simultaneously. Have you considered experiments that explore the interaction effects between multiple DVs, and if so, how do the insights change?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730206072520

thread_type:
official_review

<thread_object>
creator: reviewer_d5qm

<content>
summary:
This work conducts a thorough investigation on creating and utilizing robotics dataset for co-training robot policy with imitation learning. It demonstrates the most useful types of diversity for dataset collection and proposes insightful yet effective retrieval methods from existing datasets to improve task performance.

strengths:
1. This work tackles a popular yet significant problem concerning usage of large-scale robotics dataset.
2. The authors conduct extensive experiments on collector and retriever perspective with various DVs.
3. The paper is well-written and easy to follow.

weaknesses:
1. The experiments in Section 5.1 only consider one task *clear table*, making the conclusions less convincing. Results on more simulation tasks or clarification on the representativeness of this task should be added.
2. The co-train setting in this paper largely focuses on using generated/retrieved data from the same task as the test task (except for some results in Table 3). However, aligning the task setting between the target task and the tasks in a public large-scale dataset is difficult in most cases, especially for real-world tasks.  More experiments and analysis on co-training with different tasks can add to the practicality of this work.
3. Some of the results may be confusing. Please refer to the Question section.

questions:
1. In Table 1, it can be confusing why training and testing on only target distribution will yield such poor results, since overfitting to a rather small range of variation seems not to be that hard in imitation learning.
2. In Table 1, I am confused why co-training with varied camera poses can improve policy performance to such a great extent, because there are only several (~5) poses and they are very different. Maybe some videos and extra explanations can help clarification.
3. In Table 2 "Object spatial" part, why larger co-training distribution produces lower performance?
4. In Figure 5, I am curious whether the improved performance is attributed to the co-training setting or alignment between target and training task. For example, how is the performance if the policy is trained with retrieved Droid data without target data?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730527953228

thread_type:
official_review

<thread_object>
creator: reviewer_j7tn

<content>
summary:
This paper presents a systematic study of what factors matter most in large-scale robotics datasets for manipulation tasks. The authors introduce MimicLabs, a framework for generating controlled datasets with various dimensions of variation (DVs), and analyze dataset composition from both collector and retriever perspectives. They identify camera poses and spatial arrangements as crucial factors while finding object textures less important. The study's insights are validated on real robots and existing datasets like DROID.

strengths:
The paper addresses the crucial question of dataset composition in large-scale robotic manipulation through a systematic analysis. The two-perspective approach (collector and retriever) offers practical insights for both dataset collection and utilization. The authors conduct comprehensive experiments and clear validation in both simulation and real-world settings. The definitions and methodology based on "DV" are novel.

weaknesses:
1. Results heavily rely on MimicGen's simplified environments with basic textures and lighting. The conclusion that "texture alignment is less important" may not hold in real-world scenarios like those in DROID with complex lighting, shadows, and material properties (as in computer vision, lighting effects are often considered as a special kind of texture). Limited scene diversity compared to real datasets like DROID (8 simulated scenes vs. 52 real buildings)

2. Limited Policy Evaluation: Experiments primarily use Diffusion Policy, with only BC-RNN as comparison in MimicLabs. Performance gains might be attributed to Diffusion Policy's characteristics rather than dataset composition. Missing evaluation with modern approaches like VLA or transformer-based policies.

3. No clear metric for measuring skill similarity (e.g., is "picking a bowl" similar to "picking a cup"?) Lack of systematic handling of compound skills. Binary treatment of skill alignment without considering partial similarity.

4. Overlooked DV Interactions: No consideration of cases where a tejectory has conflicting values across different DVs (e.g., good camera pose but poor spatial arrangement). Missing analysis of trade-offs between different DVs when retrieving demonstrations. No principled approach for weighing the relative importance of different DVs when they conflict.

The limitations affect the generalizability and practical applicability of the paper's conclusions, particularly when scaling to more complex real-world scenarios or different learning algorithms.

questions:
1. How do you handle cases where a demonstration might be valuable according to one DV but less useful according to another? Is there a principled way to make such trade-offs?
2. How do your findings generalize across different imitation learning approaches? Would VLA or other transformer-based policies show similar sensitivity to DVs?
3. How does your conclusion about texture alignment being less important translate to real-world scenarios with complex lighting conditions?
4. Given that similar objects often share manipulation strategies, why not consider 'picking a cup' and 'picking a bowl' as the same skill? Would grouping such similar skills during retrieval improve policy performance or potentially harm it?
5. Since DROID contains diverse task variations across 52 buildings while your simulation covers 8 scenes, could the limited scene diversity affect your DV importance rankings? Specifically, would more diverse scenes reveal additional important DVs?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730640819713

thread_type:
official_review

<thread_object>
creator: reviewer_gs6e

<content>
summary:
This paper conducts a comprehensive empirical study on key factors in scaling robot manipulation datasets, offering valuable insights for both collectors and users.

strengths:
1. This paper collects a large-scale robot manipulation dataset in simulation with diverse dimensions of variations to enhance understanding of each factor’s impact.

2. This paper provides practical suggestions on how to collect and leverage the data, and conducts real-world experiments on the DROID dataset to verify its effectiveness.

weaknesses:
1.	In section 5.1, from the collector’s perspective, there’s only one task, “clear table”, which seems insufficient to draw conclusions (especially given that more tasks are actually used in the following sections).

2.	As illustrated in the paper, object geometry is vital to robot data. However, it is a pity that it is not included in the DVs of the experiments.

3.	Although the MimicLabs dataset generates different textures for the experiments, the textures are still primarily pure colors according to the appendix. More complex textures, such as plaid patterns or irregular textures, aren’t included.

4. There are a large number of experiments and various settings in this paper, but some of them are not explained clearly, which might be confusing (see the questions for details).

Overall, this paper conducts a nice trial to investigate the key factors in large-scale robot learning and provides many useful results and practical suggestions. However, more thorough experiments and more accurate explanations of experiments would make the conclusions more convincing.

questions:
1.	How many tasks are included in the MimicLabs dataset? Different sections use various tasks from the dataset, but I am unsure how many tasks are there in total. According to the appendix, there seem to be five, but this is not shown in the main text or images. 

2.	What exactly are the four co-training settings in Figure 2? There seems to be no exact definition of these settings, and different experiments actually use different numbers/types of co-training, making it harder to understand. It would be helpful to have a better description of all co-training settings. 

3.	Why are all the co-training datasets in section 5.1 misaligned with the target? Can’t it contain the target distribution (as I saw in some experiments in other sections)? 

4.	In section 5.3, the authors have an important finding that full retrieval might be less effective than other strategies using less data. However, a more common strategy used in other areas is to first pretrain on the full dataset and then fine-tune on the target dataset. I am wondering have the authors tried this setting?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732384824287

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to All Reviewers

comment:
We thank the reviewers for their time and effort and greatly appreciate their positive comments and insightful questions about our work.

We are delighted that reviewers found our study to be comprehensive and offer practical suggestions (Gs6e), our paper to address the crucial question of dataset composition (J7tn), our experiments to be extensive (D5QM), and our approach for testing different dataset compositions to be novel and impactful (Cfcj).

The reviewers raised a number insightful comments and questions, which we thoroughly addressed in our individual responses. Moreover, we also provided additional results, which we summarize below:

1. **Additional collector experiment** on a simulated task (`make coffee`) (**Table 9** in **Appendix K**)

2. Evaluating our retrieval strategies on the entire MimicLabs benchmark using a **transformer-based behavior cloning policy** (**Table 11** in **Appendix K**)

3. Showing the efficacy of our retrieval strategies in a **pre-training followed by fine-tuning setup**, prove further applicability of our results and evaluation framework (**Table 12** in **Appendix K**)

We have revised our paper based on feedback from reviewers (changes highlighted in blue), with additional results in the appendix as above. We sincerely hope that the reviewers will consider our rebuttal response to their concerns and additional experiments towards their final rating of our paper!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a comprehensive study on key factors that influence the effectiveness of large-scale robotics datasets for manipulation tasks. The authors introduce MimicLabs, a framework to generate synthetic datasets with controlled variations, which allows for a systematic analysis of dataset diversity. Through detailed experiments, the study highlights camera poses and spatial arrangements as being critical for dataset composition, while object textures seem to have comparatively less impact. These findings are validated across simulations, some real-world tasks, and datasets like DROID, showing up to 70% improvement in downstream policy performance in certain scenarios.

Almost unanimously, the reviewers appreciated the thorough empirical study on scaling robot manipulation datasets, the introduction of the MimicLabs framework with its controlled dimensions of variation (DVs), and the focus on key factors like camera poses and spatial arrangements for dataset composition. They also noted that the paper tackles an important and practical problem in the design of robotics datasets. Reviewers raised questions about single-task analysis from the collector's perspective, the choice of co-training versus other paradigms, multiple DVs, the role of texture, and data diversity. Through additional collector experiments, more base policy choices, and pretraining followed by fine-tuning results, many of these concerns were partially resolved.

The AC concurs with the unanimous recommendation from the committee to accept this paper. The paper raises an important and novel research question and convincingly provides insights that are valuable for the robot learning community.