Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets

Authors: Guangqi Jiang, Yifei Sun, Tao Huang, Huanyu Li, Yongyuan Liang, Huazhe Xu
Primary area: applications to robotics, autonomy, planning
Keywords: Robot Learning, Foundation Model, Representation Learning

Abstract:
The pre-training of visual representations has enhanced the efficiency of robot learning. Due to the lack of large-scale in-domain robotic datasets, prior works utilize in-the-wild human videos to pre-train robotic visual representation. Despite their promising results, representations from human videos are inevitably subject to distribution shifts and lack the dynamics information crucial for task completion. We first evaluate various pre-trained representations in terms of their correlation to the downstream robotic manipulation tasks (i.e., manipulation centricity). Interestingly, we find that the ''manipulation centricity'' is a strong indicator of success rates when applied to downstream tasks. Drawing from these findings, we propose **M**anipulation **C**entric **R**epresentation (**MCR**), a foundation representation learning framework capturing both visual features and the dynamics information such as actions and proprioceptions of manipulation tasks to improve manipulation centricity. Specifically, we pre-train a visual encoder on the DROID robotic dataset and leverage motion-relevant data such as robot proprioceptive states and actions. We introduce a novel contrastive loss that aligns visual observations with the robot's proprioceptive state-action dynamics, combined with an action prediction loss and a time contrastive loss during pre-training. Empirical results across four simulation domains with 20 robotic manipulation tasks demonstrate that **MCR** outperforms the strongest baseline by 14.8\%. Additionally, **MCR** significantly boosts the success rate in three real-world manipulation tasks by 76.9\%. Project website: robots-pretrain-robots.github.io

Decision: accept (poster)
Reviewer scores: 8, 8, 8, 3

Reviews and discussion:
<thread>
creation date:
1729552801599

thread_type:
official_review

<thread_object>
creator: reviewer_vjk6

<content>
summary:
This paper presents a representation learning approach named RPM that learns to encode images into low-dimensional states that are aligned with the proprioception state through contrastive learning. 

1. The authors showed that RPM outperforms existing learned robot representations on 20 tasks selected from 4 benchmarks. 
2. The authors proposed to use gradient heatmap as a heuristic named as "manipulation-centricity" to determine how much the learned representation contributes to the final manipulation task performance. 
3. RPM is showed to have highest manipulation-centricity than existing learned robot representations, and is evaluated with a real UR5 robot in lift, sweep and rearrange tasks.

strengths:
1. The proposed heuristics of using gradient heatmap to indicate manipulation performance is interesting. It can be linked to existing work in visual affordance prediction. 
2. Aligning the vision representation with the robot proprio state and actions is interesting, and may encourage the model to focus more on task-relevant information. 
3. The writing is easy to understand.

weaknesses:
1. The technical contribution is limited. The authors spent 4 pages between the introduction and experiment section to discuss their method. But, in fact, the only new technical point that the authors proposed is equation (1), which uses InfoNCE loss to bring visual features and proprio state/aciton closer. This looks hand-wavy. Even this innovation raises questions, as the proprio state/action only describe the robot trajectory but not the visual scenes. This is very likely to make the model overfit to the learned scenarios. 

2. The evaluation is questionable. RPM is evaluated on 20 tasks, 10 from MetaWorld, 4 from DexArt, 3 from RoboCasa, 3 from RoboMimic. However, these benchmarks combined include hundreds of tasks. Why just select these 20 particular tasks? Moreover, the "Can" task in robomimic is mentioned as challenging in this paper, but it is just a simple pick-and-place task of a moderate-size can. The evaluation raises serious questions about the generalization of this RPM method. 

2. Unnecessary discussion on how to use the human video datasets. The authors spend a considerable amount of efforts to explain the policy learning from human videos. But in the end, the conclusion is to not use human data. Human action data is abundant and contains rich and diverse behavior. If the authors' proposal is to not use them and use robot demonstrations, which in fact is the standard robot learning setup. I would prefer the paper to focus on the studied setting. 

3. Needs a more comprehensive ablation study. RPM is pre-trained with the DROID dataset, a large and relatively clean robot dataset, while many baselines that RPM compared against are not trained with DROID. 

4. In Table.1, HRP and VC-1 achieves better success rate than R3M-DROID, yet R3M-DROID's gradient heatmap focus more on the objects. According to the key finding, R3M-DROID shall perform better. 

4. The term "dynamics" is mis- and over-used, confusing in some sections.

questions:
(see weakness)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730033773842

thread_type:
official_review

<thread_object>
creator: reviewer_csn9

<content>
summary:
This paper introduces a metric (manipulation-centricity) for assessing the ability of pretrained visual encoders to model task-relevant features for robot manipulation tasks; as well as a novel pretraining objective (RPM) that incorporates the robot dynamics information available in robotics datasets. The authors first establish a correlation between the metric they define and the success rate of policies that leverage the features generated by pretrained visual encoders. Having established a positive correlation they seek to design a pretraining objective for visual encoders and evaluate its performance on their proposed metric and the downstream policy success rates. The novelty in the pretraining objective the authors introduce is a dynamics alignment term which takes the form of the InfoNCE applied to embeddings of a sequence of state action pairs and image embeddings. This novel dynamics alignment term is combined with existing behaviour cloning and temporal contrastive terms to give the full learning objective. They apply this objective function to pretrain visual encoders for manipulation tasks and demonstrate that this pretraining objective leads to improvements on their metric and downstream task performance in both simulated and real-world experiments.

strengths:
- The paper validates a pretraining objective for visual encoders that improves the performance of robot policy learning. 

- The paper introduces a novel metric for quantitatively analysing the features generated by visual encoders.

- The paper validates its claims in both simulated and real-world experiments.

- The paper includes analysis of the representations learnt in various approaches.

- The paper is concise and clearly presents the authors claims and results to validate these claims.

weaknesses:
- The paper appears to focus on pretraining and freezing visual encoders while training policies on top of these encoders. It wasn't clear if they finetune all parameters on a given task (in the simulation and real-world policy learning experiments). In general, this may be relevant as policy performance can improve with finetuning of the entire architecture. I may have missed this in the paper but it would be good to see if the pretraining results include finetuning of the overall architecture on the robot manipulation task. 

- The similarity scores for the introduced metric are quite small and rely on thresholding the binarization of the Grad-Cam output. It seems like this is a good proxy for rudimentary manipulation tasks but it doesn't necessarily seem to be a metric that will generalise to more complex settings. The metric also relies on annotations or knowledge of the task relevant features within the image. 

- The pretraining dynamics alignment term incorporates chunks of state-action pairs with a history of 3 being cited as optimal. When it comes to highly dynamic tasks it is not clear how the alignment between individual images and such a history will necessarily be beneficial. For instance a single static image does not necessarily convey rich signals for dynamics (there is some signal but it is imperfect) when compared to a history of images,  aligning a state action history and a static image seems counterintuitive to me, I question whether the improvements observed are optimal relative to other methods of incorporating dynamics information. If the dynamics varied quite a lot between demonstrations on the same robot platform I question how this will effect performance. Clearly this term is working as demonstrated in the paper but I do have reservations over its usefulness and whether it is the optimal approach to incorporating dynamics information.

questions:
Congratulations on this work and thanks for contributing it to the robot learning community. 

I wished to clarify if during policy training does the visual encoder always remains frozen and whether you finetune the entire architecture? In the first part of the paper when evaluating your introduced metric it is clear that the encoder is frozen, I wished to confirm whether this is also the case for the reported task success rates and whether you considered finetuning the entire architecture?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730682285888

thread_type:
official_review

<thread_object>
creator: reviewer_3arj

<content>
summary:
The paper proposes pre-train visual representations for robot manipulation by focusing on manipulation centricity, robot datasets (instead of human activity datasets), and auxiliary loss objectives for dynamic alignment and time contrastive learning. To measure Manipulation Centricity, they use Jaccard similarity between the binarized Grayscale CAM and SAM2’s foreground vs. background predictions. The proposed model RPM outperforms simulation and real robot tasks compared to baselines.

strengths:
1. Detailed empirical validation across diverse tasks and simulation domains, plus real robot experiments
1. well-motivated concept of manipulation centricity with clear metrics
1. ablation studies and analysis of design choices 
1. insights about benefits of robot vs human datasets for pre-training
1. Clearly written key research questions and contributions

weaknesses:
The work presents insights into the visual representations that focus on manipulation centricity and dynamic alignment perform better than existing approaches. However, there are certain assumptions whose implications are not clearly discussed.
1. DROID is chosen as the robot dataset of choice instead of other larger dataset in OXE.
1. RPM is a ResNet-based model instead of directly using / comparing to pretrained models like SAM2, etc.
1. The "manipulation centricity" does not seem to improve at similar scale across different simulators, with respect to success rate. 
1. A concise definition of manipulation centricity and how it is calculated should be present in the main paper, rather than existing only in appendix.

questions:
1. What is the tradeoff between the visual realism of the simulator vs simulator-specific training required to achieve high success rate and manipulation centricity? For example, in RoboCasa, manipulation centricity seems really low compared to others.  How to compare it across simulators?

1. How does "manipulation centricity" depend on the choice of camera intrinsics and extrinsic with respect to the robot body frame?  

1. R3M has been shown to be a poor visual representation model for robotic tasks, as compared to visual datasets [1] and pretraining image distribution matters. Why none of the baselines involve vision only representation backbones for comparison?

[1] Dasari et al, 2023. An Unbiased Look at Datasets for Visuo-Motor Pre-Training, https://data4robotics.github.io/
</content>
</thread_object>

</thread>
<thread>
creation date:
1731022954668

thread_type:
official_review

<thread_object>
creator: reviewer_wcpd

<content>
summary:
The paper first proposes manipulation centricity, a new metric for evaluating representations for robotics, and finds that it is a strong indicator of success rates in downstream robotic manipulation tasks. Building on this insight, it proposes RPM, a framework for learning visual representation for robotics from large-scale robotics dataset. Experiments in both simulation and the real world show that RPM leads to improved manipulation centricity and higher task success rate when evaluated using behavior cloning.

strengths:
- The paper is clear and well presented
- The simulation experiments are rigorous and have a lot of diversity
- The paper presents a novel metric for evaluating visual representations for robotics, and novel findings on how to integrate robotics information (states, actions) for training better representations

weaknesses:
- There could be more empirical analysis and intuition regarding how each loss affects manipulation centricity. Authors could explain intuitively how each loss contributes to better MC and visualize the gradcam of an encoder trained with each loss
- The paper lacks comparison to more state-of-the-art baselines, such as VIP/LIV, or object-centric pre-trained representations such as POCR/HODOR
- As I understand, the goal here is to make the representation pay attention to task-relevant regions e.g. robot and task-relevant objects. The authors have already demonstrated that this information can be obtained using SAM2. Have the authors considered directly using the masks as a form of supervision for training the representation? I think this could serve as a baseline, since RPM requires additional priveleged information that only a robotics dataset has, i.e. state and action, whereas we can generate segmentation masks for any general sources of data.
- The paper could benefit from more extensive real robot experiments

questions:
- Why is it still necessary to attend to the manipulator region when there is already proprioception input?
- What is the evaluation protocol in the real world? How are the environments varied between training and evaluation, and what would count as a success?
- By attending the robot and task-relevant object, technically this would allow the representation to have stronger generalization capabilities. Perhaps the author should run some visual generalization experiments to find out if RPM enables better generalization.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732253041713

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
## Review Highlight

We sincerely thank all reviewers for their thorough feedback and constructive suggestions.
We appreciate the positive comments from all reviewers on our paper's clarity and presentation. In particular, we are grateful for the acknowledge of:

* A novel manipulation centricity metric (**Reviewers 3aRj, cSN9, vJK6**)
* Innovative integration of robotics information in representation learning (**Reviewers wcpD, vJK6**)
* Comprehensive validation across simulated and real-robot tasks (**Reviewers wcpD, 3aRj, cSN9**)

The reviewers' feedback has helped strengthen our work significantly. We have addressed their questions and suggestions thoroughly in our responses.

## Reviewer concerns

We have taken the reviewers' feedback into account and responded to each question with detailed explanations and additional experiment results. For the main concerns:
* Concerns about Manipulation Centricity (Reviewers wcpD, 3aRj, cSN9, vJK6): Enhanced quantitative analysis of each component's effects on manipulation centricity, with technical details, definitions, and evaluation protocol descriptions added.
* Concerns about Empirical Evaluation (Reviewers wcpD, 3aRj, vJK6): Clarified our evaluation on 23 diverse manipulation tasks and added four new baselines (LIV[1], ImageNet[2], SAM2[3], MAE-DROID[4]), demonstrating superior performance in both simulation and real-world settings.
* Concerns about Generalization (Reviewers 3aRj, wcpD, cSN9): Validated robustness under color perturbations, varying camera viewpoints and additional finetuning experiments, while demonstrating strong generalization.
* Concerns about Technical Novelty (Reviewer vJK6): Highlighted two key contributions: systematic analysis of manipulation-centricity impact on downstream tasks, and first incorporation of state-action alignment and action prediction in robotic representation pretaining.

These responses address key concerns while providing new experimental results that strengthen our paper's contributions. We incorporate our clarifications and additional analyses in our revised manuscript.

[1] Ma et al. "LIV: Language-Image Representations and Rewards for Robotic Control"

[2] Dasari et al, 2023. "An Unbiased Look at Datasets for Visuo-Motor Pre-Training"

[3] Ravi et al. "SAM 2: Segment Anything in Images and Videos"

[4] He et al. "Masked Autoencoders are Scalable Vision Learners"
</content>
</thread_object>

</thread>
<thread>
creation date:
1733161296684

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A letter to AC: Thanks for organizing and supervising the review process and concerns about Reviewer vJK6

comment:
Dear Area Chair and Senior Area Chair,

We sincerely appreciate your time in considering this message and express our gratitude for your dedication to organizing and supervising the ICLR review process.

During the discussion period, we received positive feedback from three reviewers (**wcpD, 3aRj, and cSN9**), all providing a Rating of 8 with further discussions. These reviewers strongly acknowledged our paper's contributions and recommending its acceptance to ICLR.

Despite multiple reminders, Reviewer **vJK6** has not participated during the discussion period. Given the lack of engagement and divergent opinions from other reviewers, we are concerned about potential bias in the assessment.

For Reviewer **vJK6**'s concerns, our response:
* Provided detailed clarifications about our technical contributions (w1 & w2). These aspects of our work were specifically highlighted by the other reviewers as demonstrating strong technical novelty and thorough experimental validation.
* Addressed Reviewer vJK6's writing-related concerns (w3 & w6) with additional explanations to better convey our motivation, which have been incorporated into our updated manuscript.
* Included new ablation experiments to address the misunderstandings of our previous results (w4 & w5). As presented in our general response, we added six additional experimental studies, which received positive acknowledgment from the other reviewers.

In summary, we have thoroughly addressed all concerns raised by Reviewer **vJK6**, though unfortunately received no response. We hope this information proves helpful during the AC-reviewer discussion period.

We are grateful for your consideration and thank AC and all reviewers for their valuable contributions to the community.

Warm Regards,

Paper 793 Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a novel metric called “manipulation-centricity” (MC) for evaluating pretrained representations and demonstrates its correlation with downstream manipulation task performance. Building on these insights, it proposes RPM, a representation learning method that captures both visual features and dynamic information to enhance manipulation-centricity (MC). Through extensive experiments, the authors validate that their approach improves task success rates compared to baseline methods.

Strengths:
- Novel Metric and Concept: “Manipulation-centricity” as a quantitative metric for evaluating visual representations in robotic manipulation is innovative and well-motivated.
- Experiments: The paper provides rigorous validation across diverse tasks and simulators, demonstrating the robustness of RPM.
- Clear Writing and Structure: The paper is well-organized and presents its key ideas well.

Weaknesses
- Real-World Experiments: While the paper includes real-world evaluations, the scope of these experiments is limited. Conducting more complex tasks with diverse objects could strengthen the results.
- Missing Baseline Comparisons: As note by Reviewer wcpD, the POCR/HODOR line of work proposes pretrained visual representations for robotics that are object-centric. These could serve a strong basline for the current method. The authors have noted that the code for HODOR is not yet available; if it is released before the camera-ready deadline, I recommend running these experiments and including them as baselines.

The paper demonstrates significant novelty in proposing manipulation-centricity as a metric and aligning visual representations with robotic dynamics. It provides thorough experimental validation and offers meaningful insights into pretraining for robotic manipulation tasks. However, the limited scope of real-world experiments slightly diminishes the overall impact.

Despite these limitations, the strengths of the paper, particularly its novel contributions and rigorous empirical analysis, outweigh the weaknesses. Therefore, I recommend acceptance, with the suggestion to address the noted weaknesses in the camera-ready version.