Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets
Authors: Guangqi Jiang
Yifei Sun
Tao Huang
Huanyu Li
Yongyuan Liang
Huazhe Xu
Venue: ICLR 2025 Poster (2025)
Keywords: Robot Learning
Foundation Model
Representation Learning
Primary area: applications to robotics, autonomy, planning
TL;DR: Manipulation-centric robotic representation training on large-scale robot dataset boosts policy performance on manipulation tasks.

Abstract: The pre-training of visual representations has enhanced the efficiency of robot learning. Due to the lack of large-scale in-domain robotic datasets, prior works utilize in-the-wild human videos to pre-train robotic visual representation. Despite their promising results, representations from human videos are inevitably subject to distribution shifts and lack the dynamics information crucial for task completion. We first evaluate various pre-trained representations in terms of their correlation to the downstream robotic manipulation tasks (i.e., manipulation centricity). Interestingly, we find that the ''manipulation centricity'' is a strong indicator of success rates when applied to downstream tasks. Drawing from these findings, we propose **M**anipulation **C**entric **R**epresentation (**MCR**), a foundation representation learning framework capturing both visual features and the dynamics information such as actions and proprioceptions of manipulation tasks to improve manipulation centricity. Specifically, we pre-train a visual encoder on the DROID robotic dataset and leverage motion-relevant data such as robot proprioceptive states and actions. We introduce a novel contrastive loss that aligns visual observations with the robot's proprioceptive state-action dynamics, combined with an action prediction loss and a time contrastive loss during pre-training. Empirical results across four simulation domains with 20 robotic manipulation tasks demonstrate that **MCR** outperforms the strongest baseline by 14.8\%. Additionally, **MCR** significantly boosts the success rate in three real-world manipulation tasks by 76.9\%. Project website: robots-pretrain-robots.github.io

Published as a conference paper at ICLR 2025
MANIPULATION- CENTRIC ROBOTIC REPRESENTATION FROM LARGE- SCALE ROBOT DATASETS
ROBOTS PRE-TRAIN ROBOTS:
Guangqi Jiang'
Yifei Sun
Tao Huang Huanyu Li Yongyuan Liang"! Huazhe Xu
University of California. San Diego Tongji University
Shanghai Jiao Tong University
4 University of Maryland, College Park
Tsinghua University
A Metric for Robotic Representations
Initial
Performance
Manipulation Centricity
Indicate visual part most related robot tasks.
Manipulation Centrictt
ection
Simulation
Dynamics Alignment
Succees ltate on 4 domaine
Image inputs
Features
Chunk
Actor Loss
Robot
Predict
7-dim
MCR
Proprio,
Real-Robot
Time Contrastive Loss
Feature
36K Robotic
Policy
Trajectories
Positive
Negative
Action
Pretraining
Closed-loop Control
Performance
Figure 1: Overview, We introduce a robotic representation evaluation metric termed manipulation centricity, which exhibits a strong correlation with downstream policy performance. Accordingly, we design new pre- training method, MCR, to learn manipulation-centri representation from large-scale robotic datasets. Compre- hensive experiments both simulations and real robot validate the superiority of proposed representation.
ABSTRACT
The pre-training of visual representations has enhanced the efficiency robot learning. Due the lack of large-scale in-domain robotic datasets, prior works utilize in-the-wild human videos to pre-train robotic visual representation. De- spite their promising results, representations from human videos are inevitably subject to distribution shifts and lack the dynamics information crucial for task completion. We first evaluate various pre-trained representations in terms of their correlation to the downstream robotic manipulation tasks (i.e., manipulation cen- tricity). Interestingly, we find that the "manipulation centricity" is strong in- dicator of success rates when applied to downstream tasks. Drawing from these findings, we propose Manipulation Centric Representation (MCR), a foundation representation learning framework capturing both visual features and the dynam- ics information such as actions and proprioceptions of manipulation tasks to im- prove manipulation centricity. Specifically, DROID (Khazatsky et al., 2024) robotic dataset and leverage motion-relevant data such robot proprioceptive states and actions. We introduce novel contrastive loss that aligns visual observations with the robot's proprioceptive state-action dy- namics, combined with an action prediction loss and time contrastive loss dur- ing pre-training- Empirical results across four simulation domains with 20 robotic manipulation tasks demonstrate that MCR outperforms the strongest baseline by 14.8%. Additionally, MCR significantly boosts the success rate in three real-world manipulation tasks by 76.9%. Project website: robots-pretriin-rooots. github...........
pre-train a visual encoder the
"qqual contribution. Equual advising. Correspondence 9qj aggjiag@ucee angeuudd.d... edu..

---

Published as a conference paper at ICLR 2025
Aggregate
Robomimic
RoboCasa
MetaWorld
DexArt
R=0.93
R=0.83
75
R=0.64
R- 0.7.70
60
2-0.44
70
58
Sucoess
55
65
50
70
60
Manipuatiion Centricity
Manipulation Centricity
Manipulation Cantricity
Manipulation Centricty
Maripulation Centricity
MCR (ours)
R3M
R3M/DROID
MVP
VC11
HRP
Figure 2: Correlation between manipulation centricity and downstream performance. Our findings reveal that (I) the proposed metric of manipulation centricity strongly correlates with the downstream performance of robotic representations, and (2) using the robot dataset DROID yields greater benefits for robotic representations than human datasets. (3) These insights motivate our method, MCR, which leverages dynamics labels from the robot dataset to further enhance manipulation centricity and downstrcam performance.
INTRODUCTION
Grounding robots with generalizable visual representations, termed robotic representation, is crucial for real-world visuomotor control. Pre-training robotic representations on extensive in-domain data offers promising strategy for robotics, drawing from the success of large-scale pre-training in com- puter vision (He et al., 2022) and natural language processing (Devlin al., 2019). However, due to the scarcity of robotics data and high collection costs, recent studies have utilized everyday human operation videos (Grauman et al., 2021; Goyal et al., 2017) to pre-train visual representations for robotic manipulation, positing that knowledge behind human manipulation can inform representa- tions in question. Various levels of human manipulation knowledge - such as task semantics (Nair et al., 2022), pixel-level comprehension (Xiao et al., 2022; Majumdar et al., 2023), and physical interaction (Jia et al., 2024; Srirama et al., 2024) have demonstrated their effectiveness in bene- fiting robotic representation. A fundamental question naturally arises: (Q1) What specific features captured from human data significantly contribute to improving robotic manipulation?
To investigate this question, we conduct comprehensive evaluation of prior representations cerning their policy learning performance across various downstream simulation domains. Interest- ingly, we observe a correlation between a representation's downstream performance and its ability to capture cniiullationrrlevaan regions including robot end-effectors and task-relevant objects. To examine this correlation more formally, we introduce a metric, 'manipulation centricity', and pro- pose evaluation protocol to quantify it across different representations. The core of this protocol to measure the similarity between ground truth manipulation regions and the focus of the represen- tation. Our results indicate a strong correlation between downstream performance and manipulation centricity, as illustrated in Figure 2. These findings provide valuable insights into our first ques- tion: (A1) Manipulation centricity emerges as the key factor contributing to enhanced robotic manipulation performance.
However, human videos not only introduce inherent distribution shifts due to the numan-robot em- bodiment gap, but also lack the dynamics information essential for successful task execution. This naturally leads the second question: (Q2) Is there better dataset choice than human dataset to learn manipulation centricity in robotic representation? With the recent emergence of large- scale robot datasets (Collaboration et al., 2024; Walke et al., 2023), we hypothesize that the smaller domain gap presented by these datasets may naturally be more suitable for learning manipulation centricity. To prove this hypothesis, we re-train the prior method with a representative robot dataset, DROID (Khazatsky et al., 2024). and indeed observe significant improvements in both performance and manipulation centricity. This answers our second question: (A2) Large-scale robot datasets can be a better choice than human datasets for learning manipulation centricity.
Recognizing that robotic datasets provide more relevant information about robot embodiment tra- jectories, this raises the third question: (Q3) How to learn manipulation centricity better with large-scale robot datasets? Our starting point is the dynamics labels, including robot propriocep- tive states and actions, in the robot dataset but absent in the human dataset. We consider that these dynamics labels contain the core knowledge behind accomplishing a manipulation task, which has not been explicitly utilized in previous pre- mre-trained robotic representations from human data. To this end, we introduce a new method, Manipulation Centric Representation (MCR), designed to lever- age dynamics labels from robot dataset to improve manipulation centricity of robotic representation.
2

---

Published as a conference paper at ICLR 2025
Specifically, we propose two training objectives: dynamics alignment loss, aligning pixel representa- tions with robot state-action pairs at the same timestep, and action prediction loss, predicting robot actions from image observations. We also incorporate time contrastive learning objective (Nair et al., 2022) to encode temporal information. Integrating these objectives into the training process significantly improves manipulation centricity, leading 14.8% performance increase across four simulation domains encompassing 20 diverse robotic tasks, and a 76.9% improvement across three real-world robot tasks. This answers our third question: (A3) Effectively utilizing dynamic labels enhances the learning of manipulation centricity.
We summarize this paper in Figure 1, highlighting three key insights: (1) Manipulation centricity serves as an indicator of the effectiveness of representations for robotic control, as evidenced by the strong correlation with downstream performance. (2) Large-scale robot datasets can be a superior choice compared to human datasets for learning manipulation centricity, as indicated by significant improvements in performance and manipulation centricity when using robot datasets. (3) Effective utilization of dynamics labels from robot datasets significantly enhances the learning of manipu- lation centricity in robotic representations, demonstrated via the introduction of the MCR method, which incorporates the proposed dynamics alignment and action prediction objectives.
2 EXPERIMENTAL SETUP: EVALUATING ROBOTIC REPRESENTATIONS
This section outlines the experimental setup used to evaluate the effectiveness of pre-trained visual representations for robotic manipulation. In line with prior work (Nair et al., 2022; Xiao et al., 2022), we freeze the pre-trained encoders and utilize Imitation Learning (IL: Argall al. (2009)) for downstream policy learning. Accordingly, the quality of the visual representations is assessed based on IL performance averaged across various downstream tasks.
Evaluation protocol. To assess visual encoder Fe. which maps an RGB image I E ##xW> to continuous feature vector F we introduce a policy network Ty built atop the frozen encoder Fg. This policy network takes as input the feature vector and the robot's proprioceptive state outputting an action distribution
To ()),))))) To train the policy, a task-specific demonstration dataset DIL {T1, .T., } is collected, where each demonstration т, is trajectory consisting of a sequence of expert-level observation-actinn optimized via the Behavior Cloning (BC; Bain & Sammut (1995)) algorithm, where the optimization objective is to minimize the error between the predicted action à and the optimal action a from demonstration data. For fair comparison, trained encoder in each task. During evaluation, the policy is executed in a closed-loop manner within the environment with online feedback test its success We evaluate at least 20 episodes every fixed number of training epochs, selecting the highest success rate, The mean and standard error of success rates across three seeds are reported.
pairs т, = {Le, $t, at) Then, the policy is
employ the same BC algorithm for any used pre-
on the target manipulation task.
Simulation environments and datasets. We select total of 20 tasks across simulation environ- ments to evaluate representation quality. These tasks encompass a range of end-effectors, including grippers and dexterous hands, along with diverse robot arms and varying levels of manipulation com- plexity. Task visualizations are shown in Figure 3, with additional details provided in Appendix C.
Robomimic
RoboCasa
MetaWorld
DexArt
Figure 3: Task visualization. We consider 20 challenging and diverse manipulation tasks spanning domains.
Robomimic (Mandlekar et al., 2021) features a benchmark suite for tabletop manipulation using the Franka Emika Panda arm and parallel gripper. We select 3 challenging tasks, utilizing 200 demonstrations from the official proficient human teleoperation dataset for each task.
RoboCasa (Nasiriany et al., 2024) is realistie simulation platform focusing housing scenar- ios with a Franka Emika Panda arm. We select challenging tasks covering different kitchen scenarios and utilize 50 demonstrations for each task following their official training scripts.
3

---

Published as a conference paper at ICLR 2025
Table 1: Grad-CAM visualization. We present Grad-CAM visualizations alongside their corresponding task success rates the Square task from Robomimic and Pick Place Wall task from MetaWorld. Repre- sentations that capture the robot's end-effectors and task-relevant objects are linked to improved downstream performance. Comprehensive visualizations can be found in Appendix D.
R3M- DROID
Task
MVP
VC-I
HRP
R3M
MCR
IL. Score (%)
Square
P&& P&P Wall
MetaWorld (Yu et al., 2019) features a benchmark suite of robotic tabletop manipulation with the Sawyer robotic arm and gripper. We select 10 diverse tasks according both the difficulty categorized in Seo et al. (2022) and the evaluation suite in Nair et al. (2022), and collect 25 demonstrations via official scripted controllers for each task.
DexArt (Bao al., 2023) features benchmark suite for dexterous manipulation tasks artic- ulated objects using the XArmó arm and an Allegro hand. We select all 4 tasks and gather 100 demonstrations rollout by well-trained RL policies for each task.
Comparison methods. We consider representative methods in robotic visual representation:
MVP (Xiao et al., 2022) pre-trains Vision Transformer (VIT, Dosovitskiy et al. (2021)) using Masked Auto-Encoding (MAE, (He al., 2022)) on mix of human-object interaction videos.
VC-1 (Majumdar al., 2023) is trained similarly to MVP but also additionally incorporates igation and the ImageNet (Deng et al., 2009) dataset during pre-training.
HRP (Srirama et al., 2024) fine-tunes a pre-trained ViT predict human affordance labels, in- cluding hand posc, contact points, and active objects extracted from human videos.
R3M (Nair et al., 2022) pre-trains a ResNet (He et al., 2015) model on human videos using time contrastive learning and video-language alignment to extract temporal and semantic information.
INVESTIGATION: MANIPULATION CENTRICITY OF REPRESENTATIONS
Prior work utilizes in-the-wild human videos assist robotic representation learning, yet the domain gap between human and robotic tasks may influence the representation quality. To investigate this problem, we take initial step by visualizing these representations in simulated tasks. Interestingly, we observe that a representation' 's downstream task performance appears to correlate with its ability to capture manipulation-reles
regions. investigate this correlation more formally, we introduce metric, 'manipulation centricity', and propose evaluation protocol to measure it across different methods. Our results indicate a strong correlation between manipulation centricity and downstream task performance, which strongly guides method design in the next section. We introduce the whole pipeline the following parts and present more details in Appendix D.
Feature visualization & motivation. We weighted Class Activation Mapping (Grad- CAM; Selvaraju et al. (2017)), widely-adopted network visualization technique in computer vision, to analyze the features of existing representations. Grad-CAM highlights the regions of input im- age that are most influential the model's decision-making process. In our context, this technique reveals how the network interprets images within manipulation tasks, shown in Table 1. Specifi- cally, in the Square task, we observe that the MVP model tends to focus on irrelevant regions, such as the table, rather than the manipulator or objects, while in the Pick Place Wall task, R3M exhibits similar pattern. Both cases correspond to poor downstream performance. In contrast, representations that emphasize the robot's end-effectors and objects are associated with better downstream perfor- mance. This motivates our investigation into whether representation quality is linked to its ability to capture these manipulation--elevaa regions, a property we define as manipulation centricity.
4

---

Published as a conference paper at ICLR 2025
Measuring manipulation centricity. To quantify manipulation centricity, we compute the similarity between the regions highlighted by Grad-CAM and the ground truth regions corresponding the end-effector and task- relevant objects. The ground truth is gener- ated using the powerful video segmentation model SAM 2 (Ravi et al., 2024), with man- ual annotation of key points within the target regions. This approach enables efficient an- notation across an entire video of task execution. tation masks for demonstration data DIL across all simulated tasks, forming our evaluation dataset. As illustrated in Figure 4, manipulation centricity is then measured by averaging the Jaccard Index, widely-used similarity metric in image segmentation, between the binarized Grad-CAM outputs and the ground truth segmentation masks over the entire evaluation dataset. All the ground truth annotations are available in Appendix D.
Segmentation Mask
SAM
Jaccard
Index
+
Encoder
Figure 4: Measurement of manipulation centricity.
We apply this annotation method to create segmen-
Key findings. The aggregate results, presented in Figure 2, reveal strong correlation between manipulation centricity and downstream task performance, with Pearson correlation coefficient of R 0.93. We also evaluate this correlation within individual simulation domains, where the evaluation dataset consists of demonstrations specific to each domain. The positive correlation re- mains consistent across all domains, albeit with varying strengths. In summary, these results suggest that manipulation centricity measured on our entire evaluation dataset a reliable indicator of the effectiveness of robotic representations.
4 MCR: LEARNING MANIPULATION-CENTER REPRESENTATION
Drawing from the conclusions in Section 3, our focus shifts to improving the manipulation centricity of robotic representations in this section. This achieved by two attempts in our proposed method, Manipulation Centric Representation (MCR). First, we find that re-training existing models with large-scale robot data (i.e.. DROID (Khazatsky et al., 2024)) instead of human data can significantly improve manipulation centricity. This indicates the value of utilizing robot-specific datasets, as discussed in Section 1. Second, we introduce novel pre-training objectives that leverage the robot's state-action crramics-iiformationn provided in robot data, which is inherently absent in human datasets, to further enhance manipulation centricity, detailed in Section 4.2.
4.1 LARGE-SCALE ROBOT DATASET
Dataset processing. In recent years, several large-scale robot datasets have been introduced (Brohan et al., 2023; Walke et al., 2023; Collaboration et al., 2024; Khazatsky et al., 2024). Among these, we select the DROID dataset for our method because of its extensive scene diversity and a large volume of data. The dataset is collected using the Franka robot arm and Robotiq 2F-85 gripper via teleoperation, comprising a total of 76k trajectories. Each trajectory includes RGB images from two external Zed 2 cameras, robot proprioceptive states, and actions consisting of delta 6D poses and 1-DoF gripper actions. To ensure data quality, we filter out trajectories with fewer than 40 timesteps ensure adequate temporal information in each trajectory. Additionally, trajectories containing incomplete or singlc-word language instructions are removed, as these may indicate lower-quality interactions. After processing, we retain 36k trajectories for pre-training.
A motivating discovery. Many previous methods rely on cumann-bbjed interaction datasets to learn manipulation concepts from human behavior (Nair et al., 2022: Xiao et al., 2022; Srirama et al., 2024). However, the domain gap between human hands and robotic manipulators may inherently affect representation quality. We re-train the R3M model using the robot-specific DROID dataset, yielding variant we call R3M-DROID (equivalent to R3M*). As expected, Grad-CAM visualiza- tions in Table show that R3M-DROID better captures manipulation-relevaa titative results downstream performance, shown in Figure 2, confirms this improvement. This suggests that robot-specific data inherently benefits representation learning by narrowing the do- main gap between training and deployment environments.
regions, and quan-
5

---

Temporal contrast. We also wish the representation to encode temporal information, which has shown importance for manipulation tasks (Zhao et al., 2023). To this end, we adopt the time- contrastive learning objective from Nair al. (2022), which encourages temporally close frames in video to be closer in the embedding space than those that are temporally distant or from dif- v w, and compute the Published as a conference paper at ICLR 2025
4.2 TRAINING MCR
Unlike human datasets, robot datasets provide access to manipulator dynamics, including proprio- ceptive states and actions. While previous approaches do not fully leverage this dynamics informa- tion, we hypothesize that incorporating it will enhance manipulation centricity. To achieve this, we design two new training objectíves that explicitly leverage robot dynamics. Additionally, we adopt semantic learning loss from prior work (Nair al., 2022) retain semantic information in the representations. These objectives and their integration into our training process are detailed below.
Dynamics alignment. Our first insight is that each image observation corresponds to underlying pro- prioceptive robot state at every timestep. We aim to learn this correspondence, termed dynamics alignment, through contrastive learning. Formally, we define a state-action dynamic chunk of length at timestep
Encoder F
de = - 41 (1xx)(111111 ":=:::)((--+)------ & $1+ + The posi- tive sample for de is its corresponding RGB image I at the same timestep. while the negative samples are drawn from different timestep & within the same trajectory. During training, we randomly choose and The en- coder F is trained to discard irrelevant details from high- dimensional images and retain the essential information for manipulation. We employ the InfoNCE loss (van den Oord et al., 2019) for contrastive learning, introducing an MLP projector H to map the dynamics chunk de the same dimension as the image feature vector = (7)...
Ze
Positive
Negative Samples
Samples
H(d,)
(dd))
Projector H
Seol
Sp. Ri.
Dynamics d,
Dynamics d₁
Figure Illustration of objective Laya.
The objective function is illustrated in Figure 5 and formalized as:
((L)_H(d))))
Layn
E Elog ((s)) Hts). bEB
where S represents the negative L2 distance, and b denotes a sample from the batch B.
Action prediction. We also integrate behavior cloning (BC)-like actor into our pre-trainn pre-training frame- work, based on the idea that robotic representations should be predictive of expert-level behaviors in the dataset. The actor is implemented as shallow MLP head that maps the image feature vector 2, to the predicted robot actions àz We use mean squared error as the objective for action prediction:
Lact
MSE(a, ag).
BEB
ferent videos. For this, we sample frame triplet (Iu, Lv, Iw)) where < following loss:
((
Liel = Σ
- log ((()t==)) + ((tt)=)) + ((
bEB
where 2/th b is negative sample from different video within the batch
Oveall objective & imppemennatiinss. We train MCR with a combination of introduced objectives:
LMCR Layn + Lact + Lret-
We do not introduce additional hyperparameters for weighting these objectives, as LMCR already yields strong empirical performance. Our default encoder backbone is ResNet-50, and we use Adam (Kingma, 2015) the optimizer. The encoder is trained for 500k steps to ensure conver- gence, and we select the last checkpoint as the final model. The whole training process is used 50 hours with single NVIDIA 3090. Further training details are provided in Appendix C.

---

Published as a conference paper at ICLR 2025
Robomimic (3 tasks)
RoboCasa (3 tasks)
MetaWorid (10 tasks)
DexArt (4 tasks)
625
60.
575
65.
625
Sucossa
50
53
47.
45
00
45.0
425
40
40.0
Lewnn from Sersich
MVP
VC-1
HRP
R3M
RAM-DROID
MCR (ours)
Figure Simulation results. We evaluate MCR and baselines across different domains. Our method consis-
tently outperforms the haselines. Results are mean success rate aggregated over seeds with standard deviation.
EVALUATION AND ANALYSIS OF MCR
Our experiments are conducted in simulation and the real world to answer the following questions:
(1) Does MCR learn manipulation centricity and outperform baselines in simulation? (Section 5.1)
(2) Can the conclusions from (1) generalize to real-world manipulation tasks? (Section 5.2)
(3) Which design choices of MCR matter for learning manipulation centricity? (Section 5.3)
(4) What benefits does large-scale robot data bring to robotic representation learning? (Section 5.4)
5.1
SIMULATION RESULTS
MCR does improve manipulation centricity. We begin by presenting visualizations of Grad-CAM results for MCR in Table (see all results Table 8). Qualitatively, our representation excels at capturing manipulation-relevan regions of each task. For instance, in the Robomimic Square task, MCR focuses on the gripper, square tool, and the target area-key elements of the task. In contrast, other methods like VC-1 and MVP either fail highlight these areas, or, like R3M and HRP, only capture them partially. This improved localization quantitatively in Figure 2, where MCR significantly enhances manipulation centricity across all domains. Notably, MCR even highlights the optimal path the end-effector should follow, suggesting that representation also learns essential information related task execution.
task-relevant features is further confirmed
MCR outperforms baselines on simulation tasks. Thanks to its improved manipulation centricity, MCR delivers substantial downstream performance gains compared to the strongest baseline meth- ods across all selected domains, as visualized in Figure 6. This holds even in the DexArt domain, which involves dexterous hand as the end-effector-aa setup different from the gripper used in the DROID pre-training dataset. Moreover, in the MetaWorld domain, where prior baselines struggle and perform no better than the Soarning-foom-Sreeee augmentation implemented by Hansen et al. (2023), MCR maintains significant advantage. We attribute the poor performance of baselines in Meta World to the limited number of demonstrations, which makes it difficult for policies to leverage the pre- trained representations. In contrast, MCR duces the policy's learning burden by providing manipulation-centrie features that better fit the task. In summary, experiments suggest that pre-training with large-scale robot data, along with opti- mizing for manipulation centricity, yields representations that significantly improve performance in simulated manipulation tasks.
(LfS) method, strong baseline with data
5.2 REAL ROBOT RESULTS
Experimental setup. As visualized in Figure 7, our real-world experiments involve a UR5e arm equipped with a Robotiq 2F-85 gripper and a RealSense D435i camera for RGB image capture. We designed three tabletop manipulation tasks with different objects and manipulation skills:
Lift. The robot grips the sandbag on plate and lifts up the sky.
Sweep. The robot grasps the broom to sweep the trash from the table into the dustpan.
Rearrange. The robot picks up the on-table pot and places it at the designated spot on the stove.
The demonstrations used for BC-based policy training collected by keyboards with the human operator, with 30 collected for Lift and 40 for more difficult tasks Rearrange and Sweep. The pre- trained representations frozen during policy training, inheriting the simulation setup. For fair
7

---

Published as a conference paper at ICLR 2025
Sweep
Unde
Gripper
Rearrange
Figure Real robot setup. We design 3 real-world rohot tasks with different manipulation skills and objects.
comparison, we evaluate each method with the same sets of start-up conditions, which are unseen in demonstrations, in each task. More experimental details can be found in Appendix C.5.
MCR significantly surpasses baselines on real robot tasks. The evaluation results, shown in Table 2, demonstrate that MCR consistently outperforms the baseline methods across all tasks. In the Lift and Rearrange tasks, baseline methods often fail to grasp the object accurately, particularly when object positions were unseen in the demonstrations. deteriorates further due poor accuracy in sweeping, with the trash frequently missing the dustpan and rolling off in unintended directions. In contrast, the policies trained with our representation exhibit robust and generalizable handling of these complex tasks.
the Sweep task, their performance
Task
LIS
MVP
VC-1
R3M
MCR
VC-1
MVP
Lift
5/10
8/10
5/10
6/10
%/1
Sweep
3/10
1/10
2/10
1/10
7/10
Rearrange
2/10
3/10
2/10
4/10
7/10
R3M
MCR
All
10/30
10/30
13/30
1/300
23/30
Table 2: Real robot results. Our method MCR performs best in all Figure Grad-CAM on Rearrange. tasks. Each method is fairly assessed over 10 trials on each task.
MCR is with best manipulation centricty.
We also visualize the Grad-CAM results for each method on the challenging Rearrange task in Figure 8. Representations such as R3M fail to focus on key manipulation regions, resulting in poor performance. MVP and VC-1 perform better, as they can capture the robot arm and relevant objects on the table, but they tend to over-focus on irrelevant items such the pot. Our method avoids these pitfalls, achieving the best performance across all tasks. In conclusion, these real- world experiments underscore the effectiveness of our method in enhancing robot manipulation capabilities, significantly outperforming baseline approaches in challenging scenarios.
5.3 ABLATION STUDIES
The results presented in Table 4 indicate the effect of key design choices in our approach, offering insights and guidance when employing our mamipulation-centrre representations. All experiments are conducted on three challenging tasks: Can from Robomimic, Stick Pull from MetaWorld, and Laptop from DexArt, covering all robot arms and end-effectors prior simulation experiments.
All objectives improve manipulation centricity. We begin by evaluating the effect each training objective outlined in Equation (4) through an ablation study. Our results reveal that all objectives are essential to achieving strong downstream per- formance. In particular, the dynamics alignment and action prediction losses have the most significant effectt We attribute this their ability effectively utilize dynamics-relevant in- formation in the robot dataset, which enhances manipulation centricity. Additionally, the temporal contrastive loss also plays a crucial role by capturing the temporal dependencies the video sequences. Collectively, this analysis supports our key claim: learning manipulation-centric representations is beneficial for robotic manipulation, and our proposed objec- tives substantially improve this capability.
Table 4: Key design choices of MCR.
Ablated Components Success Rate (%)
Training Objective w/o. objective Layn
66.2 (20.8)
w/o. objective Cact
71.3 (11.2)
wlo. objective Exel
72.0
Dynamic Chunk
Lengihi 1: 3-+1 Length 3:
72.1(9999
76.8 (#2.4)
Length i: 3-+7
76.8 (#2.2)
Encoder Backbone
ResNet=: 30-518
7r..
50-434
77.9 (--717
MCR (original)
83.2 (+113)
Medium dynamic chunk length works best. Recall that dynamic chunks define the temporal horizon of robot state-action pairs utilized for modeling robot dynamics, specified in Equation (1).
8

---

Published as a conference paper at ICLR 2025
Our experiments indicate that medium chunk length of yields optimal performance. In contrast, a shorter chunk length (i.e., 1) fails to adequately capture action information, resulting in an insufficient representation of the underlying dynamics. Additionally, the size of single-state chunk considerably smaller than that of the pre-trained visual features, which can introduce noise when projected through multi-layer perceptron (MLP). On the other hand, excessively long chunks may complicate the accurate modeling of dynamics. This study highlights the necessity of effectively encoding dynamic information in representations to enhance manipulation centricity.
Larger encoders lead to better performance. Finally, we also observe a clear trend: larger en- coders consistently result in better performance. We hypothesize that larger models have a greater capacity to comprehend complex scenes and capture critical information related to dynamics and manipulation centricity. This finding is consistent with the ones in previous work (Shang et al., 2024), suggesting that our method scales well with the model size and has the potential for even greater improvements in performance if larger models are employed.
5.4 ANALYSIS ON ROBOT DATASET
Besides the studies on methodology design, we also reveal more insights into the utilization of robot dataset learning our robotic representations.
Larger dataset, better performance. We begin by examining how the size of the robot dataset affects pre-training outcomes. Specifically, reduced the data in each DROID scene from 100% to 25% and assessed the downstream performance, as shown in Figure 9. Our findings indicate that larger datasets contribute to improved representation quality. This seemingly contrasts with previous research (Dasari al., 2023), which suggested that merely increasing dataset size may not yield benefits. We remark that the key differences here are (1) our use of robot data rather than human data, and (2) the incorpo- ration of additional robot dynamics information. These factors Figure 9: Effeet of robot dataset size. enhance the scalability and effectiveness of MCR when applied to larger robot datasets. While some studies have attempted to combine human and robot data (Dasari et al., 2023; Majumdar et al., 2023), little improvement was observed, likely due to the insufficient utilization of robot dynamics. In summary, our method effectively scales with dataset size by leveraging dynamics information.
Success Rate (%)
82.5
77.5
76.0
72.5
70.0
25%
78% 100%
Dataset Scale
Greater benefits for tasks with less embodiment gap. Next, we investigate which downstream tasks benefit more from pre- training with robot data. Specifically, we examine the em-
Relsbve Change with R3M (5)
bodiment gap associated with the end-effector. The simula- tion tasks are accordingly categorized into gripper-based and dexterous hand-based tasks. As illustrated in Figure 10, both R3M-DROID and MCR outperform R3M on gripper-based tasks in terms of manipulation centricity and downstream suc- cess rate, supporting our earlier conclusions. However, perfor- mance drops in hand-based tasks, where R3M-DROID even underperforms R3M. This is likely due to the fact that all data in DROID was collected using gripper, which presents hands. To alleviate this issue, we suggest two attempts in the future: (1) leveraging dynamics from robot data better mitigate the embodiment gap, and (2) incorporating more end-effectors, such as dexterous hands, into robot datasets to enhance our manipulation-centre representations.
16
Cripper-based Tasks
Hand-based Tasks
Murks, Dartr wty.
Marg: Centrictty (MC/) Suc0ss Putio IPOI)
Success Rate 21000
Figure 10: Downstream domain gap.
embodiment gap compared to dexterous
Feature analysis. To gain more intuitive understanding of the impact of the robot dataset, we em- ploy t-SNE (Van der Maaten & Hinton, 2008) to process and visualize feature embeddings generated by the pre-trained encoder. The visualization results are presented in Figure 11. the simulation domain, R3M struggles to cluster images within individual tasks. However, this issue is partially al- leviated by using a robot dataset, as R3M-DROID exhibits improved clustering ability. Nonetheless, due to the significant domain gap between DROID and simulation environments, it still encounters difficulties in distinguishing many tasks. In contrast, our method demonstrates markedly superior clustering capabilities, indicating the importance of incorporating real-robot dynamics for effective robotic representation. Additionally, we observe that all three methods exhibit good clustering per- formance in real-world tasks, likely attributed to the more distinct visual changes between tasks

---

Published as a conference paper at ICLR 2025
R3M
R3M-DROID
MCR (ours)
R3M
R3M-DROID
MCR (ours)
age dlgn
Sim.
Real
Figure 11: t-SNE visualization. We do t-SNE visualization on 10 simulation tasks from MetaWorld and 3 real robot tasks. Each dot represents an image frame and each color indicates task. The results demonstrate that (1) our representation has the best clustering ability and (2) robot data is helpful to robotic representation.
compared to simulations. However, R3M remains inferior to the other two methods, reinforcing the critical role of robot datasets in enhancing robotic representations.
6 RELATED WORKS
Pretrained robotic representations. The development of pre- pre-trained visual representations (PVRs) has significantly improved the efficiency of downstream policy learning robotics. Notable ap- proaches include a variant of MoCo-v2 (Parisi et al., 2022) that integrates multi-layer features, MVP (Xiao et al., 2022) and VC-1 (Majumdar et al., 2023) utilizing Masked Autoencoders (He et al., 2022), and R3M (Nair et al., 2022), which employs time-contrastive objective and video- language alignment. Other works like Zheng et al. (2024) use temporal action contrastive learning, while MPI (Jia et al., 2024) focuses on predicting transition frames based on language instructions. HRP (Srirama et al., 2024) extracts affordances from large-scale human videos for enhanced gener- alization, and Theia (Shang et al., 2024) distills diverse vision models for robot learning. VIP (Ma et al., 2023) generates dense reward functions for robotic tasks. Similar to our method, RPT (Ra- dosavovic al., 2023) employs trajectory labels for representation training. In contrast, our work introduces the concept of manipulation centricity, leveraging large-scale robotic data to capture manipulation-specitf dynamics, resulting improved performance on downstream tasks.
Learning from large-scale robotic data. Recent advancements in robotics increasingly utilize large-scale datasets to enhance the capabilities of robotic systems. Collaboration et al. (2024) in- troduces the Open X-Embodiment dataset, the largest robotic dataset comprising extensive data from diverse robot embodiments across various tasks. The RT-X model (Collaboration et al., 2024), trained on this diverse dataset, shows promising results in cross-robot skill transfer generalist. Team et al. (2024) introduces Octo, a transformer based diffusion policy also trained on Open X- Embodiment, supporting flexible task and observation definitions. Additionally, Open VLA (Kim et al., 2024) is developed as a vision-language model enabling direct mapping from image inputs to continuous robot actions. Unlike previous approaches (Liang et al., 2024; Zheng et al., 2025), our work focuses on extracting dynamics and interaction information from robotic datasets to create specialized visual representations for policy learning, offering an efficient alternative generalist policies. Other related works about dynamics-awan representations and factors driving effective- ness in robotic representations are presented Appendix A.
7 CONCLUSIONS AND DISCUSSIONS
Our work introduces the concept of manipulation centricity in visual representations for robotic manipulation, revealing its crucial role in downstream task performance. By leveraging large-scale robot data and dynamics-aware learning objectives, we develop a method that significantly enhances the extraction of manipulation-centri features. This approach not only advances the state-of-the-ar in robotic manipulation across diverse tasks but also provides a new lens through which to un- derstand and evaluate representation learning in robotics. Our findings highlight the importance of aligning representation learning with the specific demands of robotic control, potentially shifting the paradigm of how we approach feature extraction for embodied agents. Looking forward, this work opens avenues for exploring multi-modal integration, such as using language instructions to learn task-aware features and further leveraging trajectory data to capture spatial-tempon robotic dynam- ics, promising to further narrow the gap between pre-trained generalized representation models and real-world robotic manipulation.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDG
This project was supported by National Key R&D Program of China (2022Z00161700) We would like thank Zhecheng Yuan, Tianming Wei, Yanjie Ze, and Yuanhang Zhang for their insightful discussions and technical supports.
REFERENCES
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and Autonomous Systems, 2009.
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15, 1995.
Chen Bao, Helin Xu, Yuzhe Qin, and Xiaolong Wang. Dexart: Benchmarking generalizable dex- terous manipulation with articulated objects. Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Ri-1: Robotics transformer for real-world control at scale. Robotics: Science and Systems (RSS), 2023.
Kaylee Burns, Zach Witzel, Jubayer Ibn Hamid, Tianhe Yu, Chelsea Finn, and Karol Hausman. What makes pre-trained visual representations successful for robust manipulation? In Conference on Robot Learning (CoRL), 2024.
Hongyi Chen, Abulikemu Abuduweili, Aviral Agrawal, Yunhai Han, Harish Ravichandar, Changliu Liu, and Jeffrey Ichnowski. Korol: Learning visualizable object feature with koopman operator rollout for manipulation. In Conference Robot Learning (CoRL), 2024.
Open X-Embodiment Collaboration, Abby O'Neill, Abdul Rehman, Abhiram Maddukuri, Ab- hishek Gupta, Abhishek Padalkar, Abraham Lee, Acom Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew Wang. Andrey Kolobov, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Rafin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang, Chenfeng Xu. Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Ed- ward Johns, Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Felipe Vieira Fru- jeri, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guangwen Yang, Guanzhi Wang. Hao Su, Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jaehyung Kim, Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang, Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang. Jordi Sal- vador, Joseph J. Lim, Junhyek Han, Kaiyuan Wang, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala, Kirsty El- lis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng. Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen, Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi "Jim" Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Mem- mel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip. Mingtong Zhang, Mingyu Ding, Minho Heo, Mohan Kumar Sri- rama, Mohit Sharma, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess. Nikhil J Joshi, Niko Suenderhauf, Ning Liu, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R Sanketi, Patrick "Tree" Miller, Patrick Yin,

---

Published as a conference paper at ICLR 2025
Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong. Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Mart 'n- Mart'in, in. Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Lin, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shu- ran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vincent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiangyu Chen, Xiaolong Wang. Xinghao Zhu, Xinyang Geng, Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yan- song Pang, Yao Lu, Yecheng Jason Ma, Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang. Yonatan Bisk, Yongqiang Dou, Yoonyoung Cho. Youngwoon Lee, Yuchen Cui, Yue Cao, Yueh-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen Zhang, Zipeng Fu, and Zipeng Lin. Open X-Embodiient: Robotic learning datasets and RT-X models. In International Conference on Robotics and Automation (ICRA), 2024.
Sudeep Dasari, Mohan Kumar Srirama, Unnat Jain, and Abhinav Gupta. An unbiased look at datasets for visuo-motor pre-training. In Conference on Robot Learning (CoRL), 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics (NAACL), 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
Christoph Feichtenhofer, Yanghao Li, Kaiming He, et al. Masked autoencoders as spatiotemporal learners. Advances in neural information processing systems, 01-35946-35995 2022.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitage et al. The" something something" video database for learning and evaluating visual common sense. In International Conference on Computer Vision (ICCV), 2017.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh K. Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Meng- meng Xu, Eric Z. Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do. Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Christian Fuegen, Abrham Kahsay Gebreselasie, Cristina González, James M. Hillis, Xuhua Huang. Yifei Huang. Wenqi Jia, Weslie Khoo, Jáchym Kolár, Satwik Kottur, Anurag Kumar. Fed- erico Landini, Chao Li, Yanghao Li. Zhengiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ra- mazanova, Leda Sari, Kiran K. Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi, Yunyi Zhu, Pablo Arbeláez, David Crandall, Dima Damen, Giovanni Maria Farinella, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawa- har, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard A. Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. Ego4d: Around the world in 3,000 hours of egocentric video. In Conference Computer Vision and Pattern Recognition (CVPR), 2021.
12

---

Published as a conference paper at ICLR 2025
Nicklas Hansen, Zhecheng Yuan, Yanjie Ze, Tongzhou Mu, Aravind Rajeswaran, Hao Su, Huazhe Xu, and Xiaolong Wang. On pre-training for visuo-motor control: Revisiting a learning- from- scratch baseline. In International Conference on Machine Learning (ICML). 2023.
Kaiming He, X. Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In International Conference on Machine Learning (ICML), 2020.
Tao Huang, Guangqi Jiang, Yanjie Ze, and Huazhe Xu. Diffusion reward: Learning rewards via conditional video diffusion. European Conference on Computer Vision (ECCV), 2024.
Tianying Ji, Yongyuan Liang, Yan Zeng, Yu Luo, Guowei Xu, Jiawei Guo, Ruijie Zheng, Furong Huang. Fuchun Sun, and Huazhe Xu. ACE: Off-policy actor-critic with causality-aware entropy regularization. In Forty-firs: first International Conference Machine Learning, 2024. https:
//openreview. net. formm? wttppummy2md0ppuu
Zeng Jia, Bu Qingwen, Wang Bangjun, Xia Wenke, Chen Li, Dong Hao, Song Haoming, Wang Dong, Hu Di, Luo Ping, Cui Heming, Zhao Bin, Li Xuelong, Qiao Yu, and Li Hongyang. Learning manipulation by predicting interaction. Roboties: Science and Systems (RSS), 2024.
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepent, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Young- woon Lee, Marius Memmel, Sungjae Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pan- nag R Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Bai- jal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Donovon Jack- son, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin, Zehan Ma. Abhiram Maddukuri, Suvir Mir- chandani, Daniel Morton, Tony Nguyen, Abigail O' O'Neill, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew E. Wang, Yilin Wu, Annie Xie, Jingyun Yang. Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg. Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph J Lim, Jitendra Malik, Roberto Martín Martín, Sub- ramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip, Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. Droid: A large-scale in-the-wild robot manip- ulation dataset. Robotics: Science and Systems, 2024.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-tanguage-action model. arXiv preprint arXiv:2406.09246, 2024.
Diederik Kingma. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: Contrastive unsupervised represen- tations for reinforcement learning. In International Conference Machine Learning (ICML), 2020.
Yongyuan Liang, Tingqiang Xu, Kaizhe Hu, Guangqi Jiang, Furong Huang, and Huazhe Xu. A generalizable policy network generator with behavior- chaviee-prompted diffusion. In The Thihty-iighth Annual Conference on Neural Information Processing Systems, 2024. URL https: htttps:/cppereeviee net forummid=pp Tm.
13

---

Published as a conference paper at ICLR 2025
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. VIP: Towards universal visual reward and representation via value- aalue-implicit pre-training. In International Conference Learning Representations (ICLR), 2023.
Arjun Majumdar. Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Pieter Abbeel, Jitendra Malik, Dhruv Batra, Yixin Lin, Olek- sandr Maksymets, Aravind Rajeswaran, and Franziska Meier. Where are we in the search for an artificial visual cortex for embodied intelligence? In International Conference on Neural Information Processing Systems (NeurIPS), 2023.
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese. Yuke Zhu, and Roberto Martín-Martinn What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning (CoRL), 2021.
Bogdan Mazoure, Remi Tachet des Combes, Thang Long Doan, Philipp Bachman, and R De- Von Hjelm. Deep reinforcement and infomax learning. In International Conference on Neural Information Processing Systems (NeurIPS), 2020.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A uni- versal visual representation for robot manipulation. In Conference on Robot Learning (CoRL), 2022.
Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. Robocasa: Large-scale simulation everyday tasks for gener- alist robots. In Robotics: Science and Systems (RSS), 2024.
Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsurprising effectiveness of pre-trained vision models for control. In International Conference on Machine Learning (ICML)), 2022.
Ilija Radosavovic, Baifeng Shi, Letian Fu, Ken Goldberg, Trevor Darrell, and Jitendra Malik. Robot learning with sensorimotor pre-training. In Conference on Robot Learning (CoRL), 2023.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00 14, 2024.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In International Conference on Computer Vision (ICCV), 2017.
Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and P. Abbeel. Masked world models for visual control. Conference on Robot Learning (CoRL), 2022.
Jinghuan Shang, Karl Schmeckpeper, Brandon B. May, Maria Vittoria Minniti, Tarik Kelestemur, David Watkins, and Laura Herlant. Theia: Distilling diverse vision foundation models for robot learning. In Conference on Robot Learning (CoRL), 2024.
Mohan Kumar Srirama, Sudeep Dasari, Shikhar Bahl, and Abhinav Gupta. Hrp: Human affordances for robotic pre-training Robotics: Science and Systems (RSS), 2024.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In International Conference on Machine Learning (ICML), 2021.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. Robotics: Science and Systems (RSS), 2024.
Aaron van den Oord, Yazhe Li. and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 018800077744 2019.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research (JMLR), 2008
14

---

Published as a conference paper at ICLR 2025
Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen- Euttscn Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning at scale. In Conference Robot Learning (CoRL., 2023.
Tete Xiao, Iliaa Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. Conference on Robot Learning (CoRL), 2022.
Guowei Xu, Ruijie Zheng, Yongyuan Liang, Xiyao Wang, Zhecheng Yuan, Tianying Ji, Yu Luo, Xiaoyu Liu, Jiaxin Yuan, Pu Hua, et al. Drm: Mastering visual reinforcement learning through dormant ratio minimization. International Conference on Learning Representations (ICLR), 2023.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-worid: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning (CoRL), 2019.
Zhecheng Yuan, Zhengrong Xue, Bo Yuan, Xueqian Wang, Wu, Yang Gao, and Huazhe Xu. Pre-trained image encoder for generalizable visual reinforcement learning. Advances in Neural Information Processing Systems, 35:1 3022-13037 2022.
Yanjie Ze, Gu Zhang, Kangning Zhang. Chenyuan Hu, Muhan Wang, and Huazhe Xu. diffu- sion policy: Generalizable visuomotor policy learning via simple 3d representations. Roboties: Science and Systems (RSS), 2024.
Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. Robotics: Science and Systems (RSS), 2023.
Ruijie Zheng, Xiyao Wang, Yanchao Sun, Shuang Ma, Jieyu Zhao, Huazhe Xu, Hal Daumé III, and Furong Huang. TACO: Temporal latent action-driven contrastive loss for visual reinforce- ment learning. In International Conference on Neural Information Processing Systems (NeurIPS), 2023.
Ruijie Zheng, Yongyuan Liang, Xiyao Wang, Shuang Ma, Hal Daumé III, Huazhe Xu, John Lang- ford, Praveen Palanisamy, Kalyan Shankar Basu, and Furong Huang. Premier-taco: Pretraining multitask representation via temporal action-driven contrastive loss. In International Conference on Machine Learning (ICML), 2024.
Ruijie Zheng, Yongyuan Liang, Shuaiyi Huang, Jianfeng Gao, Hal Daumé III, Andrey Kolobov, Furong Huang, and Jianwei Yang. VLA: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies. In The Thirteenth International Conference Learning Representations, 2025. URL https: opwnreview..eeeeeoooee net bicv.mcu11bbcdde1eee
15

---

Published as a conference paper at ICLR 2025
A ADDITIONAL RELATED WORKS
Dynamics-aware representation. Representation learning is crucial for extracting key features from image inputs in imitation learning and reinforcement learning (Xu et al., 2023; Ji al., 2024; Yuan et., al., 2022). CURL (Laskin et al., 2020) uses InfoNCE (van den Oord et al., 2019) (11 to maximize agreement between augmented observations, while CPC (Henaff, 2020) and ATC (Stooke et al., 2021) incorporate temporal dynamics into contrastive loss. DRIML (Mazoure et al., 2020) proposes a policy-dependent auxiliary objective, and KOROL (Chen et al., 2024) trains feature extractors on task-specific RGBD images during Koopman operator dynamics learning. TACO (Zheng et al., 2023; 2024) optimizes mutual information between current state-action pairs and future states. In contrast, approach trains generalized robotic representation model extract more effective dynamics-aware representations, offering improved efficiency and generalization through a train- once-for-all methodology.
Factors driving effectiveness in robotic representation. Burns et al. (2024) identifies emergent segmentation ability as a critical factor in the success of pre-trained visual models for robotic ma- nipulation. While their focus is on generalization capability, our emphasis is on the downstream learning efficiency of robotic representations, investigating this from the perspective of manipula- tion centricity. Additionally, Theia (Shang et al., 2024) highlights the importance of high entropy feature norm distribution for performance enhancement. Although they validate their findings in simulation locomotion tasks, we evaluate the effectiveness of manipulation centricity in both simu- lated and real-world robot manipulation tasks.
B TRAINING EFFICIENCY
Pre-training representations typically requires exten- sive computational resources. Methods like VC-1 and MVP have prolonged training times due to their use of MAE (Feichtenhofer et al., 2022). Even the most computationally efficient baseline, R3M, requires 120 hours single NVIDIA V100. Our method, how- ever, achieves state-of-the-an performance while re- quiring less computation. As shown in Table 5, approach has shorter training time than R3M, achieving an good balance between performance and computational efficiency.
Table 5: Computation efficiency. Training
computation requirements across methods.
Method GPU Type Training Time (h)
R3M
Tesla V100
R
MCR
RTX 3090
R0
C
MORE EXPERIMENTAL DETAILS
C.I TASK DESCRIPTIONS
We select diverse set tasks from various robotic manipulation benchmarks for evaluation. Specifically, include three tasks from Robomimic (Mandlekar et al., 2021), three tasks from RoboCasa (Nasiriany et al., 2024), ten tasks from MetaWorld (Yu et al., 2019), and four dexterous tasks from DexArt (Bao al., 2023). Detailed descriptions of each task provided below:
Can (Robomimic, E ) the task to manipulate the can using the robot's arm to perform various actions such as picking up, moving it to a different location, and orienting it in a specific way.
Lift (Robomimic, A E IR?): the task is to grasp specified item and then raise it to desired height.
Square (Robomimic, A e RT); the task is to pick up a square- nut and place it onto a rod successfully.
Close Drawer (RoboCasa, e R²): the task is to accurately close drawer.
Coffee Button Press (RoboCasa, AA E RR7:: the task is to press the button on the coffee machine to start the coffee brewing process.
Open Single Door (RoboCasa, A E IR?)): the task to open a door that singularly paneled, such as a cabinet or microwave door, which is already closed.
Assembly (MetaWorld, A R4);: the task is grasp a and position it on dowel using the gripper.
Bin Picking (MetaWorld, A E IR:); the task is to transfer disc from bin container to another.
16

---

Published as a conference paper at ICLR 2025
Button Press (MetaWorld, A R²): the task is press button using robotic arm to
activate a device.
Disassemble (MetaWorld, A E I): the task is to remove nut from a peg by picking it up,
Drawer Open (MetaWorld, A E IR:)): the task
accurately open drawer.
Hammer (MetaWorid, A E R²: the task is to drive a screw into the wall using hammer.
Pick Place Wall (MetaWorld, the puck in the designated spot.
R²): the task is to grab puck, go around a wall and put
Shelf f Place (MetaWorld, A E I): the task is grab a puck and set it shelf.
Stick Pull (MetaWorld, AA R+); the task is to use stick to pull a box by holding onto the stick.
Stick Push (MetaWorld, A E I): the task to hold a stick to push a box with it.
Bucket (DexArt, A I²²);: the task is to elevate bucket..
Faucet (DexArt, A R=²): the task is activate a faucet using rotating joint.
Laptop (DexArt, A I=: the task is to grasp the center of the display and then lift the laptop cover.
Toilet (DexArt, A E )= ) the task is to initiate the process of lifting a bigger toilet seat.
C.2 MORE GRAD-CAM VISUALIZATIONS
The Grad-CAM visualizations for each task are presented Table 8, which provides a compre- hensive comparison with other baseline methods. Notably, approach shown effectively facilitate the capture of key manipulation-centri features, thereby enhancing the model's ability to focus on the most relevant aspects of the task.
C.3 PRE-TRAINING HYPERPARAMETERS
We show our hyperparameters during the pre-training stage in Table 6. Downstream policy learning settings are introduced in Section C.5,
Table 6: Hyperparameters for MCR pre-training.
Hyperparameter
Value
Encoder type
ResNet50
Batch size
32
Learning rate
le-4
Training steps
500,000
Data augmentation
RandomResizeddCrpp (224,00.5, 1.0))
Optimizer
Adam
DROID views used
two exterior views
DROID proprioception used
cartesian and gripper position
C.4 PRE-TRAINING IMPLEMENTATION
Our codebase is built upon the implementation of R3M. Similarly, for one sample within the batch, we have 5 frames from one video. The initial and final frames are sampled from the first and last 20% of the video, respectively. We employ the same contrastive learning loss implementation, modifying only the positive and negative sample pairs. For more details, please refer to Nair et al. (2022).
To clarify further, we provide our PyTorch-like code implementation of L, as below:
actor_trum
self.cutde is the output dimension of ResNet, for example, ResNet50
is 2048
an.Sequential (nn. Linear nn. LayerrNoom (50), nn Tanh ())
50),
actor_policy - nn. Sequential action dim in our case in for DROID dataset
17

---

Published as a conference paper at ICLR 2025
Ln.ein (50, 512), ReLU (inplace-True), In.ainear (512, 512), Rn. (inglaco-Tuu))) In.Linear (512, action_dim))
The MLP part of dynamics alignment loss Lg is implemented as follows:
calculate the length of state-action dynamics chunk
inate_input_dim = 14 * self.state_chunk
4 state
+= (self.state htt.:sssat___cuk___te_ chankkchhkkkkaaeeeeeeeeee - 1) action
state_ nn. Sequentii
In.Lincar (stato_input_die 1024), nn. (), nn.Linear (1024, self.outdim)
C.5 BC IMPLEMENTATION AND SETTINGS
World and DexArt. We evaluate our model on these two domains following (Huang et al., 2024) and (Ze al., 2024). Scripted policies are used generate demonstrations in MetaWorld, and policies trained with Reinforcement Learning is used in DexArt. We generate 25 demonstra- tions per task in Meta World and 100 in DexArt, where robot end effectors and objects are initially randomized. The downstream BC policy is three-layer MLP with ReLu activations and hidden sizes of 256. A BatchNorm layer is added before the feature is input into the MLP. The policy is trained with mean squared error (MSE) loss, learning rate of 0.001, and batch size of 256.
Robomimic. We adopt the released behavior cloning implementation from RoboMimic, and use their standard imitation learning dataset. The dataset contains 200 demonstrations for each task. We only modify the code to evaluate diverse visual encoders. The downstream policy is a two-layer MLP with hidden sizes of 1024. The policy trained with MSE loss, an initial learning rate of 0,001, and batch size of 100. The learning rate is decayed with a factor of 1.
RoboCasa. We utilize the official policy learning implementation from RoboCasa. We use the BC- Transformer algorithm implemented by RoboMimic, with a RoboCasa-standard configuration as reported in the appendix of paper Nasiriany et (2024). We modify the visual observation encoder and train BC policies using the "Human-50" dataset collected by human operators. The policy is trained with an MSE loss, an initial learning rate of 0.0001, and a batch size of 16. The learning rate is decayed with factor 1.
Real world Our real-world codebase is adapted from V-D4RL. We collect demonstrations using keyboard interface, with 40 demonstrations for Sweep and Rearrange, and 30 for Lift due to its simplicity. The policy is a 3-layer BatchNormMLPP with a hidden size of 1024. We train the policy using the log-likelihood loss, with learning rate of 0.0001 and batch size of 256.
C.6 BEHAVIOR CLONING (BC) PERFORMANCE OF EACH SIMULATION TASK
We show the performance per task of our main results in table 7. Our method achieves the highest success rate in 19 out of 20 tasks evaluated, showcasing its robustness and adaptability to various scenarios.
D MORE DETAILS ON MANIPULATION CENTRICITY
Grad-CAM. Grad-CAM widely used technique for generating heatmaps of input images to identify the regions that the encoder focuses on. We utilize the PyTorch-Grad-CAl library to generate Grad-CAM figures for each visual encoder. In convolutional neural networks (CNNs), Grad-CAM generates heatmaps by backpropagating gradients through the final convolutional lay- ers, thereby highlighting important image regions. However, in Vision Transformers (ViT), the final classification is based on the class token computed in the last attention block, which means that the output is not influenced by the 14x 14 spatial channels the final layer, resulting in zero gradi- ents. To generate meaningful visualizations in VIT, it is necessary choose a layer before the final attention block. In our study, we chose the LayerNorm applied to the output of the attennttnn mechanism in the last Transformer block.
18

---

Published as a conference paper at ICLR 2025
Table 7: Main results 20 simulation tasks. Results for each task are provided in this table. A summary across domains is shown in Figure 6.
DexArt
Robumimic 130
RabocCasa
Alg Tisk
Bucket
Fancet
Laplop
Toilet
Can
Square
CloseDrawer
CoffeeBurtioPree
OpenSingleDoor
MCK (ouns)
36.7 (82.9)
38.3 (22.9)
93.3 (8229)
(52.9
68.0 (:488
96.0
(:2.3)
30,0
(21.3)
99.3 (2:12))
72.0 (13.5)
(6.35)
ESS
(1:.8)
83.3 10:10.
60..0.00
64.0
(-1.0)
(000)
cm
$2.0 (1-L.0)
(6.7
MVP
(12.9
(3.2))
81.7
80.0 (00,))
(2200
74.0
(.6.0)
14.0 (22.3)
98.0
52.7 (1:38.90
33.3 (114.5)
35.0.3000)
98.7
29.3
(82.9)
36.2 (8:29)
(65..)
(22.3)
(6000
26.0
88.7
47.3
HRP
31.7 (82.8)
36.7 (8:2.)
90.0
(35.0)
63.3
(21444)
(2.0
(23.5)
86.0
(33.)
26.0
(22.3)
91.3
(24.0)
35.3
(1:31.8)
38.0 (266.))
R5M-Droid
(2..0)
33.3 (+2.4)
80.0 (40.0)
66.7 (87.6)
54.0
54.0
96.0
(20.8)
(2.00=3.1
(-3.1)
88.7
(22.5))
51.5 (#2.55)
World
MetaWorkd
(Medium) Haamer
MetaWorkd (Iard)
MedaWorld (Very
Hlrdd)
Ale Tisk
Button
Press
Drower Open
Din Picking
Assembly
Shelf Phoce
Disaseemble
Stick Poll
Stick Push
Pick Place Wall
MCR (ours)
100.0 (1:8.))
100.0
96.7 (82.9)
(00.0)
100,0
41.7 (1:58)
(223))
86.7 (1229)
100.0 (1:8.)
91.7 (#2.9)
LBS
96.7 (12..0)
95.01 (=5.0)
81.7 (22299
95.0 (15..)
(2..0
35.0 (13500
N6.7 (8:2.0)
83.3 (25.8)
96.7 (1:2.00
85.0 (E5.0)
MVP
96.7
81.7 241.99
91.7
(2:20)
20.0 (53.10)
(5.0
75.0
96.7 (92210
76.7c.11.00
VC-1
98.3
78.3
(1:2.))
(2.0)
21.7 (52.9)
66.7
86.2
98.3
71.7
(1:2.0
71.7
21.7
63.3
35.0
43.3
71.7
58.3
HRP
98.3
96.7
81.7
R3M-Droid
(4/2.))
96.7
(1.3
(4221))
Our paper applies Grayscale CAM to measure Manipulation Centricity, where the pixel values range from 0 (black) to 255 (white), with higher values (closer to white) indicating regions that contribute more significantly to the model's decision. Furthermore, we binarize the Grayscale CAM by thresh- olding pixel values 2, setting values below 2 to 0 and those above equal to to non-zero values, thereby mitigating the effect of noise and facilitating more accurate interpretations and precise met- ric calculations.
Segment Anything Model (SAM 2, Ravi et al. (2024)) is a large-scale segmentation model. In study, utilize SAM2 segment demonstration videos of all simulation tasks. Specifically, we select the official 'base plus' model available on the SAM2 repository and adopt the implementation of the SAM2-GUI, an interactive graphical user interface to utilize pre-trained SAM2 models. We manually upload each task video and add prompt points to generate segmentation videos. Notably, foreground pixels, including robot end effectors and objects, are labeled with non- zero values, while background pixels are labeled with zero. A full list of ground truth annotations is shown in Table 8, to better illustrate manipulation centricity.
SAM 2.
Jaccard Index. We employ the Jaccard Index to quantify the similarity between Grad-CAM and SAM2 video frames, with the index ranging from 0 to 1. A higher Jaccard Index value indicates a greater degree of similarity between the two, suggesting stronger alignment between the attention maps generated by Grad-CAM and the segmentation masks produced by SAM2.
E DETAILED ANALYSIS OF DROID SUBSET USED
As described in Section 1. we perform preliminary preprocessing of the full 1.7TB RLDS dataset, yielding a subset that serves as the basis for our analysis. In this section, we present statistical characterization of this subset. Specifically, we exclude videos with fewer than 40 frames. The distribution of video lengths in the subset is illustrated in Figure 12a. Furthermore, we conduct a comprehensive statistical analysis of the language instructions accompanying the dataset, with particular focus on the frequency of common object nouns and action verbs. The results of this analysis are visualized in Figures 12b and 12c, respectively. Notably, our subset covers a large degree of data diversity, capturing a wide range of scenarios, objects, and actions, thereby ensuring the effectiveness of our method.
19

---

Published as a conference paper at ICLR 2025
2459
3459
40000
24000
3009
20000
10890
Statistics Trujectory Length.
(b) Statistics Object Noun.
Statisties
Figure 12: Statistical Analysis of the DROID Subset.
Table 8: Grad-CAM of all tasks.
R3M- DROID
Task
Annotation
MVP
VC-1
HRP
R3M
MCR
Can
Lift
Square
CoffeeButton CireeBButonCoseeDeee Drawer
OpenDoor
Assembly
BinPicking
ButtonPress
Continue
20

---

Published as a conference paper at ICLR 2025
Table 8: Grad-CAM of all tasks
R3M- DROID
Task
Annotation
MVP
VC-1
HRP
R3M
MCR
DraverOpen Disassembee
Hammer
Pick Wall
SheliPlace
StickPull
StickPush
Bucket
Faucet
Laptop
Toilet
21


Reviews:
[{"review_id": "b7dBP4H0nw", "reviewer": "Reviewer_wcpD", "summary": "The paper first proposes manipulation centricity, a new metric for evaluating representations for robotics, and finds that it is a strong indicator of success rates in downstream robotic manipulation tasks. Building on this insight, it proposes RPM, a framework for learning visual representation for robotics from large-scale robotics dataset. Experiments in both simulation and the real world show that RPM leads to improved manipulation centricity and higher task success rate when evaluated using behavior cloning.", "questions": "- Why is it still necessary to attend to the manipulator region when there is already proprioception input?\n- What is the evaluation protocol in the real world? How are the environments varied between training and evaluation, and what would count as a success?\n- By attending the robot and task-relevant object, technically this would allow the representation to have stronger generalization capabilities. Perhaps the author should run some visual generalization experiments to find out if RPM enables better generalization.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is clear and well presented\n- The simulation experiments are rigorous and have a lot of diversity\n- The paper presents a novel metric for evaluating visual representations for robotics, and novel findings on how to integrate robotics information (states, actions) for training better representations", "weaknesses": "- There could be more empirical analysis and intuition regarding how each loss affects manipulation centricity. Authors could explain intuitively how each loss contributes to better MC and visualize the gradcam of an encoder trained with each loss\n- The paper lacks comparison to more state-of-the-art baselines, such as VIP/LIV, or object-centric pre-trained representations such as POCR/HODOR\n- As I understand, the goal here is to make the representation pay attention to task-relevant regions e.g. robot and task-relevant objects. The authors have already demonstrated that this information can be obtained using SAM2. Have the authors considered directly using the masks as a form of supervision for training the representation? I think this could serve as a baseline, since RPM requires additional priveleged information that only a robotics dataset has, i.e. state and action, whereas we can generate segmentation masks for any general sources of data.\n- The paper could benefit from more extensive real robot experiments", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SVH2uKc9EA", "reviewer": "Reviewer_3aRj", "summary": "The paper proposes pre-train visual representations for robot manipulation by focusing on manipulation centricity, robot datasets (instead of human activity datasets), and auxiliary loss objectives for dynamic alignment and time contrastive learning. To measure Manipulation Centricity, they use Jaccard similarity between the binarized Grayscale CAM and SAM2’s foreground vs. background predictions. The proposed model RPM outperforms simulation and real robot tasks compared to baselines.", "questions": "1. What is the tradeoff between the visual realism of the simulator vs simulator-specific training required to achieve high success rate and manipulation centricity? For example, in RoboCasa, manipulation centricity seems really low compared to others.  How to compare it across simulators?\n\n1. How does \"manipulation centricity\" depend on the choice of camera intrinsics and extrinsic with respect to the robot body frame?  \n\n1. R3M has been shown to be a poor visual representation model for robotic tasks, as compared to visual datasets [1] and pretraining image distribution matters. Why none of the baselines involve vision only representation backbones for comparison?\n\n[1] Dasari et al, 2023. An Unbiased Look at Datasets for Visuo-Motor Pre-Training, https://data4robotics.github.io/", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Detailed empirical validation across diverse tasks and simulation domains, plus real robot experiments\n1. well-motivated concept of manipulation centricity with clear metrics\n1. ablation studies and analysis of design choices \n1. insights about benefits of robot vs human datasets for pre-training\n1. Clearly written key research questions and contributions", "weaknesses": "The work presents insights into the visual representations that focus on manipulation centricity and dynamic alignment perform better than existing approaches. However, there are certain assumptions whose implications are not clearly discussed.\n1. DROID is chosen as the robot dataset of choice instead of other larger dataset in OXE.\n1. RPM is a ResNet-based model instead of directly using / comparing to pretrained models like SAM2, etc.\n1. The \"manipulation centricity\" does not seem to improve at similar scale across different simulators, with respect to success rate. \n1. A concise definition of manipulation centricity and how it is calculated should be present in the main paper, rather than existing only in appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vr3XJqU0Qn", "reviewer": "Reviewer_cSN9", "summary": "This paper introduces a metric (manipulation-centricity) for assessing the ability of pretrained visual encoders to model task-relevant features for robot manipulation tasks; as well as a novel pretraining objective (RPM) that incorporates the robot dynamics information available in robotics datasets. The authors first establish a correlation between the metric they define and the success rate of policies that leverage the features generated by pretrained visual encoders. Having established a positive correlation they seek to design a pretraining objective for visual encoders and evaluate its performance on their proposed metric and the downstream policy success rates. The novelty in the pretraining objective the authors introduce is a dynamics alignment term which takes the form of the InfoNCE applied to embeddings of a sequence of state action pairs and image embeddings. This novel dynamics alignment term is combined with existing behaviour cloning and temporal contrastive terms to give the full learning objective. They apply this objective function to pretrain visual encoders for manipulation tasks and demonstrate that this pretraining objective leads to improvements on their metric and downstream task performance in both simulated and real-world experiments.", "questions": "Congratulations on this work and thanks for contributing it to the robot learning community. \n\nI wished to clarify if during policy training does the visual encoder always remains frozen and whether you finetune the entire architecture? In the first part of the paper when evaluating your introduced metric it is clear that the encoder is frozen, I wished to confirm whether this is also the case for the reported task success rates and whether you considered finetuning the entire architecture?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper validates a pretraining objective for visual encoders that improves the performance of robot policy learning. \n\n- The paper introduces a novel metric for quantitatively analysing the features generated by visual encoders.\n\n- The paper validates its claims in both simulated and real-world experiments.\n\n- The paper includes analysis of the representations learnt in various approaches.\n\n- The paper is concise and clearly presents the authors claims and results to validate these claims.", "weaknesses": "- The paper appears to focus on pretraining and freezing visual encoders while training policies on top of these encoders. It wasn't clear if they finetune all parameters on a given task (in the simulation and real-world policy learning experiments). In general, this may be relevant as policy performance can improve with finetuning of the entire architecture. I may have missed this in the paper but it would be good to see if the pretraining results include finetuning of the overall architecture on the robot manipulation task. \n\n- The similarity scores for the introduced metric are quite small and rely on thresholding the binarization of the Grad-Cam output. It seems like this is a good proxy for rudimentary manipulation tasks but it doesn't necessarily seem to be a metric that will generalise to more complex settings. The metric also relies on annotations or knowledge of the task relevant features within the image. \n\n- The pretraining dynamics alignment term incorporates chunks of state-action pairs with a history of 3 being cited as optimal. When it comes to highly dynamic tasks it is not clear how the alignment between individual images and such a history will necessarily be beneficial. For instance a single static image does not necessarily convey rich signals for dynamics (there is some signal but it is imperfect) when compared to a history of images,  aligning a state action history and a static image seems counterintuitive to me, I question whether the improvements observed are optimal relative to other methods of incorporating dynamics information. If the dynamics varied quite a lot between demonstrations on the same robot platform I question how this will effect performance. Clearly this term is working as demonstrated in the paper but I do have reservations over its usefulness and whether it is the optimal approach to incorporating dynamics information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "veNLhIgJUS", "reviewer": "Reviewer_vJK6", "summary": "This paper presents a representation learning approach named RPM that learns to encode images into low-dimensional states that are aligned with the proprioception state through contrastive learning. \n\n1. The authors showed that RPM outperforms existing learned robot representations on 20 tasks selected from 4 benchmarks. \n2. The authors proposed to use gradient heatmap as a heuristic named as \"manipulation-centricity\" to determine how much the learned representation contributes to the final manipulation task performance. \n3. RPM is showed to have highest manipulation-centricity than existing learned robot representations, and is evaluated with a real UR5 robot in lift, sweep and rearrange tasks.", "questions": "(see weakness)", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed heuristics of using gradient heatmap to indicate manipulation performance is interesting. It can be linked to existing work in visual affordance prediction. \n2. Aligning the vision representation with the robot proprio state and actions is interesting, and may encourage the model to focus more on task-relevant information. \n3. The writing is easy to understand.", "weaknesses": "1. The technical contribution is limited. The authors spent 4 pages between the introduction and experiment section to discuss their method. But, in fact, the only new technical point that the authors proposed is equation (1), which uses InfoNCE loss to bring visual features and proprio state/aciton closer. This looks hand-wavy. Even this innovation raises questions, as the proprio state/action only describe the robot trajectory but not the visual scenes. This is very likely to make the model overfit to the learned scenarios. \n\n2. The evaluation is questionable. RPM is evaluated on 20 tasks, 10 from MetaWorld, 4 from DexArt, 3 from RoboCasa, 3 from RoboMimic. However, these benchmarks combined include hundreds of tasks. Why just select these 20 particular tasks? Moreover, the \"Can\" task in robomimic is mentioned as challenging in this paper, but it is just a simple pick-and-place task of a moderate-size can. The evaluation raises serious questions about the generalization of this RPM method. \n\n2. Unnecessary discussion on how to use the human video datasets. The authors spend a considerable amount of efforts to explain the policy learning from human videos. But in the end, the conclusion is to not use human data. Human action data is abundant and contains rich and diverse behavior. If the authors' proposal is to not use them and use robot demonstrations, which in fact is the standard robot learning setup. I would prefer the paper to focus on the studied setting. \n\n3. Needs a more comprehensive ablation study. RPM is pre-trained with the DROID dataset, a large and relatively clean robot dataset, while many baselines that RPM compared against are not trained with DROID. \n\n4. In Table.1, HRP and VC-1 achieves better success rate than R3M-DROID, yet R3M-DROID's gradient heatmap focus more on the objects. According to the key finding, R3M-DROID shall perform better. \n\n4. The term \"dynamics\" is mis- and over-used, confusing in some sections.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)