Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Dataset

## Abstract

Abstract The pre-training of visual representations has enhanced the efficiency of robot learning.
Due to the lack of large-scale in-domain robotic datasets, prior works utilize in-the-wild human videos to pre-train robotic visual representation.
Despite their promising results, representations from human videos are inevitably subject to distribution shifts and lack the dynamics information crucial for task completion.
We first evaluate various pre-trained representations in terms of their correlation to the downstream robotic manipulation tasks (i.e., manipulation centricity).
Interestingly, we find that the “manipulation centricity” is a strong indicator of success rates when applied to downstream tasks.
Drawing from these findings, we propose M anipulation C entric R epresentation ( MCR ), a foundation representation learning framework capturing both visual features and the dynamics information such as actions and proprioceptions of manipulation tasks to improve manipulation centricity.
Specifically, we pre-train a visual encoder on the DROID (Khazatsky et al., 2024 ) robotic dataset and leverage motion-relevant data such as robot proprioceptive states and actions.
We introduce a novel contrastive loss that aligns visual observations with the robot’s proprioceptive state-action dynamics, combined with an action prediction loss and a time contrastive loss during pre-training.
Empirical results across four simulation domains with 20 robotic manipulation tasks demonstrate that MCR outperforms the strongest baseline by 14.8%. Additionally, MCR significantly boosts the success rate in three real-world manipulation tasks by 76.9%. Project website: robots-pretrain-robots.github.io.

## 1 Introduction

Grounding robots with generalizable visual representations, termed robotic representation, is crucial for real-world visuomotor control. Pre-training robotic representations on extensive in-domain data offers a promising strategy for robotics, drawing from the success of large-scale pre-training in computer vision (He et al., ) and natural language processing (Devlin et al., ). However, due to the scarcity of robotics data and high collection costs, recent studies have utilized everyday human operation videos (Grauman et al., ; Goyal et al., ) to pre-train visual representations for robotic manipulation, positing that knowledge behind human manipulation can inform representations in question. Various levels of human manipulation knowledge — such as task semantics (Nair et al., ), pixel-level comprehension (Xiao et al., ; Majumdar et al., ), and physical interaction (Jia et al., ; Srirama et al., )— have demonstrated their effectiveness in benefiting robotic representation. A fundamental question naturally arises: (Q1) What specific features captured from human data significantly contribute to improving robotic manipulation?

To investigate this question, we conduct a comprehensive evaluation of prior representations concerning their policy learning performance across various downstream simulation domains. Interestingly, we observe a correlation between a representation’s downstream performance and its ability to capture manipulation-relevant regions including robot end-effectors and task-relevant objects. To examine this correlation more formally, we introduce a metric, ‘manipulation centricity’, and propose an evaluation protocol to quantify it across different representations. The core of this protocol is to measure the similarity between ground truth manipulation regions and the focus of the representation. Our results indicate a strong correlation between downstream performance and manipulation centricity, as illustrated in Figure 2. These findings provide valuable insights into our first question: (A1) Manipulation centricity emerges as the key factor contributing to enhanced robotic manipulation performance.

However, human videos not only introduce inherent distribution shifts due to the human-robot embodiment gap, but also lack the dynamics information essential for successful task execution. This naturally leads to the second question: (Q2) Is there a better dataset choice than human dataset to learn manipulation centricity in robotic representation? With the recent emergence of large-scale robot datasets (Collaboration et al., ; Walke et al., ), we hypothesize that the smaller domain gap presented by these datasets may naturally be more suitable for learning manipulation centricity. To prove this hypothesis, we re-train the prior method with a representative robot dataset, DROID (Khazatsky et al., ), and indeed observe significant improvements in both performance and manipulation centricity. This answers our second question: (A2) Large-scale robot datasets can be a better choice than human datasets for learning manipulation centricity.

Recognizing that robotic datasets provide more relevant information about robot embodiment trajectories, this raises the third question: (Q3) How to learn manipulation centricity better with large-scale robot datasets? Our starting point is the dynamics labels, including robot proprioceptive states and actions, in the robot dataset but absent in the human dataset. We consider that these dynamics labels contain the core knowledge behind accomplishing a manipulation task, which has not been explicitly utilized in previous pre-trained robotic representations from human data. To this end, we introduce a new method, Manipulation Centric Representation (MCR), designed to leverage dynamics labels from robot dataset to improve manipulation centricity of robotic representation. Specifically, we propose two training objectives: dynamics alignment loss, aligning pixel representations with robot state-action pairs at the same timestep, and action prediction loss, predicting robot actions from image observations.
We also incorporate a time contrastive learning objective (Nair et al., ) to encode temporal information. Integrating these objectives into the training process significantly improves manipulation centricity, leading to a 14.8% performance increase across four simulation domains encompassing 20 diverse robotic tasks, and a 76.9% improvement across three real-world robot tasks. This answers our third question: (A3) Effectively utilizing dynamic labels enhances the learning of manipulation centricity.

We summarize this paper in Figure 1, highlighting three key insights: (1) Manipulation centricity serves as an indicator of the effectiveness of representations for robotic control, as evidenced by the strong correlation with downstream performance. (2) Large-scale robot datasets can be a superior choice compared to human datasets for learning manipulation centricity, as indicated by significant improvements in performance and manipulation centricity when using robot datasets. (3) Effective utilization of dynamics labels from robot datasets significantly enhances the learning of manipulation centricity in robotic representations, demonstrated via the introduction of the MCR method, which incorporates the proposed dynamics alignment and action prediction objectives.

## 2 Experimental Setup: Evaluating Robotic Representations

This section outlines the experimental setup used to evaluate the effectiveness of pre-trained visual representations for robotic manipulation. In line with prior work (Nair et al., ; Xiao et al., ), we freeze the pre-trained encoders and utilize Imitation Learning (IL; Argall et al. ()) for downstream policy learning. Accordingly, the quality of the visual representations is assessed based on IL performance averaged across various downstream tasks.

Evaluation protocol. To assess a visual encoder $\mathcal{F}_{\phi}$, which maps an RGB image $I\in\mathbb{R}^{H\times W\times D}$ to a continuous feature vector $z=\mathcal{F}_{\phi}(I)$, we introduce a policy network $\pi_{\theta}$ built atop the frozen encoder $\mathcal{F}_{\phi}$. This policy network takes as input the feature vector $z$ and the robot’s proprioceptive state $s$, outputting an action distribution $\hat{a}\sim\pi_{\theta}(\cdot|z,s)$. To train the policy, a task-specific demonstration dataset $\mathcal{D}_{IL}=\{\tau_{1},\tau_{2},...,\tau_{n}\}$ is collected, where each demonstration $\tau_{i}$ is a trajectory consisting of a sequence of expert-level observation-action pairs $\tau_{i}=\{(I_{t},s_{t},a_{t})\}_{t=1}^{T}$. Then, the policy is optimized via the Behavior Cloning (BC; Bain & Sammut ()) algorithm, where the optimization objective is to minimize the error between the predicted action $\hat{a}$ and the optimal action $a$ from demonstration data. For a fair comparison, we employ the same BC algorithm for any used pre-trained encoder in each task. During evaluation, the policy is executed in a closed-loop manner within the environment with online feedback to test its success rate on the target manipulation task. We evaluate at least 20 episodes every fixed number of training epochs, selecting the highest success rate. The mean and standard error of success rates across three seeds are reported.

Simulation environments and datasets. We select a total of 20 tasks across 4 simulation environments to evaluate representation quality. These tasks encompass a range of end-effectors, including grippers and dexterous hands, along with diverse robot arms and varying levels of manipulation complexity. Task visualizations are shown in Figure 3, with additional details provided in Section A.1.

Figure: Figure 3: Task visualization. We consider 20 challenging and diverse manipulation tasks spanning 4 domains.
Refer to caption: x3.png

- •
Robomimic (Mandlekar et al., ) features a benchmark suite for tabletop manipulation using the Franka Emika Panda arm and a parallel gripper. We select 3 challenging tasks, utilizing 200 demonstrations from the official proficient human teleoperation dataset for each task.
- •
RoboCasa (Nasiriany et al., )
is a realistic simulation platform focusing on housing scenarios with a Franka Emika Panda arm. We select 3 challenging tasks covering different kitchen scenarios and utilize 50 demonstrations for each task following their official training scripts.
- •
MetaWorld (Yu et al., ) features a benchmark suite of robotic tabletop manipulation with the Sawyer robotic arm and gripper. We select 10 diverse tasks according to both the difficulty categorized in Seo et al. () and the evaluation suite in Nair et al. (), and collect 25 demonstrations via official scripted controllers for each task.
- •
DexArt (Bao et al., ) features a benchmark suite for dexterous manipulation tasks on articulated objects using the XArm6 arm and an Allegro hand. We select all 4 tasks and gather 100 demonstrations rollout by well-trained RL policies for each task.

Comparison methods. We consider representative methods in robotic visual representation:

- •
MVP (Xiao et al., ) pre-trains a Vision Transformer (ViT, Dosovitskiy et al. ()) using Masked Auto-Encoding (MAE, (He et al., )) on a mix of human-object interaction videos.
- •
VC-1 (Majumdar et al., ) is trained similarly to MVP but also additionally incorporates navigation and the ImageNet (Deng et al., ) dataset during pre-training.
- •
HRP (Srirama et al., ) fine-tunes a pre-trained ViT to predict human affordance labels, including hand pose, contact points, and active objects extracted from human videos.
- •
R3M (Nair et al., ) pre-trains a ResNet (He et al., ) model on human videos using time contrastive learning and video-language alignment to extract temporal and semantic information.

## 3 Investigation: Manipulation Centricity of Representations

Prior work utilizes in-the-wild human videos to assist robotic representation learning, yet the domain gap between human and robotic tasks may influence the representation quality. To investigate this problem, we take an initial step by visualizing these representations in simulated tasks. Interestingly, we observe that a representation’s downstream task performance appears to correlate with its ability to capture manipulation-relevant regions. To investigate this correlation more formally, we introduce a metric, ‘manipulation centricity’, and propose an evaluation protocol to measure it across different methods. Our results indicate a strong correlation between manipulation centricity and downstream task performance, which strongly guides our method design in the next section. We introduce the whole pipeline in the following parts and present more details in Appendix B.

**Table 1: Grad-CAM visualization. We present Grad-CAM visualizations alongside their corresponding task success rates for the Square task from Robomimic and the Pick Place Wall task from MetaWorld. Representations that capture the robot’s end-effectors and task-relevant objects are linked to improved downstream performance. Comprehensive visualizations can be found in Section A.2.**
|  | Task | $\blacksquare$MVP | $\blacksquare$VC-1 | $\blacksquare$HRP | $\blacksquare$R3M | $\blacksquare$R3M-DROID | $\blacksquare$MCR | IL Score (%) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Square |  |  |  |  |  |  |  |  |
| P&P Wall |  |  |  |  |  |  |  |  |

Feature visualization & motivation.
We use Gradient-weighted Class Activation Mapping (Grad-CAM; Selvaraju et al. ()), a widely-adopted network visualization technique in computer vision, to analyze the features of existing representations. Grad-CAM highlights the regions of an input image that are most influential in the model’s decision-making process. In our context, this technique reveals how the network interprets images within manipulation tasks, as shown in Table 1. Specifically, in the Square task, we observe that the MVP model tends to focus on irrelevant regions, such as the table, rather than the manipulator or objects, while in the Pick Place Wall task, R3M exhibits a similar pattern. Both cases correspond to poor downstream performance. In contrast, representations that emphasize the robot’s end-effectors and objects are associated with better downstream performance. This motivates our investigation into whether representation quality is linked to its ability to capture these ‘manipulation-relevant’ regions, a property we define as manipulation centricity.

Figure: Figure 4: Measurement of manipulation centricity.
Refer to caption: x6.png

Measuring manipulation centricity. To quantify manipulation centricity, we compute the similarity between the regions highlighted by Grad-CAM and the ground truth regions corresponding to the end-effector and task-relevant objects. The ground truth is generated using the powerful video segmentation model SAM 2 (Ravi et al., ), with manual annotation of key points within the target regions. This approach enables efficient annotation across an entire video of task execution.
We apply this annotation method to create segmentation masks for demonstration data $D_{IL}$ across all simulated tasks, forming our evaluation dataset. As illustrated in Figure 4, manipulation centricity is then measured by averaging the Jaccard Index, a widely-used similarity metric in image segmentation, between the binarized Grad-CAM outputs and the ground truth segmentation masks over the entire evaluation dataset. All the ground truth annotations are available in Appendix B.

Key findings. The aggregate results, presented in Figure 2, reveal a strong correlation between manipulation centricity and downstream task performance, with a Pearson correlation coefficient of $R=0.93$. We also evaluate this correlation within individual simulation domains, where the evaluation dataset consists of demonstrations specific to each domain. The positive correlation remains consistent across all domains, albeit with varying strengths. In summary, these results suggest that manipulation centricity measured on our entire evaluation dataset is a reliable indicator of the effectiveness of robotic representations.

## 4 MCR: Learning Manipulation-Centric Representation

Drawing from the conclusions in Section 3, our focus shifts to improving the manipulation centricity of robotic representations in this section. This is achieved by two attempts in our proposed method, Manipulation Centric Representation (MCR). First, we find that re-training existing models with large-scale robot data (i.e., DROID (Khazatsky et al., )) instead of human data can significantly improve manipulation centricity. This indicates the value of utilizing robot-specific datasets, as discussed in Section 4.1. Second, we introduce novel pre-training objectives that leverage the robot’s state-action dynamics—information provided in robot data, which is inherently absent in human datasets, to further enhance manipulation centricity, detailed in Section 4.2.

### 4.1 Large-Scale Robot Dataset

Dataset processing. In recent years, several large-scale robot datasets have been introduced (Brohan et al., ; Walke et al., ; Collaboration et al., ; Khazatsky et al., ).
Among these, we select the DROID dataset for our method because of its extensive scene diversity and a large volume of data. The dataset is collected using the Franka robot arm and Robotiq 2F-85 gripper via teleoperation, comprising a total of 76k trajectories. Each trajectory includes RGB images from two external Zed 2 cameras, robot proprioceptive states, and actions consisting of delta 6D poses and 1-DoF gripper actions. To ensure data quality, we filter out trajectories with fewer than 40 timesteps to ensure adequate temporal information in each trajectory. Additionally, trajectories containing incomplete or single-word language instructions are removed, as these may indicate lower-quality interactions. After processing, we retain 36k trajectories for pre-training.

A motivating discovery.
Many previous methods rely on human-object interaction datasets to learn manipulation concepts from human behavior (Nair et al., ; Xiao et al., ; Srirama et al., ). However, the domain gap between human hands and robotic manipulators may inherently affect representation quality. We re-train the R3M model using the robot-specific DROID dataset, yielding a variant we call R3M-DROID (equivalent to R3M*). As expected, Grad-CAM visualizations in Table 1 show that R3M-DROID better captures manipulation-relevant regions, and quantitative results downstream performance, as shown in Figure 2, confirms this improvement. This suggests that robot-specific data inherently benefits representation learning by narrowing the domain gap between training and deployment environments.

### 4.2 Training MCR

Unlike human datasets, robot datasets provide access to manipulator dynamics, including proprioceptive states and actions. While previous approaches do not fully leverage this dynamics information, we hypothesize that incorporating it will enhance manipulation centricity. To achieve this, we design two new training objectives that explicitly leverage robot dynamics. Additionally, we adopt a semantic learning loss from prior work (Nair et al., ) to retain semantic information in the representations. These objectives and their integration into our training process are detailed below.

Figure: Figure 5: Illustration of objective $\mathcal{L}_{\mathrm{dyn}}$.
Refer to caption: x7.png

Dynamics alignment. Our first insight is that each image observation corresponds to an underlying proprioceptive robot state at every timestep. We aim to learn this correspondence, termed dynamics alignment, through contrastive learning. Formally, we define a state-action dynamic chunk of length $l$ at timestep $t$ as $d_{t}=[s_{\lceil t-\frac{l}{2}\rceil},a_{\lceil t-\frac{l}{2}\rceil},s_{\lceil
t
-\frac{l}{2}+1\rceil},\dots,s_{\lfloor t+\frac{l}{2}\rfloor}]$. The positive sample for $d_{t}$ is its corresponding RGB image $I_{t}$ at the same timestep, while the negative samples are drawn from a different timestep $k$ within the same trajectory. During training, we randomly choose $t$ and $k$. The encoder $\mathcal{F}_{\phi}$ is trained to discard irrelevant details from high-dimensional images and retain the essential information for manipulation. We employ the InfoNCE loss (van den Oord et al., ) for contrastive learning, introducing an MLP projector $H$ to map the dynamics chunk $d_{t}$ to the same dimension as the image feature vector $z=\mathcal{F}_{\phi}(I)$. The objective function is illustrated in Figure 5 and formalized as:

$$ $\vspace{-0.03in}\mathcal{L}_{\mathrm{dyn}}=-\sum_{b\in\mathcal{B}}\log\frac{e^ {\mathcal{S}(z^{b}_{t},{H(d_{t})}^{b})}}{e^{\mathcal{S}(z^{b}_{t},{H(d_{t})}^{ b})}+e^{\mathcal{S}(z^{b}_{t},{H(d_{k})}^{b})}},\vspace{-0.02in}$ (1) $$

where $\mathcal{S}$ represents the negative L2 distance, and $b$ denotes a sample from the batch $\mathcal{B}$.

Action prediction. We also integrate a behavior cloning (BC)-like actor into our pre-training framework, based on the idea that robotic representations should be predictive of expert-level behaviors in the dataset. The actor is implemented as a shallow MLP head that maps the image feature vector $z_{t}$ to the predicted robot actions $\hat{a}_{t}$. We use mean squared error as the objective for action prediction:

$$ $\vspace{-0.02in}\mathcal{L}_{\mathrm{act}}=-\sum_{b\in\mathcal{B}}\mathrm{MSE} (a^{b}_{t},\hat{a}^{b}_{t}).\vspace{-0.02in}$ (2) $$

Temporal contrast. We also wish the representation to encode temporal information, which has shown importance for manipulation tasks (Zhao et al., ). To this end, we adopt the time-contrastive learning objective from Nair et al. (), which encourages temporally close frames in a video to be closer in the embedding space than those that are temporally distant or from different videos. For this, we sample a frame triplet $(I_{u},I_{v},I_{w})$ where $u<v<w$, and compute the following loss:

$$ $\vspace{-0.03in}\mathcal{L}_{\mathrm{tcl}}=-\sum_{b\in\mathcal{B}}\log\frac{e^ {\mathcal{S}(z^{b}_{u},z^{b}_{v})}}{e^{\mathcal{S}(z^{b}_{u},z^{b}_{v})}+e^{ \mathcal{S}(z^{b}_{u},z^{b}_{w})}+e^{\mathcal{S}(z^{b}_{u},{z^{\neq b}_{u}})}} \vspace{-0.02in}$ (3) $$

where $z^{\neq b}_{u}$ is a negative sample from a different video within the batch $\mathcal{B}$.

Oveall objective & implementations. We train MCR with a combination of introduced objectives:

$$ $\mathcal{L}_{\mathrm{MCR}}=\mathcal{L}_{\mathrm{dyn}}+\mathcal{L}_{\mathrm{act }}+\mathcal{L}_{\mathrm{tcl}}.$ (4) $$

We do not introduce additional hyperparameters for weighting these objectives, as $\mathcal{L}_{\mathrm{MCR}}$ already yields strong empirical performance. Our default encoder backbone is ResNet-50, and we use Adam (Kingma, ) as the optimizer. The encoder is trained for 500k steps to ensure convergence, and we select the last checkpoint as the final model. The whole training process is used 50 hours with a single NVIDIA 3090. Further training details are provided in Appendix A.

## 5 Evaluation and Analysis of MCR

Our experiments are conducted in simulation and the real world to answer the following questions:

- (1)
Does MCR learn manipulation centricity and outperform baselines in simulation? (Section 5.1)
- (2)
Can the conclusions from (1) generalize to real-world manipulation tasks? (Section 5.2)
- (3)
Which design choices of MCR matter for learning manipulation centricity? (Section 5.3)
- (4)
What benefits does large-scale robot data bring to robotic representation learning? (Section 5.4)
- (5)
How many training computational resources does MCR require? (Section 5.5)

### 5.1 Simulation Results

MCR does improve manipulation centricity. We begin by presenting visualizations of Grad-CAM results for MCR in Table 1 (see all results in LABEL:tab:gradcam+IL_all). Qualitatively, our representation excels at capturing manipulation-relevant regions of each task. For instance, in the Robomimic Square task, MCR focuses on the gripper, square tool, and the target area—key elements of the task. In contrast, other methods like VC-1 and MVP either fail to highlight these areas, or, like R3M and HRP, only capture them partially. This improved localization of task-relevant features is further confirmed quantitatively in Figure 2, where MCR significantly enhances manipulation centricity across all domains. Notably, MCR even highlights the optimal path the end-effector should follow, suggesting that our representation also learns essential information related to task execution.

MCR outperforms baselines on simulation tasks.
Thanks to its improved manipulation centricity, MCR delivers substantial downstream performance gains compared to the strongest baseline methods across all selected domains, as visualized in Figure 6. This holds even in the DexArt domain, which involves a dexterous hand as the end-effector—a setup different from the gripper used in the DROID pre-training dataset. Moreover, in the MetaWorld domain, where prior baselines struggle and perform no better than the Learning-from-Scratch (LfS) method, a strong baseline with data augmentation implemented by Hansen et al. (), MCR maintains a significant advantage. We attribute the poor performance of baselines in MetaWorld to the limited number of demonstrations, which makes it difficult for policies to leverage the pre-trained representations. In contrast, MCR reduces the policy’s learning burden by providing manipulation-centric features that better fit the task. In summary, our experiments suggest that pre-training with large-scale robot data, along with optimizing for manipulation centricity, yields representations that significantly improve performance in simulated manipulation tasks.

Figure: Figure 6: Simulation results. We evaluate MCR and baselines across different domains. Our method consistently outperforms the baselines. Results are mean success rate aggregated over 3 seeds with standard deviation.
Refer to caption: x8.png

### 5.2 Real Robot Results

Experimental setup. As visualized in Figure 7, our real-world experiments involve a UR5e arm equipped with a Robotiq 2F-85 gripper and a RealSense D435i camera for RGB image capture. We designed three tabletop manipulation tasks with different objects and manipulation skills:

Figure: Figure 7: Real robot setup. We design 3 real-world robot tasks with different manipulation skills and objects.
Refer to caption: x9.png

- •
Lift. The robot grips the sandbag on the plate and lifts it up in the sky.
- •
Sweep. The robot grasps the broom to sweep the trash from the table into the dustpan.
- •
Rearrange. The robot picks up the on-table pot and places it at the designated spot on the stove.

The demonstrations used for BC-based policy training are collected by keyboards with the human operator, with 30 collected for Lift and 40 for more difficult tasks Rearrange and Sweep. The pre-trained representations are frozen during policy training, inheriting the simulation setup. For a fair comparison, we evaluate each method with the same sets of start-up conditions, which are unseen in demonstrations, in each task. More experimental details can be found in Section A.5.

MCR significantly surpasses baselines on real robot tasks. The evaluation results, shown in Table 2, demonstrate that MCR consistently outperforms the baseline methods across all tasks. In the Lift and Rearrange tasks, baseline methods often fail to grasp the object accurately, particularly when object positions were unseen in the demonstrations. In the Sweep task, their performance deteriorates further due to poor accuracy in sweeping, with the trash frequently missing the dustpan and rolling off in unintended directions. In contrast, the policies trained with our representation exhibit robust and generalizable handling of these complex tasks.

Figure: Table 2: Real robot results. Our method MCR performs best in all tasks. Each method is fairly assessed over 10 trials on each task.
[Uncaptioned image]: x10.png

We also visualize the Grad-CAM results for each method on the challenging Rearrange task in Figure 8. Representations such as R3M fail to focus on key manipulation regions, resulting in poor performance. MVP and VC-1 perform better, as they can capture the robot arm and relevant objects on the table, but they tend to over-focus on irrelevant items such as the pot. Our method avoids these pitfalls, achieving the best performance across all tasks. In conclusion, these real-world experiments underscore the effectiveness of our method in enhancing robot manipulation capabilities, significantly outperforming baseline approaches in challenging scenarios.

### 5.3 Ablation Studies

The results presented in Section 5.3 indicate the effect of key design choices in our approach, offering insights and guidance when employing our manipulation-centric representations. All experiments are conducted on three challenging tasks: Can from Robomimic, Stick Pull from MetaWorld, and Laptop from DexArt, covering all robot arms and end-effectors in prior simulation experiments.

**Table 3: Key design choices of MCR.**
| Ablated Components | Success Rate (%) |
| --- | --- |
| Training Objective |  |
| \cdashline1-2<br>w/o. objective $\mathcal{L}_{\mathrm{dyn}}$ | 66.2 ($\pm 0.8$) |
| w/o. objective $\mathcal{L}_{\mathrm{act}}$ | 71.3 ($\pm 1.2$) |
| w/o. objective $\mathcal{L}_{\mathrm{tcl}}$ | 72.0 ($\pm 1.2$) |
| Dynamic Chunk |  |
| \cdashline1-2<br>Length $l$: 3$\rightarrow$1 | 72.1 ($\pm 2.9$) |
| Length $l$: 3$\rightarrow$5 | 76.8 ($\pm 2.4$) |
| Length $l$: 3$\rightarrow$7 | 76.8 ($\pm 2.2$) |
| Encoder Backbone |  |
| \cdashline1-2<br>ResNet-: 50$\rightarrow$18 | 77.3 ($\pm 1.8$) |
| ResNet-: 50$\rightarrow$34 | 77.9 ($\pm 1.7$) |
| MCR (original) | 83.2 ($\pm 1.3$) |

### 5.4 Analysis on Robot Dataset

Besides the studies on methodology design, we also reveal more insights into the utilization of robot dataset in learning our robotic representations.

Figure: Figure 9: Effect of robot dataset size.
Refer to caption: x11.png

Larger dataset, better performance. We begin by examining how the size of the robot dataset affects pre-training outcomes. Specifically, we reduced the data in each DROID scene from 100% to 25% and assessed the downstream performance, as shown in Figure 9. Our findings indicate that larger datasets contribute to improved representation quality. This seemingly contrasts with previous research (Dasari et al., ), which suggested that merely increasing dataset size may not yield benefits. We remark that the key differences here are (1) our use of robot data rather than human data, and (2) the incorporation of additional robot dynamics information. These factors enhance the scalability and effectiveness of MCR when applied to larger robot datasets. While some studies have attempted to combine human and robot data (Dasari et al., ; Majumdar et al., ), little improvement was observed, likely due to the insufficient utilization of robot dynamics. In summary, our method effectively scales with dataset size by leveraging dynamics information.

Figure: Figure 10: Downstream domain gap.
Refer to caption: x12.png

Greater benefits for tasks with less embodiment gap. Next, we investigate which downstream tasks benefit more from pre-training with robot data. Specifically, we examine the embodiment gap associated with the end-effector. The simulation tasks are accordingly categorized into gripper-based and dexterous hand-based tasks. As illustrated in Figure 10, both R3M-DROID and MCR outperform R3M on gripper-based tasks in terms of manipulation centricity and downstream success rate, supporting our earlier conclusions. However, performance drops in hand-based tasks, where R3M-DROID even underperforms R3M. This is likely due to the fact that all data in DROID was collected using a gripper, which presents an embodiment gap compared to dexterous hands. To alleviate this issue, we suggest two attempts in the future: (1) leveraging dynamics from robot data better to mitigate the embodiment gap, and (2) incorporating more end-effectors, such as dexterous hands, into robot datasets to enhance our manipulation-centric representations.

Feature analysis. To gain a more intuitive understanding of the impact of the robot dataset, we employ t-SNE (Van der Maaten & Hinton, ) to process and visualize feature embeddings generated by the pre-trained encoder. The visualization results are presented in Figure 11. In the simulation domain, R3M struggles to cluster images within individual tasks. However, this issue is partially alleviated by using a robot dataset, as R3M-DROID exhibits improved clustering ability. Nonetheless, due to the significant domain gap between DROID and simulation environments, it still encounters difficulties in distinguishing many tasks. In contrast, our method demonstrates markedly superior clustering capabilities, indicating the importance of incorporating real-robot dynamics for effective robotic representation. Additionally, we observe that all three methods exhibit good clustering performance in real-world tasks, likely attributed to the more distinct visual changes between tasks compared to simulations. However, R3M remains inferior to the other two methods, reinforcing the critical role of robot datasets in enhancing robotic representations.

### 5.5 Training Efficiency

**Table 4: Computation efficiency. Training computation requirements across methods.**
| Method | GPU Type | Training Time (h) |
| --- | --- | --- |
| R3M | Tesla V100 | $\approx$120 |
| MCR | RTX 3090 Ti | $\approx$50 |

Pre-training representations typically requires extensive computational resources. Methods like VC-1 and MVP have prolonged training times due to their use of MAE (Feichtenhofer et al., ). Even the most computationally efficient baseline, R3M, requires 120 hours on a single NVIDIA V100. Our method, however, achieves state-of-the-art performance while requiring less computation. As shown in Table 4, our approach has a shorter training time than R3M, achieving an good balance between performance and computational efficiency.

## 6 Related Works

Pretrained robotic representations. The development of pre-trained visual representations (PVRs) has significantly improved the efficiency of downstream policy learning in robotics. Notable approaches include a variant of MoCo-v2 (Parisi et al., ) that integrates multi-layer features, MVP (Xiao et al., ) and VC-1 (Majumdar et al., ) utilizing Masked Autoencoders (He et al., ), and R3M (Nair et al., ), which employs a time-contrastive objective and video-language alignment. Other works like Zheng et al. () use temporal action contrastive learning, while MPI (Jia et al., ) focuses on predicting transition frames based on language instructions. HRP (Srirama et al., ) extracts affordances from large-scale human videos for enhanced generalization, and Theia (Shang et al., ) distills diverse vision models for robot learning. VIP (Ma et al., ) generates dense reward functions for robotic tasks. Similar to our method, RPT (Radosavovic et al., ) employs trajectory labels for representation training. In contrast, our work introduces the concept of manipulation centricity, leveraging large-scale robotic data to capture manipulation-specific dynamics, resulting in improved performance on downstream tasks.

Figure: Figure 11: t-SNE visualization. We do t-SNE visualization on 10 simulation tasks from MetaWorld and 3 real robot tasks. Each dot represents an image frame and each color indicates a task. The results demonstrate that (1) our representation has the best clustering ability and (2) robot data is helpful to robotic representation.
Refer to caption: extracted/5964376/figure/tsne_overall.png

Learning from large-scale robotic data.
Recent advancements in robotics increasingly utilize large-scale datasets to enhance the capabilities of robotic systems. Collaboration et al. () introduces the Open X-Embodiment dataset, the largest robotic dataset comprising extensive data from diverse robot embodiments across various tasks. The RT-X model (Collaboration et al., ), trained on this diverse dataset, shows promising results in cross-robot skill transfer as a generalist. Team et al. () introduces Octo, a transformer-based diffusion policy also trained on Open X-Embodiment, supporting flexible task and observation definitions. Additionally, OpenVLA (Kim et al., ) is developed as a vision-language model enabling direct mapping from image inputs to continuous robot actions. Unlike these approaches, our work focuses on extracting dynamics and interaction information from robotic datasets to create specialized visual representations for policy learning, offering an efficient alternative to generalist policies.

Dynamics-aware representation.
Representation learning is crucial for extracting key features from image inputs in imitation learning and reinforcement learning (Xu et al., ; Ji et al., ; Yuan et al., ). CURL (Laskin et al., ) uses InfoNCE (van den Oord et al., ) to maximize agreement between augmented observations, while CPC (Henaff, ) and ATC (Stooke et al., ) incorporate temporal dynamics into contrastive loss. DRIML (Mazoure et al., ) proposes a policy-dependent auxiliary objective, and KOROL (Chen et al., ) trains feature extractors on task-specific RGBD images during Koopman operator dynamics learning. TACO (Zheng et al., ; ; Liang et al., ) optimizes mutual information between current state-action pairs and future states. In contrast, our approach trains a generalized robotic representation model to extract more effective dynamics-aware representations, offering improved efficiency and generalization through a train-once-for-all methodology.

Factors driving effectiveness in robotic representation. Burns et al. () identifies emergent segmentation ability as a critical factor in the success of pre-trained visual models for robotic manipulation. While their focus is on generalization capability, our emphasis is on the downstream learning efficiency of robotic representations, investigating this from the perspective of manipulation centricity. Additionally, Theia (Shang et al., ) highlights the importance of high entropy in feature norm distribution for performance enhancement. Although they validate their findings in simulation locomotion tasks, we evaluate the effectiveness of manipulation centricity in both simulated and real-world robot manipulation tasks.

## 7 Conclusions and Discussions

Our work introduces the concept of manipulation centricity in visual representations for robotic manipulation, revealing its crucial role in downstream task performance. By leveraging large-scale robot data and dynamics-aware learning objectives, we develop a method that significantly enhances the extraction of manipulation-centric features. This approach not only advances the state-of-the-art in robotic manipulation across diverse tasks but also provides a new lens through which to understand and evaluate representation learning in robotics. Our findings highlight the importance of aligning representation learning with the specific demands of robotic control, potentially shifting the paradigm of how we approach feature extraction for embodied agents. Looking forward, this work opens avenues for exploring multi-modal integration, such as using language instructions to learn task-aware features and further leveraging trajectory data to capture spatial-temporal robotic dynamics, promising to further narrow the gap between pre-trained generalized representation models and real-world robotic manipulation.

## Acknowledgements

This project was supported by National Key R$\&$D Program of China (2022ZD0161700). We would like to thank Zhecheng Yuan, Tianming Wei, Yanjie Ze, and Yuanhang Zhang for their insightful discussions and technical supports.

## Appendix A More Experimental Details

### A.1 Task Descriptions

We select a diverse set of tasks from various robotic manipulation benchmarks for evaluation. Specifically, we include three tasks from Robomimic (Mandlekar et al., ), three tasks from RoboCasa (Nasiriany et al., ), ten tasks from MetaWorld (Yu et al., ), and four dexterous tasks from DexArt (Bao et al., ). Detailed descriptions of each task are provided below:

- •
Can (Robomimic, $\mathcal{A}\in\mathbb{R}^{7}$): the task is to manipulate the can using the robot’s arm to perform various actions such as picking it up, moving it to a different location, and orienting it in a specific way.
- •
Lift (Robomimic, $\mathcal{A}\in\mathbb{R}^{7}$): the task is to grasp a specified item and then raise it to a desired height.
- •
Square (Robomimic, $\mathcal{A}\in\mathbb{R}^{7}$): the task is to pick up a square-shaped nut and place it onto a rod successfully.
- •
Close Drawer (RoboCasa, $\mathcal{A}\in\mathbb{R}^{7}$): the task is to accurately close a drawer.
- •
Coffee Button Press (RoboCasa, $\mathcal{A}\in\mathbb{R}^{7}$): the task is to press the button on the coffee machine to start the coffee brewing process.
- •
Open Single Door (RoboCasa, $\mathcal{A}\in\mathbb{R}^{7}$): the task is to open a door that is singularly paneled, such as a cabinet or microwave door, which is already closed.
- •
Assembly (MetaWorld, $\mathcal{A}\in\mathbb{R}^{4}$): the task is to grasp a nut and position it on a dowel using the gripper.
- •
Bin Picking (MetaWorld, $\mathcal{A}\in\mathbb{R}^{4}$): the task is to transfer a disc from one bin container to another.
- •
Button Press (MetaWorld, $\mathcal{A}\in\mathbb{R}^{4}$): the task is to press a button using a robotic arm to activate a device.
- •
Disassemble (MetaWorld, $\mathcal{A}\in\mathbb{R}^{4}$): the task is to remove a nut from a peg by picking it up.
- •
Drawer Open (MetaWorld, $\mathcal{A}\in\mathbb{R}^{4}$): the task is to accurately open a drawer.
- •
Hammer (MetaWorld, $\mathcal{A}\in\mathbb{R}^{4}$): the task is to drive a screw into the wall using a hammer.
- •
Pick Place Wall (MetaWorld, $\mathcal{A}\in\mathbb{R}^{4}$): the task is to grab a puck, go around a wall and put the puck in the designated spot.
- •
Shelf Place (MetaWorld, $\mathcal{A}\in\mathbb{R}^{4}$): the task is to grab a puck and set it on a shelf.
- •
Stick Pull (MetaWorld, $\mathcal{A}\in\mathbb{R}^{4}$): the task is to use a stick to pull a box by holding onto the stick.
- •
Stick Push (MetaWorld, $\mathcal{A}\in\mathbb{R}^{4}$): the task is to hold a stick to push a box with it.
- •
Bucket (DexArt, $\mathcal{A}\in\mathbb{R}^{22}$): the task is to elevate a bucket..
- •
Faucet (DexArt, $\mathcal{A}\in\mathbb{R}^{22}$): the task is to activate a faucet using a rotating joint.
- •
Laptop (DexArt, $\mathcal{A}\in\mathbb{R}^{22}$): the task is to grasp the center of the display and then lift the laptop cover.
- •
Toilet (DexArt, $\mathcal{A}\in\mathbb{R}^{22}$): the task is to initiate the process of lifting a bigger toilet seat.

### A.2 More Grad-CAM visualizations

The Grad-CAM visualizations for each task are presented in Table LABEL:tab:gradcam+IL_all, which provides a comprehensive comparison with other baseline methods. Notably, our approach is shown to effectively facilitate the capture of key manipulation-centric features, thereby enhancing the model’s ability to focus on the most relevant aspects of the task.

### A.3 Pre-training Hyperparameters

We show our hyperparameters during the pre-training stage in Table 5. Downstream policy learning settings are introduced in Section A.5.

**Table 5: Hyperparameters for MCR pre-training.**
| Hyperparameter | Value |
| --- | --- |
| Encoder type | ResNet50 |
| Batch size | 32 |
| Learning rate | 1e-4 |
| Training steps | 500,000 |
| Data augmentation | RandomResizedCrop (224,(0.5, 1.0)) |
| Optimizer | Adam |
| DROID views used | two exterior views |
| DROID proprioception used | cartesian and gripper position |

### A.4 Pre-training Implementation

Our codebase is built upon the implementation of R3M. Similarly, for one sample within the batch, we have 5 frames from one video. The initial and final frames are sampled from the first and last 20% of the video, respectively. We employ the same contrastive learning loss implementation, modifying only the positive and negative sample pairs. For more details, please refer to Nair et al. ().

To clarify further, we provide our PyTorch-like code implementation of $L_{a}$ as below:

The MLP part of dynamics alignment loss $L_{d}$ is implemented as follows:

### A.5 BC Implementation and Settings

MetaWorld and DexArt.
We evaluate our model on these two domains following (Huang et al., ) and (Ze et al., ). Scripted policies are used to generate demonstrations in MetaWorld, and policies trained with Reinforcement Learning is used in DexArt. We generate 25 demonstrations per task in MetaWorld and 100 in DexArt, where robot end effectors and objects are initially randomized. The downstream BC policy is a three-layer MLP with ReLu activations and hidden sizes of 256. A BatchNorm layer is added before the feature is input into the MLP. The policy is trained with a mean squared error (MSE) loss, a learning rate of 0.001, and a batch size of 256.

Robomimic. We adopt the released behavior cloning implementation from RoboMimic, and use their standard imitation learning dataset. The dataset contains 200 demonstrations for each task. We only modify the code to evaluate diverse visual encoders. The downstream policy is a two-layer MLP with hidden sizes of 1024. The policy is trained with an MSE loss, an initial learning rate of 0.001, and a batch size of 100. The learning rate is decayed with a factor of 1.

RoboCasa.
We utilize the official policy learning implementation from RoboCasa. We use the BC-Transformer algorithm implemented by RoboMimic, with a RoboCasa-standard configuration as reported in the appendix of paper Nasiriany et al. (). We modify the visual observation encoder and train BC policies using the “Human-50” dataset collected by human operators. The policy is trained with an MSE loss, an initial learning rate of 0.0001, and a batch size of 16. The learning rate is decayed with factor 1.

Real world
Our real-world codebase is adapted from V-D4RL. We collect demonstrations using a keyboard interface, with 40 demonstrations for Sweep and Rearrange, and 30 for Lift due to its simplicity. The policy is a 3-layer BatchNormMLP with a hidden size of 1024. We train the policy using the log-likelihood loss, with a learning rate of 0.0001 and a batch size of 256.

### A.6 Behavior Cloning (BC) Performance of Each Simulation Task

We show the performance per task of our main results in table LABEL:tab:alltaskssr. Our method achieves the highest success rate in 19 out of 20 tasks evaluated, showcasing its robustness and adaptability to various scenarios.

**Table 6: Main results on 20 simulation tasks. Results for each task are provided in this table. A summary across domains is shown in Figure 7.**
|  | DexArt | Robomimic | RobocCasa |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Alg $\backslash$ Task | Bucket | Faucet | Laptop | Toilet | Can | Lift | Square | CloseDrawer | CoffeeButtonPress | OpenSingleDoor |
| MCR(ours) | 36.7 ($\pm\textbf{2.9}$) | 38.3 ($\pm\textbf{2.9}$) | 93.3 ($\pm\textbf{2.9}$) | 73.3 ($\pm 2.9$) | 68.0 ($\pm\textbf{4.0}$) | 96.0 ($\pm\textbf{2.3}$) | 30.0 ($\pm\textbf{1.2}$) | 99.3 ($\pm\textbf{1.2}$) | 72.0 ($\pm\textbf{3.5}$) | 56.0 ($\pm\textbf{3.5}$) |
| LfS | 33.3 ($\pm 5.8$) | 36.7 ($\pm 5.8$) | 83.3 ($\pm 10.4$) | 71.7 ($\pm 2.9$) | 6.0 ($\pm 0.0$) | 64.0 ($\pm 4.2$) | 4.0 ($\pm 0.0$) | 85.3 ($\pm 1.2$) | 52.0 ($\pm 4.0$) | 46.7 ($\pm 1.2$) |
| MVP | 31.7 ($\pm 2.9$) | 33.3 ($\pm 2.9$) | 81.7 ($\pm 5.8$) | 80.0 ($\pm\textbf{0.0}$) | 28.0 ($\pm 2.0$) | 74.0 ($\pm 6.4$) | 14.0 ($\pm 2.3$) | 98.0 ($\pm 2.0$) | 52.7 ($\pm 18.9$) | 33.3 ($\pm 14.5$) |
| VC1 | 30.0 ($\pm 0.0$) | 35.0 ($\pm 0.0$) | 85.0 ($\pm 0.0$) | 71.7 ($\pm 2.9$) | 44.0 ($\pm 7.0$) | 74.0 ($\pm 9.2$) | 20.0 ($\pm 3.5$) | 98.7 ($\pm 2.0$) | 29.3 ($\pm 5.8$) | 33.3 ($\pm 7.0$) |
| R3M | 31.7 ($\pm 2.9$) | 36.7 ($\pm 2.9$) | 81.7 ($\pm 5.8$) | 71.7 ($\pm 2.9$) | 50.0 ($\pm 4.2$) | 86.0 ($\pm 6.0$) | 24.0 ($\pm 1.2$) | 88.7 ($\pm 3.1$) | 47.3 ($\pm 6.1$) | 48.7 ($\pm 7.6$) |
| HRP | 31.7 ($\pm 2.9$) | 36.7 ($\pm 2.9$) | 90.0 ($\pm 5.0$) | 63.3 ($\pm 14.4$) | 42.0 ($\pm 3.5$) | 86.0 ($\pm 3.5$) | 26.0 ($\pm 2.3$) | 91.3 ($\pm 4.6$) | 35.3 ($\pm 11.6$) | 38.0 ($\pm 6.0$) |
| R3M-Droid | 35.0 ($\pm 5.0$) | 33.3 ($\pm 2.9$) | 80.0 ($\pm 0.0$) | 66.7 ($\pm 7.6$) | 54.0 ($\pm 2.3$) | 96.0 ($\pm\textbf{0.0}$) | 22.0 ($\pm 3.1$) | 88.7 ($\pm 2.3$) | 51.3 ($\pm 2.3$) | 45.3 ($\pm 7.6$) |

## Appendix B More details on Manipulation Centricity

Grad-CAM.
Grad-CAM is a widely used technique for generating heatmaps of input images to identify the regions that the encoder focuses on. We utilize the PyTorch-Grad-CAM library to generate Grad-CAM figures for each visual encoder. In convolutional neural networks (CNNs), Grad-CAM generates heatmaps by backpropagating gradients through the final convolutional layers, thereby highlighting important image regions. However, in Vision Transformers (ViT), the final classification is based on the class token computed in the last attention block, which means that the output is not influenced by the 14x14 spatial channels in the final layer, resulting in zero gradients. To generate meaningful visualizations in ViT, it is necessary to choose a layer before the final attention block. In our study, we chose the LayerNorm applied to the output of the self-attention mechanism in the last Transformer block.

Our paper applies Grayscale CAM to measure Manipulation Centricity, where the pixel values range from 0 (black) to 255 (white), with higher values (closer to white) indicating regions that contribute more significantly to the model’s decision. Furthermore, we binarize the Grayscale CAM by thresholding pixel values at 2, setting values below 2 to 0 and those above or equal to 2 to non-zero values, thereby mitigating the effect of noise and facilitating more accurate interpretations and precise metric calculations.

SAM 2. Segment Anything Model 2 (SAM 2, Ravi et al. ()) is a large-scale segmentation model. In our study, we utilize SAM2 to segment demonstration videos of all simulation tasks. Specifically, we select the official ‘base plus’ model available on the SAM2 repository and adopt the implementation of the SAM2-GUI, an interactive graphical user interface to utilize pre-trained SAM2 models. We manually upload each task video and add prompt points to generate segmentation videos. Notably, foreground pixels, including robot end effectors and objects, are labeled with non-zero values, while background pixels are labeled with zero. A full list of ground truth annotations is shown in LABEL:tab:gradcam+IL_all, to better illustrate manipulation centricity.

Jaccard Index. We employ the Jaccard Index to quantify the similarity between Grad-CAM and SAM2 video frames, with the index ranging from 0 to 1. A higher Jaccard Index value indicates a greater degree of similarity between the two, suggesting a stronger alignment between the attention maps generated by Grad-CAM and the segmentation masks produced by SAM2.

## Appendix C Detailed Analysis of DROID Subset Used

As described in Section 4.1, we perform a preliminary preprocessing of the full 1.7TB RLDS dataset, yielding a subset that serves as the basis for our analysis. In this section, we present a statistical characterization of this subset. Specifically, we exclude videos with fewer than 40 frames. The distribution of video lengths in the subset is illustrated in Figure 11(a). Furthermore, we conduct a comprehensive statistical analysis of the language instructions accompanying the dataset, with a particular focus on the frequency of common object nouns and action verbs. The results of this analysis are visualized in Figures 11(b) and 11(c), respectively. Notably, our subset covers a large degree of data diversity, capturing a wide range of scenarios, objects, and actions, thereby ensuring the effectiveness of our method.

Figure: (a) Statistics of Trajectory Length.
Refer to caption: extracted/5964376/arxiv/figure/dataset.jpg

**Table 7: Grad-CAM of all tasks.**
|  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Task | Annotation | $\blacksquare$MVP | $\blacksquare$VC-1 | $\blacksquare$HRP | $\blacksquare$R3M | $\blacksquare$R3M-DROID | $\blacksquare$MCR |
| Can |  |  |  |  |  |  |  |  |
| Lift |  |  |  |  |  |  |  |  |
| Square |  |  |  |  |  |  |  |  |
| CloseDrawer |  |  |  |  |  |  |  |  |
| CoffeeButton |  |  |  |  |  |  |  |  |
| OpenDoor |  |  |  |  |  |  |  |  |
| Assembly |  |  |  |  |  |  |  |  |
| BinPicking |  |  |  |  |  |  |  |  |
| ButtonPress |  |  |  |  |  |  |  |  |
| Disassemble |  |  |  |  |  |  |  |  |
| DrawerOpen |  |  |  |  |  |  |  |  |
| Hammer |  |  |  |  |  |  |  |  |
| PickPlaceWall |  |  |  |  |  |  |  |  |
| ShelfPlace |  |  |  |  |  |  |  |  |
| StickPull |  |  |  |  |  |  |  |  |
| StickPush |  |  |  |  |  |  |  |  |
| Bucket |  |  |  |  |  |  |  |  |
| Faucet |  |  |  |  |  |  |  |  |
| Laptop |  |  |  |  |  |  |  |  |
| Toilet |  |  |  |  |  |  |  |  |

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 4
questions: - Why is it still necessary to attend to the manipulator region when there is already proprioception input?
- What is the evaluation protocol in the real world? How are the environments varied between training and evaluation, and what would count as a success?
- By attending the robot and task-relevant object, technically this would allow the representation to have stronger generalization capabilities. Perhaps the author should run some visual generalization experiments to find out if RPM enables better generalization.
soundness: 3
strengths: - The paper is clear and well presented
- The simulation experiments are rigorous and have a lot of diversity
- The paper presents a novel metric for evaluating visual representations for robotics, and novel findings on how to integrate robotics information (states, actions) for training better representations
summary: The paper first proposes manipulation centricity, a new metric for evaluating representations for robotics, and finds that it is a strong indicator of success rates in downstream robotic manipulation tasks. Building on this insight, it proposes RPM, a framework for learning visual representation for robotics from large-scale robotics dataset. Experiments in both simulation and the real world show that RPM leads to improved manipulation centricity and higher task success rate when evaluated using behavior cloning.
weaknesses: - There could be more empirical analysis and intuition regarding how each loss affects manipulation centricity. Authors could explain intuitively how each loss contributes to better MC and visualize the gradcam of an encoder trained with each loss
- The paper lacks comparison to more state-of-the-art baselines, such as VIP/LIV, or object-centric pre-trained representations such as POCR/HODOR
- As I understand, the goal here is to make the representation pay attention to task-relevant regions e.g. robot and task-relevant objects. The authors have already demonstrated that this information can be obtained using SAM2. Have the authors considered directly using the masks as a form of supervision for training the representation? I think this could serve as a baseline, since RPM requires additional priveleged information that only a robotics dataset has, i.e. state and action, whereas we can generate segmentation masks for any general sources of data.
- The paper could benefit from more extensive real robot experiments

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. What is the tradeoff between the visual realism of the simulator vs simulator-specific training required to achieve high success rate and manipulation centricity? For example, in RoboCasa, manipulation centricity seems really low compared to others.  How to compare it across simulators?

1. How does "manipulation centricity" depend on the choice of camera intrinsics and extrinsic with respect to the robot body frame?  

1. R3M has been shown to be a poor visual representation model for robotic tasks, as compared to visual datasets [1] and pretraining image distribution matters. Why none of the baselines involve vision only representation backbones for comparison?

[1] Dasari et al, 2023. An Unbiased Look at Datasets for Visuo-Motor Pre-Training, https://data4robotics.github.io/
soundness: 3
strengths: 1. Detailed empirical validation across diverse tasks and simulation domains, plus real robot experiments
1. well-motivated concept of manipulation centricity with clear metrics
1. ablation studies and analysis of design choices 
1. insights about benefits of robot vs human datasets for pre-training
1. Clearly written key research questions and contributions
summary: The paper proposes pre-train visual representations for robot manipulation by focusing on manipulation centricity, robot datasets (instead of human activity datasets), and auxiliary loss objectives for dynamic alignment and time contrastive learning. To measure Manipulation Centricity, they use Jaccard similarity between the binarized Grayscale CAM and SAM2’s foreground vs. background predictions. The proposed model RPM outperforms simulation and real robot tasks compared to baselines.
weaknesses: The work presents insights into the visual representations that focus on manipulation centricity and dynamic alignment perform better than existing approaches. However, there are certain assumptions whose implications are not clearly discussed.
1. DROID is chosen as the robot dataset of choice instead of other larger dataset in OXE.
1. RPM is a ResNet-based model instead of directly using / comparing to pretrained models like SAM2, etc.
1. The "manipulation centricity" does not seem to improve at similar scale across different simulators, with respect to success rate. 
1. A concise definition of manipulation centricity and how it is calculated should be present in the main paper, rather than existing only in appendix.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Congratulations on this work and thanks for contributing it to the robot learning community. 

I wished to clarify if during policy training does the visual encoder always remains frozen and whether you finetune the entire architecture? In the first part of the paper when evaluating your introduced metric it is clear that the encoder is frozen, I wished to confirm whether this is also the case for the reported task success rates and whether you considered finetuning the entire architecture?
soundness: 3
strengths: - The paper validates a pretraining objective for visual encoders that improves the performance of robot policy learning. 

- The paper introduces a novel metric for quantitatively analysing the features generated by visual encoders.

- The paper validates its claims in both simulated and real-world experiments.

- The paper includes analysis of the representations learnt in various approaches.

- The paper is concise and clearly presents the authors claims and results to validate these claims.
summary: This paper introduces a metric (manipulation-centricity) for assessing the ability of pretrained visual encoders to model task-relevant features for robot manipulation tasks; as well as a novel pretraining objective (RPM) that incorporates the robot dynamics information available in robotics datasets. The authors first establish a correlation between the metric they define and the success rate of policies that leverage the features generated by pretrained visual encoders. Having established a positive correlation they seek to design a pretraining objective for visual encoders and evaluate its performance on their proposed metric and the downstream policy success rates. The novelty in the pretraining objective the authors introduce is a dynamics alignment term which takes the form of the InfoNCE applied to embeddings of a sequence of state action pairs and image embeddings. This novel dynamics alignment term is combined with existing behaviour cloning and temporal contrastive terms to give the full learning objective. They apply this objective function to pretrain visual encoders for manipulation tasks and demonstrate that this pretraining objective leads to improvements on their metric and downstream task performance in both simulated and real-world experiments.
weaknesses: - The paper appears to focus on pretraining and freezing visual encoders while training policies on top of these encoders. It wasn't clear if they finetune all parameters on a given task (in the simulation and real-world policy learning experiments). In general, this may be relevant as policy performance can improve with finetuning of the entire architecture. I may have missed this in the paper but it would be good to see if the pretraining results include finetuning of the overall architecture on the robot manipulation task. 

- The similarity scores for the introduced metric are quite small and rely on thresholding the binarization of the Grad-Cam output. It seems like this is a good proxy for rudimentary manipulation tasks but it doesn't necessarily seem to be a metric that will generalise to more complex settings. The metric also relies on annotations or knowledge of the task relevant features within the image. 

- The pretraining dynamics alignment term incorporates chunks of state-action pairs with a history of 3 being cited as optimal. When it comes to highly dynamic tasks it is not clear how the alignment between individual images and such a history will necessarily be beneficial. For instance a single static image does not necessarily convey rich signals for dynamics (there is some signal but it is imperfect) when compared to a history of images,  aligning a state action history and a static image seems counterintuitive to me, I question whether the improvements observed are optimal relative to other methods of incorporating dynamics information. If the dynamics varied quite a lot between demonstrations on the same robot platform I question how this will effect performance. Clearly this term is working as demonstrated in the paper but I do have reservations over its usefulness and whether it is the optimal approach to incorporating dynamics information.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: (see weakness)
soundness: 2
strengths: 1. The proposed heuristics of using gradient heatmap to indicate manipulation performance is interesting. It can be linked to existing work in visual affordance prediction. 
2. Aligning the vision representation with the robot proprio state and actions is interesting, and may encourage the model to focus more on task-relevant information. 
3. The writing is easy to understand.
summary: This paper presents a representation learning approach named RPM that learns to encode images into low-dimensional states that are aligned with the proprioception state through contrastive learning. 

1. The authors showed that RPM outperforms existing learned robot representations on 20 tasks selected from 4 benchmarks. 
2. The authors proposed to use gradient heatmap as a heuristic named as "manipulation-centricity" to determine how much the learned representation contributes to the final manipulation task performance. 
3. RPM is showed to have highest manipulation-centricity than existing learned robot representations, and is evaluated with a real UR5 robot in lift, sweep and rearrange tasks.
weaknesses: 1. The technical contribution is limited. The authors spent 4 pages between the introduction and experiment section to discuss their method. But, in fact, the only new technical point that the authors proposed is equation (1), which uses InfoNCE loss to bring visual features and proprio state/aciton closer. This looks hand-wavy. Even this innovation raises questions, as the proprio state/action only describe the robot trajectory but not the visual scenes. This is very likely to make the model overfit to the learned scenarios. 

2. The evaluation is questionable. RPM is evaluated on 20 tasks, 10 from MetaWorld, 4 from DexArt, 3 from RoboCasa, 3 from RoboMimic. However, these benchmarks combined include hundreds of tasks. Why just select these 20 particular tasks? Moreover, the "Can" task in robomimic is mentioned as challenging in this paper, but it is just a simple pick-and-place task of a moderate-size can. The evaluation raises serious questions about the generalization of this RPM method. 

2. Unnecessary discussion on how to use the human video datasets. The authors spend a considerable amount of efforts to explain the policy learning from human videos. But in the end, the conclusion is to not use human data. Human action data is abundant and contains rich and diverse behavior. If the authors' proposal is to not use them and use robot demonstrations, which in fact is the standard robot learning setup. I would prefer the paper to focus on the studied setting. 

3. Needs a more comprehensive ablation study. RPM is pre-trained with the DROID dataset, a large and relatively clean robot dataset, while many baselines that RPM compared against are not trained with DROID. 

4. In Table.1, HRP and VC-1 achieves better success rate than R3M-DROID, yet R3M-DROID's gradient heatmap focus more on the objects. According to the key finding, R3M-DROID shall perform better. 

4. The term "dynamics" is mis- and over-used, confusing in some sections.

Meta-review: The paper introduces a novel metric called “manipulation-centricity” (MC) for evaluating pretrained representations and demonstrates its correlation with downstream manipulation task performance. Building on these insights, it proposes RPM, a representation learning method that captures both visual features and dynamic information to enhance manipulation-centricity (MC). Through extensive experiments, the authors validate that their approach improves task success rates compared to baseline methods.

Strengths:
- Novel Metric and Concept: “Manipulation-centricity” as a quantitative metric for evaluating visual representations in robotic manipulation is innovative and well-motivated.
- Experiments: The paper provides rigorous validation across diverse tasks and simulators, demonstrating the robustness of RPM.
- Clear Writing and Structure: The paper is well-organized and presents its key ideas well.

Weaknesses
- Real-World Experiments: While the paper includes real-world evaluations, the scope of these experiments is limited. Conducting more complex tasks with diverse objects could strengthen the results.
- Missing Baseline Comparisons: As note by Reviewer wcpD, the POCR/HODOR line of work proposes pretrained visual representations for robotics that are object-centric. These could serve a strong basline for the current method. The authors have noted that the code for HODOR is not yet available; if it is released before the camera-ready deadline, I recommend running these experiments and including them as baselines.

The paper demonstrates significant novelty in proposing manipulation-centricity as a metric and aligning visual representations with robotic dynamics. It provides thorough experimental validation and offers meaningful insights into pretraining for robotic manipulation tasks. However, the limited scope of real-world experiments slightly diminishes the overall impact.

Despite these limitations, the strengths of the paper, particularly its novel contributions and rigorous empirical analysis, outweigh the weaknesses. Therefore, I recommend acceptance, with the suggestion to address the noted weaknesses in the camera-ready version.

Decision: Accept (Poster)