Thoughts and Lessons on Using Visual Foundation Models for Manipulation
Authors: Ryan Chen
Ziteng Pang
Bradly C. Stadie
Venue: Accepted by TMLR ()

Abstract: Training vision-based robotic systems from scratch is both computationally expensive and memory intensive. To mitigate these challenges, recent approaches forgo end-to-end training in favor of adopting visual representations from visual foundation models -- large scale models designed for broad task transferability. Recent years have seen numerous vision foundation models emerge, including several designed specifically for manipulation tasks. However, we still lack clear principles for what makes these models effective for robotics applications. To address this gap, we systematically evaluate vision foundation models to understand what makes them effective for offline robotic learning. We find that across eleven diverse vision encoders, a representation's ability to reconstruct edges and predict keypoints strongly correlates with its performance on manipulation tasks. Extensive correlation analysis across 21 manipulation tasks consistently shows that representations preserving edge and keypoint information achieve the highest environment success rates. These findings appear to challenge conventional wisdom about holistic reconstruction-based pretraining and offer a new lens for understanding what makes vision representations effective for robotics.

Published in Transactions on Machine Learning Research (06/2025)
Thoughts and Lessons on Using Visual Foundation Models for Manipulation
Ryan Chen
ryancher@@u..orrtuuustttee
Department of Statistics and Data Science Northwestern University
Ziteng Pang
atteggenng007@@..oorrtuvttte...
Department of Statistics and Data Science Northwestern University
Bradly C. Stadie
bstaadieShorrbaunstttreeel
Department of Statistics and Data Science Northwestern. University
Reviewed on OpenReview: https: // //pp::e//peerevveee net/ forum? id= 66mkDz Vuc
Abstract
Training vision-based robotic systems from scratch is both computationally expensive and memory intensive. To mitigate these challenges, recent approaches forgo end-to-end train- ing in favor of adopting visual representations from visual foundation models large scale models designed for broad lask transferabilityy Recent years have seen numerous vision foundation models emerge, including several designed specifically for manipulation tasks. However, we still lack clear principles for what makes these models effective for robotics applications. To address this gap, we systematically evaluate vision foundation models to understand what makes them effective for offline robotic learning. We find that across eleven diverse vision encoders, representation's ability to reconstruct edges and predict keypoints strongly correlates with its performance manipulation tasks. Extensive correlation analy- sis across 21 manipulation tasks consistently shows that representations preserving edge and keypoint information achieve the highest environment success rates. These findings appear to challenge conventional wisdom about holistic reconstructionn-beel pretraining and oller a new lens for understanding what makes vision representations effective for robotics.
1 Introduction
Consider the challenge of a robot learning to manipulate its environment directly from raw images. In contrast to image classification systems, robotic vision systems must understand not only what objects are present, but their physical properties, spatial relationships, and how they will respond interaction. Training a system from scratch to both understand visual scenes and control actions requires cnormous amounts of robot data and computation (Finn et al.) 2016 Vision foundation models offer a promising solution to this bottleneck: by leveraging visual representations pretrained large datasets, robots start with strong prior about the visual world. Prior work has shown these pretrained representations can transfer surprisingly well to robotics tasks, even when trained on out-of-domain data (Parisi et al. 2022 Building on this insight, several foundation models have been developed specifically for robotic applications
Radosavovic et al. 2022 Nair et al. 2022 Ma et al. 2022 2023 Chen et al. 2024
Despite widespread adoption of pretrained vision representations in robotics, the characteristies that make them effective for control remain largely unclear. While several works have studied different aspects of

---

Published in Transactions on Machine Learning Research (06/2025)
Gymnasium Behavior Cloning
Metaworld Behavior Cloning
0.76
0.8
0.70
Success %
0.6
0.52
0.59
0.56
Success
0.6
0.51
0.50
0.44
0.38
0.32
0,30
0.24
0.15
0,18
0.06
SAM IBOT
R3M
VP
MOCO DINO
CLIP
VC1
MAE OBI_RN MVP
SAM MOCO BOT
R3M
VIP
DINO
CLIP
MAE
VC1
MVP
OBL
RN
Encoder
Figure 1: Visual encoders exhibit substantial variability in their effectiveness when deployed in manipulation tasks with imitation learning. We examine downstream auxiliary tasks that can help explain the variability, providing insight on what is important in visual representations when trying to accomplish manipulation lasks.
these representations from inductive biases (Ze et al. 2023 Yen-Chen et al.] 2020) robustness (Burns
et al. 2024) and cross-nnviromment performance (Parisi al., 2022 Majumdar et al. 2023 we lack clear
principles for selecting and adapting them for manipulation lasks. To address this gap, conduct
systematic evaluation of nine diverse encoders encoders-ranging ranging from general-purpose robotics-specffe--aro acroos 21
manipulation tasks.
Consistent with Majumdar et al. )023), we find that no single pretrained representation uniformly outper- forms others across all tasks. However, while investigating these encoders, we made a curious discovery: there exist several pasy-to-comping diagnostic measures of a visual encoder that are strongly correlated with downstream robotic performance. Specifically, a representation's ability to capture accurate keypoints and reconstruct high-fidelity edges shows remarkably strong correlation ion with manipulation performance (Pearson Å. = 0.92, Spearman Ô2- 0.88). Visual analysis using Sobel filters reconstructions reveals these measures align with model's ability to preciscly locate objects and their boundarics, capabilities that are crucial for manipulation.
Our contributions are as follows:
1. Through extensive evaluation across 21 manipulation tasks, we identify which vision encoders con- sistently excel at robotic control. Our results show consistency across both behavior cloning and offline reinforcement learning, providing clear guidance for the robotics community on representation selection.
2. We propose easy-to-compute diagnostic measures that strongly correlate with a vision encoder's performance on downstream mannpulatiinn tasks. In particular, we show that scene reconstruction and edge reconstruction quality are remarkably associated with manipulation success, even in OOD scenarios.
3. We demonstrate Lhat. representations trained with discriminative pretraining objectives consistently outperform those trained with holistic reconstructive objectives. This finding helps resolve an ap- parent contradiction the literature between the effectiveness of supervised versus self-supervised pretraining. Namely, we show that the pretraining objective, rather than the source of supervision, what matters most for dowmstream robotic control.
2

---

Published in Transactions on Machine Learning Research (06/2025)
2
Related Works
Representations for Robotic Manipulation In robotic manipulation, visual pretraining learns trans- ferable features for downstream tasks. Parisi et al. (2022) demonstrated that visual representations can perform as well as ground-truth state representations, showing that using learned representations alone can be highly effective. Thus, with the rise of foundation models (Bommasani et al. 2021 researchers have
begun leveraging these large-scale representations to tackle initialization challenges (Finn et al. 2016).
This insight has led to the adoption of general-purpose vision models for learning manipulation skills. Ad- ditionally, vision encoders such as MVP, R3M, VIP, and VC1 (Radosavovic ct al.
2022 Nair et al.) 2022:
Ma et al. 2022 Majumdar et al. 2023) capture physical dynamics by training on robotics-adjacee datasets
(Shan et al. 2020 Goyal et al. 2017 Grauman et al. 2022 Other approaches incorporate inductive biases
for grasping and pose estimation (Ze et al. 2023
In parallel to these efforts, growing body of work focuses on learning object-centric visual representations. These approaches, including slot attention mechanism (Locatello et al. 2020 Heravi et al. 2023 and OCLR (Yoon et al. 2023), aim to decompose scenes into structured object representations using self-supervised objectives such as contrastive learning, temporal alignment, and localized masked reconstruction. In addition, entity-centric models like EIT (Haramati et al. 2024h and FCC-Difusee (Qi et. al. 2025) also leverage keypoint information on top of learning from pixels, and can produce generalizable visuomotor control policies. Our findings about keypoint correlations are aligned with intuitions behind these object and entity centric designs, however our focus is on frozen holistic pretrained representations.
Meanwhile, some studies (Hansen al. 2023 Sharma et al. 2023 suggest that training from scratch fine-tuning pretrained models can also yield strong results. In this work, we evaluate off-the-shelf vision encoders, comparing both manipulation-specifie and general-purpose models. Our goal is to develop data- driven metrics to assess representation quality in manipulation tasks, helping guide future advancements in designing pretrained vision models.
Representation Evaluation and Benchmarking in Robotics Benchmarking foundation vision models for robotic tasks has been extensively studied. Hu et al.) (0233) identified behavior cloning and inverse reinforcement learning as key paradigms for analyzing visual representations. Several studies (Burns et al.
2024 Schneider et al. 2023; Sax et al.) 2021) evaluate out-of- out-of-distribon distribution (OOD) performance of vision
encoders, while others (Paiss et al. 2022 Hu et al. 2023) focus on in-distribution performance, which aligns
closely with our work.
A key discussion in this space concerns self-supervised versus supervised vision pretraining and their effec- tiveness for manipulation tasks (Parisi et al. 2022 Burns et al. 2024). Majumdar et al. (2023) found that no single representation consistently outperforms others and introduced the VC-1 encoder, which we include in our evaluation.
Beyond benchmarking, researchers have explored the characteristics of effective representations, Wulfmeier 2021) studiod how dimensionality and disentanglement impact multitask performance, while Tomar 2023) analyzed the role of auxiliary training objectives. Both studies, along with Majundar et. al. suggest that universal learning algorithm representation excels across all robotic tasks.
ct al.
et al.
2023
Another line of work highlights the benefits of inductive biases. Burns et al. 2024]; Yen-Chen et al. (2020) demonstrated that segmentation-orient oriented inductive biases improve performance in robotics tasks. To this end, we include SAM (Krrllovv et al.) 2023) in our evaluation suite. Our work builds on these efforts by identifying key representation qualities that influence manipulation performance.
3 Background
To evaluate vision representations for robotics, we need methods that can effectively learn from offline datasets of expert demonstrations. We focus two such approaches: behavior cloning, which directly
3

---

Published in Transactions on Machine Learning Research (06/2025)
mimics expert actions, and offiine reinforcement learning, which learns value function from demonstrations. In both cases, we maintain simple architecture (Figure 2 where states and goals are processed through the vision encoder being evaluated.
Behavior Cloning Behavior cloning seeks to approximate expert behavior by learning a state-conditional action distribution Tg BC (a s). We implement this by training a three-layer MLP to regress expert actions onto scene embeddings produced by the vision encoder Φ. Specifically, for state-action pairs (sg, sampled from the expert dataset DE; we minimize ming Beg,ag~Ds | (alø(s=)) aall The state space consists of image embeddings augmented with goal image embeddings, excluding proprioception to focus evaluation visual representations.
Expert Offline Dataset
Pretrained Encoder #()
Scene
T(a d(s))
Rendering
Behavior Cloning
Expert Action
[4,422 0.37%, 8.351]
||<( [(()) acll2
T(a d(s))
Pretrained Encoder (())
Scene Rendering
Offline Reinforcement Learning
Q(d(s), a) V(o(s)
exp(B(Q((dss) - v()(s)))) (lit(a d()) -001-aa111 acll3)
Expert Action
Figure Imitation algorithms. We evaluate visual representations using offline imitation methods that learn from expert remonstrations. After training expert datasets, policies =<a)o(s)) evaluated in their respective environments.
Offline Reinforcement Learning We also evaluate representations using implicit Q-Learning (IQL) [Kostrikov et al., 2022). To maintain consistency with our behavior cloning experiments and focus purely on evaluating visual representations, use inverse Q-Learning (Garg et al.) 2021 to estimate Q-values directly from expert demonstrations without reward signals, after which implicit Q-learning estimates its V function. This creates a reward-agnostic implicit Q-learning algorithm. Like our behavior cloning experiments, we exclude proprioception data and train only on visual representations. While other offline reinforcement lcarning methods cxist, most require either reward signals (c.g., CQL [Kumar et al.) 2020). SAC-n (An) et al. 2021) ) or environment interaction (e.g., GAIL (Ho Ermon 2016)) making them unsuitable for our evaluation framework.
3.1 Environments
We aim to study the effectiveness of pretrained visual representations in how they encapsulate information about any given scene, such the acting agent, the objects in the environment, and the goal of the tasks. To this end, we select environments based on the following criteria: (a) they must be renderable into images lo encoded by visual representations; (b) solvable using only visual representations without proprioception data ensuring that the agent acts solely on visual representations; (c) encompass a diverse range of tasks with varying complexities, such as grasping, nudging, or hand gripping: and (d) have variable goal states and starting configurations to promote generality in the analysis.
4

---

Published in Transactions on Machine Learning Research (06/2025)
With these criteria in mind, arrive at 21 robotic manipulation tasks from the Fetch Suite Plappert et al. 2018). AdroitHand Suite Rajeswaran al. 2017), and Metaworld Suite (Yu et al.) 2020 We created experts that could solve these three environments with at least 95% success rate. These experts provided offline datasets for our behavior cloning and offline reinforcement learning experiments. Details about the environments and experts are discussed Appendix A Examples of environment renderings are shown Figure
Fetch Push
AdroitHand Hammer
Metaworld Coffee Pull
Metaworld Window Close
Metaworid Button Press
Figure 3: A sample rendering of some of the environments in evaluation suite.
3.2
Vision Encoders
We evaluate nine off-the-shelf, frozen vision encoders, each encoding 224 x 224 RGB renderings of environ- ment scenes into vector representations. We include both models specifically trained for manipulation lasks and foundation models trained on internet-scale datasets. The selected encoders span diverse pretraining objectives, from prediction and distillation to contrastive and holistic reconstruction approaches. A full list of considered encoders and their pretraining objectives and data sets can be found in Table 6
A fundamental aspect of vision encoders the structure of the representations they generate. Many models produce single-vector representations for an entire image. In contrast, models such SAM generate spatial feature maps corresponding to VIT patches, and are pooled with convolutions to produce holistic scene representation. These structured representations often benefit from transformer architectures for downstream tasks, for which we consider in Appendix || The remaining encoders in our evaluation suite produced single holistic vector representation for each 224x224 RGB image.
Given the spatial structure of SAM's embeddings, we initially applied a transformer-based policy to learn actions. However, as shown in Appendix [1 the MLP policy and attention policy performance did not significantly differ from each other. Since the transformer policy did not provide significant. advantage, we opted to use the MLP policy for consistency across all encoders, ensuring fair evaluation framework. The use of an MLP policy is well-established in the literature quality (Kumar et al. 2022). By maintaining a standardized evaluation methodology, our results facilitate a direct comparison of the relative effectiveness of different vision encoders for manipulation tasks.
effective method for assessing embedding
Another source variation among the encoders is the dataset used for pretraining. While all encoders were pretrained on different real-world data sets, the evaluation was conducted in simulation. Parisi et al. (2022) demonstrated that differences in pretraining data do not influence downstream simulation performance. This suggests that the encoder performance can be directly compared despite difference in training data.
4 Experimental Investigation of Vision Encoders
In this section, we evaluate the performance of different vision encoders across a diverse set of robotic tasks. We focus on understanding how different pretraining strategies, task complexities, and offline learning paradigms impact the success rates of policies for robotic tasks.
5

---

Published in Transactions on Machine Learning Research (06/2025)
4.1 Behavioral Cloning
We evaluated behavior cloning across 21 environments by regressing expert actions onto image embeddings from the vision encoders. For each policy, pass both the current scene and goal state images through the encoder. The resulting embeddings are concatenated and processed through three-layer MLP with hidden layers [256, 128, 64) to predict four-dimensional action vector. We trained policies using 2000 expert trajectorics per environment. Additional training details can be found in Appendix B
In Figure I we see that SAM representations do the best on average, even surpassing the average success rates on policies pretrained with manipulation- based representations. agreement with Hu et al. (2023) we note that the ResNet50-based representations, (R3M, VIP, MOCOv3) are quite competitive, surpassing the success rales of all VIT representations except for those of SAM representations.
Representations for Manipulation more Sample Efficient. Data efficiency has always been area of interest for robotics and in general imitation learning. Particularly for robotics, large-scale high quality data collection is time-consuning, and logistically complex. Thus finding a representation that does well in a small sample regime is useful.
To study sample efficioncy, we choose environments where the average success rate was at least 80 pereent across all representations in our behavior cloning evaluation. We identified five environments: bulton press, window open, plate slide, door close, and window close. For almost all representations, these environments be solved with near perfect accuracy with little 40 demonstrations. We examine performance after training using 5, 10 and 40 expert demonstrations.
Plate Slide
1.0
Success
0.5
10 40
5 10
40
10 40
10
10:40
dip
moco
ibot
obj_m
vip
vcl
mvp
Door Close
success %
1.0
0.5
10
1040
5am
dino
dip
moco
mae
obj_m
vip
13m
vcl
mvp
Number Trajectories
Figure Sample efficiency of representations. Behavior cloning trained on 5, 10, and 40 expert demonstrations. Representations trained for manipulation such as R3M, VIP general as they achieve higher success rates even with just 5 expert demonstrations. MoCov3 representations are also competitive in the low sample regime. Sample efficiency plots for button-press, window-open, and window-close are shown in Appendix D
sample efficient in
From Figure 4 can see that generally, pretrained representations for manipulation are more sample efficient. MoCov3 is the most sample efficient relative to general pretrained representations, as it eventually solves the environments with 100 percent success using 40 trajectories. However it is not as sample efficient as R3M, or VIP since these representations can solve the environment at a higher success rate with fewer expert trajectories. In general, representations trained specifically for manipulation tasks are more effective when sample sizes are small, which is aligned with manipulation pretraining literature (Radosavovic al. 2022
Nair et. al.] 2022).

---

Published in Transactions on Machine Learning Research (06/2025)
OOD performance correlates with ID perfor- mance In robotics, it is common for the visual scene to change, due to distractions entering the scene, or a change in lighting intensity during a tra- jectory. In the context of visuo-motor control, it is important to identify image embeddings that are ro-
om Couretation
DMO
Eee 000 NAxIs Kate
bust such visual scene perturbations. To this end,
OCL
8.20
study the out of distribution performance of the vision encoders in the Gymansium environments by considering the following out of distribution scenar- ios: 1) a nuisance object in the background at a random location, 2) random lighting intensities, 3) and combination of both nuisance object and light- also studied in Burns al. (2024.. Figure | compares the out of distribution performance of the different encoders across the listed OOD scenarios with their ID performance. The correlation weew ID behav- ior cloning and OOD behavior cloning has a Spear- man correlation of îs = 0.642 and a Pearson corre- lation of f. 0.673. The performance distributions with each type OOD setting presented in Appendix
ing intensity changes. These pert ureations
Figure 5: OOD success rates overall aligned with ID scenarios. Stronger performing encoders in the ID case also perform relatively stronger in the OOD scenario.
H
Performance distribution architecture agnostic Both ResNet50 and ViT architectures were evalu- ated across all tasks. As shown in Figure[] high-performing models exist in both architectures: ResNet-based models like VIP, R3M, and MOCO perform well, do ViT-based models like SAM and IBOT. Similarly, lower-performing models appear in both architectures, with LesNet-based OBJ RN and ViT-based MVP and MAE showing relatively weaker performance. This distribution suggests no clear performance advantage between VIT and ResNet-based architectures.
Architecture
Success %
1SD
VTT Re
0.484 0.498
0. 0.145
0.188
Table 1: Comparative analysis architecture types show that architecture type does not affect manipulation performance.
Performance distribution is dependent on pretraining objective Figure suggests an association between encoder performance in manipulation tasks and their respective pretraining objectives. Encoders pretrained with holistic reconstruction objectives, specifically VC1, MAE, MVP, and OB.J RN, utilize mean square error loss on pixels, which are aggregated across entire image scenes. These encoders also demonstrate notably worse performance in manipulation tasks. As discussed in Section 3.2) despite the variety of pre- training datasets, Parisi et al. (2022) has demonstrated that pretraining data set is not critical to downstream manipulation performance. Thus, the comparative analysis in Table 2 can be viewed as a counterfactual comparison of pretraining objectives. We further discuss the association between performance and rretrain- ing in Section 5.3] Table alsoo indicates that for models trained with a holistic reconstruction objective, VIT models have a slight edge over ResNet based models for manipulation tasks, however the performance gains see larger increase when switching from holistic reconstruction objective, to discriminative pretraining objective.
7

---

Published in Transactions on Machine Learning Research (06/2025)
Architecture
Objective
Success
Standard Deviation
ResNet
Holistic Reconstruction
0.180
0.021
ResNet
Discriminative
0.603
0.049
VIT
Holistic Reconstruction
0.360
0.086
VIT
Discriminative
0.578
0.105
Table 2: Comparative analysis of pretraining objectives show that discriminative pretraining objectives produce reprosentations that are useful for manipulation.
4.2 Behavior Cloning versus Offline Reinforcement Learning
In addition to behavioral cloning, we also evaluate the effectiveness of various visual encoders when using offline reinforcement learning. Specifically, we use our reward-agnostic version of implicit Q-learning, training each policy for 10,000 gradient steps on 2,000 expert trajectories. We chose these parameters to make success rates comparable with our behavior cloning results. Training details and the algorithm description can be respectively.
found in Appendix B and Section g
Gymnasium IQL
Metaworld IQL
0.8
0.72
0.6
0.52 0,60
0,60
0.54
0.52
Success %
0.52
0.4
0.38
0.24
0.2
0,24
0.09
0.0
SAM
IBOT MOCO DINO
CLP R3M
MAE VC1
OBL MVP
OB0
Encoder
Encoder
Figure 6: Success rates with offline reinforcement learning with different representations. In
agrecment with Figure [ we see that the reprosentations with reconstruction pretraining objectives perform worse.
A surprising pattern emerges when we compare behavior cloning and offline reinforcement learning perfor- mance across different types of pretrained representations. While one might expect offline reinforcement learning to consistently improve upon behavior cloning, we find this benefit depends strongly on how the reprosentation pretrained.
Holistic reconstruccton--aaed representations struggle with offiine RL Models pretrained with a holistic reconstruction objective show inconsistent benefits from oflline reinforcement learning for example MAE improves by 12% on Gymnasium tasks but declines by 11% on Metaworld tasks, while MVP, and VC1 both see lower success rates by using offline reinforcement learning. In contrast, representations trained with contrastive objectives (DINO, MOCOv3, CLIP) show consistent improvements under offline reinforcement learning.
Manipulation-specific representations prefer behavior cloning Even more surprisingly, representa- tions specifically pretrained for manipulation (VC1, VIP, MVP, R3M) perform worse with offline reinforce- ment learning compared to behavior cloning as shown in Figure 7 This ellect. is particularly pronounced for R3M, which excels at behavior cloning but struggles with offine reinforcement learning, consistent with findings from Hu ct al. (2023).
8

---

Published in Transactions on Machine Learning Research (06/2025)
There has been much recent investigation on when one should use behavior cloning and when one should use offline reinforcement learning, and the
Metaworld Gains with IQL
Success Race increaze with KOL
relative strengths of both methods with respect to
0.00
the expert occupancy distribution being mode cov-
ering or mode soeking (Ke ct al. 2021 Ghasemipour
eet al., 2020). In Appendix F we carry out
-0.35
small investigation of settings where offline rein- forcement learning and behavior cloning excel and
-0.00
40
C
struggle. For the purpose of our main findings, we see that there is no large difference in encoder perfor- mances due to occupancy mode behavior, regardless of which training method we select.
1000
20T
Hi
R3H
OR
Models
Figure Holistic representations trained for manipulation perform better using behavior cloning. In fact, aside from VIP which sees gain, the manipulation representations achieve lower rates of success. Difference plots for Gymnasium and Meta- world are shown side by side in Appendix
5 What
Qualities of Visual Representations Matter for Manipulation?
The goal of this section to investigate the metrics that correlate] with how well visual representations can perform in manipulation tasks. We propose two types metrics: performance in keypoint prediction and performance in image reconstruction.
Keypoint prediction is a discriminative task where the representations are probed for information on keypoints such as positions of arm joints, end effectors, objects, or goal locations the scene. We define keypoint prediction as modeling P(v] (SS), where o(S) represents the encoded scene S from the expert demonstration and represents the vector of keypoints, which can include goals positions, arm and effec- tor positions and velocities, and scene object positions. Representations excelling in keypoint prediction highlight that the fcatures embed task-relevant information like spatial relationships among key objects and understanding precise manipulation dynamics. This aligns with prior work that leverages keypoints directly for visuomotor control, either by learning keypoint- representations end-to-end (Boney et al. 2022 Daniel & Tamar 2022), ; or by extracting keypoints through pretrained anguage-guidees models to support
downstream manipulation tasks (Fang et al. 2024 Palo & Johns 2024p.
Holistic reconstruction tasks focus on using global scene representations to reconstruct current scenes scenes k-steps into the future. We define reconstruction tasks to model P(S)ø(S)). That is, we learn to recover the input distribution of scenes that are encoded by the encoder (SS). Representations that excel at high-fidelity scene reconstruction suggest that they encode information around scene understanding and model transitions. We pay particular attention to edge reconstruction. Our hypothesis is that representations capable of accurately reconst ructing fine grained edges, especially that of objects involved in manipulation, are likely to encode spatial structure that useful for control.
To evaluate our hypothesis that both metrics correlated with imitation learning success, designed suile of experiments to assess these predictive and reconstructive capabilities of various vision encoders. These experiments test the ability of encoders to extract task-relevant features and reconstruct high-fidelity image reprosentations of current and future scenes.
5.1 Keypoint Prediction is Correlated with Success Rates
Following Hu et al. 2002), we approach keypoint prediction with the linear probing method. We took the representation ions of 100 Metaworld trajectories, each rendered with 150 scenes. For each scene, the linear
We correlation as measure how useful particular metric is, providing diagnostics of visual representations for manipulation tasks.
9

---

Published in Transactions on Machine Learning Research (06/2025)
probes learned the positions and rotations of the arm joints, and the positions of objects within the scenes, such hammers, blocks, walls, and windows. The probe also learned the goal positions for each trajectory. Each linear probe uses linear head and minimizes square error of the keypoint position vectors.
Figure g shows an overall negative correlation between the log of validation loss and the behavior cloning success rates for each vision representation, The ability to predict the goal position shows a strong negative correlation between its log of prediction loss and behavior cloning success rate (Ø. = 00.227 for Spearman, Å. 00.999 for Pearson). Meanwhile, predicting the arm's key positions showed a slightly weaker correlation (e. -0.891 for Spearman, P, -0.938 for Pearson). However, while slightly smaller in magnitude, these relationships were still strongly negative. That the arm positions had the lowest correlation is expected, as successful task completion can be achieved through multiple valid arm configurations the same task goal can be reached with different joint angles and positions. In contrast, object and goal positions in the scene have a more direct, one-to-one relationship with task success, as they uniquely define the desired end state of task. When correlating with the OOD success rates, the correlations remain strong, (p. = 0.750 for Spearman, Pr -0.790 for Pearson). The graphs showing these correlations are presented in Appendix H
Arm Pos tion Prediction
Goal Position Prediction
SAN
0.7
DNO
FDM
0.6
NECD
Average Succass Rate
MAEE
0.5
BOT
2.4
0.4
0.3
0.5
0.2
022
Spearman: -0.89
Spearman: -0.93
Pearson; -0.94
Pearson -0.98
10
Arm Prediction Logs (Log)
Goal Prodiction Lass (Log)
Figure 8: Correlation between key point prediction validation loss and behavior cloning per- formance. Lower validation loss in predicting key points correlates with higher behavior cloning success rates. Roconstruction-basel encoders (MAE, VC1, and MVP) show both higher prediction losses and lower success rates. Additional correlations for object position prediction and combined key point positions, along with outlier analysis, are provided in Appendix
These correlations suggest that representations that are better at predicting positions in a scene tend to achieve higher success rates in behavior cloning. To verify the robustness of this relationship, we conducted additional analyses measure the sensitivity of these correlative values. Even when excluding SAM from our analysis, the correlations remain strong (p. -0.87 and Á. -0.903; see Appendix E.. Furthermore, DFBETA analysis (Kleinbaum et al, 1988) confirms that SAM does not unduly influence these correlations. These results strongly support the finding that a representation's position prediction ability, measured by log of validation loss, correlates with its behavior cloning performance.
5.2 Reconstructive Abilities are Correlated with Environment Success Rates
To evaluate the reconstructive abilitics of various encoders, we encode 224 224 RGB images from export trajectories and attempt to reconstruct the current scene as well as scenes 5 and 20 steps into the future. The reconstruction network uses four transposed convolution layers and combines square error loss with
10

---

Published in Transactions on Machine Learning Research (06/2025)
Scene Reconstruction
Pixel Difference Map
Edge Detection
SAM
MVP
Figure 9: Scene reconstruction quality correlates with representation performance. Visualiza- tions show reconstructed scenes, pixel-wise error maps, and Sobel edge detection for SAM (top) and MVP (bottom). SAM, which was the best-performing model, produces accurate reconstructions with clear edges. In contrast, MVP fails to capture critical scene elements.
Current Scene Reconstruction
5-Step Scene Reconstruction
SAM
DND
0.7
0.7
M/P
R3M
0.5
MOCO
IBDI
DBL MN
Average
0.3
X
Pranson: 0..45
Peansn: 0.872
0.035
0.040
0.045
0.050
0.055
0.060
0.035
0.040
0.045
0,050
0.055
0.059
Nomalzed Average Placi Difference
Normalized Average Piael Difference
Figure 10: Scene reconstruction quality correlated with behavior cloning success rate. Global scene reconstruction is strongly correlated with behavior cloning success rates. Each representation achieves similar level of reconstruction quality across the current and five-step scene reconstructions. This trend generalizes to the longer horizon, 20-step futurre scene reconstruction, which is shown in Appendix G
VGG-based perceptual loss Johnson et al. (2016). The perceptual loss proved crucial. No representation achieved high-fidelity reconstructions without it.
We evaluate reconstruction quality using two metrics: pixel-wise distance and edge preservalion via Sobel edge detection (Kanopoulos et al., 1988). between reconstructions and original scenes. Figure I0 shows that bchavior cloning success rates correlate strongly and negatively with pixel differences when reconstructing the current scene (Spearman correlation å. -0.882 and Pearson correlation å, -0.84).. In the OOD setting, the correlations also strong (Spearman correlation Ó. -0.918 and Pearson correlation Å. -0.910).
11

---

Published in Transactions on Machine Learning Research (06/2025)
The graphs showing these correlations are presented in Appendix H Similar strong correlations also hold for 5-step and 20-step future scene reconstructions, with analyses available in Appendix
The relationship between reconstruction quality and model performance is visually apparent when comparing our best and worst-performing models, demonstrated in Figure SAM's reconstruction shows sharp edges around the block and end effector, with low pixel-wise errors. contrast, MVP's reconstruction fails capture these critical scene elements, showing high pixel-wise errors and poorly defined edges, Surprisingly, pretraining, MAE, MVP, and VC1 representations perform worse at recon-
despite their reconstruction-bant struction than other pretrained models, which discuss further in Section 5.3 Additional reconstruction examples are shown in Appendix
Moving to our next experiment, we see reconstruction of edges is also correlated with success rales. To evaluate edge reconstruction quality, following Kirillov et al. (2023) we apply Sobel filters to both original and reconstructed current scenes, comparing their MSE and structural similarity index mcasure (SSIM) (Wang al., 2004). We observe strong negative correlations between edge reconstruction loss and behavior cloning success rates in Figure [1 (Pearson P, -0890 and Spearman P. 0.9188). The edge structural similarity shows similarly strong correlations (Pearson P, -09913, Spearman Å_ 0..277). This suggests that the ability to encode edge information is particularly important for downstream manipulation tasks. In the OOD setting, the correlations with edge reconstruction is strong (Pearson P, = -0.962, Spearman P. -0.910). Additionally, the OOD correlation with edge structural similarity is also strong (Pearson P. = 0.934, Spearman P.- = 0.955). The graphs showing these correlations are presented in Appendix
Edge Reconstruction
Edge Structural Similarity
SAM
TWO
0.7
R3M
0.6
0.6
VC1
BOT
2.5
0.4
0.2
62
Speammam 0
Pearson: +0.89
Pearson: 0.91
3003
3200
3100
3600
3800
4000
0.64
0.6
0.66
0.67
0.68
0.69
0.70
0.71
Reconstruction Loss
Structural
Figure 11: Edge reconstruction quality correlates with manipulation performance. Both edge reconstruction error (left) and structural similarity (right) show strong correlations with behavior cloning success rates. Interestingly, reconstruction pretrainnd models (MAE, MVP, VC1) consistently show poor edge preservation.
Our analysis of both keypoint prediction and reconstruction tasks revcals consistent pattern: representa- tions that can predict keypoints with lower validation loss and reconstruct scenes with lower errors achieve better manipulation performance. Notably, coonstruction-pretaaiie representations struggle with both key- point prediction and scone reconstruction fidelity. Mcanwhile, representations trained with discriminative objectives excel at both tasks. These results suggest that effective robotics representations should perform well on both discriminative and generative auxiliary tasks.
12

---

Published in Transactions on Machine Learning Research (06/2025)
5.3 Understanding the Impact of Pretraining Objectives Robotic Performance
There is an apparent contradiction in the vision-based roboties literature. Parisi et al. (0022) showed that the sulf-supe MoCo representations outperforms supervised ones like CLIP and ImageNet pretrained ResNets, in robotic manipulation. Yet, Burns et al. (2024) found supervised representations on ImageNet performed strongest. We propose that this apparent conflict can be resolved by reframing the compari- son: rather than supervised versus supervid, the key distinction lies in whether representations are pretrained with reconstructive or discriminative objectives.
To understand this distinction, we need to examine how these pretraining objectives fundamentally differ. Holistic reconstruction approaches, like masked autoencoding on the entire image, focus on rebuilding the global view of the image pixel by pixel, often using masked patches. Although masking creates regularized global representations, the resulting representations are embedded with inherent global reconstruction errors. These errors force downstream manipulation policies to learn from reconstruction artifacts rather than focus- ing on object or region-specific information. In contrast, discriminative approaches optimize representation spaces to capture semantic relationships, either through direct prediction of supervised signals or through contrastive learning between positive and negative pairs (Geiping et al. 2023), facilitating downstream policy learning. Table 3 classifies common vision encoders along these lines.
Encoder
Objective
Type
SAM
Segmentation Prediction
Supervised
MOCOv3
Self Contrastive
Self- Selff-Spervised
R3M
Time Language Contrastive
Sulf-Supervisod
VIP
Value function prediction
Supervised
DINOv2
Self-distillation
Self-Supervised
CLIP
Text Image Contrastive
Supervised
IBOT
Masked contrastive + self distillation
Self- Self-Supervised
MAE
Masked Image Reconstruction
Self-Supervised
VCI
Masked Image Reconstruction
Self-Supervised
MVP
Masked Image Reconstruction
Suff-Supervieed
OBJ RN
Image Reconstruction
Self- Saff-Sueervised
Table 3: Pretraining objectives of vision encoders. We provide additional details such as the pretraining
data set and architectures used in Appendix B.1
When we analyze our results through this lens of holistic reconstruction versus discriminative pretraining, clear pattern emerges. Across our experiments, representations trained with discriminative objectives, whether supervised or self-supervised, consistently outperform those trained with global reconstructive ob- models (MAE, MVP, VC1) perform poorly even at reconstruction tasks. Despite being explicitly trained to reconstruct entire images, these models struggle capture fine-grained details and edge information critical for manipulation as seen in Figures 10 and 11
jectives. Most surprisingly, occonstruction-peetaaiii
We hypothesize this performance gap likely stems from fundamental differences in how these objectives structure representation spaces. Discriminative approaches naturally emphasize meaningful feature separa- tion and semantic understanding. In contrast, global reconstructive approaches focus on overall pixel-level fidelity across the entire scene, palentially at the cost of semantic abstracttonn of individual objects. This distinction helps explain the strong performance of models like SAM and DINOv2, whose pretraining regimes are known to produce fcatures that facilitate segmentation (Kirillov et al. 2023 Oquab et al. 2023). Simi- larly, data augmentations and contrastive learming in MoCOv3 and DINOv2 help encode localized features CCen et al. 2021 Oquab et al. 2023), which proves valuable for manipulation tasks.
13

---

Published in Transactions on Machine Learning Research (06/2025)
6 Conclusion
In this work, we conducted a systematic evaluation of pretrained vision representations for robotic manip- ulation, examining their performance across 21 tasks in three environments. Our analysis reveals that the distinction between global reconotruction and discriminative pretraining objectives more crucial than the supervised versus self-supervised categorization. Representations trained with discriminative objectives con- sistently outperform those trained with global reconstructive losses across both behavior cloning and offline reinforcement learning tasks.
We identified three diagnostic metrics-keypoing prediction, scene reconstruction, and edge preserva- tion that strongly correlate with a representation's manipulation performance. In particular, scene recon- struction and edge preservation are both effective diagnostics for manipulation success, even in the presence of out of distribution perturbations to the environment. Surprisingly, models pretrained with reconstructive objectives struggled with reconstruction tasks, suggesting these pretraining objectives may not effectively caplure the fine-grained details needed for manipulation. These findings help explain seemingly contradictory results in prior work comparing supervised and supfrvuue representations.
Based on these findings, we hypothesize that future vision encoders for robotics might benefit from combining discriminative objectives with carefully designed reconstruction losses that preserve structural information. For example, incorporating perceptual losses or explicit edge preservation during pretraining might help capture the fine- gre-grainod details that prove crucial for manipulation tasks. Additionally, object-centric rep- resentations may offer advantages for manipulation tasks, as they capture semantic abstractions of objects within scenes while simultaneously maintaining high-level fidelity for image reconstruction. The landscape of vision encoders is vast, and these insights offer promising directions for further exploration.
References
Rishabh Agarwal, Marlos C Machado, Pablo Samuel Castro, and Mare G Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. In International Conference on Learning Representations, 2021.
Gaon An, Seungyong Moon, Jang- Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing syslems, 34:7436-744 7447, 2021.
Mihael Ankerst, Markus M Breunig, Pater Kriegel, and Jörg Sander. Optics: Ordering points identify the clustering structure. ACM Sigmod record, 28(2):49 60, 1999.
Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. Xiv preprint arXic:2108.. 07258, 2021.
Rimu Boney, Alexander Ilin, and Juho Kannala. Learning of feature points without additional supervision improves reinforcement learning from images, 2022. URL https://arxiv..rrg awwp://bs//10...... 079955
Kaylee Burns, Zach Witzel, Jubayer Ibn Hamid, Tianhe Yu, Chelsea Finn, and Karol Hausman. What makes pre-trained visual representations successful for robust manipulation? In 8th Annual Conference on Robot Learning, 2024.
Ricardo JGB Campello, Davoud Moulavi, and Jörg Sander. Density-based clustering based on hierarchical density estimates. In Pacific-Asia conference on knowledge discovery and data mining, pp. 160-172.
Springer, 2013.
Shizhe Chen, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Sugar: Pre-training 3d visual repre- sentations for robotics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18049 18060, 2024.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 9640 9649, 2021.
14

---

Published in Transactions on Machine Learning Research (06/2025)
Yen-Chi Chen, Christopher R. Genovese, and Larry Wasserman. A comprehensive approach to mode clus- tering. Electronic Journal of Stalislics, 10(1); 2016.
Tianhong Dai, Kai Arulkumaran, Tamara Gerbert, Samyakh Tukra, Feryal Behbahani, and Anil Anthony Bharath. Analysing deep reinforcement learning agents trained with domain randomisation. Neurocom- puting, 493:143-165, 2022.
Tal Daniel and Aviv Tamar. Unsupervised image representation learning with deep latent particles. In International Conference on Machine Learning, pp. 4644-4665. PMLR, 2022.
Xiaolin Fang, Bo-Ruci Huang, Jiayuan Mao, Jasmine Shone, Joshua B. Tenenbaum, Tomás Lozano- Lozano-Pérez, and Leslie Pack Kaelbling. Keypoint abstraction using large models for object-relative imitation learning, 2024. URL
www.://ggg/bb/////gge 23254
Chelsea Finn, Sergey Levine, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334 1373, 2016.
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. In Thirty-Fig Fifth Conference on Newral Information Processing Systems, 2021.
Jonas Geiping, Quentin Garrido, Pierre Fernandez, Amir Bar, Hamed Pirsiavash, Yann LeCun, and Micah Goldblum. A cookbook of sell-supervised learning. arXie preprinl 0rxi::2304.12 12210, 2023.
Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. divergence minimization perspective on imitation learning methods. In Conference on robot learning, pp. 1259-1277 PMLR, 2020.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, He- una Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The' something something' vidco database for learning and ovaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pp. 28420885, 2017.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 18992-19012, 2022.
Tuomas Haarnoja, Ben Moran, Guy Lever, Sandy H Huang, Dhruva Tirumala, Jan Humplik, Markus Wulfmeier, Saran Tunyauuvuakool, Noah Y Siegel, Roland Hafner, et al. Learning agile soccer skills for bipedal robot with deep reinforcement learning. Science Robotics, :(89))cad88022, 2024.
Nicklas Hansen, Zhecheng Yuan, Yanjic Ze, Tongzhou Mu, Aravind Rajeswaran, Hao Su, Huazhe Xu, and Xiaolong Wang. On pre-training for visuo-motor control: revisiting a learning-from-seree Proceedings of the 40th International Conference on Machine Learning, pp. 12511- 2023.
baseline. In
Dan Haramati, Tal Daniel, and Aviv Tamar. Entity-centric reinforcement learning for object manipulation from pixels. In The Twelfth International Conference on Learning Representations, 2024. URL https: /openreview httt/foorumiiduuxeeSiie
Negin Heravi, Ayzaan Wahid, Corcy Lynch, Pete Florence, Travis Armstrong, Jonathan Tompson, Pierre Ser- manet, Jeannette Bohg, and Debidatta Dwibedi. Visuomotor control in multi-object scenes using object- aware representations. In 2029 IEEE International Conference on Robolics and Automation (TCRA), pp. 9515 9522. IEEE, 2023.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advonces in Neural Infor- mation Processing Systems, volume 29, 2016.
Yingdong Hu, Renhao Wang, Li Erran Li, and Yang Gao. For pre-trained vision models in motor control, not all policy learning methods are created equal. In International Conference on Machine Learning, pp. 13628 13651. PML.R, 2023.
15

---

Published in Transactions on Machine Learning Research (06/2025)
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In Compuler Vision-FCCC 2016: 14th Europeon Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 11 14, pp. 694 711. Springer, 2016.
Nick Kanopoulos, Nagesh Vasanthavada, and Robert Baker. Design of an image edge detection filter using the sobel operator. IEEE Journal of solid-state circuits, 23(2):358-30) 367, 1988.
Liyiming Ke, Sanjiban Choudhury, Matt Barnes, Wen Sun, Gilwoo Lee, and Siddhartha Srinivasa. Imitation learning as f-divergence minimization. In Algorithmic Foundations of Robotics XIV: Proceedings of the Fourleenth Workshop on the Algoritlmice Foundations of Robotics 14, pp. 513-329 Springer, 2021.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pp. 4015 4026, 2023.
David G Kleinbaum, Lawrence L Kupper, Keith E Muller, and Azhar Nizam. Applied regression analysis and other maltivariable methods, volume 601. Duxbury press Belmont, CA, 1988.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning, In The Tenth International Conference on Learning Representations, TCLR 2022, Virtual Event, April 25-29, 2022, 2022.
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations, 2022. URL ltttss:/ nett/forumidd--eeeeeeeette
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offiine reinforce- ment learning. Advances in Newral Information Processing Systems, 33:1173 199779911 2020.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkorcit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Informa- tion Processing Systems, volume 33, pp. 11525-11538 Curran Associates, Inc., 2020.
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang, VIP: Towards universal visual reward and representation via value-implicit pre-training, In Deep Rein- forcement Learning Workshop NeurIPS 2022, 2022.
Yecheng Jason Ma, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. Liv: language- image representations and rewards for robotic control. In Proceedings of the 40th International Conference on Machine Learning, 2023.
Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Pierre Berges, Tingfan Wu, Jay Vakil, et al. Where are we in the search for an artificial vi- sual cortex for embodicd intelligence? Advances in Neural Information Processing Systems, 36:655-677, 2023.
Leland Mclnnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXi::1802.00 03426, 2018.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. CoRR, abs/2006. 0006609359, 2020.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. In 6th Annual Conference on Robot Learning, 2022.
Maxime Oquab, Timothóe Darcet, Théo Moutakanni, Huy Vo, Mare Szafranice, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Xiv preprint ar XxXx:2304.071990 2023.
16

---

Published in Transactions on Machine Learning Research (06/2025)
Norman Di Palo and Edward Johns. Keypoint action tokens enable in-context imitation learning in robotics, 2024. URL
19978 19578
Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupla. The unsurprising effec- tiveness of pre-trained vision models for control. In international conference on machine learning, pp. 17359-17371. PMLR, 2022.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage- regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00:. 001707, 2019.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement learning: Chal- lenging robotics environments and request for research. arXiv preprint 0rXic:1802.094400 2018.
Carl Qi, Dan Haramati, Tal Daniel, Aviv Tamar, and Amy Zhang. EC-diffuser: Multi-object manipulation via entity-centric behavior generation. In The Thirteenth International Conference on Learning Represen- tations, 2025. URL httpss//openrrevee: wett/oormmiddoop//ooooo
Lija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell, Real-world robot learning with masked visual pre-training. CoRL, 2022.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint ar Xiv: 709.10087, 2017.
Alexander Sax, Bryan Chen, Francis Lewis, Iro Armeni, Silvio Savarese, Amir Zamir, Jitendra Malik, and Lerrel Pinto. Robust policies via mid-level visual representations: An experimental study in manipulation and navigation. In Conference on Robot. Learning, pp. 2328 2248.. PMI.R, 2021.
Moritz Schneider, Robert Krug, Narunas Vaskevicius, Luigi Palmieri, and Joschka Boedecker. The surprising ineffectiveness of pre-trained visual representations for model-based reinforcement learning. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2023.
Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey. Understanding human hands in contact at internet scale. In Proceedings of the IEEE/CVE conference on computer vision and pottern recognition, pp. 9869 90089977,722 2020.
Mohit Sharma, Claudio Fantacci, Yuxiang Zhou, Skanda Koppula, Nicolas Heess, Jon Scholz, and Yusuf Aytar. Lossless adaptation of pretrained vision models for robotic manipulation. In The Eleventh Inter- national Conference on Leurning Representations (ICLR), 2023.
Manan Tomar, Utkarsh Aashu Mishra, Amy Zhang, and Matthew E Taylor. Learning representations for pixel-based control: What maiters and why? Transactions on Machine Learning Research, 2023.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-61) 612, 2004.
Markus Wulfmeeerr Arunkumar Byravan, Tim Hertweck, Irina Higgins, Ankush Gupta, Tejas Kulkarni, Malcolm Reynolds, Denis Teplyashin, Roland Hafner, Thomas Lampe, et al. Representation matters: Improving perception and exploration for robotics. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 1512-6519 TEFF, 2021.
Lin Yen-Chen, Andy Zeng, Shuran Song, Phillip Isola, and Tsung-Yi Lin. Learning to see before learning to act: Visual pre-training for manipulation. In IEEE International Conference on Robotics and Automation (ICRA), 2020.
Jacsik Yoon, Yi-Fu Wu, Heechul Bac, and Sungjin Ahn. An investigation into pre-training object-centric representations for reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning, ICMI.23. JMLR.org, 2023.
17

---

Published in Transactions on Machine Learning Research (06/2025)
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094 1100. PMLR, 2020.
Yanjie Ze, Yuyao Liu, Ruizhe Shi, Jiaxin Qin, Zhecheng Yuan, Jiashun Wang, and Huazhe Xu. H-index: Visual reinforcement learning with Land-informed representations for dexterous manipulation. NeurIPS, 2023.
18

---

Published in Transactions on Machine Learning Research (06/2025)
A Environments and Experts
Metaworld The Metaworld suite contains multiple grasping and mudging tasks, in which an end-effector manipulates objects in the environment to complete the task. Metaworld provides expert policies, from which we created expert trajectories for imitation learning. We used a total of 17 environments from Metaworld. Each task intializes to a random starting position, and a random goal position, making this a good environment for multi-goal tasks.
Fetch Suite From the Fetch suite, we evaluate two environments, FetchPush and FetchPickAndPlace In both environments, the goal is to move block to goal, either by pushing it around the table top, or by grasping the block and moving to goal location. We trained SAC policy with hindsight relabeling to create the experts.
AdroitHand From the Hand suite, we evaluated two environments, Hammer and Door. Both environments involve manipulating a 30+ DoF' hand to open a door or drive a nail into a board with a hammer. We trained a behavior cloning policy on the Farama expert trajectories to create larger expert trajectory data set. The Fetch and AdroitHand suite are collectively referred to as the Gymnasium environments due to their of the Farama gymnasium library.
19

---

Published in Transactions on Machine Learning Research (06/2025)
B Training Parameters
Each epoch in behavior cloning makes 100 gradient steps using 256 scenes for cach gradient step, for a total of 10,000 gradient steps. IQL trains with 10,000 gradient steps using 256 scenes for each step. Our IQL algorithm uses 256 samples of the transition tuple (St, a, 81+1) (without reward signals), while behavior cloning uses 256 samples in minibatches of (st, az).
Behavior Cloning
IQL
Batch Size Training Size
256
256
2000
2000
Epochs
100
Minibatches
100
Gradient Steps
10000
Actor LR.
0.0008
0.00015
Q LR
0.0003
V LR
0.0003
Expectile 7
0.7
3
Polyak T
0.05
Table 4: IQL and Behavior Cloning make the same number of gradient steps on the same size of minibatch.
We do not include any reward signal to perform the offline reinforcement learning. Instead, we use an inverse Q learning method from Garg (2021).
B.1 Model Architectures
Encoder
Objective
Type
Pretraining Data
Architecture
SAM
Segmentation Mask Prediction
Supervised
SA-1.1B
ViT-B
DINOv2
Self-distillation
delf-Supervised
ImageNet
VIT-B
CLIP
Text Image Contrastive
Supervised
WIT
VIT-B
MAE
Masked Image Reconstruction
Self- Supervised
ImageNet
VIT-B
VIP
Value function prediction
Supervised
Ego4D
ResNet50
VCI
Masked Image Reconstruction
Self- Self-Supervised
Ego Suite
ViT-B
R3M
Time Language Contrastive
Supervised
Ego4D
ResNet50
MVP
Masked Image Reconstruction
Self-Super Supervisedd
ImageNet, Robotics
VYT-B
Adja-
cent
MOCOv3
Contrastive
Self-Supervised Self-Supervised Self- Seler-Suerviied
ImageNet
ResNet50
IBOT
Contrastive + Dost-iistillation Reconstruction
ImageNet
ViT-B
OBJ RN
Objects 365
ResNet50
Table 5: Pretraining Objectives vision encoders.
20

---

Published in Transactions on Machine Learning Research (06/2025)
Encoder
Architecture
Policy VRAM (MB)
150 step runtime (seconds)
SAM
ViT-B
568.317
7.076 + 0.724
DINOv2
VIT-B
336.244
4.995 0.701
CLIP
ViT-B
573.281
2.307 0.070
MAE
ViT-B
332.972
2.393 + 0.078
VIP
ResNet50
199.581
2.347 + 0.148
VCI
ViT-B
332.972
2.397 + 0.070
R3M
ResNet50
560.989
2.372 = 0,131
MVP
ViT-B
332.972
2.409 + 0.063
MOCOv3
ResNet50
262.767
2.117 + 0.126
IBOT
ViT-B
657.525
2.401 + 0.069
OBJ RN
ResNet50
469.160
2.413 + 0.121
Table 6: All policies be trained single A10 machinos. Most policies take between to seconds to complete Irajectory and produce an action for 150 steps, however SAM and DINO take longer for their forward passes. Averages and standard deviations are reported over 5 repetitions on a single A10 machine.
C Reinforcement Learning Differences
Metaworld Gains with
with
0
0.00
-3.05
-215
D80
06 28
MM
Models
Gymnasium Gains with IQL
0.00
Success Satt Increase
0.0%
015
-0.5
635
0880
M8
SHN
MPP
Models
Figure 12: Representations trained for manipulation perform better using behavior cloning. In Gymnasium environments, MAE sees improvement while in the Metaworld suite, MAE sees decline in performance. Across the board, manipulation based representations decline in performance by using offline reinforcement learning.
21

---

Published in Transactions on Machine Learning Research (06/2025)
D Sample Efficiency
Button Press
Success
0.5
0.0
10
40
40
5 10:0
5 10 40
10 40
10 40
10 40
40
10 40
10 40
10 40
dino
dip
moco
ibot
mae
obj_
vp
r3m
vc1
mvp
Number of Trajectories
Window Close
Success %
0.0
10
10 40
5 10
10
10 40
10
10
10:0
dino
dip
moco
ibot
obj_m
vip
13m
vc1
mvp
Window Open
1.0
Success
0.5
0.0
10 40
5 10 40
5 10 40
5 10 40
10 40
10 40
5 10 40
5 10
10 40
10 40
5 10
sam
dino
dip
moco
ibot
mae
obj_m
Vip
r3m
vc1
mvp
Plate Slide
1.0
success %
10
40
10
10
10:40
dino
dip
moco
ibot
obj_m
vip
vcl
mvp
Door Close
Success %
1.0
0.0
1044 dino
5 1040 40
10
40
10:00
10
10 40
1040
10
1040
dip
objm
vip
mvp
Number of Trajectories
Figure 13: Sample efficiency of representations for behavior cloning evaluated 5, 10, and 40 expert demonstrations. Representations trained for manipulation such as R3M, VIP are more sample efficient in general as they achieve higher success rates even with just 5 expert demonstrations.
22

---

Published in Transactions on Machine Learning Research (06/2025)
E
Key Position Prediction
For objects and goals, we observe similar correlation patterns. Figure 14 keypoint prediction and behavior cloning success rates.
show a strong correlation between
WAT
MAT
Avtragge
VOCO
NOCO
Pearson: -0.87
Pearson: -0.92
Spearman: -0.77
Spearman: -0.83
0.0
0.0
10
10-3
10
10
Arm Posttion Prediction Validation
Geal Posibion Fred Prediction Validation Loss
0.6
W/2
MVP
Aunrago
VP
VP
Pearson: -0.99
Pearson: -0.90
Spearman: -0.83
Spearman: -0.82
3.0
10-4
10
10-4
30-3
Object Position Prediction Velidation
All Position Prediction Velidetion Loss
Figure 14: Correlations between prodicting positions and behavior cloning success rates. All three demon- strate strong correlations.
Position
Pearson (p-value)
Spearman (p-value)
Arm
-0.875 (0.0020)
-0.767 (0.0159)
Goal
-0.921 (0.0004)
-0.833 (0.0053)
Object
-0.886 (0.0015)
-0.833 (0.0053)
Combined
-0.904 (0.0008)
-0.817 (0.0072)
Table 7: Pearson and Spearman rank Correlations for keypoint prediction are all statistically significant the 0.05 level
SAM can be considered as an influential observation and could affect the explanatory nature of our correlation analysis. The correlations without the SAM data points are in Figure [5] Additionally, DFBETA analysis in Table show that neither of the SAM observations have high influence Kleinbaum et al. 1988
The analysis Tables 7 and g shows that removing SAM sees maximum Pearson correlation increase of 0.05 and maximum Spearman correlation increase of 0.17. While the Spearman increase of 0.17 is larger than the Pearson increase, Spearman correlation is non-parameteric method, which inherently has
23

---

Published in Transactions on Machine Learning Research (06/2025)
20
Spaaman:-..0 4.45
Figure 15: Correlations between predicting positions and behavior cloning success rates after removing SAM. Correlations remain strong.
lower statistical power compared to the Pearson correlation, thus its larger increase may not be statistically significant. The differences (Table 9 do not lie in the rejection region of 0.667 and are not considered influential.
Position
Pearson (p-value)
Spearman (p-value)
Arm
-0.903 (0.0021)
-0.857 (0.0065)
Goal
-0.973 (0.0000)
-1.000 (0.0000)
Object
-0.933 (0.0007)
-1.000 (0.0000)
Combined
-0.937 (0.0006)
-0.976 (0.0000)
Table 8: Without SAM, the Pearson and Spearman rank Correlations for keypoint prediction are also all
statistically significant but are not different from that in Table 7
Arm
Goal
Object
Combined
DFBETA
0.007
0.002
0.0003
0.007
Table 9: DFBETA analysis on the correlation coefficients by omitting SAM. All are below the threshold of 0.667, meaning SAM is not an influential point
24

---

Published in Transactions on Machine Learning Research (06/2025)
F Clustering Representations
Interostingly, we find that grasping tasks are harder to solve than mudging tasks in both behavior cloning and reinforcement learning. We define grasping tasks to be tasks where the expert uses end-effectors of the robotic arm to pick up and grip an object in the environment, while a nudging task is one where the expert uses the end-efiectors to push or drag an object without moving the effectors. Examples of grasping tasks are hammer, coffee pull, and bin picking. Nudging tasks inchude plate slide, soccer, and dial turn.
We aim to evaluate which representations benefit from using offline reinforcement learning regime. To answer this question, we study the modality of the expert action distributions, following Ghasemipour et al. (2020); Ke et al. (2021]. We view our expert data set as (ø(s), a) where the pretrained image representations o(s) define the state space, and a denotes the expert action. The state-action occupancy distribution is denoted by Pg. The expert policy is represented as π (a]p(s)) which is a conditional action distribution. We aim to learn the conditional action distribution using the expert collected trajectories with imitation algorithms. Prior work from Ghasemipour et al. (2020) differentiate between mode-covering and mode- soeking imitation algorithms that recover expert action distributions. Behavior cloning is mode covering objective, which makes it particularly difficult to recover expert action distribution that exhibits multiple modes.
We argue that the expert conditional action distributions x (alo(s)) for grasping tasks contain more modes than for nudging tasks. This is clear from the optimization objective in behavior cloning with an Lg objective, which is equivalent to minimizing a forward KL, divergence of Gaussian policy:
min (xI(T||= min E sooe log log((al((s))
This formulation corresponds to a mode-covering objective. In contrast, the optimization procedure in implicit Q learning follows an advantage-weighted regression objective, which Nair et al. (2020) Peng et al. (2019h demonstrate is equivalent to a reverse KL divergence:
min DEL (T min E s-r (((66)) log (alp(s())
Unlike behavior cloning, this objective corresponds to a mode-seeking optimization regime, making it better suited for recovering optimal actions given a multi-modal expert action distribution.
Our expert conditional action distribution are samples drawn from a contimuous state action occupancy ue and modes are detected through clustering (Chen et al. 2016). To test our hypothesis that grasping tasks exhibit. higher modality in the conditional action distribution, we discretized the representation state space by first applying UMAP (Melnnes et al. 2018) dimension reduction the representations. UMAP preserves local density structure, makes non-lincar down projections using intrinsic dimension and algebraic simplexes, and is widely used in dimension reduction and analyzing representations (Agarwal et al.) 2021 Haarnoja et al. 20244] Dai et al. (022). After applying UMAP, we use HDBSCAN (Campello et al. 2013)) to cluster the state representations. Upon discretizing the representation space, we cluster actions within each discretized state representation, and count the number of clusters in the conditional action distribution and
visualize example in Figure 17 The details of this algorithm are given in Algorithm T
We demonstrate
that dimension reduction with UMAP is not necessary in Table [I
For the results in Table 10 we use UMAP with 16 neighbors casting down to 3 dimensions. This produced clusters using HDBSCAN with the largest Silhouette values. We find that the number of negatively scored clusters is minimal with neighbors set at 16. We note that this method is sensitive to hyperparameter tuning.
When we do not reduce the dimensionality, and directly cluster with OPTICS to find approximate conditional action distribution, and then find clusters of actions, we still see that nudging tasks have less modes than grasping tasks in general across representations. The differences correlate with the gains in Figure 16 with Pearson á 0.716 and Spearman A. 0.7. However, this method may not be preferred due to the computation time due to clustering in high dimensions.
25

---

Published in Transactions on Machine Learning Research (06/2025)
Differences in Success Rate between Nudging and Grasping Tasks
BC
0.21
Diff Nudge Grasp
0.18
0.15
0.09
0.06
0.03
SAM
DINO
CLIP
MAE
MVP
R3M
VIP
VC1
MOCO
Encoders
Figure 16: IQL is better at solving grasping tasks than BC The difference in success rates between grasping and nudging tasks vary across encoders, and also by the learning method. Reinforcement learning reduces the gap in success rate between nudging and grasping tasks (blue bar). However the level of im- provement varies depending on representation.
Algorithm Determining modality with continuous expert state-actions
Require: Expert data D = ( Tuned clustering algorithm K(-): : R" + N, UMAP algorithm U + Rd
UMAP reduced embeddings u <- U(((s)
Number of clusters K (u) and labels K2 E {I,...,K} Append cluster labels to expert data D (Ø(st), at, Kt) Initialize array L to store number of modes in each cluster for c= do
Subset D to get states belonging to cluster c. De + (Ø(st), ar Kr c)
Extract all actions a from D, to get A, Get number of clusters M from conditional actions, M + K(Ac) Append L[c] +- M
Optional: Apply Ae + U(Ac)
end for
return Mean number of clusters L
Encoder d(-) Average modality for Nudging
Average modality for Grasping
SAM
3.405
4.083
0.678
DINO
2.987
3.929
0.942
CLIP
2.665
4.556
1.891
MAE
2.776
3.653
0.877
MVP
2.899
4.035
1.136
R3M
3.008
4.887
1,878
VIP
3.350
3.722
0.372
VCI
2.709
4.317
1.608
MÓCO3
4.063
5.190
1.127
Table 10: Average number of clusters in the conditional action distributions. All conditional export action distributions exhibit more modes in grasping (asks than for nudging tasks. Each number represents the average mode count from 3000 scenes representations in each of the 17 environments.
26

---

Published in Transactions on Machine Learning Research (06/2025)
Encoder o(-)
Modes π (al((()) for nudging
Modes (a)o(())) for grasping
SAM
3.81
3.88
0.07
DINO
2.75
5.89
3.14
CLIP
3.75
6.44
2.69
MAE
1.62
2.22
0.60
MVP
3.00
3.44
0.44
R3M
2.25
4.11
1.86
VIP
2.25
3.55
1.30
VC1
2.63
3.44
0.81
MОCO3
5.72
8.06
2.34
Table 11: Average number of clusters in the conditional action distributions. SAM representations form a state space that has almost the number of clusters between mudging and grasping tasks. The clustering is done by the OPTICS (Ankerst el al. 1999) algorithm. This produces moderate to strong correlations
4
(a) 3D representation of expert actions from given discretized d()) from coffee-pull (grasping), see multi-modal behavior - (alΦ(x)) in part due the end effector movement.
(b) For a given discretized d(s) from window-close (mudging), multi-modal behavior is not prominent in x (alá())) There no end effector movement, and points roughly form uniform subspace.
Figure 17: 3D representation of the action space. The (xt, y, 2) axes parameterize the arm movement, and the colors green and blue respectively parameterize the end effector being open and closed.
27

---

Published in Transactions on Machine Learning Research (06/2025)
1.0
Percentage improvement by Using IQL
0.8
X
0.6
SAM
DINO
CLIP
0.4
MAE
MVP
R3M
0.2
VC1
MOCO
0.0
0.6
1.0
1.4
1.8
Mode Difference
Figure 18: Correlations between the mode differences and the percentage improvements gained from using a mode socking objective. The corrolation is moderate and positive, which demonstrates the benefit of using mode seeking algorithms when the differences expert conditional action distribution modes are larger.
G Reconstructions
Reconstructions from various environments from Metaworld shown in Figures 19 and 20 below:
G.1 Reconstructions Correlate with Success Rates
We can also see the p-values in Table 12 and observe the correlations are significant at the 0.05 level.
Scene
Pearson (p-value)
Spearman (p-valuc)
Current
-0.805 (0.009)
-0.817 (0.007)
5-step
-0.816 (0.007)
-0.817 (0.007)
20-step
-0.809 (0.008)
-0.817 (0.007)
Table 12: The correlation between image reconstruction strongly correlated with behavior cloning success and the correlations statistically significant at the 0.01 level.
Scene
Pearson (p-value)
Spearman (p-value)
Current
-0.910 (0.0001)
-0.918 (0.0000067)
5-step
-0.932 (0.000029)
-0.873 (0.00045)
20-step
-0.931 (0.000031)
-0.873 (0.00045)
Table 13: The corrclation between image reconstruction even more strongly correlated with OOD behavior cloning success and the correlations are statistically significant the 0.01 level.
28

---

Published in Transactions on Machine Learning Research (06/2025)
assembly
button-press
coffee-pull
bin-picking
window-close
SAM
MOCO
DINO
CLIP
R3M
VIP
IBOT
Figure 19: Noo-reconstructive pretrained representations produced these reconstructions. Compared with there appears to be a higher level of fidelity when reconstructing the scene.
Figure 20
29

---

Published in Transactions on Machine Learning Research (06/2025)
assembly
button-press
coffee- pull
bin-picking
window-close
MVP
MAE
VC1
RN
Figure 20: Reconstructive pretrained representations produced these reconstructions. Compared with Figure there appears be lower level of fidelity when reconstructing the scene. In particular, the arm is reconstructed very well, and objects in the scene are poorly reconstructed, if at all.
30

---

Published in Transactions on Machine Learning Research (06/2025)
Cureent Scone Reconstruction
5-Step Sconc Reconstrucein
20 Stop Scone Rcconsbru cooon
Mer
R3M
MODO
6011
00:00
0.0
81000
0.111
5011
Figure 21: Pixel differences between reconstructions and the original scene rendering. Across the board, there is strong negalive correlation between reconstructions, future scene reconstructions, and the behavior cloning success rates.
OOD Performance
Nuisance Rosustness
Lighting Robustness
0.5
0.5
0.4
0.0
000 SHccsss
0.2
0.3
0.09
0.25
0.24
0.23
0.21
0.21
0.20
0.18
0:17
0OD Success
0.2
0.3
0.22
0.20
0.20
0.19
0.18
0.12
0.11
01
911
8.-
0.0
MOCO CLP
VC1
Frioddr
Lighting Nuisance Object Robustness
Average of OOD Sconarios
0,50
0.31
000 Succeso
0.25
0.21
0.25
0,23
0.23
0,23
0.15
000
0.30
0.12
0.13
SAM IB0T
VPP
MOCO DIO
C.IP
R3M
VC1
[R) MAT
MVP
SAM ROT MOCO DIP
CLPP
V1P
R3M
VC1
MAP
Encoder
Figure 22: Success rates accorss OOD performance is aligned with in distribution performance. Across the board, all encoders perform worse the OOD setting, but stronger encoders the ID case still perform better in the OOD setting.
31

---

Published in Transactions on Machine Learning Research (06/2025)
318
Nanan 681
635
Figure 23: Keypoints correlate with OOD success rates
BP
Ura
76
22%
22
2.2
015
016
636
355
Merskuzz dectege had batt
Memrkkcod Oitoreoe Recorstratte
Sereatzed Pyerage Noe Oftreecc for Scene Rearsituiin
Figure 24: Reconstruction quality is strongly correlated with OOD success rates
Edge Reconstruction Loss
Edge Structural Similarity
SAM
3.90
TNN
MAE
0.25
VP
Aveiage 8
BOT
ORIEN
9.20
Spearman: 13.31
Spearmart 0.95
3200
3400
3600
3800
0.66
0.70
Edge ReconstrucionLes
Similanity
Figure 25: Edge reconstruction quality and structural similarity is strongly correlated with OOD success rales
32

---

Published in Transactions on Machine Learning Research (06/2025)
Attention versus MLP For SAM
Success Rate by Environment and Architecture
Attn
Success
4.
aiembly
perppicking
butten yoees
coffee-
Baltum
taucal
hammvrr
pickplace
plat-stide
sudhoall
Rocerr
wirddw-lloa
window-sppe opont
Environment
Figure 26: Attention based policy and MLP based policies perform very similarly for SAM
Policy Architecture
Success %
Standard Deviation
MLP Attention
0.604
0.116
0.526
0.093
Table 14: The difference between MLP and Attention success rates within standard deviation of each other. This suggests the differences in success rate of MLP and Attention based policies are modest.
33


Reviews:
[{"review_id": "Uun0If2vwE", "reviewer": "Reviewer_sUmQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Comprehensive benchmarking: The study covers 21 tasks, including nudging and grasping, using both BC and offline RL, offering broad evidence of the encoders’ generality and robustness.\n- Actionable representation diagnostics: The authors propose easy-to-compute metrics (keypoint prediction, edge reconstruction) that strongly correlate with downstream policy success.\n- Clarification on pre-training objectives: By reframing ‘supervised vs. self-supervised’ into ‘discriminative vs. reconstructive,’ the authors reconcile prior contradictory results in the literature.\n\nWeaknesses:\n- Mostly simulation-based: While the tasks are diverse, validating on real hardware or under stronger domain randomization would further confirm the encoder generalization claims.\n- Lack of causal experiments: Although correlations are strong, more direct interventions (e.g., masking edges or keypoints) would help validate whether these features truly drive downstream success.\n- No new learning algorithms introduced: The main innovation is empirical rather than algorithmic, which may limit the appeal for readers looking for novel RL or representation learning methods.\n- Possible confusion between objective and architecture: Both reconstruction-based and discriminative methods use a mix of ResNet and ViT backbones, making it unclear if the observed performance differences are due to the pre-training objective or the network architecture.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a thorough empirical study of nine pre-trained visual encoders across 21 diverse robotic manipulation tasks, splitting them into nudging versus grasping categories. Through both behavior cloning (BC) and offline reinforcement learning (RL), the authors uncover strong correlations between successful downstream performance and the ability of encoders to preserve keypoints and edges in image reconstructions. A key insight is that pre-training objectives that are discriminative (supervised or contrastive) generally yield better manipulator performance than purely reconstructive strategies. The work also highlights how multi-modal action distributions (especially in grasping tasks) can affect BC vs. offline RL outcomes. Overall, the paper offers valuable practical guidance on selecting and probing pre-trained encoders for manipulation tasks, shedding light on how task-relevant structural features in representations drive success.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical:\n- Isolate backbone effects: Evaluate at least one reconstructive method on a ResNet and one discriminative method on a ViT to confirm whether performance differences are primarily tied to the objective or the architecture.\n- Strengthen causal analysis: Conduct studies (e.g., artificially mask out edges or keypoints) or domain-randomize intensively to validate that these features are crucial for success and to mitigate reliance on correlated confounds.\n\nNot critical but recommended:\n- Real-world or domain-randomized validation: Evaluate how these encoders handle real-hardware noise, lighting changes, or simulated perturbations to support robust generalization claims.\n- Detailed discussion of multi-modal behavior: Provide further analysis of how each encoder handles varying degrees of multi-modality in tasks (especially grasping), including more explicit comparisons of BC vs. offline RL outcomes.\n- Practical deployment insights: Include memory usage, inference speed, or resource overhead for each encoder to guide researchers who wish to deploy these representations at scale.", "broader_impact_concerns": "No obvious ethical or safety concerns arise.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GFZxkviaEV", "reviewer": "Reviewer_PDDW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths:\n- The paper perform extensive experiments across 21 diverse manipulation tasks, ensuring broad applicability and robustness of their findings.\n\n- The paper identifies easily measurable diagnostic metrics (edge reconstruction quality and keypoint prediction accuracy) that strongly correlate with downstream task performance. This contribution has potential implications for efficiently selecting visual encoders in robotic applications.\n\n# Weaknesses:\n- The paper lacks detailed exploration into alternative auxiliary tasks or hybrid training approaches. Given the significance of the findings, an exploration of different pre-training combinations or regularizations would provide valuable insights.\n\n- The paper primarily utilizes datasets with well-defined, structured environments. There is limited consideration of unstructured or more visually complex manipulation scenarios, which could challenge the identified correlations.\n\n- The paper heavily relies on empirical observations without providing deep theoretical analysis or insights into why edge and keypoint reconstruction correlate with manipulation success.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission investigates the effectiveness of various pre-trained visual foundation models specifically for robotic manipulation tasks, providing a evaluation across 21 diverse manipulation scenarios. The authors identify that a visual encoder's ability to reconstruct edges and accurately predict keypoints strongly correlates with downstream manipulation task success. The authors also introduce easy-to-compute diagnostic metrics (edge reconstruction and keypoint prediction accuracy) to predict the suitability of visual encoders for robotic tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Conduct OOD evaluation experiments to test the robustness and generalizability of the proposed diagnostic measures.\n\n- Perform extensive ablation studies to isolate and clarify which aspects of edge and keypoint reconstruction contribute most to the enhanced manipulation performance.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8P2mXY4lAo", "reviewer": "Reviewer_HXHd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n* Paper is clear and easy-to-follow.\n* Many recent vision encoders are benchmark, indicating the great effort of the authors.\n* In-depth analysis.\n* Proposed metrics are intuitive and make a lot of sense for assessing a representation’s quality.\n* Detailed appendix!\n\n**Weaknesses**:\n* Policy architecture: the choice of policy architecture might affect the downstream performance, and should be determined according to the representation structure (https://arxiv.org/abs/2203.05960 , https://arxiv.org/abs/2404.01220v1 ). For “holistic” representations (i.e., a single-vector representation for the entire image), an MLP might be a reasonable choice, but for more structured representations (e.g., tokens or object-centric representations), flattening the entire representation and using an MLP might not be the right choice (other architectures like deep sets, Graph Neural Networks or Transformers might be more reasonable, see above papers). “Don’t judge a fish by its ability to climb a tree” - flattening an entity/token-based representation and applying an MLP might not be a fair comparison to produce conclusions regarding the representation capabilities.\n* Benchmark: simple environments and simple tasks. There are more challenging multi-object manipulation tasks (https://arxiv.org/abs/2410.20092 , https://arxiv.org/abs/2404.01220v1 ).\n* Training data: it is worth noting that most of the pre-trained encoders are trained on real-world, natural, image data, where the pixel distribution is very different from the simulated pixels. As there are no real-world experiments, I would expect some discussion on the possible distribution shift (e.g., SAM might miss out-of-distribution objects).\n\n**Minor**:\n* I suggest removing the word “Thoughts” from the title of the paper as I find it a bit unappealing (but it is just my opinion).\n* “Keypoints” and “Key points” appear interchangeably, I recommend using a single format (e.g. Abstract and page 2 paragraph 3).\n* Broken Figure reference: Section 5.2, second paragraph, reference “Figure ??”, which I believe should be Figure 7. \n* Figure 6: it is referenced nowhere in text and also requires better explanation, the caption is not clear w.r.t what is shown.\n* Figure 7: it only shows the correlation with reconstruction and not with keypoint prediction. Did I miss something?\n* Broken Figure reference: page 10, first paragraph: “Figure ?? and 9”.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work evaluates several pre-trained vision models as feature extractors for offline robotic learning, specifically, behavioral cloning and offline RL. The main takeaway from the evaluation is that vision models that maintain important features, such as edges, in their representations and can produce meaningful keypoints, are more effective for downstream manipulation tasks. In addition, the authors propose evaluation metrics that can assist in predicting a vision encoder performance for downstream manipulation tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* The conclusion that a good vision representation that maintains edges and has the ability to produce meaningful keypoints is very intuitive, and several previous works have used this kind of representation for decision-making tasks. Crucially, in many cases, this representation is self-supervised and object-centric. I highly recommend adding a discussion or referring to this line of works as they strengthen the main arguments in this paper. \nSee: https://arxiv.org/abs/2011.14381 , https://arxiv.org/abs/2404.01220v1  , https://arxiv.org/abs/2412.18907 , https://arxiv.org/abs/2410.23254 , https://arxiv.org/abs/2403.19578 , https://arxiv.org/abs/2106.07995 \n* Question: while SAM is not “reconstruction-based” per-se, it is usually referred to as object-centric, as it provides a (meaningful) decomposition of the scene into objects. Many object-centric vision models are trained with a reconstruction objective (see the above bullet). So, in that sense, would the “reconstruction-based representations are worse” conclusion still hold? I am not sure it is fair to conclude that “reconstruction objective is bad”, but perhaps a different kind of distinction between representations is required (one example is “holistic”/single-vector versus object-centric). Page 9, second paragraph, also emphasizes that the focus should rather be on object-centric representations (like SAM) that provide better reconstructions.\n* Issues: my main two issues with this work are - (1) the choice of proper architecture to process the representations might make the conclusions invalid; (2) I think the conclusion about “reconstruction-based representations” might not necessarily be true, and it seems that the conclusion should be shaped around object-centric/non-object-centric according to the evaluated encoders.\n* Question: if the conclusion is that “predicting keypoints can indicate the encoder’s performance”, why not use keypoints as the input representation? (this is actually done in previous works such as https://arxiv.org/abs/2106.07995 and https://arxiv.org/abs/2404.01220v1 but not evaluated here)\n* Open-source code: do the authors plan to provide an open-source implementation of the benchmark and metrics for further research?\n* See “Weaknesses” and “Minor” above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers thought positively of the the revised version of the paper, and recommended acceptance.
Decision: Accept (poster)