Capturing Visual Environment Structure Correlates with Control Performance
Authors: Jiahua Dong
Yunze Man
Pavel Tokmakov
Yu-Xiong Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Robot Learning
Computer Vision
Diffusion Policy
Primary area: applications to robotics, autonomy, planning
TL;DR: We tackle the problem of visual representation selection for policy learning by proposing a practical and scalable proxy task: decoding the full environment states from visual observations.

Abstract: The choice of visual representation is key to scaling generalist robot policies. However, direct evaluation via policy rollouts is expensive, even in simulation. Existing proxy metrics focus on the representation's capacity to capture narrow aspects of the visual world, like object shape, limiting generalization across environments. In this paper, we take an analytical perspective: we probe pretrained visual encoders by measuring how well they support decoding of environment state—including geometry, object structure, and physical attributes—from images. Leveraging simulation environments with access to ground-truth state, we show that this probing accuracy strongly correlates with downstream policy performance across diverse environments and learning settings, significantly outperforming prior metrics. Our study provides insight into the representational properties that support generalizable manipulation, suggesting that learning to encode full environment state is a promising objective for visual representations for control.

Published as a conference paper at ICLR 2026
CAPTURING VISUAL ENVIRONMENT STRUCTURE COR- RELATES WITH CONTROL PERFORMANCE
Jiahua Dong
Yunze Man
University of Illinois Urbana- Champaign j Jiahuad2@ ilinoi eddu
University of Illinois Chammaainn yunzem2@1 wllinnnis s. edu
Pavel Tokmakovi
Yu-Xiong Wang
Toyota Research Institute
University of Illinois Urbana-Chan -hhaaa-hhammaiin
pavel. Tevee.ttoeen tww.akovvtr...gle.o... global
yxweillinois.es
modern simulator gives you all world states free.
MMRY Correlation Error [Lower the Better
0.12
0.115
Pepth, Lighting, plane. Sepmentation, etc.
0.101
0000
Env States
0.08
Joint Angles, pose
0,066
0.060
C
Robot States
0.04
Pose, Shape, Maderial, Semantic
0.023
GURU
CRERIT
0.00
Few-diut
Action ASE
Imageelet
Shape Blos
Sopmentation
Shate Predection Courl
Degtth
Object States
(a) Rich World States (Object, Robot, Environment)
(b) Full-state Prediction is the Best Proxy
Simulation environments provide access to full world state labels for free (left), enabling our proxy task state prediction from visual inputs. This proxy strongly correlates with downstream policy success across environments and architectures (right, results for SimplerEnv environment).
Figure 1:
ABSTRACT
The choice of visual representation is key to scaling generalist robot policics. How- ever, direct evaluation via policy rollouts is expensive, even in simulation. Existing proxy metrics focus on the representation' 's capacity to capture narrow aspects the visual world, like object shape, limiting generalization across environments. In this paper, we take an analytical perspective: we probe pretrained visual encoders by measuring how well they support decoding of environment state including geometry, object structure, and physical attributes from images. Leveraging sim- ulation environments with access to ground-trui ttrttt-thhhhh state, show that this probing accuracy strongly correlates with downstream policy performance across diverse environments and learning settings, significantly outperforming prior metrics and enabling efficient representation selection. More broadly, study provides in- sight into the representational properties that support generalizable manipulation, suggesting that learning encode the latent physical state of the environment is promising objective for control. Code and models are available at our project page.
INTRODUCTION
Although robotics has seen significant progress in developing generalist real-world manipulation
policies in recent years (Black et al., 2025; Brohan et al., 2022b; Ghosh et al., 2024), its pace of
Equal advising.

---

Published as a conference paper at ICLR 2026
advancement remains relatively slow. This is largely due to the prohibitively high cost of policy evaluations in the real world. Most works use simulation environments for model development (Li et al., 2024), but simulation policy rollouts remain significantly more expensive than standard metries used in vision or natural language processing (NLP). One particularly expensive and influential component of this process is the choice of visual representation, which modern generalist policies heavily rely on (Majumdar et al., 2023a). Recent work (Burns et al., 2024) has explored image-level proxy metrics, such as object segmentation accuracy, as a way to approximate representation quality without requiring full rollouts, but these findings remain limited in scope and generality.
More broadly, the question of what makes a visual representation useful for manipulation has received a lot of attention recently (Burns et al., 2024: Chen et al., 2021; Majumdar et al., 2023a). Among those, a growing body of work has explored reconstruction-bant -hased pretraining objectives (Hafner et al., 2020; He et al., 2022a) as visual representations for robotics (Radosavovic et al., 2022; Shridhar et al., 2023; Zhang et al., 2025b). These approaches are rooted in the idea that learning to capture the sparse state of the environment from dense visual observations leads to representations that are more aligned with the needs of control. Very recently, Qi et al. (2025) provide further support of this hypothesis by showing representations trained with behavior cloning cluster around task-relevant states.
In this paper, we study visual representations for control through the lens of their capacity to recover the full environment state from images. Inspired by recent advances in structured visual representation learning for robotics (Qi et al., 2025; Radosavovic et al., 2022; Zhang et al., 2025b), we hypothesize that representation's 's capacity decode the underlying environment state including its geometry, object structure, and physical attributes - is strong indicator of its utility for downstream control. However, obtaining such state labels for real scenes a major challenge. Instead, we capitalize on simulation environments (Todoroy et al., 2012; Yu et al., 2020), which automatically provide access to the full ground truth state of the world (see Figure 1, left).
Specifically, we propose a uniform and compact representation of arbitrary environment states together with a lightweight state prediction head that can be applied to any visual backbone in Section 3. We select nine pre-trained representations, including both those specifically designed for robotics and general purpose ones (see Section for details), for environment state regression and for policy learning and measure the correlation between the two tasks. Our study includes three simulation environments: general-purpose World (Yu et al., 2020), RoboCasa (Nasiriany et al., 2024), which features significant distribution shift between train and test samples, as well as the environment designed by Li et al. (2024) specifically to match the real world.
Across all the environments, state prediction accuracy demonstrates strong correlation with the success rate of the policies, as shown in Figure 3. It significantly outperforms all existing proxy metrics (see Figure 1), including the ones proposed by Burns et al. (2024), generalizes better across learning settings, and is substantially less computationally demanding (see Section 5). Crucially, we demonstrate that conclusions drawn from simulation reliably transfer analogous real-world tasks, establishing our proxy as a practical tool for representation selection. Finally, our analysis reveals that representational demands vary across environments, and points learning to encode the full state of the world as a promising direction for improving visual representations in control.
RELATED WORK
Representation learning for manipulation. Early methods learned to encode the image end- to-end with the policy via self-supervised or contrastive objectives applied robot observations. CURL (Laskin et al., 2020) and Act (Zhang et al., 2022) leverage contrastive losses to align visual features with control signals, demonstrating improved sample efficiency in imitation and reinforce- ment learning. Scaling up to Internet-scale egocentric video, R3M (Nair et al., 2022) trains on diverse human first-person footage to distill a frozen backbone that transfers effectively to robotic imitation tasks. Masked reconstruction paradigms like MVP (Radosavovic et al., 2022) and RPT (Radosavovic et al., 2023) further show that reconstructing masked patches of robot aideos-augmented with proprioceptive cues cues-yields task-agnostic representations that generalize across simulators and real platforms. Recently, VC-1 benchmarks dozens of vision backbones on a suite of simulated manipula- tion tasks, revealing that single off- she- model dominates across all challenges (Majumdar et al., 2023a). Most recently, (Burns et al., 2024: Gupta et al., 2024) demonstrate that Internet-scale pretraining often outperforms robotics- objectives in -iittriiittiiutti evaluations.
2

---

Published as a conference paper at ICLR 2026
Robust visual representation learning. In recent years, large- scale visual representation learning approaches have shown great robustness and generalizability in the wild (Akbari et al., 2021; Assran et al., 2023; Bao et al., 2022; Bardes et al., 2023; 2024; Chen & He, 2021; Gidaris et al., 2018; Girdhar et al., 2023; Grill et al., 2020; Joulin et al., 2016; LeCun, 2022; Li et al., 2022a; Mahajan al., 2018; Pathak et al., 2016; Tong et al., 2022; Wang al., 2023b; 2022; Xu et al,, 2021; Zellers et al., 2022; Zhang et al., 2016; Zhou et al., 2022). Supervised ResNets set the early standard for feature transfer (He al., 2016), followed by contrastive self-supervision (Chen et al., 2020; He et al., 2020)) and masked auto-encoder (He et al., 2022b). The Vision Transformer (ViT) introduces patch-based tokenization for scalable transformer encoders (Dosovitskiy et al., 2021), and subsequent self-distillation in DINOv2 yields even more generalizable features (Oquab et al., 2024). Multimodal alignment also demonstrates strong zero-shot capabilities by grounding images natural language (Li et al., 2022a;b; 2023; Radford et al., 2021). Most recently, Internet-scale generative models for image (Peebles & Xie, 2023; Rombach et al., 2022) and video (Blattmann et al., 2023; Wang et al., 2023a) domains have emerged as powerful representations of the visual world.
While these representations excel in broad-domain tasks, systematic studies of their selection for manipulation remain scant. Preliminary work examines domain shifts between Internet images and egocentric robot views (Zhan et al., 2023), but lacks actionable guidelines for choosing among dozens of models. Burns et al. (2024) propose to use segmentation ability of model as a proxy for its downstream manipulation performance, but their findings have been limited in scope and generality. Our state prediction objective addresses this gap by providing an efficient ranking of pre-trained visual representations that shows strong correlation with downstream policy performance across a wide variety of backbones, environments and learning settings.
Analysis of visual representation for control. A parallel line of work studies why certain visual encoders help manipulation. Qi al. (2025) uncovers that visual representation naturally cluster around task relevant environment states when trained with behavior cloning objectives. Similar insights occur (Jiang et al., 2025) which presents "manipulation-centte" much an embedding attends to object-contact regions. Multitask architectures like PerAct further highlight the benefit of strong visual features for manipulation tasks (Shridhar et al., 2023). Our work extends this line of research by providing a principled, state-space-level analysis of visual representations and showing which aspects of environment state drive manipulation performance.
analyzes regarding how
Evaluation of policy learning approaches. Policy-learning benchmarks span both real-world and simulation environments. Real-world benchmarks like BridgeData V2 (Walke et al., 2023), RT- 1/2 (Brohan al., 2022a; 2023) and DROID (Khazatsky et al., 2024) collect thousands of demonstra- tion trajectories across dozens of tabletop scenes on few manipulator embodiments, offering a public yardstick for behavior cloning and reinforcement learning (RL.. However, real-world data collection remains expensive and evaluations slow and challenging to reproduce. To address these limitations, simulation suites have seen great improvement in recent years, providing rapid and reproducible comparison at scale, making them the de- facto testbeds for algorithmic research (Makoviychuk et al., 2021: Yu et al., 2020; Zhang et al., 2025a; Zhu al., 2020).
Encouragingly, evidence suggests that carefully designed simulators can replace costly real- world trials during early development: SimplerEnv recreates RT-I-family policies in MuloCo (Todorov et al., 2012) and recovers their relative ranking across camera and robot variants, validating sim-to-real evaluation as a faithful proxy (Li et al., 2024). Our study builds on these insights, and further capitalizes on the fact that simulation environments provide automatics access to the full state of the environment, enabling direct evaluation of the environment-state information encoded in pretrained visual representations. This approach allows visual representations for control to be selected without expensive policy rollouts, either in the real world or in simulation.
WORLD STATE ENCODING AND PREDICTION
As proxy task for measuring the quality of visual backbones in policy learning, we regress a compact, simulator grounded state of the world from raw images, shown in Figure 2. High accuracy on this task indicates that the backbone has learned the geometric, material, and kinematic cues needed for manipulation. Below, we first describe our compact, universal state representation format and then detail the design of our state prediction head and provide the corresponding losses.
3

---

Published as a conference paper at ICLR 2026
Which visual representation benefits my robotics task?
Projection
Projected
Policy
Rollout
Trajectories
Features
Learning
Backbone Features
2D Encoder Pool
Pollcy Training: Expensive Time-consuming
ResNet
MAE
Visual Prompt
Select
CLIP
MoCov3
Feature Extraction
Strong Correlation
R3M
DINO<2
State Prediction as Proxy: Cheap Efficient
StableDif
More...
O
Projection
State Regression
Lighting
Joint
EC*
gubjaa
Visual Prompting
Rol-Sampling Projection
Shape
Material
Objees
Objjctt2
Objjctt
Figure 2: Our framework for efficient visual representation selection for control. We capitalize on the availability of ground truth world state information in simulators and propose universal, compact encoding of the states, together with a light-weight state prediction head (bottom). We demonstrate a strong correlation between our proxy objective and downstream policy performance (top).
Unified Low Dimensional States. Low-dimensional state information captures the true configuration of the environment with high fidelity: exact robot kinematics, object geometry, material type, and lighting parameters without perceptual noise. We adopt single, unified state representation that applies across all environments, so that regression eπrоr directly reflects visual representation' s capacity lask-agnostic way. We strive capture all the state information that provided by modern simulators, to allow for an out-of-the-bon applicability to new environments in the future. Specifically, our state representation comprises N₂ vectors: one environment-level vector for global states, and No object-level vectors for each object (shown in the lower part of Figure 2). This decomposition allows us to analyze per-object details (pose, shape, material) and overall scene and agent configuration (lighting, joint state, end-effector pose) in isolation.
For each object I,..., No in the scene, the object-level vector is defined as
abjj [Proee aposes Sshape: mmaa]] E 1334+33+
(1)
where Precs E R represents position and Lpeme E RR orientation (quaternion) of the object, Shape E R represent bounding-box of the object, and m E {0, 1M is one-hot vector over M materials.
The single scene-level vector defined as
8 env = [e, 9 p BR+N,++N*
where E N are lighting categories, o' E N are robot joint angles, and p E R Nee is end-effector pose. Concatenation of these vectors yields compact representation of the entire environment state
[h
obj.No, snn " D D N3
M) (1+ + Nee). (3)
Predicting these N ability to encode both object-centric and wede information. The details of the ground truth extraction for all state vectors from simulators are provided in the appendix.
+ 1 vectors from images provides a clear and quantitative metric of the backbone's
Visual-Promptee State Prediction. To provide efficient proxy metrics for representation selection, our approach outputs all the state vector dimensions in a single forward pass. However, predicting a specific object's s state in multi-object scene is inherently ambiguous unless the network is told where to look. Therefore, we design a cooul-p state prediction setting, where 2D bounding boxes of target objects are provided the model, so that the backbone can focus on relevant image regions.
Rol pooling on the backbone feature map. Given an input image x and its corresponding feature map 6(x)) H XCC, we perform region-of-interes. (Rol) average pooling inside each object bounding box bi = [21, 8/1 Z2, V2l:
Σ (a.u)Ed:
((x),, E CCC
b"
and map n² to obj, via single linear layer.
4

---

Published as a conference paper at ICLR 2026
Global encoding for environment-level factors. To capture scene-wide states (lighting, robot joints, and end-effector poses), we apply a global average pooling over all spatial locations via
W
0
1 HW
Σ
Σ
((F),,
C
Similarly, is projected to senv with linear layer. Both projections layers are optimized against the simulator ground truth, masking out the objects that are not visible in the current frame.
State Encoding Loss Function. Discrete states such as material type (M classes), lighting preset (L options), and object shape modeled as 3D box with each edge quantized into S bins are represented by one-hot vectors (cx))) E {0, 1)m*. These vectors are predicted via a softmax layer and trained with a cross-entropy loss.
Continuous states, including object position Ppose and rotation Opase: robot joint states N₂ and end- effector pose pf are standardized per dimension: 2, = de=Ks, where μi, o, are empirical mean and standard deviation from training data. The network then regresses these normalized values using an L2 loss. Continuous modeling used for these states they closely related to action planning.
Proxy Metric. To ensure broad coverage, we include all the dimensions of the state vector s in Equation 3. Specifically, for categorical states, compute classification accuracy, while for continuously states, we use the negative mean squared error (MSE) as the score, so that higher is better for all metries. After collecting scores for all models across all states, normalization per states across models. Finally, we compute the mean of the normalized scores across states obtain single evaluation score for each model. Let A denote the of states and M the set of models. The final proxy score for model m E . is defined as:
apply min-max
Tm.a minmeM Tino
Sm
(6)
A
max Tina minMeM Tmaa
aEA
where Tm,a is the raw score of model on state a.
4 EXPERIMENT PROTOCOL
In this work, we set out to analyze what makes visual representations effective for generalist ma- nipulation policies. Specifically, we evaluate the correlation between scores of various candidate objectives and the success rates of policies learned on top of wide variety of pre-trained visual representations. We validate our findings across three distinct simulation environments and further test whether conclusions transfer to the real world, where ground-truth state labels are unavailable. To this end, we show that state prediction accuracy measured in simulation reliably predicts backbone performance on real-world tasks. Next, describe the details of experiential setup.
Simulation environments. We evaluate our approach in three distinct simulation environments: MetaWorld 50 (Yu et al., 2020), RoboCasa (Nasiriany et al., 2024), and SimplerFnv (Li al., 2024). MetaWorld World 50 is a widely used benchmark for multitask robotic manipulation, offering a diverse suite of 50 object-centric challenges that rigorously test both generalization and dexterity. RoboCasa (Nasiriany et al., 2024) offers realistic household environments with everyday objects and furniture, consisting 24 atomic tasks. SimplerEnv is a lightweight and highly customizable platform that focuses on closely matching the simulation with real world setups from Google RT- 1/2 (Brohan al., 2022a; 2023) and Bridge (Walke et al., 2023). We adopt 10 different tasks from it, including 6 tasks for Google robots and 4 tasks for WidowX. Evaluating across diverse simulation domains helps ensure generalization over object categories, task complexity, and dynamics, reducing overfitting to any setting. For all environments, task. In Metaworld (Yu et al., 2020), we generate demonstrations using the provided expert policy. the official set of 50 human demonstrations per task. In SimplerEnv, collect
train the models with 50 demonstrations per
For RoboCasa, demonstrations using our own expert policy. The success rates are reported with 100 rollouts per task.
Real-world environments. To validate the sim-to-real transferability of our proxy, we reproduced two tasks from the WidowX benchmark (Li et al., 2024) on a physical Xarmó robot arm (UFACTORY) equipped with the UFactory Xarm Gripper. A RealSense D455 camera (Intel RealSense) mounted
5

---

Published as a conference paper at ICLR 2026
in the corner to provide visual input. The control frequency was set to 10 Hz with maximum end- effector speed of 0.1 m/s. Specifically, we reproduced the widowx lals__arrt____a Larrrttto___ttto ______________ pal_paatee and widowx stack__utt tasks and report their average success rate. To better match the original WidowX environment, we adjusted the camera pose and used a similar table covering (see Figure 6, left). We collected 100 demonstrations per task and report success rates over 100 rollouts.
Pretrained models. We include the following pretrained visual representations in our evaluation. Detailed description of checkpoint, training data, and implementation is in the Appendix Section D.
Supervised models. We evaluate conventional ImageNet-trained backbones, including ResNet-18 (He al., 2) and Vision Transformer (ViT-B) (Dosovitskiy al., 2021), as well as the vision encoder from CLIP model (Radford et al., 2021), which couples the same ViT architecture with contrastive language-image training objective.
Self-supervised models. For label-free representation learning we cover MoCo-v3 (momentum contrast for ViTs) (He al., 2020), MAE (masked auto-encoder) (He et al., 2022b), DINO v1/v2 (self- delff-dstillltiinn without labels) (Caron et al., 2021; Oquab et al., 2024). All three are trained on Internet images and serve as domain-agnostt baselines without robotics-specific biases.
(ianpuulttion-spcijfe models. To probe representations explicitly optimized for robotic control, we separately consider R3M (Nair et al., 2022), which learns reward-predictivv features from egocentric manipulation videos. Evaluating this encoder allows us to quantify the added value of task-aligned pre-training relative to more generic visual representations.
Generative models. Finally, explore the potential of generative models that have demonstrated remarkable capabilities in visual synthesis. Stable Diffusion (SD) (Rombach et al., 2022) learns to generate images from noise guided by text prompts, modeling diverse visual scenes that may provide useful physical priors.
Proxy objectives. We group the commonly evaluated proxy objectives into two categories: agiioonment-agnosiic tasks and environment-specifee tasks. The anvironment-agnoste ones in- clude: ImageNer Recognition Linear probing on ImageNet (Deng et al., 2009), and Shape Bias Measuring shape bias on
(Geirhos et al., 2018). The environment--pecific tasks include: Few-Shot Learning- five demonstrations per task for efficient training and evaluation (Burns et al.. 2024); Action MSE action prediction mean squared eπror loss; Segmentation Semantic segmentation performance (Burns al., 2024; Man et al., 2024); Depth Estimation Monocular depth estimation (Banani et al., 2024; Burns et al., 2024). A thorough description of these baseline proxies are detailed in the Appendix Section B.
Policy learning. We adopt multi-task diffusion policy conditioned on text prompts for policy learning from the original design (Chi et al., 2023; Ze et al., 2024), but also evaluate robustmess of our findings to different behavior cloning algorithms in Section A.2 in the appendix. A CLIP text encoder (Radford et al., 2021) embeds each instruction and fuses with visual features. Action generated via standard diffusion forward/reverse passes with classifier-fre guidance, enabling diverse task handling without retraining task-specific heads. The experiments in the main paper primarily utilize frozen visual representations; however, also report correlation results with finetuned backbones for completeness. Detailed training setup and implementation are provided in Section E.
Evaluation protocol. To provide an objective way of assessing whether proxy reliably stand in for expensive policy training when comparing visual encoders, our evaluation protocol closely follows the SimplerEnv benchmark (Li et al., 2024). We use the Mean Maximum Rank Violation (MMRV) (Li et al., 2024) metric to measure the ranking correlation between the policy success rate R2 and the state prediction proxy scores S, of a set of visual models. Specifically, we first compute the pairwise violation
Rank Violation RR R,I 1 [[SS < S2) + (R, < R,)
(7)
where i.j E M are different visual models. Then, we aggregate each policy's 's worst-case error:
N
MMRV =
1 N
Σ
max Rank Vaakaiiolaion.y. 19SSN
(8)
In addition, we report the complementary Pearson's correlation (Kadian al., 2020; Pearson, 1895) metric r r(R, S) to capture overall linear agreement. Low MMRV and high r indicate strong ranking fidelity.

---

Published as a conference paper at ICLR 2026
Metaworld
RoboCasa
0.854
Averaged Success Rate
0..650
0.930
0.603
0.605
0.600
0.590
0.292
0.205
Aparaged
0.280
0.278
6,280
0.271
0.450
0.432
0.245
0.400
0.260
Rocnet-N
R3M
WINN
MaGoug
(ineV)
DtnoV2
MAE
CLIP
50
Recnet-1
R3M
WIN
MoGoed
Dino1y
D2sn
MAE
CLP
5D
SimplerEnv (Google Robot)
SimplerEnv (WidowX + Bridge)
0.383
Averaged Suncess Rate
0.250
0.225
Aperaged Sucoess
0.350
0.300
0.282
0.300
0.323
0.200
0.150
0.120
0.155
0.107
0.237
0.246
0.216
0.050
0.200
Recadr-N IN
R3M
VIEN
McGoo3
DanoV)
DtnoV2
MAE
CLIP
Recser-it EN
HBM
VIEN
MaGawd
DineVy)
DiseV2
MAE
CLIP
Figure 3: Evaluation of diverse set of visual backhones in different robotic simulation environments. From top left bottom right: The visually simple Metaworld (a) favors more traditional ImageNet--retraiigg representations. RohoCasa (b) requires precise ohjet localization thus favors with strong object priors, whereas realistic SimplerEnv environments (c. d) benefit from real world robot data pre-trate pre-training.
EXPERIMENTAL RESULTS
Assessment of pre-trained models. Pre-trained visual representations promise to boost policy perfor- mance in robotics, where on-policy training data is scarce. Yet in practice their impact is influenced by factors such as domain shift, network architecture, and training setup. In this section, we present a systematic comparison of diverse pretrained vision encoders across multiple benchmarks, asking whether their relative performance rankings are consistent or inherently enviromment-dependerr
In Figure 3, we evaluate visual backbones across four distinct domains. In Metaworld, which features con-photorealistic observations, large-scale pretrained models such as CLIP and DINOv2 underperform. This is likely due to significant distribution shift between high-resolution Internet images used in training these models and cartoonish frames in Metaworld. pretrained models exhibit more stable performance. In RoboCasa, the combination of high-fidelin rendering and diverse scene layouts favors models like DINOv1 and DINOv2, which are particularly effective at object localization. In SimplerEnv, which is designed mimic real-world indoor scenes, R3M achieves strong performance, benefiting from its pretraining on real robot observations. These experiments show that there is single optimal representation for robot manipulation. Thus, exploring which factors influence the effectiveness of visual representations is an important problem.
contrast, ImageNet-
Proxy evaluation in simulation. We now evaluate of state regression objective a proxy for predicting the success rate of downstream policies. To this end, we first plot the correlation between the proxy score and the success rate of the corresponding policy in Figure 4. We observe that, across all 4 environments, the success in predicting the unified environment state representation proposed in Section exhibits strong correlation with the effectiveness of the polices, as indicated by low MMRV and high Pearson correlation (r) scores. The mistakes largely come from some backbones achieving virtually equal success rates (e.g. for ViT, DINOv1 and StableDiffusion in the bottom right plot), making differentiation extremely challenging.
To put these results in context, we compare against several proxy objectives and approaches for representation selection proposed in the past in Table 1, including some requiring full policy evaluation (Few-Shot) and full policy training (Action MSE). Remarkably, our policy-free approach shows stronger correlation with the success rate of the policy compared even to these privileged baselines in all the environments. Among the purely visual proxies, the segmentation metric proposed in (Burns et al., 2024) shows strong performance the RoboCasa benchmark, which requires accurate object localization, but significantly underperforms in all the other environments. This observation validates claim that focusing individual aspects of visual reasoning limits generalization.
7

---

Published as a conference paper at ICLR 2026
Metaworld
RoboCasa
MMRV 0.037
0.691
MMRV 0.010
0.760
0.65
0.32
Polloy Success Rate
0.31
0.60
0.55
4
0.27
A
0.26
0.2
0.3
0.4
0.5
0.6
0.7
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Sm
Sm
SimplerEnv (Google Robot)
SimplerEnv (WidowX Bridge) MMRV 0.069
MMRV 0,023
0.871
0.688
0.36
Pollay Success Rate
0.34
0.30
88
0.28
0.26
0.24
0.22
0.35
0.40
0.45
0.50
0.55
0.60
0.65
0.70
0.75
0.1
0.2
0.3
0.4
0.5
0.6
5m
5m
ResNet-IN 0
McCov3
DINOV2
R3M * DINOv1 A SD
MT-IN
CLIP
MAE
Figure 4:
Correlation between state prediction score and success rate policies. Our proposed proxy
task shows strong correlation and MMRV score all four different environments.
Few-Shot Action MSE
ImageNet Shape Bias Segmentation
Depth
Ours
MMRV +
Metaworld
0.069
0.081
0.070
0.175
0.167
0.109
0.037
RoboCasa
0.028
0.041
0.245
0,041
0,023
0.035
0.010
SimplerEnv (G)
0.060
0.108
0.123
0.066
0.115
0.101
0.023
SimplerEnv (W)
0.113
0.124
0.124
0.090
0.113
0.137
0.069
Average
0.068
0.089
0.141
0.093
0.105
0.096
0.035
Pearson r
Metaworld
0.650
0.016
0.587
-0.905
-0.268
0.089
0.691
RoboCasa
0.243
-0.340
0.242
-0.291
0.527
0.083
0.760
SimplerEnv (G)
0.624
-0.465
-0.660
0.633
-0.475
0.027
0.871
SimplerEnv (W)
-0.128
-0.385
-0.250
0.286
0.048
-0.483
0.688
Average
0.347
-0.294
-0.020
-0.069
-0.042
-0.071
0.753
Table I: Comparison of visual backbone selection proxies on four different simulation environments using MMRV and Pearson correlation. Our state prediction objectives show top performance across all the environments, outperforming even the methods that have direct access to policy (Few-Shot and Action MSE).
Finally, although freezing the visual backbone is more resource-efficient fine-tuning is a more typical approach in robot learning. To analyze the full potential of various backbones, we fine-tuned them in MetaWorld (Yu et al., 2020) and report the results in Figure 5. Our method still achieves high correlation for fine-tuned models. More details are provided in Appendix Section A.1
Proxy evaluation in the real world. Due to the high cost of policy evaluation in the real world, we only report results with representative subset of the visual backbones used the simulation experiments (ResNet-IN (He et al., 2016), R3M (Nair et al., 2022), VIT-IN (Dosovitskiy 2021), CLIP (Radford et al., 2021) and MAE (He et al., 2022b)). Figure 6 (right) shows the correlation between their real-world success rates and state prediction accuracy in the simulated version of WidowX. We find a strong correlation, with MMRV and r scores comparable to those observed in simulation-only experiments (Figure 4, bottom right), despite the significant domain gap.
al.,
8

---

Published as a conference paper at ICLR 2026
Finetuned Model Success Rate 5m
MMRV 0.001
0.870
Diffusion Policy Success Rate
0.76
0.74
0.72
0.70
0.68
0.65
0.70
0.75
0.80
0.85
DINOv2
*
Figure 5: Correlation for fine-tuned backbones in Meta World. Our proxy demonstrates strong
correlation with policy performance when visual backbones are fine-tuned with the policy objective.
Ppose
Apose
Sshape
Mmnat
q
pce
1
Full (Sm)
Metaworld
0.055
0.089
0.032
0.153
0.038
0.069
0.073
0.037
RoboCasa
0,014
0,017
0.011
0.024
0.032
0.027
0.031
0,010
SimplerEnv (G)
0.045
0.100
0.082
0.120
0.075
0.038
0.084
0.023
SimplerEnv (W)
0.126
0.032
0.126
0.107
0.048
0.048
0.095
0.069
Table 2: Predictive power of individual state dimension. We report MMRV for individual state
components (as defined in Section 3) across the simulation environments. They place different demands on representations, but our full approach provides versatile proxy for backbone selection.
These results demonstrate that simulation-basee state prediction is an effective proxy for real-world policy performance, reinforcing our central message that the capacity to capture state a defining characteristic of visual representations that generalize well to control.
Analysis of environment-speeific demands. Our analysis above shows that the capacity of backbone to encode the fill state of the world is mnvirommentagnnose proxy for its utility for policy learning. However, from Figure we can also observe that different environments present different representational requirements. Here study whether dissecting the effect of the individual state components can illuminate the unique demands of each domain. To this end, we conduct fine- grained analysis of the full state vector by measuring the predictive power of each individual subset of dimension in Table 2. We observe that different environments indeed present different demands for visual representations, with Metaworld and RoboCasa requiring accurate object localization in 2D) (Sshape) and SimplerEnv environments benefiting the most from accurate 3D end effector pose (p*") and 3D object orientation estimation (gpose). ). Notably, while our approach allows obtain insights into the individual demands of each manipulation environment, simply regressing the entire state
Real Robot Success Rate MMRV 3,042 ru 0.711
0.28
Dfflaaee buicy
0.24
0.20
0.30
0.40
0.50
2.00
spoon on towel
stack sube
Reshict-Ik
R3M
CUP
MAE
VTN
Figure 6: Real-world experiment setting and results. State prediction accuracy in the simulated WidowX environment strongly correlates with real-world success rates (averaged over two tasks), supporting the practical applicability of our proxy and reinforcing the link between state representa- tions and control performance.

---

Published as a conference paper at ICLR 2026
ViT-IN
Mocov3
MAE
CLIP
DINOv2
Baseline
0.683
0.671
0.648
0.765
0.767
State prediction training
0.740
0.743
0.712
0.801
0.795
Table 3: State prediction for representation learning. Joint fine-tuning with state prediction consis-
tently improves downstream policy success rates across multiple visual backbones on Meta World.
serves as an effective, versatile proxy, as indicated by the strong performance of our full method in the right most column. We provide further analysis of the effect of individual state dimensions on our method's 's performance in Section A.6 in the Appendix.
State prediction as representation learning. Beyond its utility as a proxy for model selection, fine-tuning vision encoders with our state prediction objective delivers substantial gains. We validate this by jointly fine-tuning five visual backbones with both policy learning and state regression on MetaWorld (Yu et al., 2020). Specifically, we conduct experiments on ViT-IN (Dosovitskiy al., 2021), V3 (He et al., 2020), MAE (He al., 2022b), CLIP (Radford et al., 2021) and DINOv2 (Oquab et al., 2024), using objective defined in Section 3 alongside the diffusion policy training loss.
During joint training, we attach a state-prediction head to the visual backbone and optimize a joint loss:
Cjoint = Cpoticy + À Satates
where Epolicy is the diffusion policy loss and Sstate is the regression loss. Gradients from both terms are hackpropagated through the encoder, while the policy and state heads are updated only by their respective losses. This setup encourages the backbone retain both state-relevant information and action prediction ability.
As shown in Table 3, explicitly predicting environment state improves downstream policy perfor- mance, pointing toward predictive world modeling as a promising direction for advancing visual representation learning in robotics. We provide further analysis of our auxiliary objective in Sec- tion A.7 in the Appendix.
Computational complexity. Finally, we evaluate the total computational cost (training and eval- uation) of each policy learning and proxy task, using single A A100 GPU. The results achieved in Metaworld are reported in Table 4, and demonstrate that our state prediction not only more effective as a proxy for the visual representation selection, but also more computationally efficient than all other alternatives. The comparison of results between the ResNet and ViT models also demonstrates that the computational advantages are more significant for larger models. Further details are provided in the Appendix Section C.
Few-Shot
Action MSE ImageNet Shape Bias
Segmentation
Depth
Ours
IesNet-IN ViT-IN
78 min
45 min
11 min
23 min
10 min
10 min
4 min
113 min
72 min
46 min
92 min
27 min
27 min
12 min
Table 4: Analysis of computational efficiency. We report the time for joint training and evaluation of all representation selection approaches (As defined in Section 4). Our state prediction proxy objective is not only more effective in reflecting policy learning performance, but also more computationally efficient than all the alternatives. Results are computed with a single A100 GPU.
6 CONCLUSION
In this work, we analyze what makes visual representations useful for control by leveraging simulation enviromments with access to ground-truth state. Using state prediction error proxy, we showed that representations capacity of capturing environment state correlates much more strongly with downstream policy success than existing baselines. We have also demonstrated that our conclusions hold in the real world. These results suggest simple rule of thumb for visual representation learning in robotics: the better a model understands the world, the better it can act it.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported in part by the Toyota Research Institute, NSF under Grants 2106825 and 2519216, the DARPA Young Faculty Award, the ONR Grant N00014-26- 1-2099,, and the NIFA Award 2020-67021-32777 This work used computational resources, including the NCSA Delta and DeltaAI supercomputers through allocations CIS230012, CIS230013, CIS240387, and CIS250829 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, as well as the TACC Frontera supercomputer, Amazon Web Services (AWS), and OpenAl API through the National Artificial Intelligence Research Resource (NAIRR) Pilot.
REPRODUCIBILITY STATEMENT
We have made extensive efforts to ensure the reproducibility of our work. Detailed descriptions of our experimental setup, including the unified environment state representation, state prediction objective, and evaluation metries (MMRV and Pearson correlation), are provided in Section 3 and Section 4 of the main paper. Full implementation details, including pretrained model checkpoints, training hyperparameters, and proxy objectives, documented in Appendix Sections D and E. We also provide a comprehensive analysis of additional baselines, robustness across policy algorithms, and computational complexity in Appendix Sections A-C. For simulation environments, we describe how demonstrations and ground-trute states are collected in Appendix Section D. while real-wordd experiments are specified in Section 4.2, including robot hardware, control frequency, and demonstra- tion protocol. Together, these materials ensure that all reported results can be replicated and extended by the community.
REFERENCES
Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. VATT: Transformers for multimodal self-supervised learning from raw video, audio and text. In NeurlPS, 2021. 3
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In CVPR, 2023. 3
Mohamed El Banani, Amit Raj. Kevis-Kokitt Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3D awareness of visual foundation models. In CVPR, 2024. 6, 22
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BeiT: Bert pre-training of image transformers. In ICLR, 2022. 3
Adrien Bardes, Jean Ponce, and Yann LeCun. MC-JEPA: A joint-embedding predictive architecture for supe-supe learning of motion and content features. arXiv preprint arXiv:2307.1266 2023. 3
Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. V-JEPA: Latent video prediction for visual representation learning. arXiv preprint orXiv:2404. 2400::24400..4411... 2024. 3
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. æ0: A vision-language-an aatttigggg-atiion action flow model for general robot control. In RSS, 2025.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. CoRR, 2023. 3
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov,

---

Published as a conference paper at ICLR 2026
Yuheng Kuang, Isabel Leal, Kuang-Huel Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjumath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Van- houcke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich, RT-1: Robotics transformer for real-world control at scale. arXiv preprint DrXiv:2212.06817, 2022a. 3,5
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. RT-1: Robotics transformer for real-world control at scale. In RSS, 2022b.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang- Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-2: Vision-language-aatte models transfer web knowledge to robotic control. arXiv preprint arXiv:2307. 15818, 2023. 3,
Kaylee Burns, Zach Witzel, Jubayer Ibn Hamid, Tianhe Yu, Chelsea Finn, and Karol Hausman. What makes pre-trained visual representations successful for robust manipulation? In CoRL, 2024. 2, 6, 7,
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021. 6, 21
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020. 3
Xin Chen, Sam Toyer, Cody Wild, Scott Emmons, Ian Fischer, Kuang-Huei Lee, Neel Alex, Steven H Wang, Ping Luo, Stuart Russell, et al. An empirical investigation of representation learning for imitation. In NeurIPS, 2021.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In RSS, 2023. 6
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009. 6, 22
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021. 3, 6, 8, 10
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained enns are biased towards texture; increasing shape bias improves accuracy and robustness. In /CLR, 2018. 6, 22
Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Jocy Hejna, Tobias Kreiman, Charles Xu, al. Octo: An open-source generalist robot policy. In RSS, 2024.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018. 3
Rohit Girdhar, Alaaeldin El-Nouby, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. OmniMAE: Single model masked pretraining on images and videos. In CVPR, 2023.
12

---

Published as a conference paper at ICLR 2026
Jean-Bastiens Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your latent-a new approach to self-supervised learning. In NeurlPS, 2020.
Gunshi Gupta, Karmesh Yadav, Yarin Gal, Dhruv Batra, Zsolt Kira, Cong Lu, and Tim GJ Rudner. Pre-trained -oxtat diffusion models are versatile representation learners for control. In NeurlPS, 2024.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In ICLR, 2020. 2
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 3, 6,
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020. 3, 6, 10
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022a. 2
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022b. 6, 8, 10
Yingdong Hu, Renhao Wang, Li Erran Li, and Yang Gao. For pre-trained vision models in motor control, not all policy learning methods are created equal. In ICML, 2023. 22
Intel RealSense. Intel realsense depth camera d455. https: /www://wwwwwooooooo i intelnnteile reaeeereelleeeee com/
depth- camera ddaa-..455.
Guangqi Jiang. Yifei Sun, Tao Huang. Huanyu Li, Yongyuan Liang, and Huazhe Xu. Robots pre-train robots: Manipulation-centric robotic representation from large-scale robot datasets. ICLR, 2025.
Armand Joulin, Laurens Van Der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data. In ECCV, 2016. 3
Abhishek Kadian, Joanne Truong, Aaron Gokaslan, Alexander Clegg, Erik Wijmans, Stefan Lee, Manolis Savva, Sonia Chernova, and Dhruv Batra. Sim2real predictivity: Does evaluation in simulation predict real-world performance? IEEE Robotics and Automation Letters, 5(4):6670- 6677,2020 6
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. DROID: A large-scale in-the-wild robot manipulation dataset. In RSS, 2024.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: Contrastive unsupervised representa- tions for reinforcement learning. In ICML, 2020.
Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2022-06-27. Open Review, 62(1), 2022. 3
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language-driven semantic segmentation. In ICLR, 2022a. 3
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICMI., 2022b. 3
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023. 3
Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sich, Sean Kirmani, et al. Evaluating real-world robot manipulation policies in simulation. In CoRL. 2024. 2, 5, 6, 24,
13

---

Published as a conference paper at ICLR 2026
Yecheng Jason Ma, William Liang, Vaidehi Som, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. Liv: Language-imag representations and rewards for robotic control. ICML, 2023. 20
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 2018. 3
Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Piere Pierre Berges, Tingfan Wu, Jay Vakil, et al. Where are we in the search for an artificial visual cortex for embodied intelligence? NeurIPS, 2023a. 2
Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Piers Berges, Pieter Abbeel, Jitendra Malik, Dhruv Batra, Yixin Lin, Olek- sandr Maksymets, Aravind Rajeswaran, and Franziska Meier. Where are we in the search for an artificial visual cortex for embodied intelligence? In arXiv, 2023b. 20
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo. Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108.104700 2021. 3
Yunze Man, Shuhong Zheng, Zhipeng Bao, Martial Hebert, Liangyan Gui, and Yu-Xiong Wang. Lexicon3D:: Probing visual foundation models for complex 3D scene understanding. NeurIPS, 2024. 6, 22
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3M: A universal visual representation for robot manipulation. In CoRL, 2022. 8, 16
Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. RoboCasa: Large-scale simulation of everyday tasks for generalist robots. Robotics: Science and Systems, 2024. 2, 5, 23, 25
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po- Yao Huang, Shang-War Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. TMLRR, 2024. 3, 6, 10, 21
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016. 3
Karl Pearson. Note on regression and inheritance in the case of two parents. Proceedings of the Royal Society of London, 58:240-242, 1895. ISSN 03701662. URL http://wwet jstor.
org/stabll htrssttale//// 115794. 6
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
Han Qi, Haocheng Yin, and Heng Yang. Control-oriented clustering of visual latent representation. In ICLR, 2025. 2,3
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 3, 6, 8, 10
Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training, In CoRL, 2022. 2
Ilija Radosavovie, Baifeng Shi, Letian Fu, Ken Goldberg, Trevor Darrell, and Jitendra Malik. Robot learning with sensorimotor pre-training. In CoRL, 2023. 2
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022. 3, 6
14

---

Published as a conference paper at ICLR 2026
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In CoRL, 2023. 2,3
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuloCo: A physics engine for model- control. In IROS, 2012. 2,
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VideoMAE:
Masked autoencoders are data-efficient learners for self-supervised video pre-training. In NeurlPS, 2022. 3
UFACTORY. Ufactory xarm 6 robot arm. bttps://www.uut ufactozy-xarmi warr-----a------ 6. 5
us/ wa/proddcc//
Homer Rich Walke, Kevin Black, Tony Z. Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen- Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, Abraham Lee, Kuan Fang. Chelsea Finn, and Sergey Levine. BridgeData v2: A dataset for robot learning at scale. In CoRI., 2023. 3,5
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Mod- elscope eet-tt-videe technical report. arXiv preprint arXiv:2308. 06571, 2023a.
Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Wang. Yali Wang, and Yu Qiao. VideoMAEv2: Scaling video masked autoencoders with dual masking. In CVPR, 2023b. 3
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang. Limin Wang, and Yu Qiao. Intern IntervVideo: General video foundation models via generative and discriminative learning. arXiv preprint 2rXiv:2212.031911 2022. 3
Hu Xu, Gargi Ghosh, Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. VideoCLIP: Contrastive pre-training for zero-shot video-text understanding. In EMNLP, 2021. 3
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-Worke A benchmark and evaluation for multi-task and meta reinforcement learning. In CoRL, 2020. 2, 5, 8, 10, 23, 25
Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. RSS, 2024.
Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. In CVPR, 2022.
Guanqi Zhan, Chuanxia Zheng, Weidi Xie, and Andrew Zisserman. What does stable diffusion know about the 3d scene? arXiv preprint arXiv:2310.06836, 2023. 3
Qihang Zhang, Zhenghao Peng, and Bolei Zhou. Learning to drive by watching YouTube videos: Action-conditioned contrastive policy pretraining. In ECCV, 2022.
Richard Zhang, Phillip Isola, and Alexci Efros. Colorful image colorization. In ECCV, 2016.
Shiduo Zhang, Zhe Xu, Peiju Liu, Xiaopeng Yu, Yuan Li, Qinghui Gao, Zhaoye Fei, Zhangyue Yin, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. VI.ABench: A large-scale benchmark for language-conditionne robotics manipulation with long-horizon reasoning tasks. ICCV, 2025a.
Weipu Zhang, Adam Jelley, Trevor McInroe, and Amos Storkey. Objects matter: object-centric world models improve reinforcement learning in visually complex environments. arXiv preprint ar0iv:2501. 16443, 2025b.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. iBOT: Image bert pre-training with online tokenizer. In ICLR, 2022. 3
Yuke Zhu, Josiah Wong. Ajay Mandlekar, Roberto Martín-Martin, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu. RoboSuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009, 222.222......99.,.,.. 2020. 3
15

---

Published as a conference paper at ICLR 2026
In this appendix, provide additional experiments, implementation details, and visualizations that could not be included in the main paper due to space constraints. We begin by reporting further details for the fine-tuning experiment in Figure in the main paper in Section A.I. In addition, in Section A.2, we evaluate our approach using an alternative behavior cloning algorithm to assess robustness across policy algorithms. In Section A.3, show more results for the few-shot baseline and Section A.4 reports error bar analysis and statistical robustness of our model. Furthermore, Section A.5 explores the transferability of our proxy to RoboCasa navigation tasks, and Section A.6 provides analysis of different simulator state subsets. Section A.7 details further experiments on joint learning, Section A.8 evaluates additional robotics- specific encoders, and Section A.9 analyzes our method's 's reliance on simulation versus real-world states. We provide additional discussion regarding DINOvI and v2 models in Section A. and show failure cases Section A.11. We detail the settings of other commonly used proxies in Section B and report their latency analysis in Section C. We then present details of the simulation environments used in our experiments in Section D, including how we obtain state information, determine object visibilityy, and collect expert demonstrations. Section E provides additional implementation details, Section discusses the limitation and broader impact of our work, and Section H report per-task policy evaluation results for all the experiments in Figure 3 in the paper.
A ADDITIONAL RESULTS AND ANALYSIS
A.1 DETAILS OF FINETUNING RESULTS
We notice that fine-tuning different backbones requires different hyperparameters. Thus, we conduct experiments on the most common ViT backbones in our setting. In the experiments, the learning rate of the backbone is set to 1 10 60 to stabilize the training. For the state regression proxy task, the backbone set to the same 1 x 10-6 learning rate. Additionally, we observe that all the backbones' performance improves compared with frozen parameters, but with heavily unequal improvements (DINOv2 improves a lot while MoCov3 improves minimally). Such results demonstrate that the performance ranking tends to change during backbone finetuning.
A.2 ROBUSTNESS ACROSS POLICY ALGORITHMS
To further validate our approach, we change the diffusion policy to the MLP-based policy algo- rithm (Nair et al., 2022). Specifically, we use three layers predict the action. As shown in Figure I, the algorithm change doesn't 't affect the performance correlation significantly, demonstrating the robustness of our proxy objective. Overall, the MLP prediction performs somewhat worse than the diffusion policy.
Diffusion Policy vs. Sm
MLP Vs. Sm
MMRV 0.037
0.691
0.550
MMRV
0.036 0.623
Diffusion Policy Success Rate
0.65
0.525
0.500
0.60
+
Success Rate
0.475
0.55
0.450
0.50
0.400
0.375
0.45
0.350
A
0.2
0.3
0.4
0.5
0.6
0.7
0.2
0.3
0.4
0.5
0.6
0.7
Sm
5m VIT-IN
ResNet-IN
MoCoV3
DINOv2
R3M
DINOv1
SD
V
CLIP
MAE
Figure I: The performance correlation still holds when changing the behavior cloning algorithm to MI.P algorithm, demonstrating our method's robustness.
16

---

Published as a conference paper at ICLR 2026
A.3 FURTHER RESULTS FOR THE FEW-SHOT BASELINE
In the paper, the few-shot proxy is evaluated over five episodes for efficiency, which limits its predictive power. Here we further evaluate the few-shot learning proxy using the entire evaluation set. Specifically, we take 100 episodes from the MetaWorld and SimplerEnv environments and 50 episodes from the RoboCasa environment using their default settings. As the results in Table show, this feo-shot learning variant exhibits the best correlation among all the baselines the simplistic Metaworld benchmark. However, more challenging RoboCasa and SimplerEnv, this proxy still struggles accurately capture the policy learning performance of different visual representations. Crucially, it is about 150x more expensive than our proposed approach, making it an impractical choice even for relatively simple environments like Metaworld. For reference, full training and evaluation in MetaWorld with a ResNet backbone takes about 30 hours. other words, our proxy reduces the computational cost by 99.8%.
MetaWorld
RoboCasa
SimplerEnv (G)
SimplerEnv (W)
Runtime
MMRV
Few-Shot (5 Episodes)
0.069
0.028
0.060
0.113
min
Few-Shot (Full Episodes)
0.042
0.037
0.097
0.104
~600 min
Ours
0.037
0.010
0,023
0.069
4 min
Pearson
Few-Shot Episodes)
0.650
0.243
0.624
-0.128
min
Few-Shot (Full Episodes)
0.885
-0.043
0.348
0.137
~600 min
Ours
0.691
0.760
0.871
0.688
min
Table I: Full episode evaluation the proxy task of few-shot learning. Our proxy performs better in most environments. The reported time measured on MetaWorld with a ResNet backbone.
A.4 STATISTICAL ANALYSIS
We report error bars (mean + Standard Error of the Mean, SEM) for policy performance across four environments. For each visual backbone, we conduct n independent training runs (different random seeds) and evaluate each trained policy using the same evaluation protocol in the main paper. Across all environments, the SEM is generally much smaller than the performance gaps between methods, supporting the statistical robustness our comparisons.
Backbone
MetaWorld SR
RoboCasa SR
Simpler (G) SR
Simpler (W) SR
ResNet
0.714 (=0.001)
0.303 (#0.008)
0,212 (=00088)
0.139 (#0.014)
VIT
0.706 (=0.001)
0.263 (#000055)
0.210 (=0.011)
0.174 (#0.010)
MAE
0.631 (=0.000)
0.284 (+:0008)
0.314 (=0.019)
0.074 (+0.018)
CLIP
0.616 (=0.001)
0.261 (=00006)
0.240 (=0.015)
0.115 (#0.011)
MoCov3
0.605 (=00))
0.284 (#0.005)
0.351 (=00055)
0.148 (+0.015)
DINOv2
0.598 (=0.055)
0.317 (+0.005)
0.250 (=0.01))
0.090 (+0.005)
DINOvI
0.570 (=0.002)
0.311 (+00006)
0.225 (0.011)
0.155 (+0.011)
R3M
0.538 (=.002)
0.273 (#0.005)
0.353 (0.015
0.221 (+00014)
SD
0.438 (=0007)
0.282 (+0.005)
0.325 (=00015)
0.163 (+0.017)
Table II: Error bar analysis (mean + SEM over n 5 runs) across four environments. Values are reported the thousandth.
In addition, we compute error bars for state regression in the MetaWorld environment using n = independent runs and report the results below. We report the mean regression score + Standard Error of the Mean (SEM) across runs. Consistent with the policy results, the SEM is small relative to the differences between encoders, supporting the robustness of the regression comparison.
17

---

Published as a conference paper at ICLR 2026
RecNor
VI
CLIP
MeGoos
DINO
DINOvi
RFM
50
Regression Score
0,715 (EO000))
0,706 (50,002)
(40201)
1.06 (SH1
0,327 (EILINE))
0,366 (200001))
0,554 (40001s
U.506 (400000)
(2170
Table III: Error bars for state regression World). We report regression score + SEM over 5 runs for each encoder. All values are reported to the thousandth.
A.5 TRANSFER TO ROBOCASA NAVIGATION TASKS
We also evaluate our proxy on RoboCasa navigation tasks, using exactly the same formulation as for manipulation. Despite the substantial difference in task objectives and success criteria, the proxy remains strongly correlated with navigation policy performance (Table IV), indicating that generalizes beyond manipulation-only settings.
MMRV +
Pearson's 's correlation
RoboCasa Manipulation (Ours)
0,010
0.760
RoboCasa Navigation (Ours)
0.022
0.727
RoboCasa Navigation (Segmentation)
0.037
0.436
RoboCasa Navigation (Depth)
0.043
0.172
Table IV: Proxy quality on RoboCasa manipulation vs. navigation. The same proxy, applied without modification, achieves a strong correlation with navigation policy performance, demonstrating transfer across task families.
A.6 ADDITIONAL ANALYSIS OF STATE SUBSETS
In this section, we further analyze the effect of different subsets of simulator state dimensions. We want emphasize that the state dimensions are not hand-designed: our method uses all variables di- rectly exposed by each simulator to define uniform, task-agnostic state representation. Accordingly, the per-dimension and subset ablations below are not intended show that every individual attribute contributes positively in isolation, but to illustrate how different environments emphasize distinct parts of the state and the effect of different subsets.
Full state task-critical subsets. We first compare the full state "task-critical" subset consisting only of target object states and end-effector pose. As shown in Table V, task-critical subsets do not improve correlation. In Meta MorlW they slightly reduce Pearson r, while in RoboCasa they mildly increase r at the cost of worse MMRV. Overall, full states provide more balanced and robust proxy.
Full state
Task-critical subset
Environment
MMRV +
r r
MMRV +
r
MetaWorld
0.037
0.691 0.760
0.037
0.615
RoboCasa
0.010
0.015
0.774
Table V: Full state vs. task-critical subsets. Using only task-critical variables does not consistently improve correlation, while full states remain competitive or better in terms of MMRV.
Leave-one-out ablations. To assess the contribution of individual state groups, we perform leave- one-out experiments over Meta World and RoboCasa. We remove one group at a time from full state: object position (p_pose), object orientation (q (q_pose), shape (s_shape), material (o__mt), joint angles (q_i), end-effector pose (p_ee), and lighting (1). Results in Table VI show that dropping any group generally degrades performance relative the full state; different environments are sensitive to different components, confirming that multiple state dimensions are important rather than single hand-picked subset.
Searching for best-performing subsets. Finally, we enumerate combinations state groups and select those that maximize Pearson correlation in each environment. As shown in Table VII, different
18

---

Published as a conference paper at ICLR 2026
MetaWorld
RoboCasa
Removed group
MMRV 4
r
MMRV 4
P_pose
0.7325
0,0402
0.7169
0,0163
pose
0,6358
0.0455
0.7342
0,0163
s_shape
0.7607
0.0361
0.7028
0.0186
m_mat
0.6979
0.0365
0.7797
0.0105
qj
0.6153
0.0365
0.7744
0.0153
p_ee
0.6656
0.0308
0.7375
0.0134
1
0.6656
0.0365
0.7602
0.0099
Table VI: Leave-one-out ablations state groups. Removing any state group generally harms either MMRV or Pearson r, and the most influential groups differ between environments, indicating that multiple components contribute to the proxy.
tasks favor different subsets joint and orientation states in MetaWorld vs. position and shape in RoboCasa), and there is no universal best subset across environments. While tailored subsets can slightly improve metrics in given setting, they require manual design and do not transfer. This supports our choice using the full simulator state as a simple, task-agnostic representation that scales naturally with environment complexity.
Environment
Best subset
MMRV
r
MetaWorke RoboCasa
e-ooe,, q_j
0.0245
0.8702
p_soe s_Shape
0.0110
0.8543
Table VII: Best-performing state subsets (per environment). Enumerating combinations of state groups reveals environment--peccffc optima but universal subset, reinforcing the appeal of single full-state formulation.
A.7 ADDITIONAL EXPERIMENTS ON JOINT LEARNING
In addition to the main results in Table 3, we further provide additional experiments of using a state-prediction proxy during policy learning.
Two-stage vs. joint training, In addition to joint training, we explore the performance of two-stage training. Specifically, we first fine-tune the encoder with the regression objective and then fine-tune again with the policy objective only. Table VIII shows that joint training consistently outperforms both direct fine-tuning and two-stage training across all backbones. While two-stage training improves over direct fine-tuning, it remains worse than joint training, suggesting that maintaining the auxiliary objective throughout optimization more effective than pretraining the backbone and discarding the proxy signal.
Method
VIT-IN
MoCoV3
MAE
CLIP
DINOv2
Direct fine-tuning
0.683
0.671
0.648
0.765
0.767
Joint training
0.740
0.743
0.712
0.801
0.795
Two-stage training
0.722
0.719
0.680
0.788
0.784
Table VIII: Joint two-stage of the proxy. Joint training with the proxy auxiliary loss consistently yields the best performance.
Training progression with and without the proxy. We next focus on CLIP and study how the benefit of joint training evolves over training time. Table IX reports success rates after 5, 10, 15, 20, 25, and 30 epochs for direct fine-tuning versus joint training. Direct fine-tuning improves initially but tends saturate (and even slightly fluctuate) beyond 20 epochs. In contrast, joint training not only starts from a higher performance but also continues to provide robust gains as training proceeds, yielding consistent margin over direct fine-tuning at all checkpoints.
19

---

Published as a conference paper at ICLR 2026
Method
5 ep.
10 ep.
ep.
15 ep.
20 ep.
25 ep.
30 ep.
Direct fine-tuning
0.573
0.680
0.732
0.765
0.771
0.769
Joint training 0.613
0.708
0.774
0.801
0.809
0.811
Table IX: Training progression of CLIP with and without joint learning. Success rates of CLIP under direct fine-tuning versus joint training across different epochs. Joint training consistently yields higher performance and continues to improve with longer training.
Metaworld
RoboCasa
0.744
0.32
Polity Success Rate
0.60
0.30
0.29
0.50
0.28
0.26
5m
5m
SimplerEnv (Google Robot)
SimplerEnv (WidowX + Bridge)
MMRV 0.033
r 0.875
MMRV 0.065
0.733
0.35
Policy Success Rate
0.30
#
0.15
0.25
0.05
0.35
0.40
0.45
0.50
0.55
0.60
0.65
0.70
0.75
0.1
0.2
0.3
0.4
0.5
0.6
Sm
5m
ResNet-IN
McCov3
+ DINOv2
R3M
DINOv1
SD
V
VTIN @
CUIP
MAE
VC-1
LI
Figure II: Correlation plots with rppecicc- encoders. Our proxy metric remains predictive of the downstream policy success rate.
A.8 ADDITIONAL. CORROIICS-SPECIHE ENCODERS
We evaluate two additional robotics-specific encoders, VC-1 Majumdar et al. (2023b) and LIV Ma al. (2023), under the same protocol. Both models are pretrained on large-scale roboties data and designed to capture control-relevant structure directly from images. Figure summarizes their policy success rates and correlation with proxy scores. VC-I and LIV fall the same correlation trend as our main models: higher state-prediction quality reliably corresponds to higher policy success, further supporting the generality of our proxy beyond standard image backbones.
A.9 RELIANCE ON SIMULATION ENVIRONMENT STATES
Our proxy computation in the main paper uses simulator ground states. We study how sensitive the method is to this choice and whether it can still work with noisy, estimated real-world states. We compare four variants of the proxy signal: (i) simulation proxy scores computed from ground-truth simulator states, (ii) real-world estimated proxy scores computed from estimated states (DetAny3D detections, robot states, and known object attributes), (iii) an action MSE proxy, and (iv) a depth-basee proxy. For each variant, measure its agreement with the true policy ranking using MMRV and Pearson's 's correlation (defined in the main paper). All metrics are averaged over runs.
Tab. X reports the results. Using estimated real-world states leads slightly higher MMRV and somewhat lower Pearson correlation than simulator states, but remains substantially better than the action-MSR: and basth-based baselines. Despite being noisy and incomplete, the estimated states
20

---

Published as a conference paper at ICLR 2026
(real) stack
(sim) stack
(real) carrot
(sim) carrot
Figure III: Domain gap between simulation and real-world environments. While the tasks are the same the environments differ significantly in appearance, object shapes, robot embodiment and camera poses.
still yield a proxy that correlates well with policy performance. This suggests that the effectiveness of our proxy does not critically rely on privileged simulator information, and that similar signals could be obtained from real-world perception pipelines. In addition, though The simulation and real environments differ substantially in appearance, lighting, and layout (Figure III), the correlation remains high.
MMRV І
Pearson's correlation 1
Simulation proxy scores Real-world estimated proxy Action MSE
0.041
0.711
0.047
0.532
0.055
0.347
Depth
0.080
-0.372
Table X: Effect of real-world proxies. Estimated real-world states yield proxy that is close to the simulator-basee one and clearly better than action MSE or depth.
A.10 DINOVI AND DINOV2
DINOv1 (Caron al., 2021) and DINOv2 (Oquab et al., 2024) learn different types of visual representations due to their training methodologies and data. DINOvI's self-supervised features tend be more object-centric and segmentation like, which can directly benefit tasks where identifying distinct objects is crucial (e.g. RoboCasa). DINOv2, while more powerful overall, captures wider range of visual information, including emphasizing fine-grained spatial details, which may require fine-tuning fully adapt to specific task. As experiments in section A. demonstrate, DINOv2 benefits more from fine- fine-tuning compared to DINOv1, because its richer, more general features can then be specialized to the task.
A.11 FAILURE CASES
While our proxy generally aligns well with policy performance, we have observed occasional outlier cases where it fails to predict the correct ranking. A notable example is MAE in the WidowX + Bridge setting (in Figure 4), where the aggregated proxy score overestimates its true performance.
This discrepancy arises when a model performs well on small subset of state variables that are weakly correlated with task success, leading an inflated average proxy score. For instance, MAE ranks highest in predicting pose and shape states-both identified in Table 2 as having low correlation with policy success-bie ranks poorly across more informative state variables. As a result, its high proxy score does not reflect actual performance.
While such failure modes exist, we emphasize that they are rare. Across the benchmarks we study, the proxy consistently provides reliable and interpretable signal: models with strong proxy scores typically demonstrate strong regression ability across relevant states and higher policy success. We will clarify these edge cases and their implications in the final version of the manuscript.
21

---

Published as a conference paper at ICLR 2026
B DETAILS OF BASELINE PROXIES
In this section, we detail the settings of other commonly used proxy objectives and analyze their latency bottlenecks.
ImageNet Recognition. Linear probing on ImageNet (Deng et al., 2009) a well-established benchmark for representation quality. We freeze backbones and train a k-NN classifier (Hu et al., 2023) on the ImageNet 2012 train split, using the top-1 accuracy on the validation set as a proxy.
Shape Bias. Measuring shape bias on Byylized-Imagelle (Geirhos et al., 2018) evaluates a model's reliance on structural cues rather than superficial textures. We compute the fraction of shape-based predictions on the ImageNet validation set using the same K--N classification.
Few-Shot Learning. Few-shot learning is a popular approach test the generalizability of model. Here, we use only five demonstrations per task for training efficiency and measure the final performance. The success rate at test time is used as the proxy score. Aiming for efficiency, we only use five episodes when computing the success rate for each task.
Action MSE. We use the action prediction mean squared error loss (MSE) of the behavior cloning algorithm on the validation set as a proxy of its success rate. This approach requires training the behavior cloning head till convergence, but forgoes expensive policy execution. The training setting is the same as the Few-shot Learning above.
Segmentation. Semantic segmentation performance tests spatial and contextual feature learning (Burns et al., 2024; Man et al., 2024). To build the data for each task, we generate the trajectory together with the object mask in the simulator. The dataset is divided into training and validation set for evaluation. We attach a lightweight segmentation head to the frozen encoder and report the mean Intersection-over-Unee
(mloU) on the validation split.
Depth Estimation. Monocular depth estimation probes geometric reasoning (Banani et al., 2024). Similarly to segmentation, we render the depth for each trajectory as the objective. A small depth decoder attached to each frozen backbone, and we evaluate the MSE on the validation set.
C DETAILS OF COMPLEXITY AND LATENCY COMPARISON
C.I ANALYSIS OF RESNET-IN AND VIT-IN BACKBONES
The variation in latency delta between ResNet-IN and VIT-IN across proxy tasks in Table 4 arises not only from architectural differences (e.g., ResNet vs. ViT) but also from the nature of each proxy task's training and evaluation pipeline. While the backbones are frozen, the computation time depends on several additional factors such as head architecture, batch size, data scale, and whether inference or rollout is involved. Below is a breakdown of the main factors affecting latency across task categories.
Few-Shot Policy Learning This setting involves full policy training from a few demonstrations using Diffusion Policy that includes a frozen visual encoder, text encoder, and a ID diffusion network. The total time includes both training and environment rollouts. The rollout and diffusion steps contribute significantly to the overall time, and differences between ResNet and ViT (e.g.. in feature dimensionality and processing overhead) become more pronounced in this larger pipeline.
Action MSE This setup shares the same architecture as Few-Shot (including the full diffusion model), but evaluation done purely on the validation set without rollouts. This leads to a smaller runtime overall and reduced latency gap between backbones, since the rollout overhead is removed and the backbone usage is more uniform.
ImageNet Feature extraction and evaluation over the full ImageNet validation set.
Shape Bias Requires two passes over the dataset (to isolate texture and shape cues), which effectively doubles the total runtime, and hence the latency delta between ResNet and ViT also roughly doubles.
Depth & Segmentation These are training- based dense prediction tasks. A lightweight decoder head is trained on top of the frozen encoder, and the additional runtime is dominated by the process of high-resolution feature maps being upsampled to the input resolution, and MLP predictors being applied to the large feature maps.
22

---

Published as a conference paper at ICLR 2026
In summary. the inconsistencies in the latency delta reflect the interaction between the frozen backbone and the downstream task specific computation, including head architecture, data scale, and whether inference includes rollouts or dense pixel-wir wise outputs.
C.2 TIME BREAKDOWN ANALYSIS
In addition, to have a better assessment of the time, we give the breakdown of time consumption here:
Few-Shot Learning
ResNet: Policy training (38 min), rollout evaluation (40 min)
VIT: Policy training (61 min), rollout evaluation (52 min)
Action MSE
ResNet: Policy training (38 min), evaluation (7 min)
VIT: Policy training (61 min), evaluation (11 min)
ImageNet
ResNet: Feature extraction (9 min), KNN prediction (2 min)
VIT: Feature extraction (41 min), KNN prediction min)
Shape Bias
ResNet: Feature extraction (20 min), KNN prediction (3 min)
VIT: Feature extraction (81 min), KNN prediction (11 min)
Segmentation & Depth Estimation
ResNet: Training (10 min), evaluation (0.2 min)
VIT: Training (27 min), evaluation (0.3 min)
State Regression (Ours)
ResNet: Training (4 min), evaluation (0.2 min)
VIT: Training (12 min), evaluation (0.3 min)
D SIMULATION ENVIRONMENTS
D. ENVIRONMENT DETAILS AND EXPERT POLICY
Simulation environments. We evaluate four distinct manipulation benchmarks, each with different task counts, object diversity, difficulty levels, and robot kinematics:
1. Meta World (50 tasks) (Yu et al., 2020)
Task count: 50 distinct tabletop manipulation tasks.
Object diversity: tasks involve one two CAD-style objects drawn from set of prototypical meshes (doors, drawers, windows, buttons, pegs, boxes, etc.).
Difficulty: spans easy (reach, push), medium (pick-and-place, press button), to hard (peg-in- hole, open window). Since all tasks the same setting, this benchmark is relatively easy.
Robot: Rethink Sawyer 7-DoF arm with parallel-jaw gripper.
Action control:
continuous 4-dimensional end-effector displacement (Ax, Ay, Az) plus gripper open/close command.
2. RoboCasa (24 tasks) (Nasiriany et al., 2024)
Task count: 100 everyday household tasks in kitchen scenes. We adopt the 24 atomic tasks in the experiments.
Object diversity: over 2,500 3D assets spanning 150+ object categories (cabinets, appliances, utensils, etc.).
23

---

Published as a conference paper at ICLR 2026
Difficulty: includes atomic tasks (pick, place) and composite multi-step tasks (e.g., cooking sequences), ranging from medium to high.
Robot: Rethink Sawyer 7-DoF arm (we instantiate Sawyer in all scenes).
Action control: 7-dimensional controller commanding end-effector pose (x,, y, z.yaw, pitch, roll) plus gripper open/close.
3. SimplerEnv (Li et al., 2024) (G) (10 tasks)
Task count: 10 pick tasks, including tasks for the Google Robot and 4 tasks for the WidowX.
Object diversity: different household items (tools, blocks, bottles).
Difficulty: medium, as each goal object is randomly sampled.
Robot: Fetch Robotics arm (7-DoF' articulated manipulator) for Google Robot and WidowX- 250 S 6-DoF research arm for WidowX.
Action control: 7-dimensional controller commanding end-effector pose (a, y, z.yaw, pitch, roll) plus gripper open/close.
Demonstration and state collection. As mentioned in the paper, we collect demonstrations following different strategy in each simulation environment. Specifically, for MetaWorld, we use their expert policy to generate the demonstrations. For RoboCasa, we use official human-collected demonstrations. In SimplerEny, we design the expert policy by ourselves and generate the demonstrations. Specifically, the expert policy is built with observable goal states. For each task, we write its corresponding goal- conditioned manipulation code and validate our expert policy through the success rate. Approximately, our expert policy can reach about 60% success rates.
Regarding the state information, robot-level states like joints and end-effector states can be directly gathered from the simulator. In terms of lighting, we give different lighting conditions unique class identifier The object states are stored differently across the environments. In MetaWorld, we exclude the background objects and retrieve the foreground MuJoCo bodies. In RoboCasa, the foreground objects are directly stored in the environment information. In SimplerEnv, the foreground objects are considered dynamic "actors" from the environment. Once we have the object instance, can extract its pose and material states, and the shape is defined using the united bounding box for the object's 's parts. The segmentation mask is rendered from the simulator. If the object has segmentation mask rendered, it is considered not observable in the current view. The bounding box is generated from the segmentation mask.
E IMPLEMENTATION DETAILS
E. PRETRAINED VISION MODELS
Overview. We consider diverse set of pretrained visual models widely adopted for downstream visual tasks, partieularly in robotics and embodied AI. Each model differs backbone architecture. training objective, and pretraining dataset. Table XI provides an overview.
Feature extraction details. Our feature extraction for visual backbones is split into three categories: CNN-based, ViT-based, and diffusion- The specifics are as follows:
1. CNN-based (ResNet-18).
Feature map: output of the final convolutional block (pre- (pre-pooling), size BxCxHxW. Global feature: average-pooled vector of size B C.
2. ViT-based (ViT-B).
Feature map: reshape the remaining N patch embeddings into B x C Global feature: the learned class-token embedding, shape B x C.
3. Diffusion-based (Stable Diffusion v1.5).
Feature map: extract intermediate feature maps from selected up-sampling blocks, each of shape B x C₁ x h2 x w,.
Global feature: spatially average one map into B x C₁ descriptor.
24

---

Published as a conference paper at ICLR 2026
Name
Backbone Setting
Loss Function
Pretrained Data (Size)
(Param)
ResNet-IN
ResNet-18
Supervised cross-entropy
ImageNet-Ik
12M
(~v1.MM images)
VIT-IN
ViT-B/16 (86M)
Supervised cross-entropy
ImageNet-Ik (~1.3M images)
CLIP
ViT-B
Contrastive image-text
OpenAl curated
(86M)
alignment
(~400M images)
DINOv1
ViT-B/16
Self ceff-disiillation w/o labels
ImageNet-IK (~1.3M images)
(86M)
DINOv2
ViT-B/14L.
Self-distillation with momentum teacher
Internal curated set
(86M)
(~14 images)
MoCo v3
ViT-B/16
Contrastive learning (MoCo)
ImageNet-Ik
(86M)
(~1.3M images)
MAE
ViT-B/16
Masked image reconstruction
ImageNet-Ik
(86M)
(~113M images)
R3M
R8
Contrastive w/ robot goal-conditioning
Ego4D, HM3D,
(12M)
(~100M+ frames)
U-Net w/ VIT encoder
Denoising diffusion objective
LAION-5B
(~860M)
(~5B images)
Table XI: Summary of pretrained visual models. approximate.
Param: number of parameters. Dataset size is
E.2 LICENSE OF DATASETS AND SIMULATOR USED
We list the licenses of all the simulation environments have used during our experiments.
Meta- World (Yu et al., 2020): MIT License.
RoboCasa (Nasiriany et al., 2024): MIT License.
SimplerEnv (G& W) (Li al., 2024): MIT License.
In addition, we use the official implementation and pre-trained models provided on the Huggingface, GitHub, and Pytorch Hub for ResNet-IN', VIT-IN CLIP DINOv1 DINOv2', MoCo v3', MAE R33*T and SD All of the foundation models we evaluated in our paper utilize various data sources during their pre-training phase. Please refer to their original paper for the license of the datasets they have used in pre-training their models.
E.3 ADDITIONAL DETAILS THE STATE REGRESSION PROXY
To make fair and straightforward comparison between different backbones, unify the state regression with the same resolution of the feature map. Specifically, the feature maps are interpolated x resolution and later used for the Rol-pooling. For the optimization hyperparameter, we train the model for 20 epochs with 32 batch size. We use the SGD optimizer and set the learning rate to 5e-4. The momentum is set to 0.9, and the weight decay set le-4.
http:://oosssccchcor/////ssaaa/s www.sssooootttatoteeeeeaom html
25

---

Published as a conference paper at ICLR 2026
F LIMITATION AND BROADER IMPACT
Limitation. central limitation of our approach its reliance on access to ground- truth environment state, which is readily available in simulation but not necessarily in the real world. While simulators are a widely accepted tool for efficient and reproducible evaluation robotics, they can introduce a domain gap, and some natural phenomena, like splashing water, smoke very challenging to accurately capture with existing simulators. Furthermore, our analysis limited to three simulation environments and finite of pretrained visual backbones; while diverse, this does capture the full variability of robotic tasks or vision models. Finally, although our method is computationally efficient relative baselines, still requires learning state regression heads.
Broader Impact. Our method is not directly tied to any specific deployment, and we do not anticipate immediate societal risks from the method itself. However, improved tools for evaluating and scaling robotic policies may accelerate the deployment of autonomous systems, which in turn can have significant societal implications (both positive and negative). Our reliance on simulation also raises concerns about potential mismatches between training conditions and real-world scenarios, which could lead unexpected behavior if not properly validated. We encourage researchers to complement our approach with robustness checks in real-world environments.
G USE OF LARGE LANGUAGE MODELS (LLMs)
We used Large Language Models (LLMs) only to facilitate writing, such polishing grammar, improving clarity, and suggesting alternative phrasings for text drafted by the authors. LLMs were not used for research ideation to generate core scientific content. All LLM-assisted text was reviewed and edited by the authors before inclusion.
H SUCCESS RATE BREAKDOWN
In this section, we breakdown the success rate for different backbones in Figure 3. The results are shown in Table XII, XIII, XIV and XV.
26

---

Published as a conference paper at ICLR 2026
Task
ResNer-IN
R3M
ViT-IN
MoCov3
DINOvI
DINOV2
MAE
CLIP
SD
assembly
0.65
0.19
0.51
0.44
0.47
0.33
0.47
0.37
0.11
basketball
0.18
0.12
0.15
0.10
0.09
0.06
0.14
0.07
0.05
bin-picking
0,45
0.18
0.66
0.35
0.81
0.56
0.29
0.86
0.08
box-close
0.57
0.58
0.64
0.53
0.60
0.50
0.69
0.53
0.33
presss-opooonw
0.96
1.00
1.00
0.91
0.94
0.86
0.99
0.88
0.79
button-press-topdowee
0.95
1.00
1.00
0.95
0.92
0.88
0.99
0.73
0.66
button-press
1,00
0.92
1.00
0.94
0,92
0.87
1.00
0.84
0.86
wall
1.00
1.00
1.00
1.00
1.00
0.99
1.00
0.98
0.91
coffec-button
0.95
0.95
0.98
1.00
0.86
0.97
1,00
0.72
coffee-pull
0.44
0.00
0.46
0.37
0.27
0.33
0.48
0.39
0.13
coffee-push
0.30
0.15
0.42
0.45
0.27
0.31
0.53
0.25
0.11
dial-turn
0.26
0.02
0.37
0.36
0.16
0.45
0.25
0.21
0.01
disassemble
0.43
0.23
0.50
0.39
0.40
0.09
0.45
0.38
0.12
door-close
1.00
1.00
1.00
1.00
1,00
1.00
1,00
1.00
1.00
door-lock
0.93
0.87
0.41
0.67
0.64
0.69
0.78
0.59
0.55
door-open
1.00
1.00
1.00
1.00
0.99
1.00
1.00
1.00
0.88
door-unlock
0,88
0.95
0.91
0.81
0.89
0.97
0.85
hand-insert
0.18
0.14
0.28
0.35
0.33
0.31
drawer-close
1.00
1.00
1.00
drawer-open
1.00
0.99
0.86
0.95
0.90
0.98
0.83
0.93
0.59
faucet-open
1,00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
faucet-close
1.00
1.00
1.00
0.92
0.99
0.95
1.00
0.94
0.92
hammer
0.37
0.32
0.50
0.38
0.27
0.43
0.50
0.45
0.26
handle-press-side
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
0.97
bandle-press
0.99
1.00
1.00
1.00
0.98
0.98
1.00
1.00
0.67
handle-pull-side
0.51
0.27
0.52
0.58
0.55
0.45
0.64
0.61
handle-pull
0.03
0.00
0.11
0.07
0,22
0.11
0.05
0.01
lever-pull
0.54
0.09
0.39
0.61
0.32
0.60
0.60
0.25
0.08
pick-place-wall
0.28
0.17
0.38
0.49
0.20
0.52
0.29
0.25
0.15
pick-out-offhole
0.37
0.02
0.63
0.17
0.16
0.32
0.58
0.22
0.32
pick-place
0.04
0.01
0.18
0.21
0.04
0.14
0.04
0.02
0.01
plate-slide
0.86
1.00
0.79
0.73
0.76
0.81
0.87
0.70
0.63
plate-slide-side
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
plate-slide-bat
0.99
1.00
1.00
1.00
0.95
0.97
1.00
1.00
0.99
mate-slide-backksiie
1.00
1.00
1.00
1.00
1.00
1,00
peg-insert-sie
0.28
0.21
0.19
0.19
0.40
0.18
peg-unplug-sine
0.78
0.48
0.68
0.71
0.53
0.59
0.47
0.16
0.15
0.07
0.14
0.09
0.15
0.10
0.16
0.07
0.03
stick-push
0.71
0.98
0.96
0.38
0.52
0.82
0.74
0.80
0.59
stick-pull
0.33
0.33
0.30
0.43
0.28
0.44
0.28
0.53
0.30
push
0.35
0.02
0.36
0.19
0.08
0.34
0.28
0.18
0.10
push-wall
0.47
0.07
0.72
0.70
0.45
0.60
0.38
0.36
0.28
push-back
0.31
0.05
0.38
0.19
0.22
0.31
0.25
0.27
0.11
reach
0.25
0.34
0.16
0.19
0.22
0.20
0.16
0.10
reach-wall
0.71
0.44
0.57
0.63
0,52
0.52
0.50
shelf-place
0.30
0.18
0.15
0.12
0.11
0.09
0.07
sweep-into
0.35
0.21
0.50
0.28
0.32
0.32
0.37
0.35
0.19
sweep
0.32
0.19
0.73
0.21
0.25
0.32
0.48
0.22
0.07
window-open
1.00
1.00
0.99
1.00
0.95
0.88
0.95
0.95
0.88
window-close
1,00
1.00
1.00
1.00
1,00
1.00
1.00
1.00
0.93
Table XII: Success rate breakdown for MetaWorld.
27

---

Published as a conference paper at ICLR 2026
Task
ResNet-IN
R3M
VIT-IN
MoCov3
DINOvI
DINOV2
MAE
CI.IP
SD
PoCCounteTToCab
0.02
0.00
0.00
0.00
0.00
Poonter
0.04
0.04
0.04
0.04
0.00
0.02
0.00
PnPCounterToSink
0.02
0.02
0.00
0.00
0.02
0.02
0.00
0.02
0.00
PnPSinkToCountee
0.00
0.00
0.00
0.02
0.04
0.02
0.02
0.02
0.04
PnPCounterTiMMicrwwar
0.00
0.00
0.00
0.02
0.06
0.02
0.04
0.08
0.02
MoPMicrowaeeTCCoouee
0.00
0.00
0.00
0.06
0.02
0.04
0.04
0.00
0.00
CounterToStove
0.00
0.00
0.02
0.02
0.00
0.02
0.00
0.02
0.02
PnP
PnPStoveTooContt
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.02
0.02
Senglnn
0.32
0.32
0.28
0.26
0.48
0,28
0.26
0.16
0.16
CloseSingleDoor
0.56
0.58
0.62
0.70
0.62
0.68
0.80
0.70
0.70
OpenDoubleDoor
0.16
0.12
0.30
0.18
0.30
0.12
0.30
0.12
0.32
CloseDoubleeDoor
0.60
0.48
0.64
0.62
0.74
0.54
Orawer
0.58
0.56
0.58
0.46
0.58
0.46
0.40
CraseDrawer
0.92
0.96
0.96
0.96
TurnOnSinkFaucce
0.50
0.46
0.54
0.54
0.42
0.46
JurnOftSinkFaucet
0.72
0.56
0.84
0.70
0.58
0.60
TurnSinkSpout
0.46
0.50
0.58
0.62
0.60
0.62
0.64
TurnOnStove
0.42
0.28
0.38
0.44
0.48
0.50
0.56
TurnOffStove
0.10
0.14
0.08
0.08
0.10
0.10
0.08
0.06
0.08
CoffceSetupMug
0.02
0.02
0.06
0.08
0.04
0.08
0.02
0.00
0.02
Seff
0.32
0.06
0.16
0.30
0.24
0.28
0.10
0.10
0.08
CoffeePressButton
0.24
0.24
0.28
0.26
0.24
0.14
0.16
0.34
0.22
TurnOnMicrowave
0.54
0.42
0.42
0.30
0.50
0.40
0.26
0.28
0.60
TurnOMMicrowave
0.52
0.54
0.40
0.36
0.56
0.72
0.64
0.26
0.46
Table XIII: Success rate breakdown for RoboCasa.
Task
ResNet-IN
R3M
ViT-IN
MoCov3
DINOvI
DINOv2
MAE
CLIP
SD
pick_coke,
0.35
0.53
0.33
0.58
0.37
0.31
0.34
pick_object
0.11
0.21
0.14
0.16
0.09
0.12
0.15
0.07
0.08
move near
0.05
0.07
0.07
0.10
0.06
0.05
0.09
0.10
0.04
open_drawer
0.53
0.64
0.40
0.62
0.54
0.57
0.63
0.50
0.77
close drawer
0.37
0.68
0.34
0.62
0.39
0.63
0.58
0.48
0.88
place in closed. ttp_toop__dddeeeee drawer
0.05
0.01
0.07
0.02
0.01
Table XIV; Success rate breakdown for SimplerEnv (Google Robot).
Task
ResNet-IN
R3M
VIT-IN
MoCov3
DINOv1
DINOv2
MAE
CLIP
SD
spoon_on towel
0.12
0.11
0.10
0.10
0.05
0.00
0.00
0.03
0.12
carrot on plate
0.01
0.09
0.01
0.03
0.02
0.08
0.01
0.01
0.10
stack_cube
0.05
0.11
0.03
0.02
0.04
0.00
0.03
0.08
0.02
out_eggplant_ing basket
bakket
0.38
0.60
0.50
0.28
0.53
0.26
0.1
0.36
0.38
Table XV: Success rate breakdown for SimplerEny (WidowX Bridge).
28


Reviews:
[{"review_id": "Ljn0ZKXcQq", "reviewer": "Reviewer_XhXW", "summary": "The paper proposes a training/evaluation-free proxy for selecting visual backbones for robot manipulation. It includes regressing a unified simulator state from images using the frozen backbone + MLP, then reranking backbones by this state prediction score. They show the proxy correlates strongly with policy success across 3 simulation benchmarks, and provide a real-world validation. The method is computationally cheaper than baselines, and improves policy success when used as an auxiliary objective.", "questions": "See point 3, 4 and 5 above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The method is simple and efficient. The predicted state covers information about object and scene-level variables, finally providing a single score to policy performance.\n\n* The paper evaluates across a breadth of envs, including 3 simulation envs and real-world evaluation on two tasks.\n\n* The paper uses a strong set of baselines, including few-shot, action MSE, Depth, etc.", "weaknesses": "1) The method relies on privileged information from the simulator (state + 2D object boxes) which can not be made available in the real world. It is not clear to me from the current set of experiments if the rankings would correlated if the real world env is substantially different from the simulated environment where the data is collected.\n\n2) As tasks get more complicated, the number of state variables to track will keep on increasing. For example, if the task requires picking up objects of different categories, it would require more variables to track.\n\n3) The authors should compare the real world performance of their approach with other visual backbone selection proxies. Some of these proxies (eg: Action MSE, Depth (can be obtained from the RealSense camera)) do not need privileged information from the simulator and therefore I suggest that the authors also report results when using the data from the real world.\n\n4) I would have liked to see more robotics specific visual encoders like VC-1, MVP [1], LIV [2], etc. They would have also added some diversity in terms of the dataset used in training of the vision encoder.\n\n5) The paper appears to use a baseline that closely matches the method proposed in prior work (SCR: Stable Control Representations [3]), but this connection is not acknowledged or cited; the authors should clarify this and include the appropriate reference.\n\n[1] Xiao, Tete, et al. \"Masked visual pre-training for motor control.\" arXiv preprint arXiv:2203.06173 (2022).\n\n[2] Ma, Yecheng Jason, et al. \"Liv: Language-image representations and rewards for robotic control.\" International Conference on Machine Learning. PMLR, 2023.\n\n[3] Gupta, Gunshi, et al. \"Pre-trained text-to-image diffusion models are versatile representation learners for control.\" Advances in Neural Information Processing Systems 37 (2024): 74182-74210.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pEnl7kcMOv", "reviewer": "Reviewer_ATcT", "summary": "This paper proposes state prediction from visual inputs as a fast, reliable proxy for evaluating visual representations in robotics. Instead of expensive policy rollouts, the authors measure how well a frozen encoder supports decoding of ground-truth environment state (geometry, object structure, and physical attributes). They show that state-prediction accuracy correlates strongly with downstream policy performance across multiple manipulation suites (MetaWorld, RoboCasa, SimplerEnv), is orders of magnitude faster than rollouts, and exhibits non-trivial sim-to-real correlation. The work is a simple, practical idea with useful empirical insights for representation selection and benchmarking.", "questions": "0. Could you discuss/explore more in-detail the first weakness I raised in the previous point?\n\n1. Real-world without GT state: What concrete proxy(s) would you propose—e.g., learned keypoints/pose tracking, depth recon, object state estimators—and have you piloted any?\n\n2.  Can you report correlations when decoding only task-critical subsets of state (e.g., end-effector pose + key object states) rather than the full state? This would help clarify whether the proxy is capturing the information actually needed for control, as opposed to benefiting from full-state regression capacity.\n\n3.  How does correlation change when using smaller decoders or shorter training? Understanding this would help confirm that the proxy reflects encoder quality rather than decoder strength, and that the metric remains reliable under lower-compute settings.\n\n4.  Sim-to-real outlier analysis: What explains the WidowX MAE anomaly? Can you isolate the culprit (visual domain, state scaling, or dynamics mismatch)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Strong empirical validation: Correlation holds across MetaWorld, RoboCasa, SimplerEnv with multiple seeds and error bars.\n\n- Computational efficiency: Good speedups over policy rollouts; actionable for everyday / larger-scale benchmarking.\n\n- Unified evaluation setup: Works across environments with a consistent state target.\n\n- Actionable insights: Per-dimension/attribute analyses give interpretability; sim-to-real correlation is encouraging.\n\n- Clarity: Problem framing and metrics are easy to implement and reproduce.", "weaknesses": "-Task diversity and modality coverage: While the results across manipulation benchmarks are compelling, the current evaluation is limited to manipulation-centric settings. Prior work (e.g., VC-1) has shown a form of multi-modality, where visual representations that excel in certain domains (e.g., R3M on MetaWorld) can perform poorly in others (e.g., navigation tasks in Habitat). This raises the question of whether the proposed proxy generalizes across task families with different perceptual and temporal demands (e.g., navigation, long-horizon multi-stage tasks, language-conditioned control). Including experiments or discussion around such cross-domain generalization—or clarifying the scope of applicability—would strengthen the claims and situate the method relative to known modality gaps in representation learning for robotics.\n\n- Simulator dependence: Ground-truth state access limits direct real-world deployment of the proxy; a concrete real-world surrogate (e.g., 6D pose from VLM/trackers) would elevate practicality.\n\n- Failure cases underexplored: Provide root-cause analysis for outliers (e.g., WidowX MAE)—is it state granularity, visuals, or embodiment mismatch?\n\n- Ablations on proxy design: How sensitive are correlations to the decoder capacity, training budget, or subset of state dimensions?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VuJ5OAgbMR", "reviewer": "Reviewer_vCpr", "summary": "Training robot policies, especially in the real world, are expensive.  These policies depend upon visual inputs, which generally need to be encoded before they can be fed as input to a policy network.  There are a wide variety of possible ways to encode pixels to inputs that would be useful for a policy network, and the authors aim to identify which visual encoders are most useful without having to train the expensive robot policy.\n\nThe authors posit that learning to predict underlying state representation of a robot scene from a visual representation of that scene is a good predictor how useful the resulting visual encoding is for learning control policies.  This visual encoding comes from a variety of pre-trained visual encoders like masked auto encoders, etc. As ground state information about a scene is hard to attain in the real world, the focus is on measuring this predictive effect in simulation.  \n\nThe author carefully formulate the learning problem of predicting the state from the visual simulator, with appropriate normalization and discretization applied, and details such as disambiguating which objects are targeted in the state vector by providing 2D bounding boxes for each object are handled in a systematic manner. \n\nThe process of learning to predict the state from a visual encoding is applied to a variety of different pre-trained visual learnt encoders including self-supervised, manipulation specific, and generative models. This state prediction metric is compared to a variety of alternative proxies like segmentation accuracy, etc.  The ranking via this proxy target is more accurate (Mean Maximum Rank Violation) in ranking how well the trained robotics policy will perform.  The state prediction proxy is  also more computationally efficientcompared to other proxies as learning to predict the state is a relatively low dimensional regression problem.\n\nThey then demonstrated that this ranking also transferred across the real to sim gap, and that the best visual representation in simulation also predicted best on robot policy learning performance.  They further explored which aspects of the state prediction best predicted which visualization would be best for a task.", "questions": "Have you considered mixtures of visual representations?  E.g., is it more powerful to just concatenate subsets of the representations?  Can you use your method to tell in advance which representations are likely to complement each other or make up for each others shortcomings?\n\nHow strong are the effects of initialization on how good the regression solution is?  Does the learning converge to the same point, and if not can you include error bars?\n\nPose might be highly ambiguous for many objects, e.g., cube have symmetry, vases have rotational symmetry.  Did you encounter such problems in the simulators?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "## Originality\nThe proposed metric is to my knowledge novel and useful.  It can also be a useful proxy when designing new visual encoders, as well as for quality control of the final result (and intermediate checkpoints during training), and potentially as additional auxillary loss during training.\n\n## Quality\nThe authors cast a wide net, and systematically explore the effects of many different visual representations in many robotics tasks.  They empirically demonstrate that their proxy is both more efficient and predictive than alternatives in the literature, and they demonstrate transfer to real robot tasks.\n\n## Clarity\nThe paper is clearly written, easy to follow with clear reasoning, and would be reproducible from the provided descriptions.\n\n## Significance\nBetter visual encoders are likely to help robotics, and approaches like those outlines in this paper may useful in developing that research.  It can be expanded to include other predictors like contact surfaces/other information from the simulation.", "weaknesses": "In general, I would expect that L2 losses on poses would have problems with wrapping causing large error.  I didn't notice in the paper where they tackled this potential problem.  E.g., a scene where the objects are aligned in a specific manner (lying on a table?) could have problems where close poses end up with very difference values per axis of the pose, throwing off the pose for in a set of tasks.  \n\nThe representation you can get out of a simulator will be a bit limited - it is unlikely to be useful in learning to fold a jumper as it is unlikely there will be a good \"symbolic\" target to score the outputs of the model against.  That to some degree limits the information you can get out of this proxy target, but it might still be sufficient for ranking the different representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MQUOVjaRJi", "reviewer": "Reviewer_oHEz", "summary": "The authors propose a new proxy for manipulation downstream policy success by measuring how well visual encoder pretrained backbones can predict the state of the environment (lighting conditions, robot arm joints and end effector pose, objects' positions, orientations, shapes, materials). Training such regressors is possible in simulation where ground truth state is available. For a plethora of backbone encoders, the proxy is shown to provide significantly better correlation to task performance than other methods both in simulation and on real world deployment. In addition, jointly fine-tuning visual backbones (5 different variants) for policy learning as well as state prediction is shown to  consistently offer clear improvements to policy performance on the MetaWorld benchmark.", "questions": "- Are all the tasks considered on parallel gripper pick and place tasks? \n\n- This circles back to points about what state attributes to consider, for tasks with end effector camera where the arm body is not visible how would you say things would differ, one can only predict general scene and object attributes in this situation. Would similar correlations hold in your opinion?\n\n- The other proxies seem very generic, are there no intermediate metrics or any works that provide proxies that are closer to capturing the state of the scene? \n\n- Along these lines, are there ways to combine the pre-existing metrics to see if some combination of them can challenge your approach?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well presented and thorough in situating the work within the current literature.\nThe manuscript is clear and enjoyable to read with a logical and consistent progression.\n\nThe solution presented to the proxy measure learning problem is simple and intuitive which makes it convincing. Its implementation in terms of encoding, multi-object handling, loss design and metric definition are all clear and logical.\n\nThe experimental protocol is thorough, at least for the base experiments looking into correlation quality, with many relevant benchmarks and visual encoder backbones evaluated both in simulation and in the real world. \n\nThe results appear to back the claims of the paper very elegantly.", "weaknesses": "**On the evaluation protocol:**\n- The authors state at line 302 that the protocol follows that of the SimplerEnv benchmark without any additional details or what part of the evaluation this entails, one suspects it is the averaged success rate measurements but that is not clear\n- Furthermore, the authors select the Mean Maximum Rank Violation (MMRV) metric as well as the Pearson correlation between performance and state prediction capacity as the statistical measures of correlation without any introduction, explanation or presentation of the what these metrics represent and why those and not others are utilized in the work. This leaves room for doubt as to whether the metric selection is cherry picked to amplify the paper results or objectively standard. This warrants more clarity as the choice is not explained in the main text, nor is it clarified in the appendix.\n\n**On the results:**\n- The success rates vary at most by about 15% percent between the best and worst models, with 3 out of four benchmarks having all models perform at low success rates. Each datapoint is obtained with 100 rollouts. Looking only at the MMRV and r numbers provided (and not explained as mentioned above) and average policy success rates with no standard deviations with different x axis scales, it is difficult to appreciate the statistical significance of the results.\n\n**On the analysis of individual state dimensions:**\n\nI thought this was a very nice ablation to have, yet it is not easy to decipher how it was conducted and what is interesting about the results. After some scrolling back and forth my understanding is that from the full state regressor you extract the specific predicted state entries and compute the MMRV score wrt performance. The idea being that the ones with the lowest would correlate the best with performance should they have been used on their own. In many ways this feels incomplete and perhaps the wrong question to ask:\n- The state dimensions are hand designed, and although they seem reasonable one could argue that the possibilities/variants might extend to other attributes. If the authors assume that training a regressor on the full state is the best approach, they should make a case that shows that each attribute or dimension in the state is actually contributing to correlation (at least for some backbone/benchmark combination)\n- Otherwise the authors should be looking for the best mix of attributes to consider for correlation and considering the couplings that arise both in the training of regressors as well as in establishing the correlation scores.\n- Indeed, with numbers varying quite aggressively across benchmarks for a single state dimension on its own, the only conclusion is that (l 458) \"that different environments indeed present different demands for visual representations\". But what about doing a leaving one out experiment for example to try to understand not what each brings on its own, but which are not that useful in the mix.\n- In summary, it is unclear whether all the attributes in the full state regressor are needed and how this affects the state regression quality should they be reduced/modified which can then couple back into the correlation quality.\n\n**Application demonstration asymmetry :**\n- First and foremost, though well written, the paper fails to my taste to clearly explain what are the uses of such proxies and why they are important tools. They might be cheaper to evaluate during architecture design or backbone selection for a policy but the case could be. made clearer as to their exact purpose.\n- The most interesting application beyond ranking seems to be joint fine-tuning which the authors show provides substantial performance gains across backbone models. This is a very interesting aspect of the work that is largely neglected compared to the potential interest for practitioners. This part of the work warrants more results and details.\n\nThe authors have omitted to include the use of LLMs statement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)