RobotArena $\infty$: Scalable Robot Benchmarking via Real-to-Sim Translation
Authors: Yash Jangir
Yidi Zhang
Kashu Yamazaki
Chenyu Zhang
Kuan-Hsun Tu
Tsung-Wei Ke
Lei Ke
Yonatan Bisk
Katerina Fragkiadaki
Venue: ICLR 2026 Poster (2026)
Keywords: Benchmarking
Robot Policy Evaluation
Real2Sim Translation
Vision Language Action Models
Primary area: datasets and benchmarks

Abstract: The pursuit of robot generalists, instructable agents capable of performing diverse tasks across diverse environments, demands rigorous and scalable evaluation. Yet real-world testing of robot policies remains fundamentally constrained: it is labor-intensive, slow, unsafe at scale, and difficult to reproduce. As policies expand in scope and complexity, these barriers only intensify, since defining ``success'' in robotics often hinges on nuanced human judgments of execution quality. We introduce RobotArenaInf, a new benchmarking framework that overcomes these challenges by shifting VLA evaluation into large-scale simulated environments augmented with online human feedback. Leveraging advances in vision-language models, 2D-to-3D generative modeling, and differentiable rendering, our approach automatically converts video demonstrations from widely used robot datasets into simulated counterparts. Within these digital twins, we assess VLA policies using both automated VLM-guided scoring and scalable human preference judgments collected from crowdworkers, transforming human involvement from tedious scene setup, resetting, and safety supervision into lightweight preference comparisons.
To measure robustness, we systematically perturb simulated environments along multiple axes, including textures and object placements, stress-testing policy generalization under controlled variation. The result is a continuously evolving, reproducible, and scalable benchmark for real-world-trained robot manipulation policies, addressing a critical missing capability in today’s robotics landscape. Benchmark website at \href{https://robotarenainf.github.io}{\texttt{robotarenainf.github.io}}.

ROBOTARENA 0o : SCALABLE ROBOT BENCHMARKING VIA REAL-TO-SIM TRANSLATION
Yash Jangir', Yidi Zhang's ; Kashu Yamazaki's Chenyu Zhang'; Kuan-Hsun Tu?, Tsung-Wer Ke², Lei Ke', Yonatan Bisk', Katerina Fragkiadaki 'Carnegie Mellon University, National Taiwan University
ABSTRACT
The pursuit of robot generalists, instructable agents capable of performing diverse tasks across diverse environments, demands rigorous and scalable evaluation. Yet real-world testing of robot policies remains fundamentally constrained: it is labor- intensive, slow, unsafe scale, and difficult reproduce. As policies expand in scope and complexity, these barriers only intensify, since defining "success" in robotics often hinges on nuanced human judgments of execution quality. We introduce RobotArena oc, a new benchmarking framework that overcomes these challenges by shifting VLA evaluation into large-scale simulated environments augmented with online human feedback. Leveraging advances in vision- language models, 2D-to-33 generative modeling, and differentiable rendering, our approach automatically converts video demonstrations from widely used robot datasets into simulated counterparts. Within these digital twins, we assess VLA policies using both automated VLM-guided scoring and scalable human preference judg- ments collected from crowdworkers, transforming human involvement from tedious scene setup, resetting, and safety supervision into lightweight preference compar- isons. To measure robustness, we systematically perturb simulated environments along multiple axes, including textures and object placements, stress-testing policy generalization under controlled variation. The result is a continuously evolving, reproducible, and scalable benchmark for hellwwrldd-trainee robot manipulation policies, addressing a critical missing capability in today's 's robotics landscape. Benchmark website lrobotarenaing wwweiinff.gg.u........... github.iiii
INTRODUCTION
While recent years have witnessed substantial progress in developing more capable and general robot policies, their evaluation remains a persistent challenge and lacks standardization. This problem becomes especially acute as policies grow more generalist, requiring broader and more diverse evaluation scenarios. Real-world evaluation inherently unscalable:
limited by logisties, safety concerns, and reproducibility issues, and requires significant human involvement for setup, execution, and scoring. Human operators must supervise trials and manually reset scenes, which restricts the scale and frequency of evaluations (Vincent al. 2024: Abou-Chakra et al.) 2025 Li et al. 2024c). Such manual oversight also raises concerns about consistency and faimess, particularly when baselines and new models compared under slightly different conditions.
Centralized physical evaluation provides a gold standard, where policies are tested under identical conditions, typically by submitting containers or shipping robots to shared testing site. Notable examples include the Amazon Picking Challenge Correll et al. (2016), the Open- Vocabulary Mobile Manipulation Challenge Yenamandra et al. (2023), and RoboCup Home Matamoros al. (2019). However, the high cost for both organizers and participants means such events occur infrequently. often no more than once year. In contrast, fields such as computer vision and natural language processing have advanced rapidly thanks to standardized benchmarks that provide consistent metrics, clear performance targets, and a foundation for fair comparison across methods Deng et al. (2009); Schuhmann et al. (2022). Our work is particularly inspired by LMarena Chiang al. (2024). a large- scale, crowdsourced evaluation framework that benchmarks LLMS and VLMs through direct pairwise
"Equal contribution

---

Large Scale Simulated Environments
Robote Datasets
Environmant Crcatiin ficm Real Video Evalutioon
Automanic Policy Evalastion
VIM
79.52
77.03
BROID
MJkk tie cop
78.38
77.50
Class sliding mander
Home the grece cay sight
yer shace
05.77
77.18
ROBOTARENA
Bridge V2
Fin the pab
Bau
Ponaaaation
Preferance Evalustion
Clan salarrg
IT
Fing DH
Che accont
Uler-Reconded Video
Faleedi Proticazze
Figure 1: RobotArena oo provides a scalable and extensible robot benchmarking framework by automating environment construction and evaluation. It automatically generates simulated environment seeded from real videos, deploys robot policies, and evaluates them using VL.Ms and crowdsourced workers that cast preferences between pairs of execution videos. The simulated environments are derived from both in-distribution and out-of-distribution videos, enabling rigorous tests of generalization in contemporary robot policies.
comparisons of responses the same prompt by human annotators. By aggregating thousands of such head-to-head matchups across diverse prompts, LMarena produces an Elo-style ranking that reflects collective judgments of model quality. Motivated by this success, ask: What would be the analogue of LMarena for Robotics?
RobotArena co: We introduce RobotArena Co, benchmarking framework that scales robot evaluation by deploying policies in automatically constructed simulated environments and assessing them through automatic VLM score and online human preference feedback. We first automatically translate real videos into corresponding simulation environments, building upon recent advances in rision-language models for scene understanding. 2D-to-3D generative models for asset creation. and differentiable rendering for camera pose estimation. We also introduce systematic environment perturbations, such as variations lighting, object placement, and background appearance. We then deploy VLAs in these environments and evaluate their execution trajectories using two complementary strategies: (1) absolute evaluation, in which prompted VLMs or crowdsourced human workers estimate task progress scores for each video frame, and (2) relative evaluation, in which human annotators cast pairwise preferences to execution videos from different robot policies performing the same task. We measure both in-distribution performance by testing simulation environments seeded from training videos in established datasets such as Bridge Walke et al. (2023), and out-of- distribution performance, by testing environments generated from videos outside the training set. Our initial benchmark aggregates more than 8500 preference pairs across one hundred nominal environments and hundreds of perturbations, comparing six VLAs from independent labs worldwide. To the best of our knowledge, this constitutes the largest-scale robot evaluation effort to date.
Our robot evaluations reveal several key insights. First, vision-language-action (VLA) models are highly sensitive to dataset differences: performance drops when they are tested in environments outside their training distribution, indicating that current models are not true generalists but instead specialize to the environments represented in their training data, consistent with findings in Xing jet al. (2255). Second, even within the same environment, performance degrades under perturbations, showing that robustness to distribution shifts remains open challenge. At the same time, model rankings are consistent across conditions, suggesting that architectural and data design choices lead to measurable and reproducible performance differences.
RobotArena 2o is inspired by prior efforts to design scalable robot benchmarks, particularly the seminal contributions of BEHAVIOR (Li et al. 2024) and SIMPLER (Li et al., 2024c). BEHAVIOR boasts an impressive manual effort of asset and environment creation, while SIMPLER reconstructs four real-world Bridge scenes and includes hand-designed reward functions. Compared to these ef-
2

---

forts, RobotArena oc offers a far more scalable and extensible framework by automating environment construction and evaluation.
In summary, contributions are follows:
1. We present a scalable and extensible benchmarking protocol for robotics, by coupling physics engines, real-to-sim translation and human preference feedback.
2. We introduce fully automated reality- -ommmulttion translation pipeline built upon VLMs, 2D-to-3D generative models and differentiable rendering.
3. We evaluate VLAs from labs worldwide across hundreds of environments with thou- sands of human preferences, the most extensive robot evaluation to date.
4. We present key evaluation results that reveal how current robot policies generalize- or fail vo-under distribution shifts.
Our benchmark is not without limitations. We outline these and discuss future directions and extensions. Importantly, RobotArena oo will continue to benefit from advances in physies engines and real-to-sim research. Both our benchmark environments and evaluation code will be publicly released and centrally maintained for continual support.
RELATED WORK
Evaluating Robot Policies in The Real World Evaluating robot policies in the real world in fair, comprehensive, and reproducible way remains a major challenge. Most methods are evaluated in custom lab-specific settings using proprictary hardware, task definitions, and success metries, which makes cross inosiinstitttion comparisons difficult. While standardized benchmarks exist for focused areas such as grasp prediction (Fang et al., 2020) and motion planning (Moll et al. 2014 Chamzas et al.) 2022), extending these to cover generalist skills remains complex. Standardization efforts using shared object sets (e.g.. YCB (Calli et al.) 2015), IKEA (Heo al., 2023), NIST (Kimble al. 2020), ACRV (Leitner | 2016)) help reduce some variability, but differences in hardware, camera placement, lighting, and workspace setup still hinder consistent evaluation across labs. Real-world evaluation is often time-consuee cone-conuuming and labor intensive. Human operators are typically required to supervise trials and manually reset scenes, limiting the scale and frequency of evaluations. For example, Chi et al. (2024) report manually aligning T-shaped object into 20 predefined start configurations for each rollout, a process repeated across all baselines. This reset step is non- parallelizable and presents a significant bottleneck when evaluating policies across different tasks, agents, and environments (Vincent et al.) 2024 Abbou-Chakra et al. 2025). While recent systems such as AutoEval (Zhou al. 2025) aim to automate evaluation, they are often limited in scope-e.gg c.g.. supporting only five tasks in three static real-world scenes. RoboArena Atreya et al. (2025) builds system of distribution real-worke evaluation where human users reset the scene, run the robot policies and evaluate the resulting robot executions. It targets scenes from the DROID environment and evaluates a set of policies finetuned on DROID. As real-world robot datasets and generalist policies continue to expand in scale and complexity, the need for more scalable, general, and continuously evolving evaluation framework becomes increasingly urgent.
Evaluating Robot Policies in Simulation Simulation offers scalable and safe alternative for evaluating robot policies, with numerous benchmarks like: RLBench James et al. (2020). Colos-
seum PPumacay et al.) (024), CALVIN (Mees et al. 2022). LIBERO (Liu al., (024a). Per- ACT2 (rrotz et al. (2024), Meta-World (Yu et al.) 2020). IKEA Simulation (Lee et al. 2019). and Behavior- 1K (Li et al.] 2024). However, these typically assume policies are trained and tested in the same simulated environments, potentially favoring specialist policies that exploit advantages of the closed-world environments over generalist models trained on real-worke or more general simulation data. Our approach, in contrast, uses simulation strictly an evaluation environment, independent of the policy's 's training origin. We contend that simulation is increasingly viable for policy evaluation due to improving physics engines and advancements in generative models and VLMs, which can auto- mate scene asset creation and task success detection. RobotArena oo leverages these by automatically generating diverse simulation -aase evaluation environments. The closest work, SIMPLER Li et al. (2024c), creates ligh-fidelity replicas of real scenes with human efforts, showing strong sim-to-real correlation. Unlike SIMPLER (Li et al. 2024c), RobotArena co automates both scene generation
3

---

Kabotic Datnsets
Semulatinn
Background Impaining
Deckarzesd
DRGED
dowr.
30
dnóge
the cometong
Post birimation
RYzzOn
Camer: Robot Pose Extimation
Carare Tame
Recorded Video
System Idcatilication
Kp Ki
Figure 2: Automated video-to-simulation translation in RobotArena oc. Given frame from video demonstration, we automatically create corresponding simulated environment.
and task evaluation across many more tasks and environments, and evaluates policy robustness and generalization through systematic perturbations.
Automated Real-to-Sim Translation Recent efforts real-to-sim translation, such as Phone2Proc Deitke et al. 2022), Scalable Real2Sim Pfaff al. (2025), and Re3Sim HHan al. (2025), have demonstrated high-fidelity asset generation and physics-aware reconstruction. However, these approaches typically rely multi-view captures, curated object libraries, the use of fiducial markers, making them impractical for reconstructing typical single-view robot datasets. contrast, RobotArena requires only single RGB image from static-camera demonstration. Our real-to- sim translation pipeline also differs from systems like RialTo Torne et al. (2024), which necessitates coman-in-the-logg segmentation specific calibration trajectories performed by the robot.
3 TRANSLATING VIDEOS TO SIMULATION FOR POLICY EVALUATION
RobotArena co automatically creates simula- tion environments in physics engines from video demonstrations, as shown Figure 2 Each robot demonstration is typically annotated with a language task description and per frame robot joint angle trajectories.
Fancadd
Z
Tepket
We also know the robot used and assume ac- cess to its URDF file. Our method extracts five key elements from the demonstration video: (1) the camera's 's 6- 6-DoF pose relative to the robot body frame, (2) 3D mesh reconstructions of task- relevant objects, (3) their orientations, sizes, and material properties, (4) a clean background im- age. (5) proportional-derivee Together, these components enable realistic, physics-consisten simulations derived directly from video, without requiring manual calibra- tion or curated annotations. The advantage of our modular design is long-term upgradability; each module can be replaced with stronger mod- els as real-to-sim technologies improve, contin- ually reducing error and improving benchmark fidelity.
Openning T
control gains.
Figure 3: Automated robot-camera calibra- tion through differentiable rendering of pose- conditioned 3D robot Gaussians.
4

---

Automated Robot-Camera Calibration through Differentiable Robot Rendering Robot demon- stration videos are typically uncalibrated, that is, the pose of the camera with respect to the robot's s frame is unknown. We estimate the sumera-to-robot transformation using differentiable rendering, shown in Figure [ Specifically, we first construct a joint angle-conditioned 3D Gaussian model of the robot via differentiable rendering in simulation based on its URDF file, following DR-Robot (Liu it al.
2024b), as shown in Figure Step Step I.
Given robot demonstration video annotated with per-frame joint angles, we then render the Gaussian robot model and optimize the camera's 's 3D translation and orientation to minimize a composite alignment loss with three terms: (i) an RGB loss penalizing pixel-level appearance differences, (ii) a flow loss enforcing consistency between rendered motion fields and optical flow from the video Karaev et al. (2024), and (iii) a feature loss aligning DINOv2 embeddings between rendered and observed frames. shown in Figure 3 Step 2. When calibration metadata is available (e.g.. from used for initialization; otherwise, we perform coarse grid search to provide robust starting point, as in V2 (Walke et al. 2023) and RoboMind (Wu ct al. 2024). More details on robot-camera calibration can be found in Appendix
DROID (Khazatsky al. 2024)),
Object and Scene 3D Reconstruction, Completion, and Physics Estimation We prompt Gem- ini TTam et al. (2023) to segment the robot and all task-relevant objects (Appendix C.I Each segmented image crop super-resolved with InvSR Yue al. (2024) and converted into textured 3D mesh using Hunyuan-3D (Team) (025). which, like most image-to-3D mesh generation models, reconstructs objects a canonical frame. To recover each object's correct 3D pose, we render 2D image views of the reconstructed 3D mesh and compare them against the 2D object crop using correspondence estimation from MINIMA (Ren et al. 2024). The view with the most feature matches is selected, and these correspondences are lifted to 3D using monocular depth estimate for the real image and simulated depth for the rendered view. Specifically, we first generate a relative depth map of the scene using MoGE (Wang et al.) 2024). We then derive metric scale factor by calibrating the robot arm's 's relative depth against its ground-truth depth from simulation, allowing us to unproject the masked pixels into an accurate metric-scale 3D point cloud. The final pose is then inferred via singular value decomposition (SVD) on the resulting 3D-3D correspondences (Appendix C.2). Physical and material properties for the objects are inferred by prompting Gemini and are then incorporated into the simulation environment.
To complete the scene, we generate a static background by inpainting the robot and object regions in the first video frame using the LaMa inpainting model [Suvorov et al. (202T), producing a clean backdrop for the reconstructed assets (Appendix DD. Finally, to accurately reproduce robot dynamics, we perform system identification to tune the proportional derivative (PD) controller gains, aligning simulated end-effector trajectories with those observed in the robot demonstration video (Appendix
E Apart from robot joint trajectory annotations obtained through teleoperation, our method requires no additional human supervision.
Controllable Domain Perturbations We introduce controlled perturbations to the generated environments in order to stress-test policy generalization under changes in background, scene arrangement, and color shifts. Specifically, we consider:
Background Change (ABG): Replaces the original scene background with different in- painted textures drawn from a diverse background dataset, isolating the policy's 's dependence on contextual appearance cues (Appendix G.I).
Color Shift (AColor): Alters the RGB channel configuration of the scene (c.g., converting RGB BGR), applied at intensities from 0% 100% in increments of approximately 33%, to test robustness against low-level color variation (Appendix G.2).
* Object Pose Change (AObjPose): Randomly permutes the location of objects in the scene (Appendix G.3
4 EVALUATING ROBOT TRAJECTORIES WITH HUMANS AND VLMS
We evaluate robot execution videos automatically, by prompting VLMs (Section 4. and through crowdsourcing human preferences (Section 4.2
5

---

4.1 EVALUATING ROBOT TASK PROGRESS SCORES WITH VLMs
Put the tomato pot
Task Prompt
Object State
Vision Language Model
24%
S2%
Figure 4: Left: Task progress scores computed by prompting Gemini 2.5 Pro with image frames and synchronized object and robot state sequences. Right: Example task progression. Top row shows execution frames; bottom row shows predicted progress scores.
We automate success detection and task progress evaluation at scale prompting proprietary VLMs using video and privileged simulation state information.
Given trajectory, the VLM is provided with the ordered sequence of video frames together with simulation state, including object states and robot states. The initial state explicitly included as a rero-progress reference. The model prompted assign a progress score each frame conditioned on both visual observations and state information (Figure(4). This multi-modal conditioning allows the evaluator to reason jointly about visual task completion, object configurations, and robot interactions.
To obtain a trajectory-level progress estimate, we compute the mean VLM score over the final 30% of frames in the trajectory. Intuitively, this focuses evaluation on the terminal phase of execution, where task completion (or failure) is most evident. We find that this metric aligns well with human-annotate progress scores and adopt for all subsequent analysis.
4.2 EVALUATING ROBOT PERFORMANCE WITH HUMAN PREFERENCE FEEDBACK
While automated scoring provides scalability, human preference feedback is essential capture nuanced aspects of robot behavior that numerical metrics may overlook. Our human evaluations are conducted through pairwise, double-blind comparisons of two policy execution videos obtained from the same simulation environment, under identical initial conditions and task instructions, following the protocol of Chiang et al.) (2024). For each comparison, evaluators provide two forms of feedback: (i) a preference label specifying which policy performed better overall or whether they are tied, and (ii) a free-form natural language explanation describing the rationale behind their choice. We use free-form explanations as a way to increase evaluator engagement and attention. We found that requiring written justification led to more accurate human annotations. We have included the web interface for pairwise comparisons in the Appendix and project page.
4.2.1 GLOBAL RANKING FROM PAIRWISE HUMAN PREFERENCES
We aim to compute a global policy ranking from pairwise human preferences. Let the set of N policies be II { { ... TN). and the dataset of pairwise comparisons be D, = {(P ( })}. where P PAA E (,,,0, 0,1 indicates preference for A over π, tie, or preference for πy over AA and f denotes the task on which the comparison was made. Our goal is derive global ranking R over policies, e.g., > TT > > Th. While ties are recorded for completeness, them from the ranking objective and use only decisive comparisons (#1) when fitting the model.
exclude
To aggregate pairwise judgments, we adopt the Bradley-Terry (BT) model Bradley & Terry 1952). ) standard probabilistic framework for inferring rankings from paired comparisons. The model assigns each policy latent ability score 0, 0 and defines the probability that ग, is preferred over rT as:
0:
P( > n,) 0, + 0
We estimate 0 (01....,0, O,v by maximizing the likelihood over all non-tied comparisons. This objective is concave in the log parameterization, allowing efficient optimization via standard gradient ascent. The resulting ability scores yield global ranking R. by simple sorting. To quantify uncertainty

---

DROIDSin
R120TSin
BridgeSim
Figure 5: Simulation environments RobotArena oo seeded from videos demonstrations in the datasets of Bridge, RH20T and DROID.
in the estimated rankings, compute confidence intervals for & using robust (sandwich) variance estimator. Additional details on the ranking procedure and uncertainty estimation are provided in Section
BENCHMARKING ROBOT POLICIES ROBOTARENA OO
Generated Simulation Arenas Our benchmark includes both in- and out-of-distribution environ- ments tasks for current VLAs, by selecting videos from within or outside their demonstration training sets, shown in Figure 5
BridgeSim contains simulated environments and tasks generated from robot demonstrations in the Bridge V2 dataset Walke et al. (2023), a widely used subset of the Open X-Embodiment (OXE) dataset Neill et al. (2024), frequently employed to pretrain generalist robot policies.
2. DROIDSim consists of environments and tasks created from demonstrations the DROID dataset Khazatsky et al. (2024). Unlike Bridge DROID is often excluded from pre- training pipelines for generalist policies due its higher noise levels Ma al. (2024).
3.
RH20TSim includes environments and tasks derived from the RH20T dataset Fang et al. (2023). Notably, among the candidate policies we evaluate, only SpatialVIL' backbone has been pre-trained this dataset.
Candidate Policies to Evaluate We benchmark the following open- robot policies, All policies operate with fixed egocentric camera and do not make use of wrist-mounted camera
Octo Octo Model Team et al. (2023) a transformer-basee manipulation policy pre-trained 800k demonstrations from the OXE dataset Neill et al. (2024). For our experiments, we evaluate the 93M- -arameter Octo-Bas model.
2. RoboVLM Li et al. 2024b) extends vision-language models (VLMs) into laniun-- action (VLA) policies by adding continuous action prediction head. We evaluate the variant built on the KosMos backbone Peng et al. (2023).
3. SpatialVLA Qu et al. (2025) augments standard VLAs with 3D spatial reasoning through Ego3D Position Encodings and Adaptive Action Grids. It is trained on I.I million real-world robot demonstrations.
4. CogAct Li et al. (024a) combines 7B- parameter VLM backbone with diffusion trans- former for action prediction. It is pre-trained on the large-scale OXE dataset (22.5M frames, 60 datasets, 22 robot embodiments) and further fine-tuned on smaller real-world datasets for embodiment-specifee adaptation.
For X-VLAA and To evaluation, we used the single view fine-tuned policy.
7

---

OpensPiZerm
RoboVLM Coguot
OpenP/ZZam
SpatsalVAA
Cogact
XVLA
SpatialVLA
XVLA
(%6)
Took Stoure (%6)
15
ScilgeSim
DeuiaSin
BH207Sii
BridgeSim
AOkyPosw
ACalee
ABG
(a)
(b)
Figure 7: Policy evaluation results obtained from VLMs (a) in all RobotArena environments and (b) in perturbations of BridgeSim environments.
5. X-VLA Zheng al. 2025 introduces scalable cross-embodiment VLA framework built on soft-prompted Transformer architecture. It uses embodiment-specinf learnable soft prompts while sharing a unified backbone trained with a flow-matching objective. We evaluate the 0.9B- --arammete X-VLA-0.9B model, pretrained on 290K heterogeneous episodes spanning seven hardware setups, and adapted using parameter-effictee finetuning.
6. To Black al. (0224) is ssiion-laggaage-action (VLA) foundation model built on pre-trained vision-language backbone (PaliGemma) and augmented with a flow-matching action expert for continuous control. We evaluate the open-source reimplementation, open-pi-zero allenzren (2025). which reproduces the architecture and training recipe.
5.1 ROBOT BENCHMARKING RESULTS IN ROBOTARENA DC
We show the VLA rankings derived from human pairwise preferences Figure 6 along with confidence intervals computed using robust sandwich variance estimator. We also show the automated VLM-derived evaluation results split across BridgeSim, DROIDSim, and RH20TSim in Figure (a) with Standard Error of the Mean (SEM). We also show the performance in per- turbed environments in Figure (b) with SEM for the BridgeSim for which the policies per- form the best. Human preferences align with the VLM task progress scores: Tu and X-VLA are preferred more often than Cogact, RobotVIMM Octo and SpatialVI.A. The human and VL.M rankings match exactly, indicating perfect agree- ment between automated and human evaluation.
RoboVLM
OpenPiZero
SpattelVIA
Cognet
1.5
1.1333
pelettvve Skill
L.
Figure 6: Human preference ranking of VLAS in BridgeSim environments from 8,749 pairwise comparisons.
draw the following conclusions from evaluations:
(1) Cross-dataset generalization is weak: Policies perform substantially worse in environments derived from datasets they were not trained on (c.g., DROID and RH20T), indicating current VLAs not true generalist.
(2) Model choice matters: The benchmarking results in BridgeSim and its perturbed variants clearly distinguish No and X-VLA the pop-perfornnng architectures. However, their performance is not universally superior. For example, in RH20TSim, RoboVLM (19.05%) achieves a substantially higher score than all other models, while X-VLA fails (0.00%).
(3) The "Spatial Paradox": To and X- VLA might have gained implicit 3D structure because their pre-training includes wrist-camera views. This suggests that the cross-view consistency learned from
8

---

multi-view data may provide robust spatial prior for manipulation than the current explicit 3D inductive biases used SpatialVLA.
(4) Backbone strength drives robustness: Policies with stronger VLM backbones more resilient to color perturbations. This suggests that these models rely more on invariant structural cues than on the superficial appearance features that distract less generalist models like Octo.
(5) Overfitting to specific task configurations: Performance across all evaluated policies degrades when backgrounds perturbed when object poses randomized. This decline indicates that current VLAs may overfit to the specific visual and spatial setups of their training datasets rather than achieving true semantic generalization. While 3D structural modeling provides some resilience to pose changes, the sensitivity to background suggests that policies still rely heavily on fixed environmental cues.
5.2 POLICY RANKING IN ROBOTARENA DO VERSUS THE REAL WORLD
Estimation of correlations between policy rankings RobotArena co and in the real world would require multi-task evaluations of robot policies in real-world environments that recreate the precise structures, textures and camera arrangements from environments RobotArena oo (that videos in Bridge, DROID or RH20T), task very difficult to execute accurately. free of human biases. In fact, precisely real world policy evaluation that our framework attempts to replace. However, we did test such correlation for one task, "Pul the carrot in the plate", ", by recreating the same scenes in simulation and the real world and deploying RoboVLM, Octo, and SpatialVLA both, RoboVLM and Spatial VLA succeeded in both real and simulated settings. while Octo failed in both, consistently attempting but failing to grasp the carrot, as shown in Figure
Octo SpatwwV1A
BubeVLM
OpenPiZeco
XAIAA
90
Thas Progresses Scor
Teak Progression Score
67/00
69
3D
20000
29,57 20y10
StmplerSStm4 Limss)
Btmdge
Figure 8: Comparison of Simulation-
Figure 9: Policy evaluation results in RobotArena oo Based Against beal-eerldd versus SIMPLER Li (2024c).
Robot Evaluations using Ma et al.
(2024)
5.3
ROBOTARENA DO VERSUS SIMPLER OF LI AL. (004c
In Figure compare the performance of multiple VLAs
reproduction of the four envi- ronments from the SIMPLER benchmark Li et al. (2024c), which are designed to approximate the conditions of the Bridge dataset, and our BridgeSim benchmark, which contains 70 environments derived from the same Bridge V2 dataset. All VLAs achieve substantially higher scores on SIMPLER than on BridgeSim. This discrepancy suggests that evaluation on SIMPLER may overestimate policy performance due to the very limited number of test environments and the biased selection of scenarios. In contrast, BridgeSim's broader coverage and increased diversity make it a more rigorous and reliable benchmark for evaluating current and future generalist robot policies.
6 LIMITATIONS AND FUTURE DIRECTIONS
By leveraging recent advances in reality- to to-simulation translation and crowdsoureed evaluation, RobotArena 0o provides a scalable and extensible robot benchmark. Our evaluators are not domain experts robotics researchers, but everyday end-users- the very audience that robots ultimately

---

intended to serve. The current benchmark has two main limitations. First, the policies evaluated do not yet incorporate wrist-camera inputs, which restricts the fidelity of certain manipulations. We are actively extending our reality-tosiimulatio pipeline to generate complete 3D interactive environments that will support multi-view observations. Second, current simulators still struggle to model fine-grained contact dynamies, such as inserting charger into socket. Despite progress in physics engines and automated asset generation Wang et al. (2025); Narang et al. (2022), these tasks remain difficult reproduce faithfully, highlighting crucial direction for future research. Looking forward, we believe RobotArena oo is well positioned to benefit from advances in simulation, physics engines, and environment generation, and to serve as a continually improving platform for evaluating the next generation of robotic foundation models.
7 CONCLUSION
We introduced RobotArena oc, a new benchmarking framework that scales robot evaluation through automated reality-tossimulatio translation and online human preference feedback. By coupling advances in languae models, 2D-to-3D generative models, and differentiable rendering, our framework enables the automatic construction of simulated environments directly from real-worl world videos. We demonstrated its utility by evaluating multiple VLAs across hundreds of environments and over 8500 human preference judgments, yielding the most extensive study of policy robustness and generalization to date. Our findings highlight significant lack of generalization across benchmarks, sensitivity perturbations, and systematic differences across prchitettures-revalllin both the limitations of current VLAs and the promise of scalable evaluation protocols. Looking ahead, we envision RobotArena oo continually evolving benchmark that grows alongside advances in robot learning. Future directions include expanding the range of tasks, incorporating more diverse real- world data sources, and leveraging improvements in physics engines and real-to-sim translation. By releasing both the benchmark environments and evaluation code, we aim to provide the community with an open, extensible platform for rigorous policy evaluation.
ACKNOWLEDGEMENT
We would like to acknowledge Pang-Chi (Sean) Lo for his continuous support in integrating VLA models into the benchmark and improving the environments. Due to ICLR's policy on adding new were unable to include him co-author. We would like to thank He
authors after submission, Zhu for providing the VLM prompt that we used in our VLM-derived policy evaluations. We would also like to thank Ayush Jain for general discussion regarding scene segmentation in real-to-sim pipeline. This work is funded by an Amazon AGI gift, SAFRON DARPA award HR001 1-25-33-553-,, an NSF Career award, and AFOSR Grant F-9550-23-1102257
REFERENCES
Jad Abou- Chakra, Lingfeng Sun, Krishan Rana, Brandon May, Karl Schmeckpeper, Maria Vittoria Minniti, and Laura Herlant. Real-is-sim: Bridging the sim-to-real gap with a dynamic digital twin for real-world robot policy evaluation, 2025.
03597
allenzren. open-pi-zero: Re-implementation of vision-languuag-actiin model. htttss:/ zero 2025. MIT License; open-source re-implementation of
model described in Black al. (2025).
Pranav Atreya, Karl Pertsch, Tony Lee, Moo Jin Kim, Arhan Jain, Artur Kuramshin, Clemens Eppner, Cyrus Neary, Edward Hu, Fabio Ramos, Jonathan Tremblay, Kanav Arora, Kirsty Ellis, Luca Macesanu. Matthew Leonard, Meedeum Cho. Ozgur Aslan, Shivin Dass, Jie Wang. Xingfang Yuan, Xuning Yang, Abhishek Gupta, Dinesh Jayaraman, Glen Berseth, Kostas Daniilidis, Roberto Martin-Martin Youngwoon Lee, Percy Liang, Chelsea Finn, and Sergey Levine Roboarena: Distributed real-world evaluation of generalist robot policies, 2025. URL httpt:: //arxiv/
htgg/bss/20 /206. 220..11166..1. 18123
Genesis Authors. Genesis: generative and universal physics engine for robotics and beyond, December 2024. URL httttttsss:// htt.httts:///tthheee wtt.://.ccmm/eeeeeeee. iwwrenness--mmoddieeeee AdTT:eenest
10

---

Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, et al. To: A vision-lange leroonllnngugg-aattn flow model for general robot control. arXiv preprint arXiv:2410.2466, 2024.
Ralph Allan Bradley and Milton E. Terry Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39:324, 1952. URL httpt://appii http://pp..senntticccccccccccece
orq/CorpusID: 125209803
Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M. Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. In 2015 International Conference Advanced Robotics (ICAR), pp. 510-517, 2015, doi: 10.1109/ICA..
2015.7251504.
Constantinos Chamzas, Carlos Quintero-Pena. Zachary Kingston, Andreas Orthey, Daniel Rakita, Michael Gleicher, Mare Toussaint, and Lydia E. Kavraki. Motionbenchmaker: A tool to generate and benchmark motion planning datasets. IEEE Robotics and Automation Letters, 7(2): April 2022. ISSN 2377-3774. doi: 10. 10.1 2..1100090aa222233333336.. URL http: //t. doi. org/
10. 16.1109/LRA 2021.3133603
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffiusion policy: Visuomotor policy learning via action diffusion, 2024. URL
Wei-Lin Chiang, Lianmin Zheng. Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating Ilms by human preference, 2024. URL /tLlattts///axx...tt
abs abs/2403. 04132
Nikolaus Correll, Kostas E Bekris, Dmitry Berenson, Oliver Brock, Albert Causo, Kris Hauser, Kei Okada, Alberto Rodriguez, Joseph M Romano, and Peter R Wurman. Analysis and obser- vations from the first amazon picking challenge. IEEE Trunsactions Automation Science and Engineering, 15(1):172-188 2016.
Matt Deitke, Rose Hendrix, Luca Weihs, Ali Farhadi, Kiana Ehsani, and Aniruddha Kemb- havi. Phone2Proe: Bringing Robust Robots Into Our Chaotic World. arXiv e-prints, art. arXiv:2212.04819, December 2022. doi: 10448500arXx..2222..4933
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Hao-Shu Fang, Chenxi Wang. Minghao Gou, and Cewu Lu. Graspnet- Ibillion: A large-scale benchmark for general object grasping. In 2020 IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 11441-11451 11450, 2020. doi: 10.1109/CYPR44400.2200..000 146.
Hao-Shu Fang, Hongjie Fang, Zhenyu Tang, Jirong Liu, Chenxi Wang, Junbo Wang, Haoyi Zhu, and Cewu Lu. Rh20t: A comprehensive robotic dataset for learning diverse skills in one-shot, 2023. wwtp::iww.rrgaab/////aa/ URL htt.:tppp///axll..ocr/////222... 00595
Markus Grotz, Mohit Shridhar, Tamim Asfour, and Dieter Fox. Peract2: A perceiver actor framework for bimanual manipulation tasks. arXiv preprint arXiv:2407.00277 2024.
Xiaoshen Han, Minghuan Liu, Yilun Chen, Junqiu Yu, Xiaoyang Lyu, Yang Tian, Bolun Wang, Weinan Zhang, and Jiangmiao Pang. Sem: Generating High-Fidelity Simulation Data via 3D-Photorealistic Real-to-Sim for Robotic Manipulation. arXiv e-prints, art. arXiv:2502.08645, February 2025. doi: 20.550/arii..252.0002
Minho Heo, Youngwoon Lee, Doohyun Lee, and Joseph J. Lim. Furniturebench: Reproducible real-world benchmark for long-horizon complex manipulation, 2023. URL wtt.hhtppst./.tt..e
/00b//2305.2 12821
Stephen James, Zicong Ma. David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):301-- 3026, 2020.
11

---

Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos. arXiv e-prints, art. arXiv:2410.11831, October 2024. doi: 10.48550kaxXw.2440.11555
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945 2024.
Kenneth Kimble, Karl Van Wyk, Joe Falco, Elena Messina, Yu Sun, Mizuho Shibata, Wataru Uemura, and Yasuyoshi Yokokohji. Benchmarking protocols for evaluating small parts robotic assembly systems. IEEE Robotics and Automation Letters, 5(2):883-888) 889, 2020. doi: 10.1 109/LRA.2020. 2965869.
Youngwoon Lee, Edward S. Hu, Zhengyu Yang, Alex Yin, and Joseph J. Lim. Ikea furniture assembly environment for long-horizon complex manipulation tasks, 2019. URL http//ttt////iiiiii
0ts//1911.0720
Jürgen Leitner. Adam W. Tow, Jake E. Dean, Niko Suenderhauf, Joseph W. Durham, Matthew Cooper, Markus Eich, Christopher Lehnert, Ruben Mangels, Christopher McCool, Peter Kujala, Lachlan Nicholson, Trung Pham, James Sergeant, I Liao Wu, Fangyi Zhang, Ben Upcroft, and Peter Corke. The acrv picking benchmark (apb): robotic shelf picking benchmark to foster reproducible research, 2016. Rttnhtts:/// arxiv org absp://b///a 1609. 05258
Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín Martín, Chen Wang. Gabrael Levine, Wensi Ai, Benjamin Martinez, et al. Behavior- Il: A human-centered, embodied ai benchmark with 1,000 everyday activities and realistic simulation. arXiv preprint arXiv:2403.0m 09227, 2024.
Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, et al. Cogact: foundational vision-language-acttoo model for synergizing cognition and action in robotic manipulation. arXiv preprint 1rXiv:2411. 19650, 2024a.
Xinghang Li, Peiyan Li, Minghuan Liu, Dong Wang, Jirong Liu, Bingyi Kang, Xiao Ma, Tao Kong, Hanbo Zhang, and Huaping Liu. Towards Generalist Robot Policies: What Matters in Building Aision-Languagg-AAe Actinn Models. arXiv e-prints, art. arXiv:2412.14053 December 2024b. doi: 10.48550/axXiv22412.444
Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, al. Evaluating real-world robot manipulation policies in simulation. arXiv preprint 4rXiv:2405.05941, 2024c.
Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36, 2024a.
Ruoshi Liu, Alper Canberk, Shuran Song, and Carl Vondrick. Differentiable Robot Rendering. arXiv e-prints, art, arXiv:2410.138511 October 2024b. doi: (0-445500/axX1.2211.1335
Yecheng Jason Ma, Joey Hejna, Ayzaan Wahid, Chuyuan Fu, Dhruv Shah, Jacky Liang, Zhuo Xu, Sean Kirmani, Peng Xu, Danny Driess, Ted Xiao, Jonathan Tompson, Osbert Bastani, Dinesh Jayaraman, Wenhao Yu, Tingnan Zhang, Dorsa Sadigh, and Fei Xia. Vision language models in-context value learners, 2024. hRLhttps: http///aavvvvvvv wttpa////raa/aaa/.aaoooo..o..s 2411 04549
Mauricio Matamoros, SEIB Viktor, and Dietrich Paulus. Trends, challenges and adopted strategies in robocup@ home. In 2019 IEEE International Conference Autonomous Robot Systems and Competitions (ICARSC), pp. 1-6. IEEE, 2019.
Oier Mees, Lukas Hermann, Erick Rosete- Cosete-Beas, and Wolfram Burgard. Calvin: A benchmark for language-conditionn policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters (RA-L.. 73)7732777334, 2022.
12

---

Mark Moll, Ioan Alexandru Sucan, and Lydia E. Kavraki. Benchmarking motion planning algorithms: An extensible infrastructure for analysis and visualization. IEEE Robotics & Automation Magazine, 22:96-102, 2014. htthttps: /api www.nticschoaaaat...eeo org/ Corpus htt.sschhh...c.occoooooooooo.al.
Yashraj Narang, Kier Storey, Iretiayo Akinola, Miles Macklin, Philipp Reist, Lukasz Wawrzyniak, Yunrong Guo, Adam Moravanszky, Gavriel State, Michelle Lu, Ankur Handa, and Dieter Fox. Factory: Fast contact for robotic assembly, 2022. URL https: htappp:///axiv..../ org abs /205 03532
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. 2023.
Abby O'Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x- x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 6892-6903. IEEE, 2024.
Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding Multimodal Large Language Models to the World. arXiv e-prints, art. arXiv:2306. 14824, June 2023. doi: 10.48550/arXiv.2306..4444 14824.
Nicholas Pfaff, Evelyn Fu, Jeremy Binagia, Phillip Isola, and Russ Tedrake. Scalable Real2Sim: Physics-Aware Asset Generation Via Robotic Pick-and-Place Setups. arXiv e-prints, art. arXiv:2503.00370, March 2025. doi: 00445500arxi.22500.0077..
Wilbert Pumacay, Ishika Singh, Jiafei Duan, Ranjay Krishna, Jesse Thomason, and Dieter Fox. THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation. arXiv
e-prints, art. arXiv:2402.0819) February 2024. doi: 10.445550a2Xx.22422.01999
Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Yc, Yan Ding, Zhigang Wang. Jia Yuan Gu, Bin Zhao, Dong Wang, and Xuelong Li. SpatialVLA: Exploring Spatial Representations for Visual-Language-Aette Action Model. arXiv e-prints, art. arXiv:2501.15830, January 2025. doi: 10.48550/arXi..5511.5555
Jiangwei Ren, Xingyu Jiang, Zizhuo Li, Dingkang Liang, Xin Zhou, and Xiang Bai. MINIMA: Modality Invariant Image Matching, arXiv e-prints, art. arXiv:2412.19412, December 2024. doi: 10.48550/arXx.224121142..
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprins arXiv:2210.08402. 2022.
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha. Aleksei Silvestrov, Nacjin Kong. Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution- robust large mask inpainting with fourier convolutions. arXiv preprint arXiv:2109.077 009900066,,, 2021.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11880 2023.
Tencent Hunyuan3D Team. Hunyuan3d 2.0: Scaling diffusion models for high resolution textured 3d assets generation, 2025.
Marcel Torne, Anthony Simeonov, Zechu Li, April Chan, Tao Chen, Abhishek Gupta, and Pulkit Agrawal. Reconciling Reality through Simulation: A tialttt-Sim-to-Peee Approach for Robust Manipulation. arXiv e-prints, art, arXiv:2403.0348 003499, March 2024. doi: 10.48550/arXiv.24403 03949.
Joseph A. Vincent, Haruki Nishimura, Masha Itkina, Paarth Shah, Mac Schwager, and Thomas Kollar. How generalizable my behavior cloning policy? a statistical approach to trustworthy performance evaluation, 2024. URL Rttthhttss///aatt org abs/2405. 05439
13

---

Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Estrucc, Quan Vuong, Andre He, Vivek Myers, Kuan Fang. Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning scale. In Conference Robot Learning (CoRL.), 2023.
Ruicheng Wang, Sicheng Xu, Cassie Dai, Jianfeng Xiang, Yu Deng, Xin Tong, and Jiaolong Yang. MoGe: Unlocking Accurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervision. arXiv e-prints, art. arXiv:2410.19115, October 2024. doi: 10.
2A550VarXiv24100111155
Yian Wang, Bingjie Tang, Chuang Gan, Dieter Fox, Kaichun Mo, Yashraj Narang, and Iretiayo Akinola. Matchmaker: Automated asset generation for robotic assembly, 2025. URL https:
//axxiv.orpp aba/a/a///2222 05887
Kun Wu, Chengkai Hou, Jiaming Liu, Zhengping Che, Xiaozhu Ju, Zhuqin Yang, Meng Li, Yinuo Zhao, Zhiyuan Xu, Guang Yang. al. Robomind: Benchmark multi-embodiment intelligence normative data for robot manipulation. arXiv preprint arXiv:2412.13877, 2024.
Youguang Xing, Xu Luo, Junlin Xie, Lianli Gao, Hengtao Shen, and Jingkuan Song. Shortcut learning in generalist robot policies: The role dataset diversity and fragmentation, 2025. URL
https://arxiv org/ aws abs/
2508 06426
Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin Wang, Mukul Khanna, Theophile Gervet, Tsung- Yen Yang. Vidhi Jain, Alexander William Clegg. John Turner, et al. Homerobot: Open-vocabulary mobile manipulation. arXiv preprini arXiv:2306. /1565, 2023.
Tianhe Yu, Deirdre Quillen, Zhanpeng Hc, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning. pp. 1094 110.. PMLR, 2020.
Zongsheng Yue, Kang Liao, and Chen Change Loy. Arbitrary-steps image super-resolution via diffusion inversion. arXiv preprint arXiv:2412.0900 2024.
Jinliang Zheng, Jianxiong Li, Zhihao Wang, Dongxiu Liu, Xirui Kang, Yuchun Feng, Yinan Zheng, Jiayin Zou, Yilun Chen, Jia Zeng, et al. X-vla: Soft-prompted transformer as scalable cross- embodiment Dision-languagc-accton model. arXiv preprint arXiv:2510.10277 2025.
Zhiyuan Zhou, Pranav Atreya, You Liang Tan, Karl Pertsch, and Sergey Levine. AutoEval: Au- tonomous Evaluation of Generalist Robot Manipulation Policies in the Real World. arXiv e-prints, art. arXiv:2503.2422 March 2025. doi: 25. 24278..
14

---

A WEBSITE AND CODE
The project website is available at httpp://rroootrrraaaa...g..... io and the source code be accessed http:////gghhiiooooooooaaa//////
B INFERRING CAMERA-ROBOT POSE TRANSFORMATION VIA ANALYSIS-BY- BY--YYHESIITSSS
In this optimization problem, we work with a sequence of video frames capturing the robot's motion, denoted ! where / ranges from I to T. We also have the robot's 's joint angles g' at each time t, provided by the dataset. Our task is to estimate the transformation T E SE(3), which aligns the robot's coordinate system with the camera's. s. Using T and q', we produce rendered image 4, a synthetic depiction of the robot at time t. Motion is captured through optical flow E.P estimated between consecutive video frames I and r 1. and rendered flow F, Additionally, we use the DINOv2 model to extract feature maps of DINOv2(4 from the video frames and Φ = DINOv2(4,) from the rendered images, enhancing alignment with high-level image features.
The optimization aligns the rendered and observed data using three loss terms:
1. RGB Loss: Measures the color difference between the rendered and video images each
time f:
(t) | 1
12
2. Flow Loss: Compares the rendered flow the optical flow for each pair of frames from to 1:
D F| r
3. Feature Loss: Aligns the feature maps using cosine loss at each time t:
S cos (pe, oz²)
(3)
The total loss combines these terms the sequence:
C (tg +
feart
Xllw C flow
(4)
where Xygb, Agent and Dseow are weights balancing each term. The optimal T is found by minimizing:
T argmin TESE(3)
C RELEVANT OBJECT SEGMENTATION, 3D RECONSTRUCTION, AND MATERIAL PROPERTY ESTIMATION
The creation of simulation-ready 3D assets for movable objects is accomplished through multistage pipeline. This process begins with semantic understanding of the scene using a vision language model (VLM), specifically Gemini Team et al. (2023). We prompt Gemini segment the robot and all foreground objects it interacts with. The resulting segmentation masks provide fine-grained object localization. To enhance visual details critical for reconstruction, the original image is first super-resolved, then resized to its original dimensions. The segmentation masks are applied to isolate and crop each object of interest. These object-specific image patches are further refined using second stage of super-resolution with InvSR Yue et al. (2024), producing high-fidelity inputs for 3D reconstruction. The enhanced patches are then processed by an image-to-3D model, Hunyuan-3D Team 2025), which outputs detailed meshes capturing both the geometry and texture of each object. To ensure physical plausibility within simulation, Gemini Team et al. (2023) is prompted to infer object-specific physical parameters such mass and friction coefficients. Final asset preparation includes appropriate scaling and placement within the simulated environment to ensure consistent alignment with the original scene configuration.
15

---

This section describes the method of recovering the 3D position and orientation of objects from single RGB image. The approach leverages camera parameters recovered by our previously described method, detailed in Section B and an initial depth map estimated using (MoGE Wang et al. (2024)..
C.I OBJECT SEGMENTATION
To obtain accurate localization of task-relevant entities, employ Gemini Team al. 2023 with three structured prompts tailored for different goals.
Robot Segmentation Prompt We first segment the robot, as jointly detecting all entities may result in missed or inaccurate detection of the robot. The prompt below is used to obtain the robot's segmentation mask:
Give segmentation masks for the robot in the scene. Output JSON list of segmentation masks where each entry contains the 2D bounding box in "box_2d" a descriptive text label "label", and the mask in "mask".
Foreground Object Segmentation Prompt After isolating the robot, we proceed to segment the foreground objects interacts with, along with estimating their physical properties. The following prompt is used to obtain segmentation masks and associated physical attributes:
Give segmentation masks for all important complete foreground objects on the plane which the robot interacts with. You must ignore any background objects, irrelevant surfaces, or any objects that are occluded, covered, or severely blocked by other objects.
Output a JSON list of segmentation masks where each entry contains:
"box__dd 2d" the bounding box
"mask": the segmentation mask (in image format)
"label": descriptive text label
"mass": estimated object mass in kilograms (float)
"friction": estimated friction coefficient (float) purface__ype"" of [Glass, Water, Emission, Plastic, Rough, Smooth, Reflective, Metal, Iron, Aluminium, Copper, Gold]
Task-Relevant Object Selection Prompt (conditional) If task description provided, use the following prompt to analyze the previously segmented objects, identifying those with seman- tic relevance the task and determining their functional roles (e.g., as a manipulation targer or destination):
Consider this specific task: [task), J. given the object names: nnmmect please select the object names that are relevant the task and identify their role in the task from "target" and "destination".
Output in this format exactly as: target: <target.object"" destination: Please try to identify the target object (If you can't 't find an exact match, use the closest one from given object names) and if there destination object, only output the target object.
ietisiinationoobjcctt
For example, given the task: put the pot on top of the yellow cloth and the segmented object names: sauce pan, cloth, the output would be: target: sauce pan, destination: cloth
C.2 OBJECT POSE ESTIMATION
Initial 3D Point Cloud Reconstruction With the 2D mask of the target object generated using Gemini Team et al, (2023), we unproject each masked pixel into a 3D point cloud in world coordinates. This process uses the recovered camera intrinsic and extrinsic parameters K, R, t and metric-scale depth map Duu, v). To recover metric depth maps, we compute the scale factor by comparing MoGE's Wang al. 2024) relative depth estimates of the robot arm against the simulated robot arm's in ground-truen -ruth depth; applying this factor converts the relative predictions into accurately scaled metric depth maps.
16

---

For each pixel (u, within the mask, its world-space coordinate P world is then given by
P Pworld
R. Du, v) K [t, v, 1]T t.
Simulated Viewpoint Generation and 2D Correspondence Establishment To robustly estimate the object's pose, we generate multiple synthetic views. This process begins with the generated 3D mesh of the object. To ensure dimensional consistency for the simulation, the scale of this generated original point cloud. The
mesh is aligned by utilizing its bounding box and the bounding box scale-aligned mesh is then imported into our simulation environment and initialized at reference position. In this simulated environment, multiple views are rendered by a camera programmed to navigate around this central object position. The camera's trajectory is systematically defined by varying its elevation and azimuth angles relative to the reference position.
Let
Plevels = -22,, 12,,22..... 2L..
Ovalues = {81, 02/) No = M,
For each view index i E (0....,NN 1}. we compute:
0, Ovalues ( mod No];
(6)
2: = Zlevels [i//No]]
(7)
cos 0,
1
V: ccos0: + sin 80 + +
sin 0: 2:
(8)
p: Pref + d dvv,.
where Pref the target's centroid world coordinates and is the fixed radial distance. The simulated camera is then oriented such that its viewing direction is aimed at Pref-
For each camera pose Pi, we render synthetic image:
sim Render(p:).
We then apply the MINIMA algorithm Ren et al. (2024) to establish 2D keypoint correspondences between Fa and the original masked image. This produces, for each view i, set of M, matched keypoint pairs:
C, { (K k:ng)) 1--
where kij is the j-th keypoint Him and kid its corresponding keypoint in the original image.
Optimal View Selection and 3D Keypoint Pair Generation
We select the optimal view index
argmax JE(O....NN-1)
|Cal,
and denote the corresponding rendered image by Iim- The of matched keypoints for this view is
C₁- = [k simu k:rgg) }'m
M
The optimal 2D keypoint correspondences {K k ) Specifically, for each simulated keypoint kod compute its 3D coordinate via
LM
are then lifted into 3D by unprojection.
P sim = Unproject (k*), K
sims
R
Cam- D D2
(10)
where Kim Rimm Sim the intrinsic and extrinsic parameters of the known simulated camera and DJ sim is the depth rendered in the simulation.
Similarly, for each original image keypoint orig we obtain
kij
P orig.. = Unproject k K, R. t, D( ) origi
(1I)
17

---

Figure 10: Background Inpainting Results. The top row shows the original RGB images with task- relevant objects present; the bottom row shows the corresponding inpainted images after foreground object removal, where only backgrounds remain.
where K, R and t are the recovered camera intrinsic and extrinsic parameters, and D( "orig) is the aligned MoGE Wang et al. (2024) depth at pixel (Morig Vorig)
This yields a of M 3D-3D correspondences
= ((P Parig.)) )
M
Rigid Transformation Estimation using SVD Given the set of 3D-3D keypoint correspon- dences {(P Porig.s)) our goal is to estimate the rigid transformation (R, t) that best aligns these two point sets. This transformation consists of rotation matrix R SO(3) and translation vector t E R such that:
P Porig.j R.P Ri
We employ the Singular Value Decomposition (SVD) method for this estimation, which yields the rotation R and translation that map the 3D keypoints of the simulated object to the original point cloud and thus recover the position and orientation of the generated object.
D BACKGROUND INPAINTING
We generate a static background by inpainting the robot and object regions in the first video frame with LaMa Suvorov al. (202T). We visualize the results of background inpainting for different scenes in Bridge Dataset Walke et al. 2023) in Figure
E SYSTEM IDENTIFICATION
To improve the simulation fidelity of the robot's open-loop motion, we apply system identification (SysID) to tune the proportional -ddrivattveeevvviivie (PD) controller gains, kp and kd. The goal is minimize the discrepancy between simulated and real end-effector trajectories, using only the joint position commands input. For each trajectory in dataset, extract the end-effector pose sequence and run a physics- based simulation using the same joint commands. The simulated robot is controlled with PD controller, and the gains are optimized minimize the difference in end-effector positions over time.
kp' kd" argmin kp
Σ
| x, + aresin
R 2
Rullr))
(12)
where x²! and R denote the ground- truth end-effector position and orientation from the real world dataset, x, and R2 denote those of the simulated robot, and || denotes the Frobenius norm.
18

---

Note that exclude the gripper from the system identification process, its state in the dataset typically represented by a binary open/close signal, which does not support continuous control tuning.
To search for optimal parameters, we use a gradient-free Simulated Annealing (SA) strategy. each iteration, candidate set of gains is evaluated in parallel using the Genesis simulator Authors (2024). All environments are initialized simultaneously. and the same candidate gains are applied to each. At each step:
Inverse kinematics is solved in batch produce joint targets for each trajectory.
PD control is applied in each environment using the current candidate (kp, kd).
The mean Euclidean distance between the simulated and dataset end-effector positions is computed.
The SA policy perturbs the parameters and accepts new candidates based on the change in average error.
Gains optimized 5000 steps of SA. Each iteration involves simulating all trajectories for fixed number 5000 of steps (typically ). making the process feasible in parallel with GPU acceleration. We the following parameter ranges for the search: kp [2000, 15000], kd E [10, 2000].
visual comparison for Bridge Dataset Walke et al. 2023) before and after system identification is shown Figure [1] where the dataset (red) and simulated (blue) end-effector trajectories are overlaid on the XY planc.
before the idemiiittiionn
ofler the identifization
(c) before the syskem idemification
after the system inletificattinn
Figure Inferring robot control gains for matching robot trajectories in reality and simulation. The end-effector trajectory. projected onto the XY plane, is shown for both the pre-identi pre-identification (a.c) and dost-identification (b,d) stages. After system identification, the simulated trajectory (blue) aligns closely with the recorded dataset trajectory (red), whereas significant deviations are observed prior to identification.
F BridgeSim ENVIRONMENT VISUALIZATIONS
We present additional visualizations of results from our Real2Sim pipeline in Figure 12
G ENVIRONMENT PERTURBATIONS
To quantify policy robustness under realistic visual and spatial variability, we systematically introduce four controlled environment perturbations.
G.1 BACKGROUND CHANGE (ABG)
For each scene, we replace the original background with five different inpainted backgrounds generated by Reality-to-Simulate pipeline and keep all foreground objects exactly the same, that for every scene we have five 'background flipped' variants that differ only in their contextual appearance, illustrated in Figure
19

---

Figure 12: Visualizations of a subset of BridgeSim Environments
Figure 13: Visualizations of subset of DROIDSim Environments
G.2 COLOR SHIFT (ACOLOR)
For each scene, leave the foreground (objects, lighting, dynamics) completely unchanged and apply only low-level color perturbation to the background by remapping its RGB channels to BGR and blending back at four intensity levels. Concretely, for every background pixel with the original
20

---

Figure 14: Background Change Example. The top-left image shows the original image without background perturbations.
Figure 15: Color Shift Example. The lefumost image shows the original image without color perturbations.
color vector [Rt G, BJ, we compute:
R
B
C'(a) (1 (1)a)
G
+
G
B
R
where
(0.00, 0.33, 0.66, 1.00}
corresponds respectively 0%, 33%. 66% and 100% BGR swap intensity. This yields four "color-
swapped" background variants per scene, as illustrated in Figure I5
G.3 OBJECT POSE CHANGE (AORJPOSE)
For each scene containing N distinct assets with original world-space positions )N we perform N independent random permutations (x( 1.V (including the identity permutation for the original layout). For permutation k; we reassign asset the position of asset π (8) (i), yielding
(k)
(())
i == L.....,
This procedure generates N scene variants that differ only in object arrangement, illustrated in Figure [6
21

---

Figure 16: Object Position Perturbation Example. The top-left image shows the original setup without perturbation.
Froms Progression
Framme Task Comp
Brams
Figure 17: Example VLM-generated task evaluation curves on base environment. Left panels: Representative frames sampled at low- and progress points. Right panels: VLM-assigned completion score (y-axis) across frame ID (x-axis).
H AUTOMATED TASK PROGRESS SCORING WITH VLMS
To obtain dense, fine-grained task progress signals, we use structured evaluation prompt with large VLsion-L Model (VLM), specifically Gemini. The prompt provides: (i) the natural language task instruction, (ii) structured robot and object state information, and (iii) sequence of visual frames capturing the scene evolution. Concretely, we present both the initial and current scene states-ae each decomposed into visual observations, and action trajectory history. This multimodal formulation enables the VLM to jointly reason over visual evidence and structured state transitions when estimating task progress. The full prompt is provided below.
22

---

PROGRESS.EVAL.PROMP """You are an expert zoboticist. The robot performing the task of "(instruction)'.
Your task is to evaluate the robot's progress towards completing the overall task by analyzing the provided initial/current scene states and robot action trajectory.
***** Input *****
This is the initial state of the scene.
(1) Visual Observations:
(2) Object Layout:
finit.bbjettinnfomaa)
(3). Contact information.
{init contact information]
From the initial state, the robot has made progress represented by the following trajectory:
(bistory.information)
Such progress results in the current state of the scene.
(1) Visual Observations: sitrretttiiuall.beerreaaa
(2). Object Layout:
courrent.object. inforratifoo
Contact Information:
(eurrent contact information)
***** Output *****
Based on the above information, please do the following: ## Reflection
Evaluate the overall progress of the robet toward completing the by analyzing the trajectory and comparing initial and current scene state. Are the subgoals reasonable? Is the robot making meaningful progress tewards the final goal? ## Progress Score
Predict task completion percentage between and 100.
In the initial state, the task completion percentage is 0. If all necessary steps are completed, the task complction percentage is 100. The score should increase when meaningful subgeal is completed. The score should decrease if action undoes degrades previously completed meaningful subgoal. The score should also decrease if the robot's s action causes SEVERE damage to other objects in the environment (minor disturbance is acceptable and can be ignored). In this part, just provide sentence.
and
concise explaining
Make sure you use 'Hl Reflection' and 'll Progress Score' as section headers in your response. Flease use EXACTLY ONE number between 0 and 100 for the Progress Score.
We present the results for generative value learning based progressive score prediction in Figure 17
18
I VLM EVALUATIONS
In the RH20TSim environment, Octo achieves significantly higher VL.M score than the other models. Qualitative analysis reveals this is due to a minimalistic motion pattern; Octo just slowly lowers its arm directly onto the workbench. This simple trajectory succeeds coincidence, as the target object happens to be positioned in the arm's 's path. This shortcut solution scores higher than the baseline policies, which engage in more extensive but ultimately unsuccessful exploration.
HUMAN EVALUATIONS
To assess the reliability of our automated task success metrics, we conducted a validation study by comparing them against human judgments collected through study. We developed web-based
23

---

Figure 18: Example VLM LLM-generated task evaluation curves on perturbed environments. Top: A high-progress moment immediately after the object lift, for which the VLM predicts a completion score of approximately 60%. Bottom: An inconsequential action with no strong effect task progress, for which the VLM predicts low completion score of approximately 10%.
Score
frome Comp
Figure 19: Example VI.M-generaed task evaluation curves on perturbed environments. Top: A successful pick-and-place execution-aafe -ateer the object lift the VLM score climbs steadily and correctly shows task completion. Bottom: An unsuccessful trajectory with completion score remaining low, demonstrating the VLM's capacity to detect failure to complete the task.
24

---

interface that presented annotators with two side-by-side videos, each showing different policy attempting the same task described by a natural language instruction. For each pair, participants were first asked to provide descriptions of each robot's attempt and then judge which policy performed better or was a tie. An example of this interface is shown in Figure 20 To ensure high data quality, participants had to pass qualification quiz. by correctly evaluating at least 8 out of 10 video pairs. These pairs were intentionally chosen with clear performance differences, making them straightforward judge.
From a total pool of 70 environments, we collected 8749 pairwise comparisons on the Amazon Mechanical Turk (AMT) platform, with each participant assigned random subset evaluate. All participants were compensated according to the platform's 's guidelines, ensuring a diverse pool of evaluators and promoting reliable, unbiased annotations.
Welcome to Robot Evalustion Study
Whate Expect
teers eonrie ee Sarri mooe do iocuts
Study Overview
Thank hi opeiing now skoig lovalie tolp aknndue cheee pocces sbed aspuirrar tanu.
yeure agjor
uodecaa,
kotwns the poiny.
alteriing dcoorg clems, untove yeu des porfurmen therr pastar ralciate
Qudiiootton
The onllecter
besd as oulyy proceed
callly stngs ty pars rocy willee and can modes ecampe belw
provess
please Partcipet
Figure 20: Human evaluation interface. Participants viewed policies side-by-side and selected the one that performed better or if it was a tie.
GLOBAL RANKING FROM PAIRWISE PREFERENCES
Let IT {m...... n denote the set of policies, and let the dataset of pairwise preferences be
D, = [(PrA
where PRA,T E f-1, 1, 0, 1} indicates a preference for πA over TB (1), a tie (0), or TB over TA (-1), and identifies the task which the comparison was made.
We model the probability of each comparison using the Bradley-Terry (BT) model. Let 0; > be the latent ability, score, of policy n; For any pair of policies (Mz Ty); the probability that m; is preferred to πī is given by:
0:
P( > πT))
0, + 0
0j
P(m, > nz)
8, + 0
Maximum Likelihood Estimation. ing the likelihood of observed human preferences:
The parameters 0 (01,... [N0 are estimated by maximiz-
Σ [I[P., = log P(T, Ty) 1[P., -1]log P(T, T.)]
(
This objective is concave in the natural log- parameterization log 0,, and find the maximum efficiently using Newton Raphson optimization algorithm. We can use other iterative algorithms as well.
25

---

Confidence Intervals and Sandwich Variance. To quantify uncertainty in the estimated abilities, we can compute standard errors using robust (sandwich) estimator. Let B, = log 0g denote the log-ability of policy n; and 3 be the MLE.
The score function is:
ac Σ U(3)
(I[P,) = - > m1))) xij
0
((J)),;
where Xij is a vector with +1 at position i, -1 at j. and 0 elsewhere.
The observed Fisher information (negative Hessian) is:
82C
P(T, TT)(1 P(T, x nz)) x.)x (KJ)ED,
Σ
0303T
The outer product of the score vectors
S = Σ ( Unr(d) (1[Pr, = 1] P(T, > 1T;i8)=x X,g
(HJ)E,,
The robust (sandwich) covariance estimator is then given by:
V sandwich
. -H'S
Finally, to center the scores and remove the global offset (since adding constant to all 3r does not change relative preferences), we compute:
V A V sandwich AT, A IN N I-IN-
The (1 - a:) confidence interval for each policy score is:
CI, = 8r 21-0/2 Via
where 21 o/2 is the standard normal quantile.
Global Ranking. A global ranking R over policies be obtained by ordering the estimated ability scores 0, (or centered log-scores 3,). Approximate rankings can also account for uncertainty: if the confidence intervals of two policies do not overlap, one can confidently assert that one is preferred over the other according human evaluations.
K MEASURING CORRELATION BETWEEN VLM AND HUMAN SCORES
Image
Iaage: Robet State
Imxage Object Stute
6
Sexre
CT Seax Jazuat
thamaa
Figure 21: Measuring Correlation between VLM and Human scores.
Figure shows the correlation between VLM-derived scores and human ground-truth scores under four input settings: images only, robot state only, object state only, and combined robot and object states. Each point represents a trajectory, with the dashed red line indicating the regression and the gray diagonal representing perfect agreement. In all cases, VLM scores correlate positively with human evaluations, with stronger alignment when structured state information is provided, particularly when both robot and object states are available.
26

---

L ETHICS STATEMENT
This work complies with the ICLR Code of Ethics and established research integrity principles. No human subjects, personal, sensitive data were used for training or evaluating policies; all human involvement in the benchmarking framework is limited to online preference annotation of video content from robot simulation, with participants recruited and compensated through recognized platforms according to their guidelines. Annotator tasks were designed to be non-intrusive, requiring only high-level performance comparisons in a double-blind interface, and a qualification quiz was used to promote reliable, unbiased evaluations. Research artifacts, algorithms, and datasets are used compliance with their licenses, and no intellectual property or privacy concerns are raised by the experimental protocol. The study sought to anticipate and mitigate unfair outcomes and measured bias and robustness across diverse input sources, including systematic stress-tests for failure modes. Limitations concerning simulation realism and generalization beyond the studied benchmarks are seknowledged as discussed in the main text and the limitations section. The full evaluation framework and results will be released for broad, equitable access by the research community.
M REPRODUCIBILITY STATEMENT
Significant effort has been devoted to ensuring that all results in this paper reproducible. Details of the
translation pipeline, environment and object generation, evaluation method- ology (including VLM progress scoring, human preference infrastructure, and global policy ranking algorithm), as well as hyperparameter settings, are provided in the main text and in Appendices B-G. The codebase for environment construction, simulation, and scoring, along with evaluation logs and benchmark scenarios, will be anonymously released as supplementary materials. Source datasets, simulation assets, and parameter optimization procedures are fully documented in the Appendix and will be indexed for direct replication. Further, systematic perturbation experiments and robustness analyses can reproduced using the public release. The aim is to enable independent verification and extension of all primary claims and experimental findings using the open-source pipeline described in the manuscript.
N LLM USAGE STATEMENT
In the development of this manuscript, Large Language Models (LLMs) were used as general-purpor assistive tools for grammar checking, rewording, preparation of summary paragraphs, and technical editing. No sections of the manuscript, experimental results, or research claims were generated solely by an LLM. All original research content, analyses, and experiment design were authored by the stated researchers. When LLMs contributed to polishing language or format, the outputs were reviewed, edited, and verified for accuracy to ensure compliance with the ICLR Code of Ethics, integrity, and originality. The authors take full responsibility for all content presented here and confirm that no LLM --M-eneertted text constitutes plagiarism, fabrication, or any form of scientific misconduct. LLMs are not considered authors and have not contributed to research ideation, methodology, or core scientific findings
27


Reviews:
[{"review_id": "LJSLWnD5ZF", "reviewer": "Reviewer_i6zo", "summary": "This paper introduces a scalable framework for evaluating VLA policies in robotics. It automatically converts real-world robot demonstration videos into simulated environments using advances in VLMs, 2D-to-3D generative modeling, and differentiable rendering. The benchmark assesses robot policies through automated VLM-based scoring and human preference judgments, enabling large-scale, reproducible evaluations without manual setup. Experiments reveal limited cross-dataset generalization and robustness in current models, while confirming consistent performance rankings.", "questions": "Please see the first paragraph that I wrote above in the Weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "- The pipeline that transforms real-world robot demonstration videos into simulated environments is very nice and generally useful.\n- The study conducts, according to the authors, the most extensive evaluation of generalist robot policies to date.", "weaknesses": "Real-to-sim pipeline is very nice. But for example, how is it better than testing VLAs on a bunch of different simulation environments? That would also make sure to include some out-of-distribution domains, wouldn't it? And why is this work framed as a policy evaluation work? It looks like a real-to-sim method, and it deserves credit for that contribution (more for it than for policy evaluation, because real-to-sim is more general).\n\nI think the paper has a good potential, but the following improvements (or addressing these comments) would make it much stronger:\n- Wrist camera is a common input, but the current work is missing them as acknowledged by the paper.\n- Reliance on generative models may introduce reconstruction artifacts and physics inaccuracies.\n- Benchmark scope limited to manipulation tasks: no locomotion, mobile, or multi-agent settings.\n- Evaluation depends on a specific VLM backbone (Gemini). This harms accessibility. It would be nice to have open-source variants that are more accessible to research labs.\n- Domain perturbations are hand-designed and may not capture natural distribution shifts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EUBSQZu0L5", "reviewer": "Reviewer_kdpq", "summary": "In this work, the authors introduce a real-to-sim translation framework named RobotArena to tackle the challenges in real-world robot policy benchmarking with the following two aspects: 1) an automated pipeline for converting real-world videos into simulated environments via VLM-based scene understanding and differentiable rendering. 2) an evaluation protocol combining VLM-based progress scoring and crowdsourced human preference feedback. The authors conduct extensive experimental results on hundreds of scenarios to demonstrate the effectiveness of proposed framework on VLA-based policies.", "questions": "Regarding the real-to-sim translation pipeline design: \n1. Although the fully automated reality-to-simulation translation pipeline is novel, could the authors provide more details on how error propagation across these stages is quantified? For instance, how do inaccuracies in VLM-based segmentation or 3D mesh generation impact the final simulation fidelity?\n\nRegarding the evaluation methodology and potential biases:\n1. The hybrid evaluation combining VLM scoring and human preferences is compelling. Yet, how do the authors mitigate biases inherent in both methods? \n\nRegarding the experimental design and generalization claims: \n1. The large-scale experiments reveal significant cross-dataset performance drops. Could the authors discuss whether the observed failures are due to visual domain shifts or task complexity differences? \n2. How might the benchmark be extended to include dynamic scenarios to better assess true generalization?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The automated real-to-sim translation pipeline that the paper introduces is innovative.\n2. The hybrid assessment method integrating VLM-based scoring with human preference feedback is comprehensive.", "weaknesses": "1. The simulation environment cannot accurately reproduce fine-grained physical interactions (e.g., plug insertion, deformable object manipulation), limiting evaluation fidelity for precision tasks.\n2. The multi-stage pipeline may accumulate errors, but the paper lacks quantitative analysis of error propagation across stages.\n3. The benchmark primarily focuses on static-camera, table-top manipulation tasks from datasets like Bridge and DROID, lacking coverage of dynamic scenarios, mobile navigation, or multi-object interactions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CU9auDRzu9", "reviewer": "Reviewer_f9Cq", "summary": "This paper presents a scalable benchmarking framework for evaluating real-world–trained vision-language-action (VLA) robot policies. Traditional real-world testing is costly, slow, and unsafe, while existing simulation benchmarks lack the realism to assess models grounded in real demonstrations. The proposed method bridges this gap by automatically converting real robot videos into simulated digital twins using advances in vision-language modeling, 2D-to-3D generation, and differentiable rendering. Within these simulations, robot policies are evaluated through both automated vision-language-model scoring and large-scale human preference judgments collected online. The framework also introduces systematic environment perturbations to test robustness, enabling reproducible, human-aligned, and scalable evaluation of general-purpose robot manipulation agents.", "questions": "1. I am curious about the evaluation results of the pi series VLA models on the proposed benchmark.\n2. Can you provide more detailed information of your sim-real alignment experiments? For example, what are the controllers and control frequencies of the simulation and real environments, since controller type and frequency can sometimes cause different motions in simulation and real world?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a comprehensive and automated pipeline that seamlessly bridges real-world robot data and simulation, enabling scalable and reproducible evaluation of vision-language-action models.\n2. It conducts the largest cross-lab evaluation to date, providing unprecedented insights into the generalization capabilities and limitations of current generalist robot policies under diverse distribution shifts.\n3. The paper is well-written, presenting a complex technical system with conceptual clarity and a logical narrative that is easy to follow.", "weaknesses": "My main concerns lie in the proposed real2sim pipeline, including:\n\n1. The 3D assets are primarily from some 3D generation models, which may produce meshes with different shape with real objects, or have implausible collisions. Moreover, the physical parameters are given by some LLMs, which can also result in some implausible physical movements in simulation.\n2. Inpainting background makes the camera viewpoint in this evaluation process remain the same as, or close to, that in the original video. However, we often require other camera or multiple cameras for policy inference, e.g., the wrist/first/third view cameras.\n3. The pipeline requires system identification of robots, which can cause much extra data collection and human efforts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sSfOZltkSM", "reviewer": "Reviewer_dV9J", "summary": "This paper provides a fairly comprehensive benchmarking system for evaluating VLA models for robotics. The contributions of this work are:\n\n- A benchmarking protocol for robotics in simulation. This includes VLM-based task evaluation and preference modeling.\n- An automated video to simulation pipeline.\n- Extensive evaluation of many VLA models.\n- Evaluation results across in-distribution and out-of-distribution cases.", "questions": "- There doesn’t seem to be a discussion on the evaluation of diverse language instructions from users in this work. Can the authors comment on this?\n- Related works:\n    - Could the authors discuss other works that similarly provide a real to sim pipeline, such as https://arxiv.org/pdf/2403.03949? I think it would be helpful to understand how much of this work is really an aggregation of existing approaches vs. novel contributions that the reader can take away.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- This work provides a powerful framework for the robotics community to develop on top of. This kind of work can really make an impact on scaling the data necessary for VLA-style models to succeed.\n- The breadth of this work is quite remarkable, from automating video-to-simulation to evaluation across many of the open-source VLAs to extensive evaluation results.\n- This work fills in significant gaps in an end-to-end pipeline that would make robot training closer to the approaches taken for training LLMs.\n- The presentation of this work is well done, with thoughtful and descriptive figures and clearly robust evaluation.", "weaknesses": "- Overall, the work is quite strong, but there could be some clarifications that could strengthen the claims in the paper.\n- On the claims of comprehensive evaluation, the authors should comment on the evaluation of diverse language instructions from users, as this is a reasonable failure mode of VLAs (e.g. https://arxiv.org/pdf/2411.18676)\n- The authors should make it clear what aspects of the pipeline are truly new vs. aggregations of existing approaches. For instance, how original is the approach for real-to-sim? The work doesn’t need to be completely novel, but it would have been nice to see the authors comment on how this approach draws from prior work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)